I made a lot of progress today.
It turned out the block culling was the easiest thing to do with the biggest payoff. I did end up changing the block coordinates to y-up.
Plus a lot of other little changes and ideas along the way.
Now I need to check this into SVN and put a .zip on Google Code.