Hi Warren,
I can only image that your D800 is broken in some way. Even used as a DX camera, it's 16 meg will out shine the D300's 12 meg with ease, and it's not due to the sensor size. The pardalote picture used in my reply is cropped one to one pixels, and so I can seriously say that its limit is firstly the user's technique and secondly the lens being used. However, I often don't get pictures that will go one to one with the camera, probably only one in five will do this on a tripod, so it's always pleasing when it happens. It's never going to happen free hand with shutter speeds under 2,000th of a second and maybe not even then.
You mention the eventual D300 replacement in your reply. The other possibility is the D810. When this first came out, I was surprised by how little seemed to have changed in its specs when compared with the D800. However, I was quite wrong to doubt the strength of the change. Lots of things are new, the sensor, the focus detection; which is from the D4 and the best in the business, and the shutter. I suspect that it's the last item that will interest you. It has very little vibration indeed and is extremely quiet.
This is important when considering your problems with the D800, which has quite a kick to it, in my view. This kick is easily stopped by cupping you hand down the side of the lens, to take up the shock of the shutter. People who are not pixel peepers probably won't notice the difference, but if you are keen to get every last drop of detail from your gear, give it a try. The memorial shot of Tom Oliver on the main web site has Tom showing how it's done.
You might wonder why all Nikons up to the D800 have had quite a strong shutter vibration. Well, they are metal shutters and this allows you to sync at 320th of a second. My friends, who all have Canons, find 160th is their reliable sweet spot, possibly due to their cloth shutter, though I'm no expert in this. You also gain the advantage that the metal construction gives to their shutter life. My D300 has in excess of 350,000 actuations and I'm still waiting, with curiosity, to see when it will bite the dust.
So after all this I would suggest a couple of experiments to try, if you haven't done them already. Use the D8OO on the tripod to take pictures using a cable release or the self timer. Then try with the lens cupped in you hand. Change the shutter speeds to see if this causes much change in your results.
If all else fails, try the D810. To my sorrow, I believe it's quite a lot better camera than the D800 in many ways. Maybe Graham Cam, who has one (and had a D800) , might add a word to two about this sometime. Meanwhile, if you look at Brian O'Leary's pictures, they'll give you an idea of what the D810 can do.
Cheers,
les