I've downgraded to esxi 5.0u2
Based on this post (http://communities.vmware.com/message/2145026#2145026), I finally have the screen working, if not any acceleration.
Try adding this to your boot options under grub:
fglrx.modeset=0
This seems to work while adding just
nomodeset
doesn't.
Now I just have to figure out the acceleration part. It seems like the fglrx kernel module isn't actually being loaded.