Okay, let's try this again.
I just checked this procedure/instructions outlined below using ESXi 5.0U1 623860 installed on the physical ESXi host using ESXi 5.1a 838463 as the nested ESXi VM and then loaded up a Windows 2008 R2 SP1 x64 VM running on the nested ESXi 5.1a VM and the x64 Windows 2008 R2 SP1 VM worked fine (although much slower than if all were running on a physical server of course).
---------
Assuming the required physical CPUs in your physical ESXi 5.0 host have EPT capability (required for Intel CPUs) or RVI capability (required for AMD CPUs), then make sure you have:
vhv.allow = TRUE
inserted into the /etc/vmware/config file of your phyiscal ESXi 5.0 host (this is required in order to get nested ESXi hosts to work but no reboot of the physical ESXi 5.0U1 host should be necessary for this setting to get engaged).
Create a VM for the ESXi 5.1a host but initially choose "Other x64" o/s type and choose E1000E as the NIC type but don't power it on yet.
Once the VM is created and still powered off, change the O/S type to ESX 5.x (which should now be exposed and available to choose).
Also in the ESXi 5.1a VM, under Edit Settings, choose the Option Tab and under CPU/MMU Virtualization Settings choose the bottom choice.
At this point, you may or may not need a special CPU mask from J Mattson here in this community forum to insert into the VMX file of the nested ESXi 5.1a VM (right now where I'm at I only have the AMD settings rather than the Intel settings available to me but it may likely work without any special CPU masks so just keep going).
Make sure you have assigned at least 2 vCPUs and 2GB of memory to the nested ESXi 5.1a VM (required to get ESXi 5.1a to install).
Boot your nested ESXi 5.1a VM and install ESXi 5.1a into the VM.
After the nested ESXi 5.1a VM has had ESXi installed, boot the ESXi 5.1a VM and configure it to your liking.
When you create a vSwitch in the nested ESXi 5.1 VM, if you want the VMs that run on the nested ESXi 5.1a VM to have outside network visibility, you'll need to engage Promiscuous Mode on the vSwitch that holds the VM Port Group as well as the vSwitch that holds the Management Port of the nested ESXi 5.1a VM (it's a choice under Edit and then choose the Security Tab for the vSwitch and change the setting for Promiscuous Mode to Accept from the default of Reject). If you just have one vSwitch for everything (VM, Managaement and VMotion/VMKernel) then you only need to change to Promiscuous Mode on that one vSwitch.
Install your 64 bit Windows VM onto the nested ESXi 5.1a VM.
Remember, none of this is going to work with running x64 Windows VMs on your nested ESXi 5.1a VM unless you have EPT/RVI capability built into your physical CPUs. Note before anyone asks -- no Intel CPU with the word "Core 2" in the model name will have EPT capability. Server based Intel CPUs generally need to be Nehalem family or above in order to have EPT capability. AMD server based CPUs need to be 23xx series Opteron CPUs or higher with at least B3 stepping or higher for the 23xx series Opterons in order to have RVI capability that allows this to work.
Whew, back to the future with ESXi 5.0u1 on the physical server. Where's my Delorean?
Datto