What exactly are you trying to accomplish, Guest clustering? I suppose you are aware of these guidelines and documents already, check them carefully anyways:
Many points can be applied to non-Windows clustering too.
It wasn't clear from your post which modes you used, but in this case you need to set the RDM in physical compatbility mode and each VM needs a dedicated virtual SCSI controller in physical bus sharing mode mode well. Add the RDM through the GUI on your first VM and don't manually create the mapping file with vmkfstools for something.