Note: this work was done around vSphere 5.0. Will double-check results with vSphere 5.1 but I suspect it will be similar.
I've done some work around the vSphere Authentication Proxy now; still not confident I've got it down pat, but I have some interesting observations that might be useful to someone out there.
The first step with Authentication Proxy is to replace the rui certificate, as with the other services. The cert resides in C:\ProgramData\VMware\vSphere Authentication Proxy\ssl. As with Autodeploy, I had to run the following command after replacing the certificate:
"C:\Program Files (x86)\VMware\vSphere Authentication Proxy\cam-register.exe" -r -a <vcenter server IP> -u Username -p Password -s "C:\ProgramData\VMware\vSphere Authentication Proxy\vmconfig-cam.xml"
When you restart the Proxy Adapter service, you should see some content appearing in C:\ProgramData\VMware\vSphere Authentication Proxy\logs\camadapter.log, with no error messages. Amongst the entries, you should see it deleting the certificate in IIS, and re-adding it. It does this on every startup. Go ahead and verify that you can actually see the new certificate in IIS (edit the SSL binding of the site and view the cert - or just browse to https://<vcenter>:51915/ssl (should get a successful SSL connection, with a "You do not have permission..." message displayed on screen.)
Unlike the other services, Authentication Proxy has an affinity for a combination of DER-encoded certificates. The VMware documentation seems to call this "Windows format", though I'm not sure where the authors got that idea from. To cover the bases, the other certificates in use by Authentication Proxy should all be in both "CRT" (base64-encoded) and DER (non-base64-encoded) format. The service seems to automatically create DER-encoded copies of the ca.crt certificate, but not the certificates in the "trust" folder.
As expected, there appears to be some interaction between Authentication Proxy and Autodeploy. The default Authentication Proxy "ca.der" certificate in the Authentication Proxy SSL folder is the same one used by Autodeploy. I'm not quite sure yet how these two services deal with different installation orders etc. I'm also not entirely sure how this ca.der certificate is used, but I suspect it's to allow individual hosts to validate the certificate chain that signed the Authentication Proxy cert. In my case, I replaced it with the custom CA certificate I created for my Autodeploy instance above; this worked. After some testing, it seems that using the root CA cert here worked as well.
A third building block is the C:\ProgramData\VMware\vSphere Authentication Proxy\trust folder. According to the documentation, this folder should contain the Authentication Proxy certificate if you replace it with a custom-signed cert. This didn't make much sense to me, so I placed my CA certificate in there. This worked; either my Root CA or the Autodeploy CA worked, presumably because this allows the Authentication proxy to trust the host certs issued by Autodeploy. Leaving the trust folder empty caused Authentication Proxy to "break" - the service started, but hosts would throw an "access denied" message if you attempted to join them to the domain. Note that after you restart the Proxy Adapter service, you should now see a message in camadapter.log along the lines of "Add trust certificate <folderpath>"
Finally, it seems that the Authentication Proxy automatically retrieves the certificates from each host connected to vCenter and places them in the ssl folder. It does this the moment a new host is connected. For example, if you join a new host, you might see a new file called "host-28.der/host-28.crt" appear in the folder. I suspect it uses this certificate to determine if it's a legitimate host, and that this somehow ties up to one of the two CA certificates in the ssl and trust folders.
So, in summary, I did the following, which seems to work:
- Replaced the rui certificate
- Replaced the ca.der cert in the SSL folder with either my root CA or the issuing CA I generated
- Added my CA certificate (in DER format) to the "trust" folder (any name will do, it parses *.* upon startup)
- Restarted the Authentication Proxy services
- Ran the cam-register command
- Restarted the Authentication Proxy services again for good measure
As I said, I'm not 100% confident I've got it fully figured out, but the above seems to work for me so far - my hosts are issued with trusted certificates automatically by AutoDeploy, and I can join them to the domain using the Authentication Proxy service (which now uses a trusted cert itself) without importing any certificates.