While upgrading my home-lab I ran into an issue which results in a rollback of the VMware SSO installation/upgrade. Although this issue is also reported on the VMware Support Insider blog, I choose to also document it on my blog.
After starting the the upgrade of the Windows vCenter Server the upgrade suddenly (specifically when importing Lookup Service Data) stops and automatically rolls back.
The message displayed by the installer is: “Simple Install Setup Wizard ended prematurely because of an error. Your system has not been modified. To install this program at a later time, run Setup Wizard again. Click the Finish button to exit the Setup Wizard.”
The vim-sso-msi file is displayed and show messages similar to:
Action 9:04:17: PostInstallScripts. Importing data from previous version... Action 9:04:38: PostInstallScripts. Importing Lookupservice data... CustomAction DoUpdateAndMigrateTasks returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox) Action ended 9:04:43: InstallFinalize. Return value 3. MSI (s) (EC:84) [09:04:43:378]: User policy value 'DisableRollback' is 0 MSI (s) (EC:84) [09:04:43:378]: Machine policy value 'DisableRollback' is 0
The SSO upgrade will fail if there is a mismatch between the certificate for the 5.1 SSO service and a related registry key that indicates if the system is DNS-enabled. The issue occurs on some vCenter 5.1 systems running with default self-signed certificates.
First check the DNS name of your SSL certificate. The default location for the SSL certificate is C:\ProgramData\VMware\VMware VirtualCenter\SSL, the self-signed certificate is called rui.crt. Check the Subject Name and/or Subject Alternative name for the correct DNS name:
Also check the value in HKEY_LOCAL_MACHINE\Software\VMware, Inc.\VMware Infrastructure\SSOServer\FQDNIp. If there’s an IP address configured here, change the value the DNS name as displayed by Subject Name and/or Subject Alternative name. If you already ran the installation before editing the registry, also delete the %ProgramData%\Vmware\CIS (probably C:\ProgramData\VMware\CIS) folder.
After this change restart the installation, which will continue without a problem. A detailed explanation of this issue is available VMware KB Article 2060511. Read this article before applying any change to fully understand the issue and resolution.