Azure Service Fabric Resource Deployment Failure Exit Code -532462766 - azure

Question

After creating a vanilla service fabric cluster through the Azure portal, I am getting this error on the VM Scale Set. It has happened using a range of different cluster names, sizes and VM types.
Full error:
{
"status": "Failed",
"error": {
"code": "ResourceDeploymentFailure",
"message": "The resource operation completed with terminal provisioning state 'Failed'.",
"details": [
{
"code": "VMExtensionHandlerNonTransientError",
"message": "Handler 'Microsoft.Azure.ServiceFabric.ServiceFabricNode' has reported failure for VM Extension 'Test_ServiceFabricNode' with terminal error code '1009' and error message: 'Enable failed for plugin (name: Microsoft.Azure.ServiceFabric.ServiceFabricNode, version 1.0.0.33) with exception Command C:\\Packages\\Plugins\\Microsoft.Azure.ServiceFabric.ServiceFabricNode\\1.0.0.33 \\ServiceFabricExtensionHandler.exe of Microsoft.Azure.ServiceFabric.ServiceFabricNode has exited with Exit code: -532462766'"
}
]
}
}
This is before I am trying to publish my user-code to the cluster, so not really sure what can be causing this. Using default settings for most things. Initial capacity 3 (test cluster). Bronze Durability/Reliability
EDIT:
Seeing this issue when connecting to one of the VMs in remote desktop.
Application: ServiceFabricExtensionHandler.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.Azure.ServiceFabric.Extension.Core.AgentException
at Microsoft.Azure.ServiceFabric.Extension.Core.CertificateUtility.LoadClientCertificate(System.String, System.String)
at Microsoft.Azure.ServiceFabric.Extension.Core.CertificateUtility.LoadCertificateWrapper(Microsoft.Azure.ServiceFabric.Extension.Core.Models.CertificateSettings)
at Microsoft.Azure.ServiceFabric.Extension.Core.Models.HandlerSettings.AllowAccessToCerts()
at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.ValidateDeployment(Microsoft.Azure.ServiceFabric.Extension.Core.Models.HandlerSettings)
at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.InstallService()
at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.InstallServiceWithRetry()
at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.Enable()
at Microsoft.Azure.ServiceFabric.Extension.Handler.Program.Main(System.String[])

Answer 1

The issue is that Service Fabric can't find the certificate you have configured. There are a couple common reasons for this:
The wrong certificate thumbprint is provided. Assuming you are using KeyVault, make sure you aren't using the ID portion of the KeyVault URL as your cert thumbprint.
There is a hidden unicode character at the beginning of your cert thumbprint. This is common if you got your thumbprint from Windows cert viewer dialog as when you copy/paste from that dialog it inserts a hidden character at the beginning. Resolution is to first copy to notepad (or any other ASCII editor).
There are two ways you can troulbeshoot this if you RDP to one of the VMs:
Check the Azure guest agent logs at C:\WindowsAzure\Logs\WaAppAgent.logs. You will be looking for entries relating to installing the Microsoft.Azure.ServiceFabric.Extension, and then something like "Cannot find certificate with thumbprint "xxxx" ...". Make sure the thumbprint is what you expect, and that the thumbprint doesn't start with a question mark "?xxx" which indicates the hidden unicode character.
Open the Service Fabric Admin event logs (eventvwr -> Applications and Services Logs -> Microsoft-ServiceFabric). You should see error entries related to failing to load a certificate.