We have a worker process running in IIS (Windows Server 2019) that makes an https request to a kafka resource in AWS in the background. In order for this process to work, we have to import certificates from Amazon into the local computer cert store so that the ssl connection can be established and it will trust the certificate presented from the resource in AWS. We have imported the cert chain and gotten this to work with the following caveats:
The service account specified for the AppPool identity has to be a local admin
OR
We have to run some process, ANY process (other than w3wp.exe that's already running), as the service account.
I've dealt quite a bit with certs, permissions, etc, but what is really odd about this one is that if I open notepad, for example, as the service account, then as long as notepad is running the process works - meaning it reads the cert and is able to establish the ssl connection. If I close notepad, the process then fails, and I get the following error:
ssl://b-2.kafkamsk.abc123.c2.kafka.us-west-2.amazonaws.com:9094/bootstrap: SSL handshake failed: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (add broker's CA certificate to the Windows Root certificate store) (after 72ms in state CONNECT)
(text in the URL replaced with abc123 for obfuscation)
Note that we don't need to access the private key of the cert, we simply need to read the cert from the cert store. The CA cert has been imported into "Trusted Root Certification Authorities." The intermediate cert has been imported into "Intermediate Certification Authorities" and the leaf cert has been imported into the personal store - all of these in the Local Computer cert store. And again, it all works if we simply login and run some process as the service account, even without it being a local admin.
I've run sysinternals procmon while executing the process that fails, and see several access denied attempts on HKEY_USERS.DEFAULT\Software\Microsoft\SystemCertificates, and those do not appear if I have notepad running as the service account when attempting the process. I've given explicit permissions to the service account on those registry keys, but it has not helped.
For obvious security reasons, we don't want our service account to be a local admin on the box, and for operations/administration reasons we don't want to rely on having some other process running as this account just to make this work.
It probably works when notepad is run as its triggering the user profile to be loaded, terminating notepad causes the profile to be unloaded. If you were to set LoadUserProfile to true on the app pool it would probably work - BUT this is not a good idea.
I think the proper solution is related to the permissions for IIS AppPool\AppPoolName on the certificate itself (which boils down to a file on disk with an ACL somewhere). Grant the app pool rights via the Certificates MMC or powershell and it should work.
if you download the latest .pem file from https://curl.se/docs/caextract.html, you can specify that file in the code with:
"ssl.ca.location",caCertLocation
Event Hubs Demo from Azure here:
https://github.com/Azure/azure-event-hubs-for-kafka/blob/master/quickstart/dotnet/EventHubsForKafkaSample/Program.cs
Related
Our APIs are exposed through Azure Cloud Services (Web Role). We use certificate uploaded to Cloud Service for encryption of data stored in backend. During certificate rotation, We uploaded new certificate by updating *.CSCFG file
Our CSCFG
<Certificate name="encryptcertificatethumbprint" thumbprint="NewThumbprint" thumbprintAlgorithm="sha1" />
When we login into Portal > Cloud Service > Certificates Tab. We could see both Old Certificate and New certificate.
Unfortunately some of the records in backend system are not re-encrypted with new rotated certificate. When our App reads data from backend system, it fails to decrypt with message "The secret could not be decrypted. Check that certificate with serial number 'XXXXX-XXX' is installed. The inner cryptographic exception: 'Keyset does not exist" since old certificate is not present in webrole instance.
Instead of doing another deployment with both certificates, we remote into web role instance and installed old certificate under Personal > Certificate
After installing old certificate, Application still unable to find certificate by thumbprint in Local Machine certificate store. We tried restarting webrole, still application unable to pick certificate.
Few Questions, can anyone help (without any deployment to address this issue)
What will be certificate location on webrole instance, where we can upload certificate. Our logic of certificate search based on Serial number in LocalMachine and then also Current User. Though we uploaded certificate in LocalMachine, Still webrole unable to pickup certificate
What will be Current User certificate store i.e. what will be account on which webrole runs
If the certificate gets expired, will certificate be returned ?
I haven't tried to accomplish your specific goal, but this information may help get you closer.
The Azure guest agent installs certificates as the system account in the CurrentUser store, so any interactions you want to do with those certs will need to be as the system account. To do this you can download Sysinternals Suite and run psexec to launch a cmd prompt under System context. Then in that new cmd prompt you can run certutil, mmc.exe, etc:
psexec -i -d -s cmd
certutil -store -user My
You should be aware that whatever you do (ie. installing a new certificate) will be temporary.
See https://learn.microsoft.com/en-us/archive/blogs/kwill/windows-azure-disk-partition-preservation for information about when the Windows partition (D drive) will be rebuilt, meaning you will lose your manually installed certificates.
Unexpected role recycles (ie. WaHostBootstrapper exits or WaIISHost/WaWorkerHost crashes) will cause the guest agent to uninstall and then reinstall all certificates.
Issue got resolved. After giving permission to "Network Service".
Right Click on Certificate > All Task > Manage Private Key > Add Network Service Account to get access to certificate.
I am creating a VPN in Azure, and created self-signed certificate in the following places:
Local Computer:
Personal/Certificates: Issued To: FQDN name is the certificate's name
Trusted Root Certification Authorities/Certificates:
manually copied from Personal
I have configured the VPN in Azure and it is downloaded and extracted and the vpn client is installed successfully, however, when I run the client I received the following error:
A certificate could not be found that can be used with this Extensible
Authentication Protocol. (Error 798)
Screenshot:
The error seems suggesting the certificate is NOT found, does the name matter?
Should I change the cert's name from Azurecert to FQDN which is the name in my local computer?
Thank you for your help in advance.
UPDATE:
I removed the existing certification in Azure's configuration and re-added back with the same FQDN name shown in local computer's certificates, I redownloaded the client, I removed the existing installed vpn client and re-installed the new one, I receive the same error message. So it seems name is not the root cause?
UPDATE2:
The procedure I've followed:
Create Self Signed Certificate with the FQDN name on local laptop;
New-SelfSignedCertificate –DnsName NV-RXIE.novantas.pri -CertStoreLocation “cert:\LocalMachine\My”
Add the self-signed certificate as a trusted certificate authority, Copy the new cert to Trusted Root Certificate Authorities
export the cert and open it, copy the cert part and paste into the
VPN setting – Root certificates, Public Certificate Data
Download the VPN client and install it on laptop, run it
Connect, failed with:
A certificate could not be found that can be used with this Extensible
Authentication Protocol. (Error 798)
When you try to connect to an Azure virtual network by using the VPN client, except for exporting the root certificate public key .cer file to Azure, each client computer that connects to a VNet using Point-to-Site must have a client certificate installed. You generate a client certificate from the self-signed root certificate and then export and install the client certificate. If the client certificate is not installed, authentication fails.
This problem occurs if the client certificate is missing from Certificates - Current User\Personal\Certificates.
You could follow this solution to fix this issue. For more information about how to install the client certificate, see Generate and export certificates for point-to-site connections.
In case anyone runs into this issue at some stage, I had installed a new root cert that worked for 2 out of 3 VPN gateways fine. The third kept giving a 798 error even though the certs were correct and in the right place.
To fix the Error 798, I did the following:
reset the gateway in Azure Portal. (support & troubleshooting on VPN gateway blade)
remove the VPN configuration from my pc (win10)
reboot pc (just to be safe)
download and reinstall the VPN client from the Azure Portal again (from Point-to-site configuration on Azure VPN gateway in question)
Once done, I could then connect without any issues. Tested on several different users.
My guess is that if you are adding / removing the Root certs it might need you to reinstall the VPN client on your computer after the gateway has the new root cert configuration.
Hope that helps.
In addition to the answer by Nancy Xiong:
If you are still having problems with this error you can try the following
Run certmgr.msc
Go to Personal->Certificates
Right-click your certificate
All Tasks->Export
Choose Yes: Export private key
Accept default options until you reach a step where you must enter a password
Enter a password, and continue until you have exported your certificate
Repeat this process if you have more than one certificate
Locate your certificates in the Windows file explorer
Right-click->Install
Select Current User for the Store Location
Accept default options, and enter the certificate password when prompted
When asked which Certificate Store to place the certificate in, select Place all certificates in the following store
Click 'Browse' and select your Personal store
This should now work.
In rare circumstances you may find that this solution will only work for a short time (usually failing the next time you reboot). In this case you may need to follow these additional steps
Boot your computer into BIOS Configuration
Disabled any settings for Intel VTX and Intel VTD
Restart your computer
Retry the steps above
In addition to the answer by Peter Morris, I have did the below mentioned steps which helped in resolving this issue.
Delete the client certificate from your PC. This can be done by searching Manage User Certificates right click on the certificate -> Delete
Now, Reinstall your certificate, Accept default options, and enter the certificate password when prompted.
When asked which Certificate Store to place the certificate in, select Place all certificates in the following store
Click 'Browse' and select your Personal store
Accept all the default options which comes next and click finish.
The above steps helped me to resolve this issue
For me, I got this error because my previous cert has expired after 1 year.
I just deleted the old cert and followed this to create a new one:
https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-certificates-point-to-site
I am developing a windows service application that will run on customer PC/servers and access a Web API endpoint hosted in an Azure Website. It needs to authenticate the user, and I would prefer not storing credentials on the customer's machine. So, I've landed on client certificates to authenticate the users. I have this working against a local, non-Azure Website IIS instance with self-signed certificates. However, I'm unable to get it working in an Azure Website.
As far as I can tell, there are two issues that I'm not finding much documentation on:
How do I install my own CA certificate in the Trusted Root of the Website instance(s)? Or will this only work with CA certificates that are already trusted?
How do I enable "Accept Client Certificates" for this application? In IIS you do this under "SSL Settings". Documentation indicates that modifying the system.webServer/security/access node of app.config will accomplish this, but obviously you can't do that in Websites. Documentation for websites suggests this node is unlocked for use in web.config, however adding that node results in an error "The page cannot be displayed because an internal server error has occurred.", even if custom errors is off.
For Azure web sites vs web roles client authentication options are rather limited. Websites don't let you run programs with elevated permissions, which is required for making IIS changes and storing certificates into the trusted root.
There's a way to configure you website to always (you don't get the benefit of making it optional as with IIS 'Accept' configuration) request client certificate. This feature is currently only available through Azure management REST API, you can't access it through the portal UI. You can find more information here.
Essentially you turn on clientCertEnabled website setting to true. The mechanics of this option are different from traditional client authentication where server needs to have a CA certificate with which the client cert is signed in its trusted root. The server doesn't run any validation on the client certificate, the application needs to run the cert check itself, which comes in a request header "X-ARR-ClientCert". GetClientCertificate() extension method on HttpRequestMessage will parse it automatically.
Alternatively, you can host your Web API as a web role. That gives access to running startup tasks with elevated permissions that allows access to trusted root and making IIS configurations, more details/examples here. You can either copy the CA certificate to the app folder or upload to the user store via Azure portal so that it is available for copying over to the trusted root in a startup task. IIS changes can be made via “Microsoft.Web.Administration” library available as NuGet package through ServerManager class.
For question 2, here's a blog post on how to install client certificates on Azure Websites: http://azure.microsoft.com/blog/2014/10/27/using-certificates-in-azure-websites-applications/
For question 1, you can't install your own CA certs as trusted root certificates, but if you have certs from a CA that's already trusted then you can use them without any issues.
I'm trying to remove a certificate (via the Management Portal) that is due to expire on 16/07/2014 days and I'm getting this error:
The certificate I'm trying to delete was automatically generated 'CN=Windows Azure Tools' and was used for Remote Access.
I also tried to configure the cloud services to use new certificates, but still no success.
Looked at Fiddlers log and indeed there in an internal server error:
My questions are:
Is there any way I can delete this certificate?
If I can't delete it, what happens when it expires?
What I really want to avoid is to have to redeploy the production services because of this certificate.
Florin, you cannot delete a certificate in production use, but you have two options for this issue:
The easiest option is to just ignore it. The RDP cert is only used for password encryption and it doesn't matter if the cert is expired or not since Azure does not check the expiration date when doing the encryption/decryption.
You can upload a new certificate and then modify the RDP configuration to use that new certificate. After this you can delete the old certificate. This is essentially the same process you would use for an HTTPS certificate.
I am building a site that needs to call a 3rd party web service over https. The service requires an X509 certificate for authentication.
I can successfully call the service and get a response back using a console application, or a web application running in visual studio web server, this suggests to me that the code, and certificate provided by the 3rd party are all ok.
As soon as I deploy the site in IIS, it cannot connect to the service, I am getting "The request was aborted: Could not create SSL/TLS secure channel" error. I spent a couple of hours trying to figure this out, assuming it was permissions related as IIS app pool running under network service etc.
The certificate is installed in the local machine store, not current user, and I have granted network service (also added everyone is desperation!) using the winhttpcertcfg tool and manually in MMC > Manage Private Keys, so I am confident that the network service account has access to the certificate.
I also used the FindPrivateKey tool and checked the physical windows file and permissions look ok! I tried using identity impersonate in the config, using my windows account and that doesn't make a difference.
I am pulling my hair out with this one, can anyone give me any suggestions on what to try next?
Cheers