We have an Azure Application Gateway V2 setup with WAF feature enabled. We ran into trouble when suddenly, the health of all our backend pools (3 in our case) started showing unhealthy and the error was being shown when we checked backend health:
Backend server certificate expired. Please upload a valid certificate.
Now, the same certificate is applied on application gateway and on the backend pool servers/VMs. We had to switch to non-SSL port for our backend servers to make the problem go away.
I am not able to pinpoint the problem as if any certificate is expired when applied to backend pool VM should give the same error once applied to the application gateway itself. Can someone help to identify the actual root cause of the problem and why it is showing error only at the backend pool level?
Very important thing is that the certificates being shown in browser when the backend pool URL is opened is showing still more than 2 months before it expires.
Thanks.
I can't comment as I have not enough reputation point but this might be connected to the fact that Sectigo's legacy AddTrust External CA Root certificate expires on May 30, 2020
https://support.sectigo.com/articles/Knowledge/Sectigo-AddTrust-External-CA-Root-Expiring-May-30-2020
In my case, the browser was also showing that everything is ok, but HttpClient responsible for doing the healtheck was not able to connect to the endpoint. What is interesting is the fact that I wasn't able to hit the endpoint with HttpClient running on dotnet core 2.1 but on dotnetcore 3.1 everything was fine
Same thing happened with my web app (roughly at the same time as reported by you) hosted on Azure which is also behind an Application Gateway V2 setup with WAF enabled. Taking a restart of the web app resolved the error.
I'm not sure what the underlying issue is with Application Gateway at the moment, but a restart of the webapp worked for one of my apps, however an other one stayed unhealthy. I could successfully connect to the webapp from the local server however, just not through the app gateway.
What worked for me in the end was removing the server from the backend pool and re-adding it. Once it was added back, the Backend health changed to 'Healthy'.
After checking via openssl, it was identified that the intermediate certificate was revoked by the CA and it was causing the issue.
To check whether certificate is valid or not, Online Certificate Status Protocol can be used or CRI file can be downloaded and the certificate can be checked in that file. The second method has little drawback that CRI files take a little time at the CA end to be updated.
Related
I'm using an Azure Application Gateway v2 to route traffic to a backendpool containing VMs running some docker container hosting an aspnet core webapi. The application is listeing in port 443.
The gateway listener is configured to accept HTTPS connections. A pfx certificate has also been added.
The HTTP setting of the gateway is configured as follow:
I've provided, hopefully, the correct root certificate for the setting. I've exported the certifacte as described here:
https://learn.microsoft.com/de-de/azure/application-gateway/certificates-for-backend-authentication
All VMs from the backend pool use the same certificate as the gateway listener.
When calling the api using a C# application or open a specific url within a browser, the connections seems to be trusted.
But I get an error when calling the api using postman with endabled SSL cerificate verification.
Error: unable to verify the first certificate
A check on sslhopper.com also indicates an issue within the chain.
I got further information doing a chekc on digicert.com.
TLS Certificate is not trusted
The certificate is not signed by a trusted authority (checking against Mozilla's root store). If you bought the certificate from a trusted authority, you probably just need to install one or more Intermediate certificates. Contact your certificate provider for assistance doing this for your server platform.
I'm really stuck in resolving the issue. All helpful hints are very welcome.
The production environment, running on a Windows Server, wihtout the Gateway but the same certificates is valid. So the issue must be somewhere in the gateway configuration.
The issue is resolved. The crucial clue was delivered from a blog post.
http://blog.repsaj.nl/index.php/2019/08/azure-application-gateway-certificate-gotchas/
Gateway V2: the importance of the certificate chain
I did a new export of the pfx certificate including the whole chain and uploaded it to the gateway.
If the cert is not trusted, then you need to check the Application Gateway Listener. Make sure that you are not using Basic listener.
If you are using multi-site listener, make sure that the host name which you have mentioned matches with the certificate CN
I have hosted an MVC application in Azure App Service. The application is not creating any problem when i am testing it in local host. I can easily connect to the Dynamics-365 through the SDK and can do easily CRUD operation.
But when i am trying to login which is hosted in Azure it is giving that error though am not getting this error if i restart the app service. If i restart the app service in every morning then i am not getting this error. But i don't want to restart the app service daily.
Here the same question also has been asked but there is not any solution which has been marked as Solved. Time zone difference between client and server has been discussed here but the part which confused me when i am restarting the app service it is working.
How to solve this issue? I am thinking about app service Auto Healing. Should this work?
Any solution would be appreciated.
Thanks in advance.
From your description of Question it looks like connection pool issue. You probably are not closing connection after your transaction is completed. Try force closing connection (threads) that might solve your issue.
OR
The clock on the Server hosting client application was out of sync with the server having the service.
This is almost always because of a server time skew. The remote server
and the client's system time must be within (typically) 10 minutes of
each other. If they are not, security validation will fail.
I'd check azure service bus and find out what their server time is, and
compare that to your server time.
WCF gives an unsecured or incorrectly secured fault error
https://social.msdn.microsoft.com/Forums/vstudio/en-US/ae4172dd-e215-4a72-b927-d164ce4cc318/an-unsecured-or-incorrectly-secured-fault-was-received-from-the-other-party?forum=wcf
I have a seemingly intermitted issue with a site I'm hosting on Azure.
The site works for most but we're getting a few reports that when people try to access the site via a link (in an email for example) the site throws an ERR_CONNECTION_RESET error in Chrome.
The site is an S1 App Service
The site running is .Net Core 2.1 (MVC)
It has Let's Encrypt certificate applied to it (and this shows in the browser)
It is set to be Https only and Always On in Azure's SSL and Application Settings blades respectively.
I've had a look around and it feels like maybe this is an SSL issue but I'm not sure.
I've done a re-bind of the cert.
Can anyone offer any advice? I've not encountered this before.
Update
An update from our clients suggest it could be happening as our domain is newly registered.
Specifically, it could be a reputation/categorisation issue coming in to play with the threat protection they utilise on their networks.
I will look into submitting an update to the relevant sites to see if it helps.
We are developing an ASP.NET WebAPI that is hosted in IIS and authenticates using client certificates with iisClientCertificateMappingAuthentication.
In our on-premises development environment we have tested IIS 7-8 and Windows 7-8/Server 2012, all of which work fine.
We have set up a demo environment in Azure with a VM running Windows Server 2012 R2 and IIS 8. Using this configuration we have yet to successfully pass through IIS client certificate validation, where we consistently receive an HTTP 403.16 error.
The service is consumed by a custom iOS app that we have verified is sending the client certificate, which was expected since my understanding of the error implies IIS is unable to validate the certificate that it received.
The Client Certificate Authority is installed in the Trusted Root Certification Authorities and the Client Authentication Issuers stores for the Local Computer.
Pretty much all of the resources we can find on this issue propose the solution here: http://social.technet.microsoft.com/Forums/en-US/fae724e8-628e-45a5-bf39-6e812d8a1a70/40316-problem-in-iss8-on-mp-in-dmz?forum=configmanagerdeployment
where it's suggested we add a registry setting for ClientAuthTrustMode. This has not resolved the issue for us; we also did not need to do it for any of our local testing which involved the exact same OS and IIS versions.
We've spent days on this with no progress and were hoping someone may have some insight into this issue. Is there any default configuration that we have yet to come across to enable this form of authentication for VMs in Azure? It seems as though IIS on a VM in Azure is unable to actually validate against the CA in Trusted Root. One idea I had was that perhaps the certificate is being stripped from the request before it's routed to IIS, but again, that seems unlikely given my understanding of the error code.
Has anyone gotten a setup like this to work?
I happened upon this knowledge base article, maybe it can help you
http://support.microsoft.com/kb/2802568
I am building a site that needs to call a 3rd party web service over https. The service requires an X509 certificate for authentication.
I can successfully call the service and get a response back using a console application, or a web application running in visual studio web server, this suggests to me that the code, and certificate provided by the 3rd party are all ok.
As soon as I deploy the site in IIS, it cannot connect to the service, I am getting "The request was aborted: Could not create SSL/TLS secure channel" error. I spent a couple of hours trying to figure this out, assuming it was permissions related as IIS app pool running under network service etc.
The certificate is installed in the local machine store, not current user, and I have granted network service (also added everyone is desperation!) using the winhttpcertcfg tool and manually in MMC > Manage Private Keys, so I am confident that the network service account has access to the certificate.
I also used the FindPrivateKey tool and checked the physical windows file and permissions look ok! I tried using identity impersonate in the config, using my windows account and that doesn't make a difference.
I am pulling my hair out with this one, can anyone give me any suggestions on what to try next?
Cheers