I have set up a server using IIS on Windows Service 2016 using python FastAPI with uvicorn. Sending individual querying works well, but I had been moving to testing the API with parallel queries using k6. I had been sending queries with 30 VU across 1min with a random sleep between, resulting in around 2.1 requests/sec. However I noticed that the service had been having sporadic 502.3 errors about 15% of the time.
The error codes tagged to it were: 12030 and 12152. According to https://learn.microsoft.com/en-us/windows/win32/winhttp/error-messages:
ERROR_WINHTTP_CONNECTION_ERROR
12030
The connection with the server has been reset or terminated, or an incompatible SSL protocol was encountered. For example, WinHTTP version 5.1 does not support SSL2 unless the client specifically enables it.
ERROR_WINHTTP_INVALID_SERVER_RESPONSE
12152
The server response cannot be parsed.
The failure percentage seems to scale with higher number of requests per second.
I had checked the httperr logs under C:\Windows\System32\LogFiles\HTTPERR, but only saw Timer_ConnectionIdle which I read to be not an issue.
How else can I troubleshoot these error 502.3 to see what the issue is?
UPDATE 2022/12/20:
Managed to get the FRT for one of the occurrences. How to proceed to troubleshoot? It does seem to be just indicate 502.3 error
Event: MODULE_SET_RESPONSE_ERROR_STATUS
ModuleName: httpPlatformHandler
Notification: EXECUTE_REQUEST_HANDLER
HttpStatus: 502
HttpReason: Bad Gateway
HttpSubStatus: 3
ErrorCode: 2147954552
Related
The errors are as shown in the image... I followed the guide and downloaded the newest tools, but the error still persists... I'm on school wifi right now, and since I'm new to this, I have no idea on how to change Azure's environment variables on an existing project... I cannot sign in in the first place. I was just following through the guide :(
Please check the below steps if they helps to work around:
Error in Azure portal is HTTP response code 503 Service Unavailable
This situation happens due to network connectivity or service available issues.
The better approach is to retry the operation and if the issue persists, contact Azure Support as referenced here.
Alternative ways to solve this error is finding the cause by navigating to Diagnose and solve problems to know the root cause of 503 error as there can be multiple reasons for this error.
Please check the below causes related to 503 error issue if they helps to work around:
request taking a long time
application crashing due to an exception.
average response time is long
Function App is also an app service so app service enforces limits on the number of outbound connections
Error in Browser / Postman is 502 - Web server received an invalid response while acting as a gateway or proxy server.
As this 502 error, you were addressed in MSFT Q&A.
Normally, 502 error occurs when HTTP is placed instead of HTTPS in the connection but I know that the Azure Functions endpoint look like in this format
http://<APP_NAME>.azurewebsites.net/api/<FUNCTION_NAME>
And 502 error occurs when maximum timeout exceeds the value. Please check the timeout value of the function app and the logs, metrics of requestTime, responseTime in Application Insights and if it is the cause, increase the timeout value.
References:
Troubleshooting Reason for a 502 Error
How do I fix this 502 Error on my Azure Function?
Got a bit of a weird issue occurring with an ASP.NET Web API (.Net Framework 4.8) application running on IIS 8.5.
The application will run responding to requests in < 1 sec and will suddenly spike to > 1 sec.
At the same time a large number of requests appear in the HTTPERR log with the error Request_Cancelled.
Failed request tracing shows the request is hitting 1 second in AspNetHttpHandlerEnter. This would lead me to think that the application itself is performing poorly, however, monitoring doesn't show any performance issues elsewhere.
Eventually these cancelled requests cause the DB connection pool to run out of connections.
Is there any way to get more info about what might be causing the Request_Cancelled errors in a Web API application?
Server 1 is sending an xml message via IIS to Server 2.
Server 2 receives it, and send back an acknowledgment message to Server 1.
Upon receipt of that message, Server 1 sends the next message in the queue.
However, Server 1 intermittently (4/5 times a week) does not receive the acknowledgment message (we tested the issue and proved that Server 1 is sending the acknowledgment message).
The IIS logs for the time is is occurring tells us there's an error 1236 (sc-win32-status 1236 - which means "The network connection was aborted by the local system").
We're at a loss as to what is causing this or how to fix it. Interested to see if anyone has come across an issue like this before...
How did you prove that Server 2 is sending the acknowledgement message -- through network tracing on Server 1, or some other means? Logs within the software may not be enough. Barring anything bad going on at the networking level, it is possible that one of the sides is having an exception, and aborting the connection as a result. The application pools may be auto-recycling due to IIS recycle rules, and although IIS should properly handle it a pool re-start, maybe something did not occur as expected. When one pool starts, and the other one is processing the final requests on shutdown, maybe there is some locking going on, not expecting two processes running at the same time.
Every day at about 3:00PM-4:00PM GMT the response times start to increase (no memory increase or CPU increase)
There is a azure availability test going to server every 10 minutes.
As this is a dev site there is no traffic to it other than me (at the odd time) and the availability test
I log to a variable internally the startup time and this shows that the site is not restarting
The first request via a browser when this starts happening is very slow (2 minutes - probably some timeout).
After that it runs perfectly. That seems like the site is shutting down and then starting up on first request, but the pings are keeping it alive so the site is not shutting down (as far as I know)
On the odd log entry I get - I seem to be getting 502 errors - but I can't confirm this as the FEEB logs are usually off at this time.
FREB logs turn off automatically after 1 hour and as this is the middle of the night for me (NZDT) - I don't get a chance to turn on.
See attached images - as you can see the response times just increase at same time
Ignore the requests where they are above 20 - thats me going to it via browser
I always check the azure dashboard BEFORE viewing site in browser
Just got this error (from web browser randomly - keep accessing the same page:
502: The specified CGI application encountered an error and the server terminated the process.
Other relevant Info (Perhaps):
I initially had the availability test ping going to a ping endpoint /ping that only returned a 200 and empty string when I noticed this happening
It now points to the sites homepage to see if it changed anything - still the same.
Assuming the database is not the issue as the /ping endpoint doesn't touch the database - just a straight controller return.
Internal Exception handling is catching nothing
Service: Azure Free Web App (Development)
There are no web jobs or timed events on this site
Azure Dashboard Initial
Current tests:
Uploading as new site to a Basic 1 Small
Restarting dev site 12 hours before issues (usually 20 hours before)
Results:
Restarting free web-app 12ish hours before issue - same result at same time - so its not the app slowly overloading or it would me much later
Basic 1 Small: no problems - could it be something with the dev server ?
Azure Dashboard From Today
Observations:
Same behavior with /ping endpoint (just return empty string 200 Ok) and Main home page endpoint (database lookups [w/caching] / razer)
If anyone has any ideas what might be going on - I would very much appreciate it
:-)
Update:
It seems to of stopped (on its own) about 11/1/2016 1:50:49 AM GMT - my internal timestamp says it restarted - and then the errors started again same time as usual. Note: no-one is using the app. The basic 1 Small Server is still going fine.
Sorry I can't add anymore images (not enough rep)
By default, web apps are unloaded if they are idle for some period of time, which could cause the web site slow response during this period of time. Besides, this article is about troubleshooting HTTP "502 Bad Gateway" error or a HTTP "503 Service Unavailable" error in Azure web apps, you could read it. And from the article we could know scaling the web app could mitigate the issue.
Background / Issue
Having a strange issue running a Ghost blog on Azure. The site seems to run fine for a while, but every once in a while, I'll receive a 500 error with no further information. The next request always appears to succeed (in tests so far).
The error seems to happen after a period of inactivity. Since I'm currently just getting set up, I'm utilizing an Azure "Free" instance, so I'm wondering if some sort of resource conservation is causing it behind the scenes (which will be allevaited when I upgrade).
Any idea what could be causing this issue? I'm sort of at a loss for where to start since the logs don't necessarily help me in this case. I'm new to NodeJS (and nodeJS on Azure) and since this is my first foray, any tips/tricks on where to look would be helpful as well.
Some specific questions:
When receiving an error like this, is there anywhere I can go to see any output, or is it pretty much guaranteed that Node actually didn't output something?
On Azure free instances, does some sort of resource conservation take place which might cause the app to be shut down (and thus for me to see these errors only after a period of inactivity)?
The Full Error
The full text of the error is below (I've turned debugging on for this reason):
iisnode encountered an error when processing the request.
HRESULT: 0x2
HTTP status: 500
HTTP reason: Internal Server Error
You are receiving this HTTP 200 response because system.webServer/iisnode/#devErrorsEnabled configuration setting is 'true'.
In addition to the log of stdout and stderr of the node.exe process, consider using debugging and ETW traces to further diagnose the problem.
The node.exe process has not written any information to stderr or iisnode was unable to capture this information. Frequent reason is that the iisnode module is unable to create a log file to capture stdout and stderr output from node.exe. Please check that the identity of the IIS application pool running the node.js application has read and write access permissions to the directory on the server where the node.js application is located. Alternatively you can disable logging by setting system.webServer/iisnode/#loggingEnabled element of web.config to 'false'.
I think it might be something in the Azure web config rather than Ghost itself. So look for logs based on that because Ghost is not throwing that error. I found this question that might help you out:
How to debug Azure 500 internal server error
Good luck!