I have ISS Server hosting Dash app.
What I am noticing in the browser is that app starts but doesn't return anything back. Looking at the HTTP logs, it shows HTTP 500 error with sc-win32-status 258. Timetaken value from HTTP logs is around 100seconds.
There are HTTP 200 with same time taken value...so that tells me its not only timeout issue. I didn't change the default timeout value in IIS (which I think is more than 5 mins?)
Trying to figure out what is causing this intermittent timeout issue.
Thanks!
The issue was in the timeout duration in IIS > FastCGI Settings > Process Model > Activity Timeout.
The operation was taking long time then it was set to communication with IIS.
Increasing the timeout to 180seconds seemed to fix the issue for me.
Related
We have a blue, green site and server farm set up for zero downtime deployment which works fine but we are seeing in the web logs for the site that does the routing that every now and then the wrong server is being sent requests.
2023-02-19 09:50:05 /DocumentView DID=23932&SERVER-ROUTED=LIVE-GREEN
2023-02-19 09:50:09 /FileDownload FID=50516&SERVER-ROUTED=LIVE-BLUE
2023-02-19 09:50:13 /Publish DocID=9154358&SERVER-ROUTED=LIVE-BLUE
2023-02-19 09:50:15 /SiteView DID=23932&SERVER-ROUTED=LIVE-GREEN
In this instance it tries to send two requests to Live-Blue which is set to Unhealthy and the site Stopped. The Health Test interval is set to 1 seconds (at the moment).
It's not happening a lot(maybe once every 25k request) but is very annoying to those that do get the 502 error message. (502.3 - Bad Gateway: Forwarder Connection Error (ARR).)
Anyone have any ideas on how to fix or diagnose the issue?
I have set up a server using IIS on Windows Service 2016 using python FastAPI with uvicorn. Sending individual querying works well, but I had been moving to testing the API with parallel queries using k6. I had been sending queries with 30 VU across 1min with a random sleep between, resulting in around 2.1 requests/sec. However I noticed that the service had been having sporadic 502.3 errors about 15% of the time.
The error codes tagged to it were: 12030 and 12152. According to https://learn.microsoft.com/en-us/windows/win32/winhttp/error-messages:
ERROR_WINHTTP_CONNECTION_ERROR
12030
The connection with the server has been reset or terminated, or an incompatible SSL protocol was encountered. For example, WinHTTP version 5.1 does not support SSL2 unless the client specifically enables it.
ERROR_WINHTTP_INVALID_SERVER_RESPONSE
12152
The server response cannot be parsed.
The failure percentage seems to scale with higher number of requests per second.
I had checked the httperr logs under C:\Windows\System32\LogFiles\HTTPERR, but only saw Timer_ConnectionIdle which I read to be not an issue.
How else can I troubleshoot these error 502.3 to see what the issue is?
UPDATE 2022/12/20:
Managed to get the FRT for one of the occurrences. How to proceed to troubleshoot? It does seem to be just indicate 502.3 error
Event: MODULE_SET_RESPONSE_ERROR_STATUS
ModuleName: httpPlatformHandler
Notification: EXECUTE_REQUEST_HANDLER
HttpStatus: 502
HttpReason: Bad Gateway
HttpSubStatus: 3
ErrorCode: 2147954552
Every day at about 3:00PM-4:00PM GMT the response times start to increase (no memory increase or CPU increase)
There is a azure availability test going to server every 10 minutes.
As this is a dev site there is no traffic to it other than me (at the odd time) and the availability test
I log to a variable internally the startup time and this shows that the site is not restarting
The first request via a browser when this starts happening is very slow (2 minutes - probably some timeout).
After that it runs perfectly. That seems like the site is shutting down and then starting up on first request, but the pings are keeping it alive so the site is not shutting down (as far as I know)
On the odd log entry I get - I seem to be getting 502 errors - but I can't confirm this as the FEEB logs are usually off at this time.
FREB logs turn off automatically after 1 hour and as this is the middle of the night for me (NZDT) - I don't get a chance to turn on.
See attached images - as you can see the response times just increase at same time
Ignore the requests where they are above 20 - thats me going to it via browser
I always check the azure dashboard BEFORE viewing site in browser
Just got this error (from web browser randomly - keep accessing the same page:
502: The specified CGI application encountered an error and the server terminated the process.
Other relevant Info (Perhaps):
I initially had the availability test ping going to a ping endpoint /ping that only returned a 200 and empty string when I noticed this happening
It now points to the sites homepage to see if it changed anything - still the same.
Assuming the database is not the issue as the /ping endpoint doesn't touch the database - just a straight controller return.
Internal Exception handling is catching nothing
Service: Azure Free Web App (Development)
There are no web jobs or timed events on this site
Azure Dashboard Initial
Current tests:
Uploading as new site to a Basic 1 Small
Restarting dev site 12 hours before issues (usually 20 hours before)
Results:
Restarting free web-app 12ish hours before issue - same result at same time - so its not the app slowly overloading or it would me much later
Basic 1 Small: no problems - could it be something with the dev server ?
Azure Dashboard From Today
Observations:
Same behavior with /ping endpoint (just return empty string 200 Ok) and Main home page endpoint (database lookups [w/caching] / razer)
If anyone has any ideas what might be going on - I would very much appreciate it
:-)
Update:
It seems to of stopped (on its own) about 11/1/2016 1:50:49 AM GMT - my internal timestamp says it restarted - and then the errors started again same time as usual. Note: no-one is using the app. The basic 1 Small Server is still going fine.
Sorry I can't add anymore images (not enough rep)
By default, web apps are unloaded if they are idle for some period of time, which could cause the web site slow response during this period of time. Besides, this article is about troubleshooting HTTP "502 Bad Gateway" error or a HTTP "503 Service Unavailable" error in Azure web apps, you could read it. And from the article we could know scaling the web app could mitigate the issue.
This week we have started getting 502 errors on our Web App, these are random; some times they happen when there is consistent load other times they happen with even a single request.
I have checked event viewer and there is no application crash, also don't have any really slow requests in IIS logs. I had auto-heal enabled, which is now disabled. I have also enabled auto scale and even with 4 instances running I get 502 error every once in a while. There is no log entry for this 502 in IIS logs, so I am guessing something upstream is returning this response, I just don't know why its doing that and why its so random.
This maybe related to platforms other than ColdFusion.
IIS 6 Log reports "time-taken" much longer (30 minutes) than 120 seconds set in Connection Timeout for several requests to ColdFusion page.
I assume that ColdFusion was unresponsive at the moment. I would like IIS to stop the request rather than wait this long.
Is there an IIS setting that would force this?
Not really because iis is no longer handling the request once it has been passed to cf. You could try playing with application pool timeout and see if you can get that to throw an error.
This scenario can also be considered as the slow HTTP DoS attack when caused by the client. IIS doesn't provide much protection against it (at least for slow POST body) because Microsoft considers it a protocol bug, not an IIS weakness. Although I think in this case it is your server doing it to itself.
Things to check:
You didn't mention whether it is the request that is slow or the
server's response. You could try tweaking your
MinFileBytesPerSec parameter if it's the response that is slow. By
default it will drop the connection if the client is downloading at
less than 240 bytes per second.
Remember, that 120 second IIS timeout is an idle timeout. As long as the client sends or receives a few bytes inside 120 seconds, that timer will keep getting reset.
You didn't mention if this long wait is happening on all pages or always in a few specific ones. It is possible that your CF script is making another external
connection, e.g. CFQUERY, which is not subject to CF timeouts, but to the timeouts
of the server it is connecting to. Using the timeout attribute inside CFQUERY may prevent this.
You also didn't mention what your Coldfusion settings are. Maybe the IIS timeout setting is being ignored by the Coldfusion
JRUN Connector ISAPI filter, so you should check the settings in
Coldfusion Administrator. Especially check if Timeout Requests
after has been changed. If its still at the default of 60
seconds, check your code to see if it has been overridden there, e.g.
<cfsetting requestTimeOut = "3600">
Finally, there is the matter of the peculiar behavior of CF's requestTimeout that you might have to workaround by replacing some cfscript tags with CFML.