I am not sure if this is the case, but I think I have the following problem.
After scale in happens some requests gets 5XX response. I am not sure if 5XX appears because of scale in, but it appears so. And what I think happens is that after scale in rule was applied my app instance is shut down before some requests got appropriate response.
Could it be the case? In case yes, maybe somebody knows the solution?
I am using Azure.
Update: currently I am getting 502 Bad Gateway, but the strange thing is that App Service is not showing any failures
My guess it the failure happens in IIS and it does not even reach my app. Could it be the case?
Related
I am receiving more tan 10.000 bad request in Application Insights of Azure, I know some details of the transaction but I can not find a thread to pull, and understand why those head requests are getting a 404 error, how can I troubleshoot those requests?
The problem can be resolved by making a valid "/" path in you application that returns a 200 response. Alternatively look at #RithwikBojja-MT refs.
The reason for this is actually that Azure pings the root path of your applications if you have the "Always-On" feature enabled.
I have looked at the request trace for several requests that resulted in the same outcome.
What will happen is I'll get a HttpModule="iisnode", Notification="EXECUTE_REQUEST_HANDLER", HttpStatus=500, HttpReason="Internal Server Error", HttpSubstatus=1013, ErrorCode="The pipe has been ended. (0x6d)"
This is a production API. Fewer than 1% of requests get this result but it's not the requests themselves - I can reissue the same request and it'll work.
I log telemetry for every API request - basics on the way in, things like http status and execution time as the response is on its way out.
None of the requests that get this error are in telemetry which makes me think something is happening somewhere between IIS and iisnode.
If anyone has resolved this or has solid thoughts on how to pin down what the root issue is I'd appreciate it.
Well for me, what's described here covered the bulk of the issue: github.com/Azure/iisnode/issues/57 Setting keepAliveTimeout to 0 on the express server reduced the 500s significantly.
Once the majority of the "noise" was eliminated it was much easier to correlate the remaining 500s that would occur to things I could see in my logs. For example, I'm using a 3rd party node package to resize images and a couple of the "images" that were loaded into the system weren't images. Instead of gracefully throwing an exception, the package seems to exit out of the running node process. True story. So on Azure, it would get restarted, but while that was happening requests would get a 500 internal server error.
I am running the angular application in Microsoft Teams and deploying it on Azure.
But it's not running up as it is looking for the robots933456.txt route.
It wasn’t the case before today.
Please guide me on how to proceed further.
On running the app the logs shows:
The error was natively on my side nothing to do with the above but pretty much seen and raised by people so thought of sharing below:
After doing some research figured out I can safely ignore this message. /robots933456.txt is a dummy URL
the path that App Service uses to check if the container is capable of serving
requests. A 404 response simply indicates that the path doesn't exist, but
it lets App Service know that the container is healthy and ready to respond
to requests.
https://github.com/MicrosoftDocs/azure-docs/issues/32472
How is possibile to fallback to another API service if the first try fails with status code of 429 or 500?
Consider this situation:
The gateway should first try a microservice host, but if it timeouts or answer a non 2xx status code it should try a next one.
This is not supported in Express Gateway out of the box. If a request is failing, you'll get a failed request.
Right now you can specify multiple urls for a serviceEndpoint that will be used, but in a round robin manner; therefore you'll still get an error first, before trying again with the second service.
Express Gateway could — and maybe should — handle such case. In meantime I'd suggest you to look to other alternatives offered by your infrastructure. Consul could be the way to go.
We have a node dyno processing small API requests, ~10/second. All requests complete in under 0.5s
Once every few days, dyno starts giving H12 Request timeout errors on all requests. We couldn't discover the cause. Restarting fixes it.
How to make Heroku automatically restart the dyno on a H12 Request timeout threshold, e.g. more than 5/second?
As ryan said H12 Request timeout means that Heroku's load balancers are sending a request to your app but not getting a response in time (heroku has a max response time of 30 seconds). Sometimes a request is just intense to calculate or an inefficient DB query is delaying the response.
Yet the root of the problem does not necessary mean an application error on your side.
In our case we have multiple web dynos handling requests in parallel. Now and then one of those dynos produces H12 (timeouts) while all others are running flawless. So we can completely rule out all application problems. A restart of the affected dyno helps with a high probability, as your application lands on a different physical server whenever it is restarted (at least with a high probability).
So Heroku has "bad servers" in their rotation! And now and then your code will land on one of those bad servers. I cannot say if one has a "noisy neighbor" problem. I also asked Heroku how to prevent that and the only response that I got was to pay for dedicated performance dynos, which is quite dissatisfying...
H12 Request timeout means that Heroku's load balancers are sending a request to your app but not getting a response.
This can happen for lots of reasons, since the app is already working you can likely rule out configuration issues. So now you are looking at the application code and will have to inspect to the logs to understand whats happening. I'd suggest using one of their logging apps like papertrail so you can have a history of the logs when this happens.
Some things it could be, but not limited to:
Application crashing and not restarting
Application generating an error, but no response being sent
Application getting stuck in event loop preventing new request
Heroku provides some documentation around the issue that might help in debugging your situation
https://devcenter.heroku.com/articles/request-timeout
https://help.heroku.com/AXOSFIXN/why-am-i-getting-h12-request-timeout-errors-in-nodejs