I'm hosting my web application on azure web app by using suave and following the recommended guide.
My app was running like a charm until some days ago something went horribly wrong:
The process threw and exception and crashed, but HttpPlatformHandler failed to restart the process and the eventlog was full of the following errors:
Process '0' failed to start. Port = 30590, Error Code = '-2147023897'.
and
Maximum rapid fail count per minute of '10' exceeded.
I got this error for hours and in this time my application responded with
502: The specified CGI application encountered an error and the server terminated the process.
The only way to solve the problem was to manually restart the web application.
I don't understand why the platform handler was unable to restart the process.
A successful start looks like the following:
Process '5516' started successfully and is listening on port '29676'.
So what's up with this Process '0'?
Related
Got a bit of a weird issue occurring with an ASP.NET Web API (.Net Framework 4.8) application running on IIS 8.5.
The application will run responding to requests in < 1 sec and will suddenly spike to > 1 sec.
At the same time a large number of requests appear in the HTTPERR log with the error Request_Cancelled.
Failed request tracing shows the request is hitting 1 second in AspNetHttpHandlerEnter. This would lead me to think that the application itself is performing poorly, however, monitoring doesn't show any performance issues elsewhere.
Eventually these cancelled requests cause the DB connection pool to run out of connections.
Is there any way to get more info about what might be causing the Request_Cancelled errors in a Web API application?
I have a web application running as a service on an Ubuntu EC2 Instance. As of the past 24 hours, the application has been crashing randomly 2-4 hours after running with the message attached in the image below. The error is:
[nodemon] app crashed - waiting for file changes before starting...
I have run into this error before but usually, it is a syntax error and it will not allow me to actually start the application. In this case, the app functions normally for several hours before crashing. I have no idea where to even start as there's nothing above it that looks like it could be causing the crash. The only thing is it looks like the website receives 3 Get / Requests before the server can respond then it crashes. Most of the posts I've found online about this also block the application from running and don't mention the fact that the app runs normally then crashes.
Any help would be greatly appreciated.
Thanks!
Error Log from Journalctl
It looks like a silent error. I would try to log every input (e.g. http request and timeouts) with timestamp and also log the crash with time. When a crash occurs I would compare the time to events happening right before.
Also check your /var/log/ if the programm was terminated by the system or another programm.
I meet a problems when deploy my code to Azure Funtions using App Service Plan. My function is long running (about 4 minutes, for business logic) and when I call it by Post man, it response 502.
The specified CGI application encountered an error and the server
terminated the process.
In this time my function is still running, I monitor on the portal of Function Apps and it return a success response after 4 minutes, but I meet error and can not receive the response when call from client.
Response Message & Status Code
Response Header
I read below link and see that may be the limitation is from Azure App Service Plan, but how to config ASP to solve this error ?
Some information:
- Azure Functions V2.
- App Service Plan S1 Standard, 2 instances.
The specified CGI application encountered an error and the server terminated the process
I was having a similar issue, after debugging it turned out i had an unhandled exception inside a Parallel.Foreach Loop. The parallel loop did not have any try/catch within it, and as a result having an exception thrown bubbled up to the app domain killing and restarting the web app, with this same error showing:
"The specified CGI application encountered an error and the server terminated the process."
You should check all your asynchronous threads for error capturing
We have deployed a ASP.NET Core application on Windows Server 2012 R2. Everything is working except when sometimes the customers experience site crashes with 502.5 - Process Failure error. They have said it stays for 1-5 minutes.
From EventViewer I can see these errors:
Failed to start process with commandline '".\<appname>.exe" ', ErrorCode = '0x80070002'.
From the errors in EventViewer I can say that the error occurred twice this weak.
It seems to be a common problem that others also have experience it but I still don't know the reason why it is happening in my case.
We have this application deployed on some other machines and I have never seen that happens.
My guess is that the Rapid Fail Protection feature of the IIS triggers and restarts the pool. It is setup with the defaults - 5 crashed for 5 minutes. I have checked our logs for errors and saw that error a lot:
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[0]
An unhandled exception has occurred: The antiforgery token could not be decrypted.
System.InvalidOperationException: The antiforgery token could not be decrypted. ---> System.Security.Cryptography.CryptographicException: The key {259b6327-0ab1-4dff-95ed-48f568879fab} was not found in the key ring.
After following this https://learn.microsoft.com/en-us/aspnet/core/publishing/iis#create-a-data-protection-registry-hive and did some tests locally (where I also had that) the error is still there.
Questions:
Can this be the reason for 502.5 error ? Can I disabled (or extend the crashes count per minutes) the Rapid Fail Protected ? What will happen if I do?
according to the the IIS documentation the rapid fail protection once activated leads to the deactivation of an application pool if a "failure" occurs. However, I could not find the definition of the "failure" case. In my web application I have a special exception that I would like the IIS to consider it as a "failure".
Does anyone have an idea? Thanks
This appears to have a list, for Server 2003 at least: http://web.archive.org/web/20130511004652/http://technet.microsoft.com/en-us/library/cc787273(WS.10).aspx
The WWW service shuts down an application pool whenever a worker
process in the application pool fails often enough to equal or exceed
the Rapid-Fail Protection (RFP) interval time window (for example:
five failures in five minutes). The WWW service detects failure
whenever:
A worker process does not start within the startup time limit.
A worker process does not shut down within the shutdown time limit.
A worker process shuts itself down because of a fatal error and sends
the WWW service an error code.
A worker process fails to respond to a ping message.
The WWW service detects that a worker process is sending non-standard
communications (the worker process may have been taken over).
(updated with archive.org to fix broken link, and replicated detail here)
The documentation for configuring rapid fail protection alludes to a "failure" meaning a worker process crash.
Through experimentation I've noticed that you should expect something like the following in Windows Event Application Logs for a w3wp.exe crash:
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/1/ROOT
Process ID: 2628
Exception: System.SomeUnhandledException
Indeed with rapid fail protection enabled with the default configuration, 5 such events within 5 minutes of each other cause the application pool to stop, and you'll see a further Windows Event Application Log similar to:
Application pool 'my-test-application-pool' is being automatically
disabled due to a series of failures in the process(es) serving that
application pool.