I have a fairly simple API environment running on an IIS site. The site is mostly configured with defaults and serving on average 4-6 request per second. All it does is take in information passed in through GET request parameters and stores it in a database.
As I said the site has an average load of 4-6 requests per second and right now crashes pretty often. I'm trying to reconfigure the site to handle more of a load, but I'm not hugely familiar with IIS configuration. Does anyone have advice on how to debug this issue or how to better configure IIS for a session-less API environment? I can confirm that the default logging doesn't give me any useful errors, but if desired I can post some here.
Edit:
This is the standard error I see in the Application Event Log in regards to the error
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec4aa8e
Exception code: 0xc0000374
Fault offset: 0x00000000000c40f2
Faulting process id: 0xbc0
Faulting application start time: 0x01cddcf5147d18cc
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 5256ddd1-48e8-11e2-9c4a-0050560f015f
So far we have setup load balancing since the original post to help us out, but still see the error happening.
Related
our Api app is in UAT on Azure with service plan (Standard 3 large). What should we do if App Availability is Zero. It is getting slow response or timeout issue. When i restart the application it is up to normal. (We are using Parallel Language programming.(Async/Await)
How to find the route cause from it for slowness issue.
Ensure that Always On feature is enabled.
Such problems may be caused by application level issues, such as:
network requests taking a long time
application code or database queries being inefficient
application using high memory/CPU
application crashing due to an exception
You could enable web server diagnostics to fetch more details on the issue.
Detailed Error Logging - Detailed error information for HTTP status codes that indicate a failure (status code 400 or greater). This may contain information that can help determine why the server returned the error code.
Failed Request Tracing - Detailed information on failed requests, including a trace of the IIS components used to process the request and the time taken in each component. This can be useful if you are attempting to improve web app performance or isolate what is causing a specific HTTP error.
Web Server Logging - Information about HTTP transactions using the W3C extended log file format. This is useful when determining overall web app metrics, such as the number of requests handled or how many requests are from a specific IP address.
Also, Azure Application Insights collects telemetry from your application to help analyze its operation and performance. You can use this information to identify problems that may be occurring or to identify improvements to the application that would most impact users. This tutorial takes you through the process of analyzing the performance of both the server components of your application and the perspective of the client: https://learn.microsoft.com/en-us/azure/application-insights/app-insights-tutorial-performance
Ref: https://learn.microsoft.com/en-us/azure/app-service/app-service-web-troubleshoot-performance-degradation
I'm hosting my web application on azure web app by using suave and following the recommended guide.
My app was running like a charm until some days ago something went horribly wrong:
The process threw and exception and crashed, but HttpPlatformHandler failed to restart the process and the eventlog was full of the following errors:
Process '0' failed to start. Port = 30590, Error Code = '-2147023897'.
and
Maximum rapid fail count per minute of '10' exceeded.
I got this error for hours and in this time my application responded with
502: The specified CGI application encountered an error and the server terminated the process.
The only way to solve the problem was to manually restart the web application.
I don't understand why the platform handler was unable to restart the process.
A successful start looks like the following:
Process '5516' started successfully and is listening on port '29676'.
So what's up with this Process '0'?
according to the the IIS documentation the rapid fail protection once activated leads to the deactivation of an application pool if a "failure" occurs. However, I could not find the definition of the "failure" case. In my web application I have a special exception that I would like the IIS to consider it as a "failure".
Does anyone have an idea? Thanks
This appears to have a list, for Server 2003 at least: http://web.archive.org/web/20130511004652/http://technet.microsoft.com/en-us/library/cc787273(WS.10).aspx
The WWW service shuts down an application pool whenever a worker
process in the application pool fails often enough to equal or exceed
the Rapid-Fail Protection (RFP) interval time window (for example:
five failures in five minutes). The WWW service detects failure
whenever:
A worker process does not start within the startup time limit.
A worker process does not shut down within the shutdown time limit.
A worker process shuts itself down because of a fatal error and sends
the WWW service an error code.
A worker process fails to respond to a ping message.
The WWW service detects that a worker process is sending non-standard
communications (the worker process may have been taken over).
(updated with archive.org to fix broken link, and replicated detail here)
The documentation for configuring rapid fail protection alludes to a "failure" meaning a worker process crash.
Through experimentation I've noticed that you should expect something like the following in Windows Event Application Logs for a w3wp.exe crash:
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/1/ROOT
Process ID: 2628
Exception: System.SomeUnhandledException
Indeed with rapid fail protection enabled with the default configuration, 5 such events within 5 minutes of each other cause the application pool to stop, and you'll see a further Windows Event Application Log similar to:
Application pool 'my-test-application-pool' is being automatically
disabled due to a series of failures in the process(es) serving that
application pool.
When I run my WorkerRole C# application on Azure, after a while waworkerhost.exe crashes due the following exception:
Application: WaWorkerHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Runtime.CallbackException
Stack:
at System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
I have an application that generates load to a webserver. I don't care about the actual response, but i want to control the number of requests made per second.
Therefore i have a Timer that fires every second and generates a number of requests. I have tried the following options:
Parallel.For with WebRequests
For loop with ASync WebRequests
For loop with ThreadPool.QueueUserWorkItem(do
webrequest)
When the number of requests increase, the exception occurs (8+ req/sec). The same exception for all three options. When I run the role in local DevelopmentFabric all three options work just fine. If someone could give me some pointers on what might be going wrong I appreciate it. If you have other ideas to generate this type of load from Azure and C#, please share your thoughts.
The author answered the question in the comment to the original post, but for better visibility, I'm reporting it to here:
Turn out to be an IntelliTrace issue, see
http://social.msdn.microsoft.com/Forums/en-ZA/windowsazuretroubleshooting/thread/543da280-2e5c-4e1a-b416-9999c7a9b841:
...
After redeploying my solution with Intellitrace disabled, the issues
where resolved, and my WorkerRole stayed healthy.
Have a lot of troubles on production server. Some routing cause crashing of Application Pool with event id 1011:
Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1011
Date: 1/21/2009
Time: 9:08:17 AM
User: N/A
Computer: xxxxxxxxxxxxx
Description:
A process serving application pool 'DefaultAppPool' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '3788'. The data field contains the error number.
8007006d
I have a few very hard hours for me before I found a problem.
Thanks to Tess Ferrandez and her blog post I found it.
Always double check Your multithreaded code in asp.net application. When Unhandled exceptions occurs application pool crashes and it's damn hard to find WHY.
Tess's blog was a little advanced for me. I had to search around for quite a bit before I found the right articles that helped me debug my dump files. This article will help others who want to debug their crashing asp.net application pools but don't know how to start.