We have a web application that runs on 6 web servers with HAProxy as the load balancer. We use web deploy to sync our IIS and application across all web servers. Starting January some of customers starting reporting application slow downs. After a lot of work we found that request coming to IIS at random times get stuck in BeingRequest state of IIS Web Core. I am attaching a screenshot from one of my server. Any insight into the issue will be really appreciated.
Thanks,
Fahad
I have a similar issue but my requests are stuck in a different part of IIS. If anyone has any input there you may also find it useful - Debugging requests which are 'stuck' in an IIS worker process
In the screenshot you give your oldest request is 16s old - do they stay in forever or are they just very slow to process? If they don't process is the oldest request in the queue always exactly the same URL and if so can you trigger the issue with that URL?
If they do eventually process (or even if they don't) a good first step for you would be to run Failed Request Tracing/Logging - you can configure it directly in inetmgr at site level. Looking at the compact output will give you an overview of if your requests are being sent on loops around IIS or if there is anything triggering that you wouldn't expect in the life of the request.
If they do eventually process also look into utilisation exhaustion - maybe IIS is just crawling as it's struggling for CPU/RAM/IO - check out the usual suspects.
Related
I'm currently experiencing some hangs on production environment, and after some investigation I'm seeing a lot of request queued up in the worker process of the Application Pool. The common thing is that every request that is queued for a long time is a web api request, I'm using both MVC and Web API in the app.
The requests are being queued for about 3 hours, when the application pool is recycled they immediately start queueing up.
They are all in ExecuteRequestHandler state
Any ideas for where should I continue digging?
Your requests can be stalled for a number of reasons:
they are waiting on I/O operation e.g database, web service call
they are looping or performing operations on a large data set
cpu intensive operations
some combination of the above
In order to find out what your requests are doing, start by getting the urls of the requests taking a long time.
You can do this in the cmd line as follows
c:\windows\system32\inetsrv\appcmd list requests
If its not obvious from the urls and looking at the code, you need to do a process dump of the w3wp.exe on the server. Once you have a process dump you will need to load it into windbg in order to analyze what's taking up all the cpu cycles. Covering off windbg is pretty big, but here's briefly what you need to do:
load the SOS dll (managed debug extension)
call the !runaway command
to get list of long running threads dive into a long running thread
by selecting it and calling !clrstack command
There are many blogs on using windbg. Here is one example. A great resource on analyzing these types of issues is Tess Ferrandez's blog.
This is a really long shot without having first hand access to your system but try and check the Handler mappings in IIS Manager gui for your WebApi. Compare it with IIS settings of your DEV or any other Env where it works.
IF this isnt the issue then do a comparison of all other IIS settings for that App.
Good luck.
I deployed node js server to Azure WebApp, and it worked fine. But, I see that sometime the response time is very slow. Also, I see that somewhere above 500 request/second the server start to fail handling request, and I see it use only 15% CPU. Now, I checked and the server return 500 error because the pipe is busy (by the win32 error code). That's why I was wondering if there is something I can change in the IISNode config to improve the server request capacity.
I already enabled the AlwaysOn feature, and also I add a check in Pingdom to keep the site alive. Also, I already changed nodeProcessCountPerApplication to 0 so it use all the available process.
Thank you,
Omer
One thing you can do is enable Always On. Without it, when your site hasn't been visited for 20 minutes the site gets taken down. Then the next time someone makes a request to your site Azure Web Apps warms up (re-sets up) your site but this process takes a few seconds.
Note that Always On is only available for sites in Basic, Standard, or Premium SKUs
Also, check out this page for tips on debugging Node.js apps in Azure Web Apps: https://azure.microsoft.com/en-us/documentation/articles/web-sites-nodejs-debug/
I have a Website hosted on Windows Azure. This website is a custom ASP.NET MVC 4 site hosted as a shared web site instance. Within the past couple of days, I've started to get large spikes in CPU Time. These spikes have been sustained and have caused my web site to get suspended. However, I'm not sure how to determine the cause of these spikes. Here is what I've done so far:
I attempted to look at the diagnostics via the Windows Azure FTP drop. I did not see anything there.
I reviewed my Google Analytics to see if there was anything out of the ordinary. The site had 20 visitors yesterday. So nothing crazy.
How can I identify the culprit of the the CPU spike? Once it spikes, it just sits there for hours. I'm not sure what would cause this.
Thank you
Have you tried running your site on your local box and simulating your visitor traffic, exercising all your website's features?
Testing locally is 1000's of times easier and more revealing than trying to debug a site that's running live.
If you still can't find anything wrong when running locally, consider using logging and tracing to strategic points in your site so that you can see how often, and how long it takes for your site to execute complex operations.
I have a web application that's consuming a WCF service. Both are slow on warmup after IIS reset or app pool recycle. So, as a possiible solution I installed Application Warm-Up for IIS 7.5 and set it up for both web site and wcf service.
My concern is, it doesn't seem to make any difference - first time I hit the site it still takes long time to bring it up. I checked event logs, there are no errors. So I'm wondering if anything special needs to be done for that module to work.
In IIS manager, when you go into the site, then into Application Warm-Up, the right-hand side has an "Actions" pane. I think you need the following two things:
Click Add Request and add at least one URL, e.g. /YourService.svc
Click Settings, and check "Start Application Pool 'your pool' when service started"
Do you have both of these? If you don't have the second setting checked, then I think the warmup won't happen until a user hits the site (which probably defeats the purpose of the warmup module in your case).
There is a new module from Microsoft that is part of IIS 8.0 that supercedes the previous warm-up module. This Application Initialization Module for IIS 7.5 is available a separate download.
The module will create a warm-up phase where you can specify a number of requests that must complete before the server starts accepting requests. Most importantly it will provide overlapping processes so that the user will not be served by the newly started process before it is ready.
I have answered a similar question with more details at How to warm up an ASP.NET MVC application on IIS 7.5?.
After you have fixed possible software/code optimizations allow me to suggest that each and evey code needs processing via hardware cpu. And our server skyrocketed in performance when we went to a multicore cpu and installed more GIGS of ram and connected UTP-6 cable insetad of standard UTP 5e cable onto the server... That doesnt fix your problem but if you are obsessed with speed as much as us, then you will be interested in the various dimensions that bottleneck speed.
Can anyone tell me, is there a way to run a process in IIS shared hosting service.
Suppose, the scenario is like "I want to send emails to a list of email id's after everywhere 3 hrs", so the challenge here is the process should not be invoked by a HTTP link. It should be automatic.
I think we can do this by IIS worker processes.
Also this all will be happening on a shared server(like GoDaddy) in IIS7, .NET 3.5
Please anyone give me a direction.
Thanks in advance.
This question was asked ages ago, but for what it's worth - I ended up using Hangfire to handle my long running tasks in my ASP app.
You can easily configure it for shared hosting and then for a dedicated server if you can scale up / out according to your needs.
It's super easy to use, just follow the doc step by step.
Cheers,
Andrew
You should write and run this as a Windows service, assuming you have access to install a service.
You could run a background worker thread from your asp.net code-behind but the problem is that IIS will terminate the thread after it is idle for a relatively short period of time. I tried going this route, trying to geocode a list of addresses (800+, from a SharePoint list) and IIS kept timing out my thread and stopping it. We ended up going with adding events to the SharePoint list that would geocode when the item was changed/added to the list.
One other option you could look into is using Windows Workflow API, it was designed for this kind of thing.