well this has been bugging me for a couple of days on and off. I am at a clients site where they have a number of bespoke, written in house, services running on a Windows 2008R2 IIS server. The problem is that a couple of these services keep hanging, they are stuck in a “Stopping” state and the only way to kill them off is to open process explorer and kill the threads. Before anyone says anything about using ‘runas’, or logging on as the local admin, or the service owner, etc we’ve been through all of that.
The problem lies with the executable itselfs. The development team, in another country are going to look at this but it will take 4-5 months minimum, and we’re not certain they’ll get it right then.
I have a Powershell script to check the services on a regular basis which has the ability to ensure the services are running and if not, the force a stop and restart of the service, then it sends an email to confirm the actions. However with these specific services mentioned it can do nothing. They can’t be killed in task manager, taskkill, or process explorer (unless one kills the threads) it just says access denied. It is possible to change the permissions in process explorer and kill it but that’s a lengthier process than killing the threads.
To make things a little more difficult I can’t use the process name as on this server there are two other websites using an exe with the same name, just in a different folder.
What I’m after is a way to find and kill the threads of a processID, which I’ve already obtained via the script I have, so the rest of the script can complete the task of restarting the said service. At the moment this service dies on an inconsistent basis throughout the day and night, and the support guys have to RDP onto the server, open process explorer, find the offending process and kill the threads off then restart the services. A bit too much hassle for these already over worked guys especially if we can get powershell to do it automatically.
Hope someone can help on this. Thanks in advance.
Low level thread handling is likely to require native Win32 API usage. Powershell might help with P/Invoke, but the process is going to be complex. For starters, find out if the following tools can be used to identify the stuck thread. Maybe you can combine this info with some Sysinternals tools like handle.exe to find out what really blocks the thread.
The .Net framework has some tools available via System.Diagnostics.Process namespace. A list for threads for named process is available like so,
$ps = [diagnostics.process]::getProcessesByName("iexplore")
$p = $ps[0]
$p.Threads[0]
Full documentation is in MSDN. There is no method for killing a thread, but this should be kind of starting point for identifying the stuck one.
Another a way is to use WMI to get win32_thread data like so,
$threads = gwmi win32_thread
The output is quite different and some filtering is needed. Some examples are available. Another a WMI solution attempt might be based on Win32_process that has Terminate method.
Related
Are there any limitations on creating separate processes from an Azure Web Site (specifically, from a continuous Web Job)? I have an executable that often (about %20 of the time) stalls and eventually fails with exit code -1073741819 (access denied? or access violation?), but only when run as a separate process. If this work is retried later, it eventually succeeds (usually on the first retry).
When instead I call this logic directly via a .NET method call (so within the same process and app domain), the code succeeds 100% of the time. The same code also always succeeds when run locally, even when it creates a separate process.
Is there anything going on at the Azure Web Sites/Web Jobs level that I should be aware of, such as using Windows job objects or other security mechanisms to limit the creation or runtime of spawned processes? If not, any suggestions on how to diagnose further what might be going wrong? (I believe remote desktop to a web site isn't possible; anything else that would help "see" what's failing, such as whether there's a WER dialog appearing?)
In case it matters, the logic (in both cases) includes P/Invoking custom native code, and the web site I'm using is Always On, x64, Basic pricing tier.
#David Ebbo, thanks for the suggestion. I used it to help isolate, and I ultimately found this was non-determinism in the code made more likely in the Azure Web Sites environment but not 100% restricted to that context.
I'm in the process of designing an application that will run on a headless Windows CE 6.0 device. The idea is to make an application that will be started at startup and run until powered off. (Basically it will look like a service, but an application is easier to debug without the complete hassle to stop/deploy/start/attach to process procedure)
My concern is what will happen during development. If I debug/deploy the application I see no way of closing it in a friendly and easy way. (Feel free to suggest if this can be done in a better/user friendly way) I will just stop the debugger and the result will be WSACleanup is not called.
Now, the question. What is the consequence of not calling WSACleanup? Will I be able to start and run the winsock application again using the debugger? Or will there be a resource leak preventing me to do so?
Thanks in advance,
Jef
I think that Harry Johnston comment is correct.
Even if your application has no UI you can find a way to close it gracefully. I suppose that you have one or more threads in loops, you can add a named manual reset event that is checked (or can be used for waits instead of Sleep()) inside the loop condition and build a small application that opens the event using the same name, sets it and quits. This would force also your service app to close.
It may not be needed for debugging, but it may be useful also if you'll need to update your software and this requires that your main service is not running.
I have a Windows Azure Web Site that recently got temporarily suspended due to CPU usage quotas. That's fine, I'd normally let it reset and continue on. But CPU usage has been extremely high for the last 12 hours now and doesn't seem to be stopping despite the site being suspended. Because it is suspended, there's been no data out, no incoming requests to process, nothing. So what the heck is suddenly using all of this CPU power? This site has been running just fine, well under quota, for months without any deployments/code changes.
In lieu of determining the cause of the high CPU usage, I'm more concerned with simply getting it to stop now. As it is, every time the CPU quota resets it immediately gets suspended again since usage is still so ridiculously high.
Is there some way I can kill the process/site? The Stop/Restart buttons are missing from the Azure Management portal (I'm guessing because it is suspended), but despite being suspended for so long now it is still consistently eating up CPU. (On the activity graph, CPU usage spiked sometime yesterday afternoon and has been plateaued since.)
This is very odd, and not what I would expect to see happen. You have a few options to try:
Attempt to use PowerShell to stop the web site. You can download the Windows Azure PowerShell Cmdlets at http://www.windowsazure.com/en-us/downloads/. The Stop-AzureWebsite cmdlet is what you'll want to use. If you are unfamiliar with the PowerShell cmdlets there are some examples and info on Microsoft's website.
Put in a support ticket if you have a support plan. If you don't have a support plan you can also put in a post on the MSDN Forums which is seen by support folks more than they are here I think. There is no SLA or such with the free accounts, but they see these and do investigate. Just might take some time.
If you for sure want to stop it one option is to attempt to delete it if that option is still available to you in the portal. If you don't see this in the portal either then there is a PowerShell CmdLet Remove-AzureWebsite which does this operation. This will also take out the code and data for the site unless that data is persisted outside of the web site environment (like to a database). This might be your last resort and hopefully you have the content of the site backed up or as part of source control somewhere. If not, attempt to get to it using FTP.
Add an app_offline.htm to the root of the site it will cause iis to stop the application.
http://weblogs.asp.net/scottgu/archive/2005/10/06/426755.aspx
It does sound like you have a piece of code in a loop though are you using threads in the site?
Edit
As I have never run into this situation I don't know if your ftp actions are also blocked
Can anyone tell me, is there a way to run a process in IIS shared hosting service.
Suppose, the scenario is like "I want to send emails to a list of email id's after everywhere 3 hrs", so the challenge here is the process should not be invoked by a HTTP link. It should be automatic.
I think we can do this by IIS worker processes.
Also this all will be happening on a shared server(like GoDaddy) in IIS7, .NET 3.5
Please anyone give me a direction.
Thanks in advance.
This question was asked ages ago, but for what it's worth - I ended up using Hangfire to handle my long running tasks in my ASP app.
You can easily configure it for shared hosting and then for a dedicated server if you can scale up / out according to your needs.
It's super easy to use, just follow the doc step by step.
Cheers,
Andrew
You should write and run this as a Windows service, assuming you have access to install a service.
You could run a background worker thread from your asp.net code-behind but the problem is that IIS will terminate the thread after it is idle for a relatively short period of time. I tried going this route, trying to geocode a list of addresses (800+, from a SharePoint list) and IIS kept timing out my thread and stopping it. We ended up going with adding events to the SharePoint list that would geocode when the item was changed/added to the list.
One other option you could look into is using Windows Workflow API, it was designed for this kind of thing.
I am trying to profile a memory guzzling biztalk orchestration. This orchestration is using plenty of maps with a few custom script functoids.
I wish to identify the lifetime of the custom objects created by orchestration and how are they promoted in the Garbagage collection. Am not able to use the CLR profiler for the same.
Looking out for pointers to identify the objects that are getting promoted in the Garbage collection.
Hey Pari, here are a couple of things to try. The first, I have personally used the Ants Profiler on BizTalk processes before, here's a link:
http://www.red-gate.com/supportcenter/Content.aspx?p=ANTS%20Profiler&c=knowledgebase%5CANTS%5FProfiler%5CKB200801000222.htm
Second, you can also do some spelunking with the Windows Perfmon, but it won't be as granular as the object level, but it will tell you if you are having troubles with promotions, the large heap, etc.
You should be able to get this information using Perfmon in Windows. The basic ones you are probably after are in the .NET CLR Memory object. Unfortunately you will have to add them for all of the BizTalk processes, as the naming convention does not allow you to see the host name here. You will see them listed here, and in the log, as BTSNTSvc, BTSNTSvc#1, BTSNTSvc#2, etc. It will take a little extra work to identify which one is the process you are interested in.
The counter that will allow you to identify the correct process is the the Process\ID Process counter, again for each BizTalk process. This will allow you to connect the PID from the process when it starts, to the PID in the Perfmon logs later.
The last step is to create a new host, if you haven't already, and isolate the orchestration to it. That way it is the only thing running in that BizTalk process. After that you can open up Windows Task Manager and view all of the BTSNTSvc.exe processes that are running. Start with the new process off, check the PIDs listed in Task Manager, and then turn on the new host. The new PID is the one assigned to the host you just turned on. Record the PID that the new process is generated with and use it to identify which process in the Permon logs you are interested in. Unfortunately you will have to repeat this step every time you want to take measurements.
One last thing to mention, when you turn your Perfmon log on it will only record the hosts that are on at the time. So you will want to turn it on after you turn the host with the orchestration on.
You may want to also check out the objects in the Biztalk:MessageAgent, as there are some good memory counters in there for BizTalk with the actual names associated with them. It's not as granular as what you are looking for.
I have also heard of an orchestration profiler, but I have never used it. You might give it a shot:
http://www.codeplex.com/BiztalkOrcProfiler