My .net application process will not stop running. I can use ANTS to profile it, but all examples talk about increasing memory and new instances. How do I find out what is preventing the application from exiting.
I have done snapshots while the application is running properly and then a second snapshot after it is "closed" but is still running as a process. What should I be looking for?
Related
I just started working on a legacy Spring Boot application, and I noticed it was not shutting down smoothly, required a hard kill to close it.
I did a thread dump and I see Spring is launching lots of threads that just don't want to die.
Ok, says I, must be the #Async's and #EnableAsync, we have a number of those to handle the initialization. I removed all of those, but no change.
Then I thought it might be Micrometer - we do a lot of instrumentation using #Timed, but I removed those and no change to the thread dump.
I searched for any instances of any kind of executor but nothing turns up.
What could be starting all of these threads and QuartzScheduler_Workers?
I am a newbie and trying to figure out how process monitoring works with JXcore. I saw the documentation but need few steps in order to make my server application starting multithreaded and monitored properly.
Thanks in advance!
I'll try to explain it to you. There is no shame to be a newbie! :)
JXcore offers you two types of application monitoring.
1) One of them is Process Monitor and this is a process, which runs as separate instance. Your applications may subscribe to it for being monitored. Monitor verifies them in regular intervals, and if it finds that your application is gone it tries to relaunch it. For example, if your application servers http and should be online all the time - Process Monitor will ensure, that it is really running.
The fastest way to start to monitor your application is to:
launch the monitor: > jx monitor start
launch your application with automatic subscription to the monitor: > jx monitor run app.js
After that, when your application crashes, Process Monitor will restart it. You can test it by just killing your application's process.
Process monitor also gives you information about currently monitored processes. You can browse to see the list of them:
http://127.0.0.1:17777/json
2) Second type of a monitoring feature is process and thread recovery. With Process Recovery you can achieve the same as with the Process Monitoring, so there is no reason to use them both at the same time.
Another scenario could be:
Let's say you have a multithreaded application and only to recovering it's threads is enough.
Your application is launched with a command:
jx mt-keep:3 app.js
which means, that you run it with 3 threads.
To enable Thread Recovery is enough to subscribe to process.on('restart') event like this:
process.on('restart', function (cb) {
process.release();
cb();
});
Remember, to call cb() callback. As you probably saw it in the docs, the thread will not restart until you invoke this callback. Before restart, you may back-up things etc.
Basically that's it. Feel free to play with it!
I have a large web app that runs on our two live servers. Part of our server side C# code calls a third party app to do a task for us.
That task works most of the time, but at a certain point it stops working until the AppPool is recycled.
This all happens in w3wp.exe, so I can see it running in process monitor like this (when it is not working),
Thread Create
Access the file PreviewGenerator.exe
Hive unloaded (this is the registry)
Thread Exit
And like this when it is working,
Thread Create
Access the file PreviewGenerator.exe
Process Start
Does heaps of stuff with PreviewGenerator.exe including reading / writing / registry, etc.
Process Exit
Hive unloaded
Thread Exit
How can I debug what is going on in my AppPool and why starting a separate process is not working some of the time?
I found the best thing to do was to create a separate app pool for my application in IIS and set an upper limit for the amount of RAM it could use. Also I found it useful to turn on the 'Generate Recycle Event Log Entry' items under the app pool settings.
You can then go to the system event log and filter out the items with a source of 'WAS' to understand what is going on in the app pools, when they are restarting and when they stop from being idle etc.
I think the main problem in our case is that the IIS box was running out of memory. Tuning the app pools and adding some extra RAM seems to have solved it.
I try to use the winDBG (adplus) to dump the w3wp process.
When I run this command adplus.vbs -hang -quiet -p ****, I found it create a folder with a big size file, and the size was growing. Then suddenly, the big size file disappeared and the process re-start again. Does anyone know about it?
Best Regards,
Yongwei,
Colin is right; in effect, you're racing against IIS as it is recylcing the application pool. As you're snapping your process snapshot, you're either hitting a memory-threshold for recycling, or health checks are perceiving the process to be hung and instituting a recycle (possibly due to ADPlus locking the process)
Here's how I would modify your application pool characteristics prior to attempting your next capture. You only need these changes in effect for as long as it takes to capture your dump:
Turn off memory-based recycling limits (physical and virtual)
Turn off the idle timeout limit (if it's on)
Disable both Pinging and Rapid Fail Protection
In effect: you need to turn off all of the features that try to keep your app pools running well. Capturing a memory snapshot takes time (as you know).
I would also recommend checking out ProcDump (http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx) from the SysInternals guys. It was just released last month, and it makes process memory captures a bit easier. An article on using it to capture the W3WP is here: http://blogs.msdn.com/webtopics/archive/2009/08/08/using-procdump-exe-to-monitor-w3wp-exe-for-cpu-spikes.aspx
I hope this helps!
I can only imagine the memory usage of the w3wp process got to much which triggered an app pool recycle, which means restarting w3wp.
I ran into an issue with an IIS web app shutting down an idle worker process! The next request would then have to re-initialize the application, leading to delays.
I disabled the IIS shutdown of idle worker processes on the application pool to resolve this. Are there any issues associated with turning this off? If the process is leaking memory, I imagine it is nice to recycle the process every now and then.
Are there any other benefits to having this process shutdown?
I'm assuming that you're referring to IIS 6.
Instead of disabling shutdown altogether, maybe you can just increase the amount of time it waits before killing the process. The server is essentially conserving resources - if your server can stand the resource allocation for a process that mostly sits around doing nothing, then there isn't any harm in letting it be.
As you mentioned, setting the auto-recycling of the process on a memory limit would be a good idea, if the possibility of a memory leak is there.