I'm not asking how to find the cause of the crash. Actually there is no crash at all. I can't exclude the possibility of memory leaking but the executable passed Valgrind analysis in the stress testing. However, when it's running in the cloud, with much load, it gradually consumed much memory. Devop had to use kill -6 pid to kill the process and generated a core dump file, then restarted it. With that core dump, what are the good tools and technics to help you locate which part of the code contributed to the very high memory consumption? Thanks!
Related
If we want to debug memory-related issues of a process then we have to start the process using Valgrind. Are there any other tools using which we can analyze a process that is already running in the embedded system?
For example, a process will be started by the embedded system on bootup. The memory consumption of the process is increasing gradually. I don't want to kill and start the process with Valgrind, I want to inspect the existing process. Are there any tools that can help here?
I think we can try with /proc/pid/maps, but not sure how we can understand the anonymous allocations in /proc/pid/maps file.
According to https://azureossd.github.io/2015/09/18/taking-a-crash-dump-of-nodejava-process-using-the-procdump-on-azure-webapp/
we could collect the memory dump of node.exe with ProcDump, where we can use WinDBG to open it.
Are there any tools or maybe WinDBG extensions including its commands that could help us investigate high memory?
I am looking to analyze memory leaks for the web app deployed in azure.
Referring to following urls
https://blogs.msdn.microsoft.com/kaushal/2017/05/04/azure-app-service-manually-collect-memory-dumps/
https://blogs.msdn.microsoft.com/kaushal/2017/05/04/azure-app-service-manually-collect-memory-dumps/
we were able to extract memory dump and analyze them. but since we were not able to inject the LeakTrack dll / enable memory leaks tracking when collecting the dump, we are getting message that leak analysis was not performed due to not injecting the dll on performing memory analysis.
please suggest how to find out memory leakages from analyzing the dump in this scenario.
As you said, DebugDiag currently can't create reflected process dumps, and ProcDump doesn't have a way to inject the LeakTrack dll to track allocations. So, we could get around by working with both tools.
We can simply go to the Processes tab in DebugDiag, right click the process, and chose "Start Monitory for Leaks."
We can do that by scripting DebugDiag and ProcDump to do the individual tasks we've set out for them.
Once we have the PID of the troubled process, we can use a script to inject the LeakTrack dll into the process. With the PID known and the script created, we can launch DebugDiag from a command line.
Such as:
C:\PROGRA~1\DEBUGD~1\DbgHost.exe -script "your LeakTrack dll path" -attach your PID
For more detail, you could refer to this article.
Here is also the reference case.
I'm running a Node.JS web application that works fine for a few hours and then at some random point in time, the V8 heap suddenly starts growing very quickly without a reason and about 40 minutes later, this growth usually stops and the process continues running normally.
I'm monitoring this with nodetime:
What could be the cause of this? Is it a memory leak in my program or perhaps a bug in V8?
There is no way of knowing what the issue by what you provided, but there's a 99.99% chance the problem is inside / fixable in your code.
The best tools I've found for debugging memory issues with Node.js is https://github.com/bnoordhuis/node-heapdump, you can set it up to dump a certain intervals, or by default it listens to USR2 signal, so you can send kill -s USR2 to the pid of your process and get the snapshot.
Then you can use Chrome Inspector to load the heap into it's profiling tool and start inspecting.
I've generally found the issues to be around holding on to external requests too long.
I have a C/C++ application that crashes only under heavy loads. I normally use valgrind and gprof to debug memory leaks and profiling issues. Failure rate is like about 100 in a million runs. This is consistent.
Rather than reproduce the traffic to my application, can I superficially limit the resources available to the debug build of the application running within valgrind somehow?
ulimit can be used from bash to set hard limits on some resources.
Note that in Linux only some of the memory ulimits actually work.
For example, I don't think ulimit -d which is supposed to limit the data segment (which I think is RSS) really works.
As I recall from my experience with trying to keep Evolution (the email client) under control, ulimit -v (the virtual memory) was the only one that worked for me.
It sounds like it could be a race condition - have you tried the 'helgrind' valgrind tool?