I have an ASAN instrumented 'deamon' process that always runs in my system. I see the memory leaks are reported only when the process exits. Is there anyway that i can ask LSAN to dump the leak reports without having to kill the process ? Is there any signal that i can send to process , so that it will detect and dump the leaks ?
Use __lsan_do_leak_check:
// Check for leaks now. This function behaves identically to the default
// end-of-process leak check. In particular, it will terminate the process if
// leaks are found and the exitcode runtime flag is non-zero.
// Subsequent calls to this function will have no effect and end-of-process
// leak check will not run. Effectively, end-of-process leak check is moved to
// the time of first invocation of this function.
// By calling this function early during process shutdown, you can instruct
// LSan to ignore shutdown-only leaks which happen later on.
Related
I have a process that seems to be leaking memory. The longer the process runs, the more memory it uses. That is in spite of the fact that the process consists primarily of a loop that iteratively calls a function which should not preserve any data between calls. When I use valgrind to check for leaks, everything comes back a-ok. When the process eventually exits after running for a few hours, there is a substantial delay at exit, which all leads me to believe that memory is being allocated in that function and not freed immediately because it is still referenced. The memory is then subsequently freed on exit because that reference is eventually freed.
I'm wondering if there is a way with valgrind (or some other linux-compatible tool) to do a leak check between two code checkpoints. I'd like to get a leak report of all memory that was allocated but not freed between two code checkpoints.
I wrote an article on this a few years back.
In short, you include valgrind.h and then you can use macros like
VALGRIND_DO_LEAK_CHECK
Alternatively you can attach gdb and issue the 'monitor leak_check' command. This can be incremental. See here
I'm having trouble on distinguishing how thread crashes differ from a process crash.
Is there even any difference?
Yes, there is a difference. A thread can exit while the process remains active. For example, a thread can exit either normally or by throwing an exception out of its stack, and it stops execution, but other threads in the process will continue to execute normally, and the process heap space will continue to be allocated. If the process crashes - or if it exits normally - all threads not already terminated will terminate and all memory allocated by the process will be released.
I have installed a handler (say, crashHandler()) which has a bit of file output functionality. It is a linux thread which registers for SIGSEGV with the crashHandler(). File writing is requred, as it stores the stack trace to persistent storage.
It works most of the times. But in a specific scenario, the function (crashHandler()) executes the function partly (I can see logs) and then device reboots. Can someone help me with a way to deal with such ?
The first question to ask here is why the device rebooted. Normally having an ordinary application crash won't cause a kernel-level or hardware-level reboot. Most likely, you're either hitting a watchdog timer before the crash handler completes (in which case you should extend the watchdog timeout - do NOT reset the timer from within the crash handler though, as then you're risking problems in the crash handler itself preventing a reboot), or this is pid 1 and it's crashing within the SIGSEGV handler, causing a kernel panic due to pid 1 (init) dying.
If it's the latter, you need to be more careful with what you do in that crash handler. Remember, you just crashed. You know memory is corrupt, but you don't know how it's corrupt. It may be corrupt in ways that affect the crash handler itself - e.g. if you corrupt the heap metadata, you may be unable to allocate memory without crashing for real this time. You should keep what you do in that handler to a bare minimum - in particular, avoid calling any library functions that are not documented as being async-signal-safe and avoid using any complex (pointer-containing) data structures or dynamically allocated memory. For the highest level of safety, limit yourself to just fork() and exec()ing another process that will use debugger APIs (ptrace() and /proc/$PID/mem) to perform memory dumps or whatever else you might need.
I have a Fortran program that runs a series of identical calculations on a number of different input data. After doing these calculations the code then always writes a GNUplot script that does some diagnostic plotting (nothing too difficult) and runs it using execute_command_line in Linux.
This usually works well, but after some time I think there must be a memory leak of some kind that works cumulative, because the GNUplotting becomes slower and slower. At some point it virtually stalls.
My question is therefore: Is it possible to interrupt the call to execute_command_line using the keyboard without killing the main Fortran program? Needless to say, CTRL-C kills everything, which is not what I want: I want the main program to continue.
I have been playing with the optional flag wait=.true. but this does not help.
Also, I know that the memory leak has to be fixed (or whatever the cause is), but for now I would like to first see the diagnostic output.
The only solution I have been able to come up with is kind of a workaround:
Modify the shell script so that it
runs the Fortran program in the background: ./mpirun prog_name options &
gets the PID of this proces: proc_PID=$!
waits for the process: wait $proc_PID
traps an interrupt signal: trap handler SIGINT
lets the handler send a SIGURS1 signal: function handler() { kill -SIGUSR1 $proc_PID }
modify the Fortran code so that it catches the SIGUSR1 signal and does what you want with it. For example by having a look here.
By running the mpi process in the background you avoid killing mpirun with SIGINT, which cannot be trapped but you send instead a SIGURS1, which is properly propagated to the mpi processes where it can be handled with directly.
As a side note, however, I realized that this will not solve my problem as my problem was related to an external call to gnuplot using execute_command_line. Since I had a cumulative memory leak, at some point this call started taking for ever because memory resources became scarcer. So the only thing I could have done is manually killing the gnuplot process.
Better, of course, was fixing the memory leak, which I did.
I'm trying to clean up specific memory objects created by a specific thread (hence only accessible to that thread). The only way for me to achieve that is to switch to that particular thread when freeing that memory block.
This is how I allocated the specific memory context:
http://imagebin.ca/img/S6mwZBFu.jpg
This is what I attempted to do:
alt text http://imagebin.ca/img/DeTe9Z6h.jpg
I have originally added the memory context creation and destruction in a manner like the following:
int Thread2::main()
{
CudaMemoryContext *theCudaObj = new CudaMemoryContext();
while(!TerminateStatus())
{
...
}
delete theCudaObj;
return 0;
}
However, this approach is not working very well, i.e. the program crashes right when I'm cleaning up on the "delete theCudaObj;" line. I'm wondering if I can switch active threads when cleaning up, or allocate the CUDA context to be accessible by both threads so that I can clean up and access it with ease through both threads. Thanks in advance for suggestions.
How is Thread#1 destroying Thread#2? It's typically best to signal a thread to terminate itself and not use TerminateThread().
Your original approach looks like the right way to go about things - when the thread is signaled to terminate it stops looping and cleans up any allocated memory.
In the context where you signal a thread to terminate, be sure you wait for it to exit before allowing the application to exit. Premature exit could have been causing your crash. Run with the debugger attached and set to break when exceptions are thrown to diagnose.