I work in a MnC software company. My task is to fix memory leaks in the software. I am using valgrind memcheck tool. I used 'valgrind --leak-check=yes --log-file=vg.log '. I found that valgrind is showing same memory leak multiples times in different sizes in vg.log and the vg.log file has 2 million lines. As a result I am not able to identify which leak has highest size. I mean which one is the biggest leak. Do you have any idea to solve this issue? I want same memory leak should come once in vg.log with aggregate size.I am using val3.12.0 version of valgrind. I am using Red Hat Enterprise Linux Workstation release 6.5.
The leaks are dumped in order of increasing size. Therefore the last leak is the biggest one. Each leak description looks something like
XX bytes in B blocks are definitely lost in loss record R of N
where
XX is the total size of the leak for that callstack
B is the number of times there is a leak with that callstack. For instance, if the total is 4096 bytes in 4 blocks that means that the call happens 4 times and each leak is 1024 bytes
R is the running count of the leaks
N is the total number of different leak callstacks
Actually valgrind has a --num-callers option which can used to club different similar memory leaks.
Related
Recently we had memory spike issue in one of our .Net core 2.0 application. When we analyze the memory dump, we found around 2GB of free space which is not released.
Following is the statistics from windbg tool.
Statistics:
MT Count TotalSize Class Name
000000e32eaf4ca0 15404 2584481422 Free
Can you please let me know what could be the reason for memory not getting released?
At least for the following reasons:
.NET thinks that your program needs that memory again soon
The free parts of the memory do not make up a complete block to give back to the OS
Regarding 1.), .NET might keep the memory which it got from the operating system, because giving it back to the OS and then reclaiming it from the OS has an impact on performance. If .NET thinks your program will request memory in the near future, it could simply "cache" it for you.
Regarding 2.), that's best explained in an example. Assume that your application requested 10 MB. .NET might take a 50 MB block from the OS. The application then requests 1 MB. .NET puts it into the existing 50 MB block. The 10 MB get freed. However, .NET cannot give 49 MB back to the OS, because giving memory back to the OS can only be done at the same size in which it was given.
.NET operates directly on VirtualAlloc() and VirtualFree(), therefore the possibilities are somewhat limited.
If you look at the Count column, you'll see that your 2.5 GB are not contiguous. It is split into 15000 fragments. Your program might suffer from memory fragmentation. Memory fragmentation can occur on the small object heap due to pinned handles and on the large object heap because that one does not get compacted.
As per this page, for versions 2.6 and 2.7 (http://www.timestored.com/kdb-guides/memory-management)
2.6 Unreferenced memory blocks over 32MB/64MB are returned immediately
2.7 Unreferenced memory blocks returned when memory full or .Q.gc[] called
But in both versions, there is a significant difference between used and heap space shown by .Q.w[]. This difference only grows as I run the function again. In 2.6, a difference could occur due to fragmentation (allocating many small objects) but I am not confident it accounts for this big of a difference. In 2.7, even after running .Q.gc[], it shows a significant difference. I would like to understand fundamentally the reason for this difference in the two versions as highlighted below.
This is behavior I am seeing in 2.6 and 2.7:
2.6:
used| 11442889952
heap| 28588376064
2.7 (after running .Q.gc[])
used| 11398025856
heap| 16508780544
Automatic Garbage collection doesn't clear small objects (<32MB) . In that case manual GC call is required. If your process has lot of unreferenced small objects then that will add up in heap size and not to used size.
Second, since KDB allocates memory in power of 2, that makes the difference between used and heap memory. For ex. if a vector requires 64000 bytes, it will be assigned a memory block of size 2^16 = 65536 bytes. And boundary cases makes this difference huge, for ex. if vector requires 33000 bytes (just over 2^15) it will be allocated 65536 bytes (2^16).
Following site has good explanation of GC behavior:
http://www.aquaq.co.uk/q/garbage-collection-kdb/
In windows, I can check the largest contiguous free blocks via 'feature memstats'. But this does not work on linux. I am currently working on a project which needs to load a very large matrix(59107200*17). And I ran into 'out of memory' error. Is there any way to check this and other memory infomation in matlab on linux?
Have you tried memory function? I think it should be available on Linux as well.
Added
By the way, 59107200*17 array of doubles require about 8 Gb of memory (each double number occupies 8 bytes). Do you have it?
Recently we've discovered that our node.js app is most probably having some memory leak (memory consumption showed in htop is growing and growing). We've managed to isolate small amount of code into separate script which is still causing memory leak and now trying to hunt it down. However we're having some troubles with analysing and understanding our test results gathered by htop tool and this v8 profiler: http://github.com/c4milo/node-webkit-agent
Right after script start htop is showing following memory consumption:
http://imageshack.us/a/img844/3151/onqk.png
Then app is running for 5 minutes and I'm taking heap snapshots each 30 secs. After 5 mins results are following:
Heap snapshots sizes:
http://imageshack.us/a/img843/1046/3f7x.png
And results from htop after 5 mins:
http://imageshack.us/a/img33/5339/2nb.png
So if I'm reading this right then V8 profiler shows that there is no serious memory leak, but htop shows that memory consumption grew up from 12MB to 56MB! Can anyone tell where is this difference coming from? And why even at the beginning of tests htop shows 12MB vs 4MB showed by profiler?
htop author here. You're reading the htop numbers right. I don't know about the V8 profiler, but on the issue of "12MB vs 4MB" at the start, it's most likely the case that V8 is accounting only your JS data, while htop accounts the entire resident memory usage of the process, including C libraries used by V8 itself, etc.
I have been running a crypto-intensive application that was generating pseudo-random strings, with special structure and mathematical requirements. It has generated around 1.7 million voucher numbers per node in over the last 8 days. The generation process was CPU intensive, with very low memory requirements.
Mnesia running on OTP-14B02 was the storage database and the generation was done within each virtual machine. I had 3 nodes in the cluster with all mnesia tables disc_only_copies type. Suddenly, as activity on the Solaris boxes increased (other users logged on remotely and were starting webservers, ftp sessions, and other tasks), my bash shell started reporting afork: not enough space error.
My erlang Vms also, went down with this error below:
Crash dump was written to: erl_crash.dump
temp_alloc: Cannot reallocate 8388608 bytes of memory (of type "root_set").
Usually, we get memory allocation errors and not memory re-location errors and normally memory of type "heap" is the problem. This time, the memory type reported is type "root-set".
Qn 1. What is this "root-set" memory?
Qn 2. Has it got to do with CPU intensive activities ? (why am asking this is that when i start the task, the Machine reponse to say mouse or Keyboard interrupts is too slow meaning either CPU is too busy or its some other problem i cannot explain for now)
Qn 3. Can such an error be avoided? and how ?
The fork: not enough space message suggests this is a problem with the operating system setup, but:
Q1 - The Root Set
The Root Set is what the garbage collector uses as a starting point when it searches for data that is live in the heap. It usually starts off from the registers of the VM and off from the stack, if the stack has references to heap data that still needs to be live. There may be other roots in Erlang I am not aware of, but these are the basic stuff you start off from.
That it is a reallocation error of exactly 8 Megabyte space could mean one of two things. Either you don't have 8 Megabyte free in the heap, or that the heap is fragmented beyond recognition, so while there are 8 megabytes in it, there are no contiguous such space.
Q2 - CPU activity impact
The problem has nothing to do with the CPU per se. You are running out of memory. A large root set could indicate that you have some very deep recursions going on where you keep around a lot of pointers to data. You may be able to rewrite the code such that it is tail-calling and uses less memory while operating.
You should be more worried about the slow response times from the keyboard and mouse. That could indicate something is not right. Does a vmstat 1, a sysstat, a htop, a dstat or similar show anything odd while the process is running? You are also on the hunt to figure out if the kernel or the C libc is doing something odd here due to memory being constrained.
Q3 - How to fix
I don't know how to fix it without knowing more about what the application is doing. Since you have a crash dump, your first instinct should be to take the crash dump viewer and look at the dump. The goal is to find a process using a lot of memory, or one that has a deep stack. From there on, you can seek to limit the amount of memory that process is using. either by rewriting the code so it can give memory up earlier, by tuning the garbage collection setup for the process (see the spawn options in the erlang man pages for this), or by adding more memory to the system.