I am developing some software in C. One of the targets in my makefile runs a test, and I use time to record the runtime and valgrind to check for memory leaks. When I invoke one of the smaller tests directly from the command line, I get this.
$ time valgrind bin/test data/gff3/example1.gff3 data/gff3/example2.gff3
==27268== Memcheck, a memory error detector
==27268== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==27268== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for copyright info
==27268== Command: bin/test data/gff3/example1.gff3 data/gff3/example2.gff3
==27268==
==27268==
==27268== HEAP SUMMARY:
==27268== in use at exit: 0 bytes in 0 blocks
==27268== total heap usage: 4,001 allocs, 4,001 frees, 152,648 bytes allocated
==27268==
==27268== All heap blocks were freed -- no leaks are possible
==27268==
==27268== For counts of detected and suppressed errors, rerun with: -v
==27268== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 4 from 4)
real 0m1.460s
user 0m1.330s
sys 0m0.110s
However, when I invoke the test from my makefile (using the same exact command), I get this.
$ make mem
time valgrind bin/test data/gff3/example1.gff3 data/gff3/example2.gff3
==27265== Memcheck, a memory error detector
==27265== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==27265== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for copyright info
==27265== Command: bin/test data/gff3/example1.gff3 data/gff3/example2.gff3
==27265==
==27265==
==27265== HEAP SUMMARY:
==27265== in use at exit: 0 bytes in 0 blocks
==27265== total heap usage: 4,001 allocs, 4,001 frees, 152,648 bytes allocated
==27265==
==27265== All heap blocks were freed -- no leaks are possible
==27265==
==27265== For counts of detected and suppressed errors, rerun with: -v
==27265== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 4 from 4)
1.32user 0.10system 0:01.55elapsed 91%CPU (0avgtext+0avgdata 214752maxresident)k
24inputs+8outputs (1major+18568minor)pagefaults 0swaps
There seem to be line break issues with the output of the time program at the bottom, as well as some additional output I didn't see when invoking directly from the command line. I've seen this before when I'm trying to time a job I've run with nohup.
Why is this happening and what can I do to get consistent results?
time is a bash builtin. The output from your Makefile looks like the output of /usr/bin/time. The easiest way to get consistent results is to run '/usr/bin/time' instead of time on the command line.
Related
let main =
print_endline "Hello world"
Here is a simple OCaml program ^
When I compile with ocamlc, and then run this program in valgrind:
==12457== Memcheck, a memory error detector
==12457== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==12457== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==12457== Command: ./a.out
==12457==
Hello world
==12457==
==12457== HEAP SUMMARY:
==12457== in use at exit: 7,391,113 bytes in 30 blocks
==12457== total heap usage: 56 allocs, 26 frees, 7,470,447 bytes allocated
==12457==
==12457== 256 bytes in 1 blocks are definitely lost in loss record 15 of 27
==12457== at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==12457== by 0x11D6BC: caml_stat_alloc (in /usr/bin/ocamlrun)
==12457== by 0x13627C: caml_executable_name (in /usr/bin/ocamlrun)
==12457== by 0x13C5C4: caml_main (in /usr/bin/ocamlrun)
==12457== by 0x11CDB1: main (in /usr/bin/ocamlrun)
==12457==
==12457== 3,936,288 bytes in 1 blocks are possibly lost in loss record 27 of 27
==12457== at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==12457== by 0x11D601: caml_alloc_for_heap (in /usr/bin/ocamlrun)
==12457== by 0x13EF42: caml_init_major_heap (in /usr/bin/ocamlrun)
==12457== by 0x12EFF5: caml_init_gc (in /usr/bin/ocamlrun)
==12457== by 0x13C43D: caml_main (in /usr/bin/ocamlrun)
==12457== by 0x11CDB1: main (in /usr/bin/ocamlrun)
==12457==
==12457== LEAK SUMMARY:
==12457== definitely lost: 256 bytes in 1 blocks
==12457== indirectly lost: 0 bytes in 0 blocks
==12457== possibly lost: 3,936,288 bytes in 1 blocks
==12457== still reachable: 3,454,569 bytes in 28 blocks
==12457== suppressed: 0 bytes in 0 blocks
==12457== Reachable blocks (those to which a pointer was found) are not shown.
==12457== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==12457==
==12457== For lists of detected and suppressed errors, rerun with: -s
==12457== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)
When I compile with ocamlopt, and then run this program in valgrind:
==12562== Memcheck, a memory error detector
==12562== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==12562== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==12562== Command: ./a.out
==12562==
Hello world
==12562==
==12562== HEAP SUMMARY:
==12562== in use at exit: 7,080,897 bytes in 27 blocks
==12562== total heap usage: 27 allocs, 0 frees, 7,080,897 bytes allocated
==12562==
==12562== LEAK SUMMARY:
==12562== definitely lost: 0 bytes in 0 blocks
==12562== indirectly lost: 0 bytes in 0 blocks
==12562== possibly lost: 3,936,288 bytes in 1 blocks
==12562== still reachable: 3,144,609 bytes in 26 blocks
==12562== suppressed: 0 bytes in 0 blocks
==12562== Rerun with --leak-check=full to see details of leaked memory
==12562==
==12562== For lists of detected and suppressed errors, rerun with: -s
==12562== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
Why is there memory leaks in the program compiled with ocaml? Are these just fake memory leaks? Do I have to worry about this?
I believe the OCaml GC is much wiser about what memory is acessible than an outside tool that doesn't know how OCaml works. Such tools need to make very general assumptions that can easily be wrong.
There are almost no leaks reported, just "possible" leaks (whatever that means). You could imagine the OCaml bytecode runtime allocates a small amount of global storage at startup. Calling this a "memory leak" is not correct, in my opinion. It would make no sense to release the memory at exit since this is just unnecessary work.
There is no leak shown for the native runtime (compiled with ocamlopt). So there doesn't seem to be anything that needs to be explained.
So, in sum, I wouldn't worry about this. It isn't an error so much as sensationalistic terminology in the output of the memory checker. If these were C programs there might be more reason to be concerned.
(However it still makes no sense to release global memory just before exit. Hence many C programs will report similar spurious "leaks" by this analysis.)
The bytecode interpreter is "leaking" a 256 bytes string containing the executable name. Since the name is owned during the whole lifetime of the program, that does not really matter.
And since the "leak" is in the bytecode interpreter, it doesn't happen for the natively compiled program.
More generally using valgrind is not really useful for a garbage collected language (outside of compiler development), since the garbage collector will gobble all memory. It is better to use a memory profiler that knows about the GC like statmemprof.
ALL,
igor#IgorReinCloud ~/dbhandler/Debug/dbhandler $ valgrind --demangle=yes dbhandler --log-file=memcheck.log --leak-check=full
==28275== Memcheck, a memory error detector
==28275== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==28275== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==28275== Command: dbhandler --log-file=memcheck.log --leak-check=full
// Some errors logged on screen
Unknown long option 'log-file'
// Some more errors logged on screen
Usage: dbhandler [-h] [--verbose]
-h, --help show this help message
--verbose generate verbose log messages
==28275==
==28275== HEAP SUMMARY:
==28275== in use at exit: 1,549,966 bytes in 16,372 blocks
==28275== total heap usage: 177,899 allocs, 161,527 frees,
10,849,729 bytes allocated
==28275==
==28275== LEAK SUMMARY:
==28275== definitely lost: 0 bytes in 0 blocks
==28275== indirectly lost: 0 bytes in 0 blocks
==28275== possibly lost: 3,408 bytes in 32 blocks
==28275== still reachable: 1,464,270 bytes in 15,670 blocks
==28275== of which reachable via heuristic:
==28275== length64 : 4,872 bytes in 81 blocks
==28275== newarray : 2,096 bytes in 51 blocks
==28275== suppressed: 0 bytes in 0 blocks
==28275== Rerun with --leak-check=full to see details of leaked memory
==28275==
==28275== For counts of detected and suppressed errors, rerun with: -v
==28275== Use --track-origins=yes to see where uninitialised values come from
==28275== ERROR SUMMARY: 28 errors from 8 contexts (suppressed: 0 from 0)
However,
igor#IgorReinCloud ~/dbhandler/Debug/dbhandler $ valgrind --help | grep log
--time-stamp=no|yes add timestamps to log messages? [no]
--log-fd=<number> log messages to file descriptor [2=stderr]
--log-file=<file> log messages to <file>
--log-socket=ipaddr:port log messages to socket ipaddr:port
igor#IgorReinCloud ~/dbhandler/Debug/dbhandler $
So what the problem is? Why the messages are not going to the log file as it should? Am I missing an option here?
TIA!
Put the options to Valgrind before the program you want it to check. It assumes options after the program name are options to the program, not Valgrind.
Hence, you have:
valgrind --demangle=yes dbhandler --log-file=memcheck.log --leak-check=full
but you should be using:
valgrind --demangle=yes --log-file=memcheck.log --leak-check=full dbhandler
or even use -- to state that you've finished the options to Valgrind:
valgrind --demangle=yes --log-file=memcheck.log --leak-check=full -- dbhandler
The Unknown long option log-file message probably comes from dbhandler and not Valgrind. It's also why you're getting the 'usage' message from dbhandler.
I'm learning Rust and I was playing with Box, so I tried checking leaks with valgrind but it shows that there are no allocations on the heap:
$ rustc -C opt-level=0 raii.rs
$ valgrind ./raii
==3850== Memcheck, a memory error detector
==3850== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==3850== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==3850== Command: ./raii
==3850==
5
Changed:10
==3850==
==3850== HEAP SUMMARY:
==3850== in use at exit: 0 bytes in 0 blocks
==3850== total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==3850==
==3850== All heap blocks were freed -- no leaks are possible
==3850==
==3850== For counts of detected and suppressed errors, rerun with: -v
==3850== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
valgrind --leak-check=full ./raii shows exaclty the same result.
Here's the Rust code:
fn main() {
let mut _box2 = Box::new(5i32);
println!("{}", _box2);
*_box2 = 10i32;
println!("Changed:{}", _box2);
{
let _box3 = Box::new(4i32);
}
}
Some other info:
$ rustc -V
rustc 1.8.0 (db2939409 2016-04-11)
$ valgrind --version
valgrind-3.10.1
$ uname -a
Linux 3.19.0-59-generic #65~14.04.1-Ubuntu SMP Tue Apr 19 18:57:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Why is that? I thought Box allocates the variable on the heap. Also, this example shows almost the same code and Valgrind shows allocations there.
If I add the following code:
let _stack = (1u64, 2u64, 3u64);
let _heap = Box::new((4u64, 5u64, 6u64));
println!("Memory used by stack: {} bytes", std::mem::size_of_val(&_stack));
println!("Memory used by heap: {} bytes", std::mem::size_of_val(&_heap));
It prints exactly what I expected:
$ ./raii
Memory used by stack: 24 bytes
Memory used by heap: 8 bytes
In the second case, the tuple was placed on the heap and a pointer (8 bytes) was pushed to the stack.
In the first case tuple was placed on the stack, so it takes 24 bytes.
Valgrind seems to be able to count heap allocations from other programs:
$ valgrind echo "test"
==4575== Memcheck, a memory error detector
==4575== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==4575== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==4575== Command: echo test
==4575==
test
==4575==
==4575== HEAP SUMMARY:
==4575== in use at exit: 0 bytes in 0 blocks
==4575== total heap usage: 30 allocs, 30 frees, 3,681 bytes allocated
==4575==
==4575== All heap blocks were freed -- no leaks are possible
==4575==
==4575== For counts of detected and suppressed errors, rerun with: -v
==4575== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
Rust 1.32
As of Rust 1.32, the default allocator for an executable is now the system allocator, so you don't need to set anything by default.
Previous versions
Rust uses jemalloc as the allocator when you build a binary. The way that jemalloc is currently compiled does not contain the Valgrind hooks needed.
If you need the ability to track allocations with Valgrind, you can switch to the the system allocator if you are using nightly Rust.
See also:
Why does Valgrind not detect a memory leak in a Rust program using nightly 1.29.0?
I am trying to understand whether or not my application is leaking.
When running my application, I periodically run pmap and observe:
mapped: 488256K writeable/private: 90144K shared: 0K
mapped: 488260K writeable/private: 101912K shared: 0K
mapped: 488256K writeable/private: 102708K shared: 0K
mapped: 488260K writeable/private: 105112K shared: 0K
I run top and observe:
VIRT RES SHR
488260 17684 3020
488256 20060 3032
488256 22700 3032
488256 26132 3032
488256 28772 3032
488256 31880 3032
The increase in "RES" and in "writeable/private" is what makes me suspect a leak. However, running valgrind I do not detect any major leak, and when I abort execution I consistently
see about 20Mb reachable memory:
==19998==
==19998== HEAP SUMMARY:
==19998== in use at exit: 20,351,513 bytes in 974 blocks
==19998== total heap usage: 329,404 allocs, 328,430 frees, 34,562,346 bytes allocated
==19998==
==19998== LEAK SUMMARY:
==19998== definitely lost: 63 bytes in 4 blocks
==19998== indirectly lost: 0 bytes in 0 blocks
==19998== possibly lost: 4,679 bytes in 76 blocks
==19998== still reachable: 20,346,771 bytes in 894 blocks
==19998== suppressed: 0 bytes in 0 blocks
==19998== Rerun with --leak-check=full to see details of leaked memory
==19998==
==19998== For counts of detected and suppressed errors, rerun with: -v
==19998== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
I run valgrind --tool=massif and also see 20Mb flat:
QUESTION: Can someone please explain why valgrind and massif tell me that my program uses 20Mb memory consistently, but top and pmap tell me usage is growing?
In order to understand why top shows increase for your process you also need to analyze memory allocations in your program with valgrind --pages-as-heap=yes. You will understand why there is increase. It is the way top measures memory consumption of a process. http://valgrind.org/docs/manual/ms-manual.html#ms-manual.not-measured. And with valgrind --pages-as-heap=yes you will see where these allocations are done in you program
I am in the process of learning C from Zed Shaw's excellent Learn C the Hard Way.
I am running Debian Sid with kernel version 3.9.0-vanillaice amd64 (I compiled it) with glibc 2.17 (installed from Debian's repo's).
I am on Exercise 4 in the book.
Here is my code:
#include <stdio.h>
//This program is purposefully messed up.
int main()
{
int age = 10;
int height;
printf("I am %d years old.\n");
printf("I am %d inches tall.\n", height);
return 0;
}
Valgrind should produce something like this:
$ valgrind ./ex4
==3082== Memcheck, a memory error detector
==3082== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==3082== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for copyright info
==3082== Command: ./ex4
==3082==
I am -16775432 years old.
==3082== Use of uninitialised value of size 8
==3082== at 0x4E730EB: _itoa_word (_itoa.c:195)
==3082== by 0x4E743D8: vfprintf (vfprintf.c:1613)
==3082== by 0x4E7E6F9: printf (printf.c:35)
==3082== by 0x40052B: main (ex4.c:11)
==3082==
==3082== Conditional jump or move depends on uninitialised value(s)
==3082== at 0x4E730F5: _itoa_word (_itoa.c:195)
==3082== by 0x4E743D8: vfprintf (vfprintf.c:1613)
==3082== by 0x4E7E6F9: printf (printf.c:35)
==3082== by 0x40052B: main (ex4.c:11)
==3082==
==3082== Conditional jump or move depends on uninitialised value(s)
==3082== at 0x4E7633B: vfprintf (vfprintf.c:1613)
==3082== by 0x4E7E6F9: printf (printf.c:35)
==3082== by 0x40052B: main (ex4.c:11)
==3082==
==3082== Conditional jump or move depends on uninitialised value(s)
==3082== at 0x4E744C6: vfprintf (vfprintf.c:1613)
==3082== by 0x4E7E6F9: printf (printf.c:35)
==3082== by 0x40052B: main (ex4.c:11)
==3082==
I am 0 inches tall.
==3082==
==3082== HEAP SUMMARY:
==3082== in use at exit: 0 bytes in 0 blocks
==3082== total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==3082==
==3082== All heap blocks were freed -- no leaks are possible
==3082==
==3082== For counts of detected and suppressed errors, rerun with: -v
==3082== Use --track-origins=yes to see where uninitialised values come from
==3082== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 4 from 4)
$
Mine mentions nothing about the errors:
$ valgrind ./ex4
==4098== Memcheck, a memory error detector
==4098== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==4098== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==4098== Command: ./ex4
==4098==
I am -16776056 years old.
I am 0 inches tall.
==4098==
==4098== HEAP SUMMARY:
==4098== in use at exit: 0 bytes in 0 blocks
==4098== total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==4098==
==4098== All heap blocks were freed -- no leaks are possible
==4098==
==4098== For counts of detected and suppressed errors, rerun with: -v
==4098== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)
I am assuming you are using the gcc compiler.
I just gave that a little testing and my valgrind did report the errors only when I had the program built with -O0. With optimizations on (-O3) valgrind did not report the errors. I suppose that would be because the compiler is able to remove the uninitialized jump because the value never changes and is removed entirely. Since you are doing that for exercise you might try to figure out which compiler options work for you. In case you don't know yet what that is, look for "Debug" or "Release" build and switch to "Debug".
HTH,
Moose