Correlation between malloc_stats and /proc/pid/stat - linux

I am working on a embedded linux system. I understand what info malloc_stats and /proc/pid/stats provide. I want to know how the info printed by malloc_stats is related to the memory usage info provided by /proc/stats.
Background is that I want to instrument each thread in my app to check for memory leaks.Malloc_stats prints useful info but cant be used programatically./proc//task/ has useful info but I am unable to correlate it to the heap memory used by the current thread.

Have you overlooked the mallinfo() library function? It's where malloc_stats() gets its information from.
To answer the question directly: The data in /proc will reflect the total memory usage of the process, including slack space between memory allocations and free memory, as well as memory that's being used which wasn't allocated through malloc() at all (e.g, the stack, global/static variables, etc). malloc_stats() will break that down into what's actually allocated and what isn't.

Related

Linux: Tracking allocated memory over time

I want to generate a graph of the allocated memory for a particular PID over time for which I am currently using a custom script that uses an strace log. From the strace log, I am aggregating the memory allocation changes from mmap, munmap, and, brk system calls.
I was wondering, however, if there is a better and more matured solution to do this (measure/graph the lifetime of memory allocations for a process)
I believe what you are looking for is a tool called massif visualizer (a part of Valgrind) which allows you to view graphed memory allocation for specific processes over time and is still actively maintained.

How to determine the real memory usage of a single process?

How can I calculate the real memory usage of a single process? I am not talking about the virtual memory, because it just keeps growing. For instance, there are proc files like smaps, where you can get the mappings of a process. But this is virtual memory and the values of that file just keeps growing for running process. But I would like to reflect the real memory usage of a process. E.g. if you plot the memory usage of a process it should represent the allocations of memory and also the freeing of memory. So the plot should be like an up and down movement instead of a linear function, that just keeps growing for a running process.
So, how could I calculate the real memory usage? I would appreciate any helpful answer.
It's actually kind of a complicated question. The two most common metrics for a program's memory usage at the OS level are virtual size and resident set size. (These show in the output of ps -u as the VSZ and RSS columns.) Roughly speaking, these tell the total memory the program has assigned to it, versus how much it is currently actively using.
Further complicating the question is that when you use malloc (or the C++ new operator) to allocate memory, memory is allocated from a pool in your process which is built by occasionally requesting an allocation of memory from the operating system. But when you free memory, the memory goes back into this pool, but it is typically not returned to the OS. So as your program allocates and frees memory, you typically will not see its memory footprint go up and down. (However, if it frees a lot of memory and then doesn't allocate it any more, eventually you may see its rss go down.)

Does linux core dumps have thread cpu usage information

Since I'm fairly new to linux and core dumps, I'm not sure what kind of information is stored in core-dumps. It makes me wonder if there is a GDB command to retrieve CPU % usage of threads from a Core dump file. Like the CPU % usage you get from 'top' command. Would be also nice to get memory usage too.
I'm rephrasing the question from my previous posting to stay more focused to the answer I'm looking for.
Reference : How to diagnose a python process chewing CPU in linux
Thanks.
No, it's not possible to obtain info about the CPU usage from a coredump.
The coredump is just the snapshot of the memory of the process at death-time. Any dynamic history is not available: CPU make/model/frequency, system load, number of other processes, kernel scheduling info, etc.
As a side effect, you DO get the memory usage information, as long as you know the memory available on the system that generated the coredump: since the coredump is the memory of the process, the more memory the process used, the bigger the coredump (generally speaking, there are exceptions like regions of memory not included in the codedump).
A core dump is a copy of the crashed process's address space (memory). You can use it to see how much memory the process was using (and you can examine all the data in its memory at the time it crashed), but it doesn't contain any information about CPU usage.
For the future, you can collect this easily enough -- have your process periodically collect memory usage for each thread, and when debugging, hunt for that variable in the core.

Memory Debugging

Currently I analyze a C++ application and its memory consumption. Checking the memory consumption of the process before and after a certain function call is possible. However, it seems that, for technical reasons or for better efficiency the OS (Linux) assigns not only the required number of bytes but always a few more which can be consumed later by the application. This makes it hard to analyze the memory behavior of the application.
Is there a workaround? Can one switch Linux to a mode where it assigns just the required number of bytes/pages?
if you use malloc/new, the allocator will always alloc a little more bytes than you requested , as it needs some room to do its housekeeping, also it may need to align the bytes on pages boundaries. The amount of supplementary bytes allocated is implementation dependent.
you can consider to use tools such as gperftools (google) to monitor the memory used.
I wanted to check a process for memory leeks some years ago.
What I did was the following: I wrote a very small debugger (it is easier than it sounds) that simply set breakpoints to malloc(), free(), mmap(), ... and similar functions (I did that under Windows but under Linux it is simpler - I did it in Linux for another purpose!).
Whenever a breakpoint was reached I logged the function arguments and continued program execution...
By processing the logfile (semi-automated) I could find memory leaks.
Disadvantage: It is not possible to debug the program using another debugger in parallel.

How does the amount of memory for a process get determined?

From my understanding, when a process is under execution it has some amount of memory at it's disposal. As the stack increases in size it builds from one end of the process (disregarding global variables that come before the stack), while the heap builds from another end. If you keep adding to the stack or heap, eventually all the memory will be used up for this process.
How does the amount of memory the process is given get determined? I can only imagine it depends on a bunch of different variables, but an as-general-as-possible response would be great. If things have to get specific, I'm interested in linux processes written in C++.
On most platforms you will encounter, Linux runs with virtual memory enabled. This means that each process has its own virtual address space, the size of which is determined only by the hardware and the way the kernel has configured it.
For example, on the x86 architecture with a "3/1" split configuration, every userspace process has 3GB of address space available to it, within which the heap and stack are allocated. This is regardless of how much physical memory is available in the system. On the x86-64 architecture, 128TB of address space is typically available to each userspace process.
Physical memory is separately allocated to back that virtual memory. The amount of this available to a process depends upon the configuration of the system, but in general it's simply supplied "on-demand" - limited mostly how much physical memory and swap file space exists, and how much is currently in use for other purposes.
The stack does not magically grow. It's size is static and the size is determined at linking time. So when you take enough space from the stack, it overflows (stack overflow ;)
On the other hand, the heap area 'magically' grows. Meaning that when ever more memory is needed for heap, the program asks operating system for more memory.
EDIT: As Mat pointed out below, the stack actually can increase during runtime on modern operating systems.

Resources