I have a dump of a process that was running with user-stack-traces flag on. I am trying to analyze a leak from WinDbg. Using instructions here I am able to see top allocations grouped by allocation sizes, list all allocations with specific size, display stack of an allocation using allocation address.
Is there a way to display top allocations grouped by stack? (By 'top' I mean highest contributors to total heap size or total allocation count.) All the information is already in the dump, I just need the right WinDbg extension. I would be surprised if no one wrote such an extension so far.
Related
The amount of memory used by my program reported by top is more than twice the amount reported in /proc/[pid]/statm. What explains this difference?
From the "man" page:
https://man7.org/linux/man-pages/man5/proc.5.html
/proc/[pid]/statm
Provides information about memory usage, measured in pages.
The columns are:
size (1) total program size
(same as VmSize in /proc/[pid]/status)
resident (2) resident set size
(inaccurate; same as VmRSS in /proc/[pid]/status)
shared (3) number of resident shared pages
(i.e., backed by a file)
(inaccurate; same as RssFile+RssShmem in
/proc/[pid]/status)
text (4) text (code)
lib (5) library (unused since Linux 2.6; always 0)
data (6) data + stack
dt (7) dirty pages (unused since Linux 2.6; always 0)
Some of these values are inaccurate because of a kernel-
internal scalability optimization. If accurate values are
required, use /proc/[pid]/smaps or
/proc/[pid]/smaps_rollup instead, which are much slower
but provide accurate, detailed information.
ALSO: VMsize is the "address space" that the process has in use: the number of available adresses. These addresses do not have to have any physical memory attached to them. Attached physical memory is the RSS figure.
Q: So what specific values did you have questions about?
Please see my comments below, and please refer to these links:
man top
man proc
Increase of Virtual Memory without increase of VMSize
Short answer:
RES in top (RSS is ps), this is the sum of memory that is mapped to physical pages of memory.
VMsize is the "address space" that the process has in use: the number of available adresses.
"VMSize" addresses do not (necessarily) have any physical memory attached to them.
Attached physical memory ("Resident memory") is the RSS figure.
It would also be helpful to see your actual values from top, /proc/nnn/stat and from ps.
Here's a great article (one of many):
Process Memory Management in Linux -
Baeldung.com
VSZ Memory
VSZ is short for Virtual Memory Size. It’s the total amount of
memory a process may hypothetically access.
It accounts for the size of the binary itself, any linked libraries,
and any stack or heap allocations.
When a process is started, VSZ memory becomes RSS memory...
RSS Memory
As opposed to VSZ, RSS, also short for Resident Set Size, is a measurement that shows how much RAM has been allocated to a
process during its execution
But, it isn’t actually an accurate measurement for RAM usage for a
process since it doesn’t take into account libraries already loaded in
memory...
Because RSS shows the whole memory output, even for information that
has already been preloaded, if we add up everything in the processes
list, we’d get more memory than the total memory allocated.
I hope that all helps ... at least a little.
If you've still got questions, please copy/paste some actual values from your system, so we can discuss specific examples.
I would like to compute the peak memory usage of my OCaml program when it is running in compiled form as native code. I considered using the stats API in the Gc module, but it seems to return a snapshot at the time it is called. Is there some information in the Gc module or some other module that I can use to get peak heap usage just before my program terminates?
You can get the current size of the major heap using Gc.stat, see the live_words field, multiply it by the word size in bytes to get the size in bytes (8 in a 64 bit system). It doesn't really matter but you can also add the size of the minor heap to the calculation, which is available via Gc.get () see the minor_heap_size field (again in words).
You can create an alarm with Gc.create_alarm to check the size of the heap after each major collection to get the maximum size ever used.
In Android Studio Profiler, there are two places which display the size of native memory occupied by an app.
The first place is in the horizontal bar. Profiler documentation describes this as: "Native: Memory from objects allocated from C or C++ code".
The second place is the app heap dump, Native Size column. In documentation it is described: "Native Size: Total amount of native memory used by this object type (in bytes)"
In my case, the horizontal bar displays 30.12 MB and the heap dump "native size" column displays around 9.28 MB (which is the sum of all objects in app heap that have non-zero native size column)
Why are these two sizes different?
For the "horizontal bar" if you look closely you'll realize that the size of the memory used by graphics is 0. reason for this is that on some devices, bitmaps and other graphical assets are handled in the native memory. So extra memory other than the 9.28mb is most likely these graphical assets.
I've got a node.js application where the RSS memory usage seems to keep growing despite the heapUsed/heapTotal staying relatively constant.
Here's a graph of the three memory measurements taken over a week (from process.memoryUsage()):
You may note that there's a somewhat cyclical pattern - this corresponds with the application's activity throughout each day.
There actually does seem to be a slight growth in the heap, although it's nowhere near that of the RSS growth. So I've been taking heap dumps every now and then (using node-heapdump), and using Chrome's heap compare feature to find leaks.
One such comparison might look like the following (sorted by size delta in descending order):
What actually shows up does depend on when the snapshot was taken (eg sometimes more Buffer objects are allocated etc) - here I've tried to take a sample which demonstrates the issue best.
First thing to note is that the sizes on the left side (203MB vs 345MB) are much higher than heap sizes shown in the graph. Secondly, the size deltas clearly don't match up with the 142MB difference. In fact, sorting by size delta in ascending order, many objects have be deallocated, which means that the heap should be smaller!
Does anyone have any idea on:
why is this the case? (RSS constantly growing with stable heap size)
how can I stop this from happening, short of restarting the server every now and then?
Other details:
Node version: 0.10.28
OS: Ubuntu 12.04, 64-bit
Update: list of modules being used:
async v0.2.6
log4js v0.6.2
mysql v2.0.0-alpha7
nodemailer v0.4.4
node-time v0.9.2 (for timezone info, not to be confused with nodetime)
sockjs v0.3.8
underscore v1.4.4
usage v0.3.9 (for CPU stats, not used for memory usage)
webkit-devtools-agent v0.2.3 (loaded but not activated)
heapdump v0.2.0 is loaded when a dump is made.
Thanks for reading.
The difference you see between RSS usage and heap usage are buffers.
"A Buffer is similar to an array of integers but corresponds to a raw memory allocation outside the V8 heap"
https://nodejs.org/api/buffer.html#buffer_buffer
I need to find out how many pages of memory a process allocates?
Each page is 4096, the process memory usage I'm having some problems locating the correct value. When I'm looking in the gome-system-monitor there are a few values to choose from under memory map.
Thanks.
The point of this is to divide the memory usage by the page count and verify the page size.
It's hard to figure exact amount of memory allocated correctly: there are pages shared with other processes (r/o parts of libraries), never used memory allocated by brk and anonymous mmap, mmaped file which are not fetched from disk completely due to efficient processing algorithms which touch only small part of file etc, swapped out pages, dirty pages to-be-written-on-disk etc.
If you want to deal with all this complexity and figure out True count of pages, the detailed information is available at /proc/<pid>/smaps, and there are tools, like mem_usage.py or smem.pl (easily googlable) to turn it into more-or-less usable summary.
This would be the "Resident Set Size", assuming you process doesn't use swap.
Note that a process may allocate far more memory ("Virtual Memory Size"), but as long as it don't writes to the memory, it is not represented by physical memory, be it in RAM or on the disk.
Some system tools, like top, display a huge value for "swap" for each process - this is of course completly wrong, the value is the difference between VMS and RSS and most likely those unused, but allocated, memory pages.