I have used chrome with debug build on Linux for my experiment.
I ran chrome browser with debug build to run some web page, which is kind of loading lots of images.
To validate, i used "cat /proc/render-pid/statm" to check the memory.
To understand the allocation pattern, I have run with "valgrind --tool=massif --vgdb=yes --vgdb-error=0".
I see the RSS (2nd value in statm) increase from initial 133129 pages (before loading images) to 176562 pages(after loading images). And valgrind snapshot shows the memory allocated. (More than 140MB). If you calculate from statm its around 165M (Excluding Shared memory)
Next I freed up every image ( cleared up the internal caches ).
Now the Valgrind snapshot shows only 4MB of memory.
statm RSS value reduce to 161916 pages.
(I agree that Valgrind might take memory. But I have similar behavior in case of running alone. )
What I couldnt get is :
statm shows that total allocated in heap is around 165M, where as valgrind shows 140M
After freeing, valgrind shows around 4M where as statm shows increase in memory of 27 from the start of value.
So how to find where the memory is allocated??
NOTE:
valgrind doesnt show memory allocated via MMAP and SBRK (With the parameters which I passed) . But as per my understanding (With simple test program), the allocations via MMAP / SBRK will increase "Data" which 6th value in Statm output.
Hence I have ruled out that possibility.
Also i have not used "tc-malloc".
Since I am running chrome with valgrind, i see humongous amount of memory allocated. So generally I see only differences.
I calculated the private memory using RSS Memory - Shared Memory (2nd - 3rd value in statm output)
The values from statm output:
Initial Value : 370434 133129 9450 918 0 289639 0
Peak Value: 412588 175355 10145 918 0 331197 0
End Value (After freeing): 407473 161916 9573 918 0 326678 0
Related
The amount of memory used by my program reported by top is more than twice the amount reported in /proc/[pid]/statm. What explains this difference?
From the "man" page:
https://man7.org/linux/man-pages/man5/proc.5.html
/proc/[pid]/statm
Provides information about memory usage, measured in pages.
The columns are:
size (1) total program size
(same as VmSize in /proc/[pid]/status)
resident (2) resident set size
(inaccurate; same as VmRSS in /proc/[pid]/status)
shared (3) number of resident shared pages
(i.e., backed by a file)
(inaccurate; same as RssFile+RssShmem in
/proc/[pid]/status)
text (4) text (code)
lib (5) library (unused since Linux 2.6; always 0)
data (6) data + stack
dt (7) dirty pages (unused since Linux 2.6; always 0)
Some of these values are inaccurate because of a kernel-
internal scalability optimization. If accurate values are
required, use /proc/[pid]/smaps or
/proc/[pid]/smaps_rollup instead, which are much slower
but provide accurate, detailed information.
ALSO: VMsize is the "address space" that the process has in use: the number of available adresses. These addresses do not have to have any physical memory attached to them. Attached physical memory is the RSS figure.
Q: So what specific values did you have questions about?
Please see my comments below, and please refer to these links:
man top
man proc
Increase of Virtual Memory without increase of VMSize
Short answer:
RES in top (RSS is ps), this is the sum of memory that is mapped to physical pages of memory.
VMsize is the "address space" that the process has in use: the number of available adresses.
"VMSize" addresses do not (necessarily) have any physical memory attached to them.
Attached physical memory ("Resident memory") is the RSS figure.
It would also be helpful to see your actual values from top, /proc/nnn/stat and from ps.
Here's a great article (one of many):
Process Memory Management in Linux -
Baeldung.com
VSZ Memory
VSZ is short for Virtual Memory Size. It’s the total amount of
memory a process may hypothetically access.
It accounts for the size of the binary itself, any linked libraries,
and any stack or heap allocations.
When a process is started, VSZ memory becomes RSS memory...
RSS Memory
As opposed to VSZ, RSS, also short for Resident Set Size, is a measurement that shows how much RAM has been allocated to a
process during its execution
But, it isn’t actually an accurate measurement for RAM usage for a
process since it doesn’t take into account libraries already loaded in
memory...
Because RSS shows the whole memory output, even for information that
has already been preloaded, if we add up everything in the processes
list, we’d get more memory than the total memory allocated.
I hope that all helps ... at least a little.
If you've still got questions, please copy/paste some actual values from your system, so we can discuss specific examples.
I would like to compute the peak memory usage of my OCaml program when it is running in compiled form as native code. I considered using the stats API in the Gc module, but it seems to return a snapshot at the time it is called. Is there some information in the Gc module or some other module that I can use to get peak heap usage just before my program terminates?
You can get the current size of the major heap using Gc.stat, see the live_words field, multiply it by the word size in bytes to get the size in bytes (8 in a 64 bit system). It doesn't really matter but you can also add the size of the minor heap to the calculation, which is available via Gc.get () see the minor_heap_size field (again in words).
You can create an alarm with Gc.create_alarm to check the size of the heap after each major collection to get the maximum size ever used.
When running a Java app (in YARN) with native memory tracking enabled (-XX:NativeMemoryTracking=detail see https://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html and https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html), I can see how much memory the JVM is using in different categories.
My app on jdk 1.8.0_45 shows:
Native Memory Tracking:
Total: reserved=4023326KB, committed=2762382KB
- Java Heap (reserved=1331200KB, committed=1331200KB)
(mmap: reserved=1331200KB, committed=1331200KB)
- Class (reserved=1108143KB, committed=64559KB)
(classes #8621)
(malloc=6319KB #17371)
(mmap: reserved=1101824KB, committed=58240KB)
- Thread (reserved=1190668KB, committed=1190668KB)
(thread #1154)
(stack: reserved=1185284KB, committed=1185284KB)
(malloc=3809KB #5771)
(arena=1575KB #2306)
- Code (reserved=255744KB, committed=38384KB)
(malloc=6144KB #8858)
(mmap: reserved=249600KB, committed=32240KB)
- GC (reserved=54995KB, committed=54995KB)
(malloc=5775KB #217)
(mmap: reserved=49220KB, committed=49220KB)
- Compiler (reserved=267KB, committed=267KB)
(malloc=137KB #333)
(arena=131KB #3)
- Internal (reserved=65106KB, committed=65106KB)
(malloc=65074KB #29652)
(mmap: reserved=32KB, committed=32KB)
- Symbol (reserved=13622KB, committed=13622KB)
(malloc=12016KB #128199)
(arena=1606KB #1)
- Native Memory Tracking (reserved=3361KB, committed=3361KB)
(malloc=287KB #3994)
(tracking overhead=3075KB)
- Arena Chunk (reserved=220KB, committed=220KB)
(malloc=220KB)
This shows 2.7GB of committed memory, including 1.3GB of allocated heap and almost 1.2GB of allocated thread stacks (using many threads).
However, when running ps ax -o pid,rss | grep <mypid> or top it shows only 1.6GB of RES/rss resident memory. Checking swap says none in use:
free -m
total used free shared buffers cached
Mem: 129180 99348 29831 0 2689 73024
-/+ buffers/cache: 23633 105546
Swap: 15624 0 15624
Why does the JVM indicate 2.7GB memory is committed when only 1.6GB is resident? Where did the rest go?
I'm beginning to suspect that stack memory (unlike the JVM heap) seems to be precommitted without becoming resident and over time becomes resident only up to the high water mark of actual stack usage.
Yes, at least on linux mmap is lazy unless told otherwise. Anonymous pages are only backed by physical memory once they're written to (reads are not sufficient due to the zero-page optimization)
GC heap memory effectively gets touched by the copying collector or by pre-zeroing (-XX:+AlwaysPreTouch), so it'll always be resident. Thread stacks otoh aren't affected by this.
For further confirmation you can use pmap -x <java pid> and cross-reference the RSS of various address ranges with the output from the virtual memory map from NMT.
Reserved memory has been mmaped with PROT_NONE. Which means the virtual address space ranges have entries in the kernel's vma structs and thus will not be used by other mmap/malloc calls. But they will still cause page faults being forwarded to the process as SIGSEGV, i.e. accessing them is an error.
This is important to have contiguous address ranges available for future use, which in turn simplifies pointer arithmetic.
Committed-but-not-backed-by-storage memory has been mapped with - for example - PROT_READ | PROT_WRITE but accessing it still causes a page fault. But that page fault is silently handled by the kernel by backing it with actual memory and returning to execution as if nothing happened.
I.e. it's an implementation detail/optimization that won't be noticed by the process itself.
To give a breakdown of the concepts:
Used Heap: the amount of memory occupied by live objects according to the last GC
Committed: Address ranges that have been mapped with something other than PROT_NONE. They may or may not be backed by physical or swap due to lazy allocation and paging.
Reserved: The total address range that has been pre-mapped via mmap for a particular memory pool.
The reserved − committed difference consists of PROT_NONE mappings, which are guaranteed to not be backed by physical memory
Resident: Pages which are currently in physical ram. This means code, stacks, part of the committed memory pools but also portions of mmaped files which have recently been accessed and allocations outside the control of the JVM.
Virtual: The sum of all virtual address mappings. Covers committed, reserved memory pools but also mapped files or shared memory. This number is rarely informative since the JVM can reserve very large address ranges in advance or mmap large files.
We use a Tool (Whats Up Gold) to monitor memory usage on a Linux Box.
We see Memory usage (graphs) related to:
Physical, Real, Swap, Virtual Memory and ALL Memory (which is a average of all these).
'The ALL' Memory graphs show low memory usage of about: 10%.
But Physical memory shows as 95% used.
Swap memory shows as 2% used.
So, do i need more memory on this Linux Box?
In other words should i go by:
ALL Memory graph(which says memory situation is good) OR
Physical Memory Graph (which says memory situation is bad).
Real and Physical
Physical memory is the amount of DRAM which is currently used. Real memory shows how much your applications are using system DRAM memory. It is roughly lower than physical memory. Linux system caches some of disk data. This caching is the difference between physical and real memory. Actually, when you have free memory Linux goes to use it for caching. Do not worry, as your applications demand memory they gonna get the cached space back.
Swap and Virtual
Swap is additional space to your actual DRAM. This space is borrowed from disk space and once you application fill-out entire DRAM, Linux transfers some unused memory to swap to let all application stay alive. Total of swap and physical memory is the virtual memory.
Do you need extra memory?
In answer to your question, you need to check real memory. If your real memory is full, you need to get some RAM. Use free command to check the amount of actual free memory. For example on my system free says:
$ free
total used free shared buffers cached
Mem: 16324640 9314120 7010520 0 433096 8066048
-/+ buffers/cache: 814976 15509664
Swap: 2047992 0 2047992
You need to check buffer/cache section. As shown above, there are real 15 GB free DRAM (second line) on my system. Check this on your system and find out whether you need more memory or not. The lines represent physical, real, and swap memory, respectively.
free -m
as for free tool analisys about memory lack in linux i have some opinion proved by experiments (practice)
~# free -m
total used free shared buff/cache available
Mem: 2000 164 144 1605 1691 103
you should summarize 'used'+'shared' and compare with 'total'
other columns are useless just confuse and nothing more
i would say
[ total - (used + shared ) ] should be always at least > 200 MB
also you can get almost the same number if you check MemAvailable in meminfo :
# cat /proc/meminfo
MemAvailable: 107304 kB
MemAvailable - is how much memory linux thinks right now is really free before active swapping happens.
so now you can consume 107304 kB maximum. if you
consume more big swappening starts happening.
MemAvailable also is in good correlation with real practice.
I am trying to understand my embedded linux memory usage.
By using the top utility and the process file /proc/meminfo I can see how much virtual memory a process is using, and how much physical memory is available to the system. But it would seem for any given process the virtual memory can be very much higher than the used physical memory. As this is an embedded system memory swapping is disabled.(SwapTotal = 0)
How is linux calculating the free physical memory? As it doesn't seem to be accounting for everything allocated in the virtual memory space.
MemFree in /proc/meminfo is a count of how many pages are free in the buddy allocator. This buddy allocator is the fundamental unit of physical memory allocation in the kernel; however there are a lot of ways pages can be returned to the buddy allocator in time of need - for example, freeing empty SLABs, discarding cache/buffer RAM (even if this means invalidating PTEs in a running process), or as a last resort, swapping things out.
In fact, MemFree is generally controlled to be only 5-10% of total physical RAM, with any extra free RAM being co-opted into cache as time goes on. As such, MemFree alone is a very incomplete view of the overall memory situation.
As for the virtual memory (VSIZE) of a given process, this refers to the sum total of the sizes of all mapped memory segments in the process's address space. However, not all of these will be physically present - some may be paged in upon first access and as such will not register as memory in use until actually used. The resident size (RSIZE) is a more accurate view, as it only registers pages that are mapped in right now - although this may also not be accurate if a given page is mapped in multiple virtual addresses (which is very common when you consider multiple processes - shared libraries have the same physical RAM mapped to all processes that are using that library)
Try using htop. You will have to install it sudo apt-get install htop or yum install htop, whatever.
It will show you a more accurate representation of memory usage.
Basically, it comes down to "buffers/cache".
free -m
Look at the free column in the buffers/cache row, this is a more accurate representation of what is actually available.
total used free shared buffers cached
Mem: 3770 3586 183 0 112 1498
-/+ buffers/cache: 1976 1793
Swap: 7624 750 6874