This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Linux memory: buffer vs cache
When using command like ps/free to monitor memory on a Linux box, we can see a statistic called buffered memory and another called cached memory. I have searched Internet but cannot find a consistent answer for their differences. Appreciate if anyone could give me any hints.
BTW: I am debugging a program's memory usage pattern, so understanding the concepts are useful for my development.
thanks in advance,
Lin
Buffers are the I/O buffers whereas cached is the page cache.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 7 years ago.
Improve this question
I found at several places that Linux uses pages and a paging mechanism but I didn't find anywhere where this file is or how to configure it.
All the information I found is about the Linux swap file / partition. There is a difference between paging and swapping:
Paging moves pages (a small frame which contains a piece of data - usually 4 KB but can vary between different OS's) from main memory to a backbend storage, happens always as a normal function of the operating system.
Swapping moves an entire process to storage and happens when the system is memory stressed or on windows 8 when a new application is hibernating.
Does Linux uses it's swap file / partition for both cases?
If so, how could I see how many page are currently paged out? This information is not there in vmstat, free or swapon commands (or that I fail to see it).
Or is there another file used for paging?
If so, how can I configure it (and watch it's usage)?
Or perhaps Linux does not use paging at all and I was mislead?
I would appreciate if the answers will be specific to red hat enterprise Linux both versions 6 and 7 but also a general answer about all Linux's will be good.
Thanks in advance.
On Linux, the swap partition(s) are used for paging.
Linux does not respond to memory pressure by swapping out whole processes. The virtual memory system does demand paging, page by page. Under extreme memory pressure, one or more processes will be killed by the OOM killer. (There are some useful links to documentation in the first NOTE in man malloc)
There is a line in the top header which shows swap partition usage, but if that is all the information you want, use
swapon -s
man swapon for more information.
The swap partition usage is not the same as the number of unmapped pages. A page might be memory-mapped to a file using the mmap call; since that page has backing store in the file, there is no need to also write it to a swap partition, and the system won't use swap space for that. But swap partition usage is a pretty good indicator.
Also note that Linux (unlike Windows) does not allocate swap space for pages when they are allocated. Instead, it adds the new page to the virtual memory map without any backing store. and allocates the swap space when the page needs to be swapped out. The consequence (as described in the malloc manpage referenced earlier) is that a malloc call may succeed in allocating virtual memory, but a subsequent attempt to use that virtual memory may fail.
Although Linux retains the term 'swap partition' as a historical relic, it actually performs paging. So your expectation is borne out; you were just thrown by the archaic terminology.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question appears to be off-topic because it lacks sufficient information to diagnose the problem. Describe your problem in more detail or include a minimal example in the question itself.
Closed 8 years ago.
Improve this question
My Tcl script aborts, saying that it is unable to realloc 2191392 bytes. This happens when the script is kept for a longer execution duration, say, more than 10 hours. My Tcl script connects to devices using telnet and ssh connections and executes/verifies some command outputs on devices. The Linux machine has enough RAM 32GB, and ulimit is unlimited for process, data, file size. My script process doesn't eat up more memory, but the worst case is < 1GB. I just wonder why memory allocation is failed having plenty of RAM.
That message is an indication that an underlying malloc() call returned NULL (in a non-recoverable location), and given that it's not for very much, it's an indication that the system is thoroughly unable to allocate much memory. Depending on how your system is configured (32-bit vs. 64-bit; check what parray tcl_platform prints to find out) that could be an artefact of a few things, but if you think that it shouldn't be using more than a gigabyte of memory, it's an indication of a memory leak.
Unfortunately, it's hard to chase down memory leaks in general. Tcl's built-in memory command (enabled via configure --enable-symbols=mem at build time) can help, as can a tool like Electric Fence, but they are imperfect and can't generally tell you where you're getting things wrong (as you'll be looking for the absence of something to release memory). At the Tcl level, see whether each of the variables listed by info globals is of sensible size or whether there's a growing number of globals. You'll want to use tools like string length, array exists, array size and array names for this.
It's also possible for memory allocation to fail due to another process consuming so much memory that the OS starts to feel highly constrained. I hope this isn't happening in your case, since it's much harder to prevent.
I have tried to search this topic on google and this site but I can't find a proper answer.
I am trying to allocate a big continuous block of memory (a few MB) at a set physical address during the Linux booting process. But I am still not clear where I should place my "alloc_bootmem" function. I am running Linux on an ARM processor.
AFAIK, there is a way to create a driver which contains a call to "alloc_bootmem" and then compile that driver directly to into the kernel.
Another method is to add "alloc_bootmem" somewhere in the Linux kernel source.
The last method that I think exists is to create a settings file like boot.rc?(not sure) so that during booting Linux will reserve the memory I want allocated.
If there is a clear way or a link to an answer to this question, I really would appreciate everyone's help. The basic question is "where should I call "alloc_bootmem" so it will work during booting?"
Thanks,
Shahril
Take a look at: http://lwn.net/Kernel/LDD3/ chapter 8 it explains the memory allocation for early booting stages.
Further information about booting memory allocation can be found here:
https://www.kernel.org/doc/gorman/html/understand/understand022.html
This feature is used for allocating large memory chunks during the system boot up and it uses the physical rather than virtual memory. After MMU is up and running there is no possible way of accessing the memory AFAIK
If you are looking for a large continues memory allocation you should probably use different allocator take a look at:
http://lwn.net/Articles/396702/
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 2 years ago.
Improve this question
I have a linux x86 application that makes use of various third-party shared-object libraries. I suspect these libraries are leaking memory (since it can't possibly be my code ;-)
I tried the trusty valgrind, but it died a horrible death because one of the third-party libraries is using an obscure x86 instruction that valgrind doesn't implement.
I found a recommendation for DUMA and gave it a try (using the LD_PRELOAD technique to bring DUMA in at run-time), but it aborted complaining about a free operation on memory that wasn't allocated via DUMA (almost certainly by some constructor of a static object in one of the previously mentioned third-party libraries).
Are there other run-time-linkable (or otherwise not requiring a recompilation/relink) tools around that will work on linux?
Give Dr. Memory a try. It is based on DynamoRIO and shares many of the features with Valgrind.
The TotalView debugger (or, more precisely, its Memscope) has a feature set similar to the one of Valgrind.
You can also try Electric Fence (original author's link) (the origin of DUMA) for buffer overflows or touch-after-free cases (but not for memleaks, though).
In 2020, to find memory leaks on Linux, you may try:
Address Sanitizers
For both GCC(above 4.8) and Clang (above 3.1), the address sanitizer can be used, it's great
the tool has been proved useful in large projects such as Chromium and Firefox.
It's much faster than other old alternatives like Valgrind.
ASan will provide very detailed memory region information, which is very helpful for analysis of the leak.
The drawback for ASan: You need to build your program with the option -fsanitize=address; The extra memory cost is much bigger.
TCmalloc
TCmalloc can be both used with LD_PRELOAD or directly link to your program. The result can be visualized with the pprof program, it has both beautiful web UI and consoles text mode, I suggest using it if address sanitizer is not applicable in your environment(If you have a very old compiler or your PC have very limited memory to run ASan).
TCmalloc is also used in large-scale production and proved to be robust.
Linux Perf tools and BCC
Linux perf tools can also be used to find memory leaks, it's a tool based on sampling. So it can not be precise, but it's still a great tool to help us analyze the usage of memory.
There is also a script from bcc's tools.
./memleak -p $(pidof allocs)
Trace allocations and display a summary of "leaked" (outstanding)
allocations every 5 seconds
./memleak -p $(pidof allocs) -t
Trace allocations and display each individual allocator function call
./memleak -ap $(pidof allocs) 10
Trace allocations and display allocated addresses, sizes, and stacks
every 10 seconds for outstanding allocations
./memleak -c "./allocs"
Run the specified command and trace its allocations
./memleak
Trace allocations in kernel mode and display a summary of outstanding
allocations every 5 seconds
./memleak -o 60000
Trace allocations in kernel mode and display a summary of outstanding
allocations that are at least one minute (60 seconds) old
./memleak -s 5
Trace roughly every 5th allocation, to reduce overhead
The pros of such tools: We don't need to rebuild our program, so it's handy for analyzing some online services.
Heapusage is a simple run-time tool for finding memory leaks on Linux and macOS. The output logging format for leaks is quite similar to Valgrind, but it only logs definite leaks (i.e. allocations not free'd at termination).
Full disclosure: I wrote Heapusage for usage in situations when Valgrind is inadequate (high performance applications, and also for CPU architectures not supported by Valgrind).
Hello I developed a multi-threaded TCP server application that allows 10 concurrent connections receives continuous requests from them, after some processing requests, responds them to clients. I'm running it on a TI OMAP l137 processor based board it runs Monta Vista Linux. Threads are created per client ie 10 threads and it's pre-threaded. it's physical memory usage is about %1.5 and CPU usage is about %2 according to ps, top and meminfo. It's vm usage rises up to 80M where i have 48M (i reduced it from u-boot to reserve some mem for DSP). Any help is appreciated, how can i reduce it??.(/proc/sys/vm/.. tricks doesn't help :)
Thanks.
You can try using a drop in garbage collecting replacement for malloc(), and see if that solves your problem. If it does, find the leaks and fix them, then get rid of the garbage collector.
Its 'interesting' to chase these kinds of problems on platforms that most heap analyzers and profilers (e.g. valgrind) don't fully (if at all) support.
On another note, given the constraints .. I'm assuming you have decreased the default thread stack size? I think the default is 8M, you probably don't need that much. See pthread_attr_setstacksize() if you haven't adjusted it.
Edit:
You can check the default stack size with pthread_attr_getstacksize(). If it is at 8M, you've already blown your ceiling during thread creation (10 threads, as you mentioned).
Most VM is probably just for stacks. Of course, it's virtual, so it doesn't get commited if you don't use it.
(I'm wondering if thread's default stack size has anything to do with ulimit -s)
Apparently yes, according to
this other SO question
Does it rise to that level and stay there? Or does it eventually run out of memory? If the former, you simply need to figure out a way to have a smaller working set. If the latter, you have a memory leak and need to fix it.