The parent process fails with errno=12(Out of memory) when it tries to fork a child. The parent process runs on Linux 3.0 kernel - SLES 11. At the point of forking the child, the parent process has already used up around 70% of the RAM(180GB/256GB). Is there any workaround for this problem?
The application is written in C++, compiled with g++ 4.6.3.
Maybe virtual memory over commit is prevented in your system.
If it is prevented, then the virtual memory can not be bigger than sizeof physical RAM + swap. If it is allowed, then virtual memory can be bigger than RAM+swap.
When your process forks, your processes (parent and child) would have 2*180GB of virtual memory (that is too much if you don't have swap).
So, allow over commit by this way:
echo 1 > /proc/sys/vm/overcommit_memory
It should help, if child process execves immediately, or frees allocated memory before the parent writes too much to own memory. So, be careful, out of memory killer may act if both processes keep using all the memory.
man page of proc(5) says:
/proc/sys/vm/overcommit_memory
This file contains the kernel virtual
memory accounting mode. Values are: 0: heuristic overcommit (this is
the default) 1: always overcommit, never check 2: always check, never
overcommit
In mode 0, calls of mmap(2) with MAP_NORESERVE are not checked, and
the default check is very weak, leading to the risk of getting a
process "OOM-killed". Under Linux 2.4 any nonzero value implies mode
1. In mode 2 (available since Linux 2.6), the total virtual address space on the system is limited to (SS + RAM*(r/100)), where SS is the
size of the swap space, and RAM is the size of the physical memory,
and r is the contents of the file /proc/sys/vm/overcommit_ratio.
More information here: Overcommit Memory in SLES
fork-ing requires resources, since it is copy-on-writing the writable pages of the process. Read again the fork(2) man page.
You could at least provide a huge temporary swap file. You could create (on some file system with enough space) a huge file $SWAPFILE with
dd if=/dev/zero of=$SWAPFILE bs=1M count=256000
mkswap $SWAPFILE
swapon $SWAPFILE
Otherwise, you could for instance design your program differently, e.g. mmap-ing some big file (and munmap-ing it just before the fork, and mmap-ing it again after), or more simply starting at the beginning of your program a popen-ed shell, or a p2open-ed one or making explicitly the pipe-s to and from it (probably a multiplexing call à la poll would also be useful), and later issue commands to it.
Maybe we could help more if we had an idea of what your program is doing, why does it consume so much memory, and why and what is it forking...
Read Advanced Linux Programming for more.
PS.
If you fork just to run gdb to show the backtrace, consider simpler alternatives like recent GCC's libbacktrace or Wolf's libbacktrace...
A nicer solution on Linux would be to use vfork or posix_spawn (as it will try to use vfork if possible): vfork "creates new processes without copying the page tables of the parent process", so it will work even if your application uses more than 50% of RAM available.
Note that std::system and QProcess::execute also use fork under the hood, there is even a ticket about this problem in Qt framework: https://bugreports.qt.io/browse/QTBUG-17331
Related
When I searched about "What happens if malloc and exit with not free?", I could find answers saying "Today, OS will recover all the allocated memory space after a program exit".
In that answer, what is the meaning of "recover"?.
OS just delete it's PCB and page table when process exit doesn't it?
Are there additional tasks OS has to do for complete termination of process?
When a program starts, the OS allocates some memory to it. During the programs execution the program can request more blocks of memory from the OS and it can release them as well when it doesn't need them any more. When the program exits, all the memory used by it is returned to the OS.
The pair malloc()/free() (and their siblings and equivalents) do not interact with the OS1. They manage a block of memory (called "heap") the program already got from the OS when it was launched.
All in all, from the OS point of view it doesn't matter if your program uses free() or not. For the program it's important to use free() when a piece of memory is not needed to let further allocations succeed (by reusing the freed memory blocks).
1 This is not entirely true. The implementation of malloc() may get more memory blocks from the OS to extend the heap when it is full but this process is transparent to the program. For the program's point of view, malloc() and free() operate inside a memory block that already belongs to the program.
The operating system allocates and manages memory pages in a process. As part of the process cleanup at exit, the operating system has to deallocate the pages assigned to a process. This includes the page tables, page file space, and physical page frames mapped to logical pages. This takes is complicated because multiple processes may map to the same physical page frames which requires some form of reference counting.
The heap is just memory. The operating system has no knowledge whatsoever of process heaps. Inside malloc (and similar functions), there will be calls to operating system services to map pages to the process address space. The operating system creates the pages but does not care what the pages are used for.
If you do malloc's without corresponding free's your process will keep requesting more and more pages from the operating system (until you reach the point where the system will fail to allocate more pages). All you are doing is screwing up the heap within your own process.
When the process exits, the operating system will just get rid of the the pages allocated to the heap and your application's failure to call free will cause no problem to the system at all.
Thus, there is a two level system at work. malloc allocates bytes. The operating system allocates pages.
Why is memory consumption jumping unpredictably as I step through a program in the gdb debugger? I'm trying to use gdb to find out why a program is using far more memory than it should, and it's not cooperating.
I step through the source code while monitoring process memory usage, but I can't find what line(s) allocate the memory for two reasons:
Reported memory usage only jumps up in increments of (usually, but not always exactly) 64 MB. I suspect I'm seeing the effects of some memory manager I don't know about which reserves 64 MB at a time and masks multiple smaller allocations.
The jump doesn't happen at a consistent location in code. Not only does it occur on different lines during different gdb runs; it also sometimes happens in illogical places like the closing bracket of a (c++) function. Is it possible that gdb itself is affecting memory allocations?
Any ideas/suggestions for more effective tools to help me drill down to the code lines that are really responsible for these memory allocations?
Here's some relevant system info: I'm running x86_64-redhat-linux-gnu version 7.2-64.el6-5.2 on a virtual CentOS Linux machine under Windows. The program is built on a remote server via a complicated build script, so tracking down exactly what options were used at any point is itself a bit of a chore. I'm monitoring memory usage both with the top utility ("virt" or virtual memory column) and by reading the real-time monitoring file /proc/<pid>/status, and they agree. Since this program uses a large suite of third-party libraries, there may be one or more overridden malloc() functions involved somewhere that I don't know about--hunting them down is part of this task.
gdb, left to its own devices, will not affect the memory use of your program, though a run under gdb may differ from a standalone run for other reasons.
However, this also depends on the way you use gdb. If you are just setting simple breakpoints, stepping, and printing things, then you are ok. But sometimes, to evaluate an expression, gdb will allocate memory in the inferior. For example, if you have a breakpoint condition like strcmp(arg, "string") == 0, then gdb will allocate memory for that string constant. There are other cases like this as well.
This answer is in several parts because there were several things going on:
Valgrind with the Massif module (a memory profiler) was much more helpful than gdb for this problem. Sometimes a quick look with the debugger works, sometimes it doesn't. http://valgrind.org/docs/manual/ms-manual.html
top is a poor tool for profiling memory usage because it only reports virtual memory allocations, which in this case were about 3x the actual heap memory usage. Virtual memory is mapped and made available by the Unix kernel when a process asks for a memory block, but it's not necessarily used. The underlying system call is mmap(). I still don't know how to check the block size. top can only tell you what the Unix kernel knows about your memory consumption, which isn't enough to be helpful. Don't use it (or the memory files under /proc/) to do detailed memory profiling.
Memory allocation when stepping out of a function was caused by autolocks--that's a thread lock class whose destructor releases the lock when it goes out of scope. Then a different thread goes into action and allocates some memory, leaving the operator (me) mystified. Non-repeatability is probably because some threads were waiting for external resources like Internet connections.
I am using ioctl() to read data from a block device (scsi.)
I have noticed that when I read 1024 sectors, ioctl finishes without a problem. When I read 2048, after a few long moments it returns ENOMEM (errno=12) which is not even listed on the list of possible errors (see http://man7.org/linux/man-pages/man2/ioctl.2.html)
I have tripple checked that I am passing proper buffer size, so this cannot be the case - no buffer overrun.
How can I learn the largest buffer size to be read using ioctl then?
Edit 1
Some additional information may become helpful:
Enterprise Linux Enterprise Linux Server release 5.3 (Carthage)
Red Hat Enterprise Linux Server release 5.3 (Tikanga)
2.6.18-128.el5
The error occurs as the default behaviour of Linux memory management, which is "overcommiting". This means that the kernel claims to allocate memory successfuly, but doesn't actually allocate the memory until later when you try to access it. If the kernel finds out that it's allocated too much memory, it kills a process with "the ENOMEM (Out Of Memory) killer" to free up some memory. The way it picks the process to kill is complicated, but if you have just allocated most of the memory in the system, it's probably going to be your process that gets the bullet.
To change this behaviour try the following :
echo "2" > /proc/sys/vm/overcommit_memory
On Linux 2.6 and later, it is possible to modify the kernel's behavior so that it will not "overcommit" memory. Although this setting will not prevent the OOM killer from being invoked altogether, it will lower the chances significantly and will therefore lead to more robust system behavior. This is done by selecting strict overcommit mode via sysctl:
sysctl -w vm.overcommit_memory=2
I have written a program which works on huge set of data. My CPU and OS(Ubuntu) both are 64 bit and I have got 4GB of RAM. Using "top" (%Mem field), I saw that the process's memory consumption went up to around 87% i.e 3.4+ GB and then it got killed.
I then checked how much memory a process can access using "uname -m" which comes out to be "unlimited".
Now, since both the OS and CPU are 64 bit and also there exists a swap partition, the OS should have used the virtual memory i.e [ >3.4GB + yGB from swap space ] in total and only if the process required more memory, it should have been killed.
So, I have following ques:
How much physical memory can a process access theoretically on 64 bit m/c. My answer is 2^48 bytes.
If less than 2^48 bytes of physical memory exists, then OS should use virtual memory, correct?
If ans to above ques is YES, then OS should have used SWAP space as well, why did it kill the process w/o even using it. I dont think we have to use some specific system calls which coding our program to make this happen.
Please suggest.
It's not only the data size that could be the reason. For example, do ulimit -a and check the max stack size. Have you got a kill reason? Set 'ulimit -c 20000' to get a core file, it shows you the reason when you examine it with gdb.
Check with file and ldd that your executable is indeed 64 bits.
Check also the resource limits. From inside the process, you could use getrlimit system call (and setrlimit to change them, when possible). From a bash shell, try ulimit -a. From a zsh shell try limit.
Check also that your process indeed eats the memory you believe it does consume. If its pid is 1234 you could try pmap 1234. From inside the process you could read the /proc/self/maps or /proc/1234/maps (which you can read from a terminal). There is also the /proc/self/smaps or /proc/1234/smaps and /proc/self/status or /proc/1234/status and other files inside your /proc/self/ ...
Check with free that you got the memory (and the swap space) you believe. You can add some temporary swap space with swapon /tmp/someswapfile (and use mkswap to initialize it).
I was routinely able, a few months (and a couple of years) ago, to run a 7Gb process (a huge cc1 compilation), under Gnu/Linux/Debian/Sid/AMD64, on a machine with 8Gb RAM.
And you could try with a tiny test program, which e.g. allocates with malloc several memory chunks of e.g. 32Mb each. Don't forget to write some bytes inside (at least at each megabyte).
standard C++ containers like std::map or std::vector are rumored to consume more memory than what we usually think.
Buy more RAM if needed. It is quite cheap these days.
In what can be addressed literally EVERYTHING has to fit into it, including your graphics adaptors, OS kernel, BIOS, etc. and the amount that can be addressed can't be extended by SWAP either.
Also worth noting that the process itself needs to be 64-bit also. And some operating systems may become unstable and therefore kill the process if you're using excessive RAM with it.
I'm using Slackware 12.2 on an x86 machine. I'm trying to debug/figure out things by dumping specific parts of memory. Unfortunately my knowledge on the Linux kernel is quite limited to what I need for programming/pentesting.
So here's my question: Is there a way to access any point in memory? I tried doing this with a char pointer so that It would only be a byte long. However the program crashed and spat out something in that nature of: "can't access memory location". Now I was pointing at the 0x00000000 location which where the system stores it's interrupt vectors (unless that changed), which shouldn't matter really.
Now my understanding is the kernel will allocate memory (data, stack, heap, etc) to a program and that program will not be able to go anywhere else. So I was thinking of using NASM to tell the CPU to go directly fetch what I need but I'm unsure if that would work (and I would need to figure out how to translate MASM to NASM).
Alright, well there's my long winded monologue. Essentially my question is: "Is there a way to achieve this?".
Anyway...
If your program is running in user-mode, then memory outside of your process memory won't be accessible, by hook or by crook. Using asm will not help, nor will any other method. This is simply impossible, and is a core security/stability feature of any OS that runs in protected mode (i.e. all of them, for the past 20+ years). Here's a brief overview of Linux kernel memory management.
The only way you can explore the entire memory space of your computer is by using a kernel debugger, which will allow you to access any physical address. However, even that won't let you look at the memory of every process at the same time, since some processes will have been swapped out of main memory. Furthermore, even in kernel mode, physical addresses are not necessarily the same as the addresses visible to the process.
Take a look at /dev/mem or /dev/kmem (man mem)
If you have root access you should be able to see your memory there. This is a mechanism used by kernel debuggers.
Note the warning: Examining and patching is likely to lead to unexpected results when read-only or write-only bits are present.
From the man page:
mem is a character device file that is an image of
the main memory of the computer. It may be used, for
example, to examine (and even patch) the system.
Byte addresses in mem are interpreted as physical
memory addresses. References to nonexistent locations
cause errors to be returned.
...
The file kmem is the same as mem, except that the
kernel virtual memory rather than physical memory is
accessed.