I have written a program which works on huge set of data. My CPU and OS(Ubuntu) both are 64 bit and I have got 4GB of RAM. Using "top" (%Mem field), I saw that the process's memory consumption went up to around 87% i.e 3.4+ GB and then it got killed.
I then checked how much memory a process can access using "uname -m" which comes out to be "unlimited".
Now, since both the OS and CPU are 64 bit and also there exists a swap partition, the OS should have used the virtual memory i.e [ >3.4GB + yGB from swap space ] in total and only if the process required more memory, it should have been killed.
So, I have following ques:
How much physical memory can a process access theoretically on 64 bit m/c. My answer is 2^48 bytes.
If less than 2^48 bytes of physical memory exists, then OS should use virtual memory, correct?
If ans to above ques is YES, then OS should have used SWAP space as well, why did it kill the process w/o even using it. I dont think we have to use some specific system calls which coding our program to make this happen.
Please suggest.
It's not only the data size that could be the reason. For example, do ulimit -a and check the max stack size. Have you got a kill reason? Set 'ulimit -c 20000' to get a core file, it shows you the reason when you examine it with gdb.
Check with file and ldd that your executable is indeed 64 bits.
Check also the resource limits. From inside the process, you could use getrlimit system call (and setrlimit to change them, when possible). From a bash shell, try ulimit -a. From a zsh shell try limit.
Check also that your process indeed eats the memory you believe it does consume. If its pid is 1234 you could try pmap 1234. From inside the process you could read the /proc/self/maps or /proc/1234/maps (which you can read from a terminal). There is also the /proc/self/smaps or /proc/1234/smaps and /proc/self/status or /proc/1234/status and other files inside your /proc/self/ ...
Check with free that you got the memory (and the swap space) you believe. You can add some temporary swap space with swapon /tmp/someswapfile (and use mkswap to initialize it).
I was routinely able, a few months (and a couple of years) ago, to run a 7Gb process (a huge cc1 compilation), under Gnu/Linux/Debian/Sid/AMD64, on a machine with 8Gb RAM.
And you could try with a tiny test program, which e.g. allocates with malloc several memory chunks of e.g. 32Mb each. Don't forget to write some bytes inside (at least at each megabyte).
standard C++ containers like std::map or std::vector are rumored to consume more memory than what we usually think.
Buy more RAM if needed. It is quite cheap these days.
In what can be addressed literally EVERYTHING has to fit into it, including your graphics adaptors, OS kernel, BIOS, etc. and the amount that can be addressed can't be extended by SWAP either.
Also worth noting that the process itself needs to be 64-bit also. And some operating systems may become unstable and therefore kill the process if you're using excessive RAM with it.
Related
I am using Linux kernel 2.6.38, and I am running a process that allocates 4GB of memory, and I have a 4GB of ram available, so when I run my application it allocates around 0.5GB from swap space. however, my application runs for a very long time and accesses data on the swap space several times.
(Edited)
To clarify what I am doing:
I am running Linux 2.6.38, with 4 GB of RAM.
without running any applications, the system is occupying around 500MB of RAM.
I created a simple application that allocates 4GB of memory and seeks across allocated memory and changes the values of that memory many times (loop of 10 iterations).
it is obvious that I will need the swap space in order for application to run.
when I run my application, the swap space keeps accumulating and becomes full after few iterations, and the process is killed.
after the process is killed the swap space remains full as well.
I tested my application on a more recent kernels and it works fine, the swap space does not accumulate.
is this a bug on this kernel version (2.6.38)? is there a fix to it?
There's no memory leak.
You're assuming that when your application needs more memory than what's available, parts of it is written to swap. This is not necessarily true.
The system may (and generally will) write other, completely unrelated processes to swap, because they're not currently in use.
Since this swap space does not belong to your application, it will remain in use after your application exits.
This swap space may further stay in use for a long time since Linux doesn't preemptively load them back when there's free RAM.
I'm not sure my response will answer your question but I asked myself a similar question a while back.
To summarise when Linux allocates memory (RAM/SWAPP) it only frees it when it's needed. That means even after the process has terminated the allocated memory will remain until another process needs the space.
However if you want to free the SWAPP you can do it manually
sudo swapoff -a
Do not forget to turn it back on
sudo swapon -a
You can find more information at that link and that one
I have a Ubuntu 12.10 (kernel 3.9.0-rc2) installation running on VMWare. I've given it 512MB RAM.
cat /proc/meminfo shows:
MemTotal: 507864 KB
MemFree: 440180
I want to use the swap (for some reason) so I wrote a C program which allocates a 500MB array (using malloc()) and fills it with junk. However, the program gets killed before it can fill the entire array and a message "Killed" appears on the screen.
I wanted to ask if this is normal behavior and what is the reason behind this? In my opinion, the swap should be used because the free RAM is insufficient.
Edit: I did not mention that I have 1GB swap. cat /proc/swaps shows:
/dev/sda5 Size: 1046524 Used: 14672.
The "Used" amount increases when I run the memory-eating program. But as you can see, a lot of swap is leftover. So why did the program have to be 'Killed'?
So I couldn't find a valid answer. I have a temporary solution:
I had modified the Virtual Machine settings to give 512MB RAM to the VM. Now I reverted back to 2GB and ran 5 parallel programs each consuming 500MB. Thankfully, all of them run and the swap gets used.
I just needed to use the swap for a project on swap management.
It also matters how you have written your C program to allocate the memory and what are the compiler flags. For example, if you are statically allocating memory (such as double A[N][N]), the behaviour is different from dynamically allocating it: (such as using malloc/calloc). Static allocations are limited by the memory model of the compiler (medium, small etc, often can be specified). Perhaps, a good starting point is :
http://en.wikipedia.org/wiki/C_dynamic_memory_allocation
Does this help?
The parent process fails with errno=12(Out of memory) when it tries to fork a child. The parent process runs on Linux 3.0 kernel - SLES 11. At the point of forking the child, the parent process has already used up around 70% of the RAM(180GB/256GB). Is there any workaround for this problem?
The application is written in C++, compiled with g++ 4.6.3.
Maybe virtual memory over commit is prevented in your system.
If it is prevented, then the virtual memory can not be bigger than sizeof physical RAM + swap. If it is allowed, then virtual memory can be bigger than RAM+swap.
When your process forks, your processes (parent and child) would have 2*180GB of virtual memory (that is too much if you don't have swap).
So, allow over commit by this way:
echo 1 > /proc/sys/vm/overcommit_memory
It should help, if child process execves immediately, or frees allocated memory before the parent writes too much to own memory. So, be careful, out of memory killer may act if both processes keep using all the memory.
man page of proc(5) says:
/proc/sys/vm/overcommit_memory
This file contains the kernel virtual
memory accounting mode. Values are: 0: heuristic overcommit (this is
the default) 1: always overcommit, never check 2: always check, never
overcommit
In mode 0, calls of mmap(2) with MAP_NORESERVE are not checked, and
the default check is very weak, leading to the risk of getting a
process "OOM-killed". Under Linux 2.4 any nonzero value implies mode
1. In mode 2 (available since Linux 2.6), the total virtual address space on the system is limited to (SS + RAM*(r/100)), where SS is the
size of the swap space, and RAM is the size of the physical memory,
and r is the contents of the file /proc/sys/vm/overcommit_ratio.
More information here: Overcommit Memory in SLES
fork-ing requires resources, since it is copy-on-writing the writable pages of the process. Read again the fork(2) man page.
You could at least provide a huge temporary swap file. You could create (on some file system with enough space) a huge file $SWAPFILE with
dd if=/dev/zero of=$SWAPFILE bs=1M count=256000
mkswap $SWAPFILE
swapon $SWAPFILE
Otherwise, you could for instance design your program differently, e.g. mmap-ing some big file (and munmap-ing it just before the fork, and mmap-ing it again after), or more simply starting at the beginning of your program a popen-ed shell, or a p2open-ed one or making explicitly the pipe-s to and from it (probably a multiplexing call à la poll would also be useful), and later issue commands to it.
Maybe we could help more if we had an idea of what your program is doing, why does it consume so much memory, and why and what is it forking...
Read Advanced Linux Programming for more.
PS.
If you fork just to run gdb to show the backtrace, consider simpler alternatives like recent GCC's libbacktrace or Wolf's libbacktrace...
A nicer solution on Linux would be to use vfork or posix_spawn (as it will try to use vfork if possible): vfork "creates new processes without copying the page tables of the parent process", so it will work even if your application uses more than 50% of RAM available.
Note that std::system and QProcess::execute also use fork under the hood, there is even a ticket about this problem in Qt framework: https://bugreports.qt.io/browse/QTBUG-17331
If I go to WHM and see my server's memory usage, it says that only 16% of memory is in use.
But when I connect to server using SSH and run command "free -m" then it shows that 80% is in use. Why is that? I want to know exact memory usage of all applications running like MySQL, Apache e.t.c.
How do I view that?
Thanks
As they say, "It's Complicated".
Linux uses unused memory for disk buffering and caching. It speeds things up. But you may need to look at the -/+ buffers/cache line of free.
'ps' can show you, for any given process, or for all processes, the %cpu, %mem, cumulative cpu-time, rss (resident set size, the non-swapped physical memory that a process is using), size (very approximate amount of swap space that would be required if the process were to dirty all writable pages and then be swapped out), vsize (virtual memory usage of entire process (vm_lib + vm_exe + vm_data + vm_stack)), and much much more.
For any given process, you can cat /proc/$PID/status -- it's human readable -- and check out the VmSize, VmLck, VmRSS, VmData, VmStk, VmExe, VmLib, and VmPTE values, along with others...
But that's just for starters... Processes can allocate memory but not use it. (Memory can be allocated, but the memory pages are not created/issued until they're actually used. That whole on-demand thing.)
Processes can map in hardware space, showing up as using a large quantity of memory that's not actually coming from system RAM. (X-servers are known to sometimes do this. It's some wonky stuff involved kernel drivers...)
There's the executable, which is usually a memory-mapped file. Meaning that parts that are swapped-in are taking up RAM, but when swapped out it never takes up swapfile space.
Processes can have other memory-mapped files...
There's shared-memory libraries, where the same RAM pages are used by multiple programs concurrently.
So we have to ask, as far as memory goes, what exactly counts and what doesn't?
Linux noob question:
If I have 500MB of RAM, and 500MB of swap space, can the OS and processes then use 1GB of memory?
In other words, is the total amount of memory available to programs and the OS the total of the physical memory size and swap size?
I'm trying to figure out which SNMP counters to query, but need to understand how Linux uses virtual memory a little better first.
Thanks
Actually, it IS essentially correct, but your "virtual" memory does NOT reside beside your "physical memory" (as Matthew Scharley stated).
Your "virtual memory" is an abstraction layer covering both "physical" (as in RAM) and "swap" (as in hard-disk, which is of course as much physical as RAM is) memory.
Virtual memory is in essention an abstraction layer. Your program always addresses a "virtual" address, which your OS translates to an address in RAM or on disk (which needs to be loaded to RAM first) depending on where the data resides. So your program never has to worry about lack of memory.
Nothing is ever quite so simple anymore...
Memory pages are lazily allocated. A process can malloc() a large quantity of memory and never use it. So on your 500MB_RAM + 500MB_SWAP system, I could -- at least in theory -- allocate 2 gig of memory off the heap and things will run merrily along until I try to use too much of that memory. (At which point whatever process couldn't acquire more memory pages gets nuked. Hopefully it's my process. But not always.)
Individual processes may be limited to 4 gig as a hard address limitation on 32-bit systems. Even when you have more than 4 gig of RAM on the machine and you're using that bizarre segmented 36-bit atrocity from hell addressing scheme, individual processes are still limited to only 4 gigs. Some of that 4 gigs has to go for shared libraries and program code. So yer down to 2-3 gigs of stack+heap as an ADDRESSING limitation.
You can mmap files in, effectively giving you more memory. It basically acts as extra swap. I.e. Rather than loading a program's binary code data into memory and then swapping it out to the swapfile, the file is just mmapped. As needed, pages are swapped into RAM directly from the file.
You can get into some interesting stuff with sparse data and mmapped sparse files. I've seen X-windows claim enormous memory usage when in fact it was only using up a tiny bit.
BTW: "free" might help you. As might "cat /proc/meminfo" or the Vm lines in /proc/$PID/status. (Especially VmData and VmStk.) Or perhaps "ps up $PID"
Although mostly it's true, it's not entirely correct. For a particular process, the environment you run it in may limit the memory available to your process. Check the output of ulimit -v as well.
Yes, this is essentially correct. The actual numbers might be (very) marginally lower, but for all intents and purposes, if you have x physical memory and y virtual memory (swap in linux), then you have x + y memory available to the operating system and any programs running underneath the OS.