I have a program that collects various kstat information on our Solaris systems and, now that we've introduced Linux into our data center, I'd like to do the same for Linux.
I'm having trouble, however, finding equivalents for many of the kstats. I was wondering if there is a library or utility that mimics kstats for the Linux environment. Even a partial implementation would be helpful.
As of right now, I've been parsing files in /proc but finding the right information has been hit or miss. For example, kstat has the following data:
unix::vminfo
swap_alloc
swap_avail
swap_free
swap_resv
In Linux, you have the entries "SwapTotal" and "SwapFree" but
a) It appears that swap_free actually corresponds to "SwapTotal" and swap_avail corresponds to "SwapFree"
b) I can't find values for swap_avail (Maybe SwapTotal minus SwapFree?) now swap_resv
Any ideas?
I'm not aware of a Linux kstat implementation but anyway, you are first facing a terminology issue here.
The Solaris kstats swap statistics you are referencing are using "swap" to mean the whole virtual memory, i.e. the swap areas plus a large part of the RAM.
On the other hand, the Linux SwapTotal and SwapFree statistics are only related to the swap area (i.e. on disk).
Another issue is Linux overcommit memory allocation so a memory reservation counter might not be maintained and wouldn't be useful anyway.
There is this meminfo documentation take 2 article on LWN which describes all fields from /proc/meminfo and says the following about SwapTotal and SwapFree:
SwapTotal: total amount of swap space available
SwapFree: Memory which has been evicted from RAM, and is temporarily
on the disk
There is also some discussion at http://kerneltrap.org/node/4097.
Perl version:
https://github.com/zfsonlinux/linux-kstat
"This is an implementation of the Sun::Solaris::Kstat Perl module
for Linux ZFS. It should behave identically to the Solaris version."
Ruby version:
https://www.rubydoc.info/gems/linux-kstat/Linux/Kstat
"The Kstat class encapsulates Linux kernel statistics derived from /proc/stat."
Related
I am using GNU Clisp to compute a very big matrix represented as a hash table of hash tables. The ultimate hash values being stored are single floats.
The program seems to run out of memory after a while and I am thinking I need to
change the variable type somehow of the ultimate values so as to use less memory
have the operating system allocate more memory
somehow use virtual memory from the hard drive; or some combination thereof.
Any suggestions? I did a lot of searches and could not find anything.
You can use short-float - they are immediate on all platforms that CLISP supports.
Depending on your platform you might want to use the -m option to allocation more memory, but I don't think this make any difference on a modern platform - CLISP will allocate all it needs as it goes, up to the physical + swap.
virtual memory (swap) should be enabled using OS. Note that it is very slow compared to physical RAM, so it should be relied upon judiciously.
So is meminfo the best way to say there is X free memory? I ask this question because my company states that this is not an accurate way to represent free memory. In fact, it is the sum of memfree, cached, buffers, slab and if on a vmware box, the vmware ballon is added from /proc/vmmemctl. thoughts?
Regarding the vmware host case, I had a similar problem with one or our servers, see https://serverfault.com/questions/435307/redhat-linux-server-paging-sum-of-res-rss-buffers-cached-total-who-is-u
I think the problem with meminfo is that it doesn't clearly show the memory used by vmmemctl. Since vmmemctl is a driver, the memory it uses doesn't appear also in top/ps.
I believe that /proc/meminfo is accurate (but I can't prove that), and more generally the /proc pseudo-file system gives you a lot of information about the system (and each process).
Of course it gives information about the kernel providing it (so the kernel running inside your VMware virtual machine).
If you are concerned by a specific process, see also this answer
I want to get the map of allocated memory in RAM running on Linux.
I am looking to find the utilization of memory at a given time as well as specifics of allocation in terms of user process and kernel modules and kernel itself.
This is very very hard to get right because of shared memory, caching and on demand paging.
You can indeed use /proc/PID/maps as the previous answer stated to learn that glibc, as an example, occupies a certain range of virtual memory in a certain process but part of that is shared between all processes mapping that library (the code section), part isn't (the dynamic linker tables, for example). Part of this memory space might not be in RAM at all (paged to disk) anc opy on write semantics might mean that the memory picture at the next moment might be very different.
Add do that the sophisticated use Linux makes in caching (the page and buffer caches being the major ones) which part of which can be evicted at the kernel whim (cache IO buffers which are not dirty) but some cannot (e.g. tmpfs pages) and it gets really hairy really quickly.
In short - no one good answer to get a true view of what uses RAM and for what in a Linux system. The best answer I know is pagemap and related tool. read all about it here: http://lwn.net/Articles/230975/
You can find it out by checking ever process memory mapping
cat /proc/<PID>/maps
and for overall memory state
cat /proc/meminfo
This question is related to Java Refuses To Start - Could Not Resrve Enough Space for Object Heap and should be easy enough to figure out. However; my searches haven't yielded anything useful.
Essentially we have 2 32 bit OS's (RedHat & SuSE) on different machines with the same hardware. Both use the same JVM both executing the same command line. RedHat works perfectly fine but SuSE reports there isn't enough Memory.
We just need to know if this is a limitation of the version of SuSE we're using or if it's something else.
'cat /proc/version' gives us:
Linux version 2.6.5-7.244-bigsmp (geeko#buildhost) (gcc version 3.3.3 (SuSE Linux)) #1 SMP Mon Dec 12 18:32:25 UTC 2005
'uname -a' gives us the following on BOTH types of machines:
UTC 2005 i686 i686 i386 GNU/Linux
The JVM memory limit is related the largest free contiguous block available, not the amount of free memory. The limit varies from about 1.4 GB to a bit over 2.0 GB, and depends on where your operating system puts various things in memory. I don't know the particulars of where Redhat or Suse load stuff into memory, but it could be that suse is mapping some library to an address in the middle of RAM, where Redhat might map it at the end (speculating).
And remember that your actual memory usage in java is more than what you specify for Xmx. The other memory settings also affect the size of your heap (like permgen). So it could also be that the perm space on Suse has a larget default than on Redhat.
Also, depending on the memory allocation profile of your application, you might get away with a smaller heap size and different garbage collecting options. There are some details here (http://java.sun.com/performance/reference/whitepapers/tuning.html) and other places. For example, if you allocate a lot of small, temporary blocks, you'll want different GC settings than if you have a lot of bit, long-lived objects.
Regarding the linked question, why not just use Redhat? That might be a simplistic solution, but I guarantee it's going to fix your problem faster than deeply delving into the arcane world of java tuning and OS memory management :P
Firstly, you are crazy to be running a 32-bit OS when you have this much address space pressure. Migrate to a 64-bit JVM on 64-bit Linux. How much time have you wasted already trying to diagnose this problem which you must have suspected from the outset would go away with the larger address space of a 64-bit system ?
Secondly, it's well known that out of all the Linux vendors Red Hat has the most kernel engineers on staff and makes some serious tweaks for the kernels in their RHEL products. These often include patches for large workloads like yours (well, it's a large workload for a 32-bit system, it's nothing special on 64-bit). So there's some chance the reason ultimately is that RHEL has other customers doing the same crazy stuff as you and you're benefiting from work they did to support those customers.
Finally though, since I suspect you're going to insist on trying to find a way to do this on 32-bit SuSE I will point out that Linux offers a variety of address space trade-offs on 32-bit x86, and it's possible (but not certain) that your SuSE systems just have a different trade-off selected. If you can bring up the configuration of the running kernels (often in /boot/config....) then you can compare settings like HIGHMEM.
The conventional option until a few years ago was 2:2 split, that is userspace is limited to 2GiB of address space, an easy solution to program and it has decent efficiency but in this scenario obviously you can't have your requested heap since it would leave no space for the program text, stack etc. More recently the trend has been for 3:1 (similar to the Windows /3GB switch) which expands userspace address space at the cost of cramming the OS kernel itself into less space which potentially causes its own problems. This might work, but it would be very cramped so I also wouldn't be surprised if it didn't work for your jobs. Finally newer Linux kernels also offer an option where you get 4GiB 32-bit userspace, which might be enough to make your jobs run reliably, at a significant performance cost since then obviously userspace and kernel addresses can't co-exist.
To try this you'd need a new kernel. You may be able to just install one provided by SuSE (see if they offer others to choose from, e.g. a "PAE" option) or you may have to compile your own, in which case it probably invalidates your support contract.
But really, you should just go with option 1, switch to a 64-bit JVM and put your feet up.
How does free calculate used memory and why does it differ from what /proc reports?
# cat /proc/*/status | grep VmSize | awk '{sum += $2} END {print sum}'
281260
But free says:
# free
total used free shared buffers cached
Mem: 524288 326488 197800 0 0 0
Who is right? Is 281260kb memory used or 326488kb?
The title asks: "How does free calculate used memory?"
Answer: It asks the OS, which has to keep track of that to do its job.
More specifically, it asks the memory management subsystem. As sheepsimulator notes in the comments, the Linux kernel exposes all kinds OS maintained data in the /proc virtual filesystem, but every full service OS has to keep track of them kind of data, so it is a small matter to provide an API for free to use.
The question asks: "Why does this differ from adding up the VmSize reported for all processes?"
Answer: There are at least to thing going on here
Linux will promise memory to a program without actually allocating it. When you do char *p=new(1024*1024*1024*sizeof(char)); the kernel doesn't go out to get you a gigabyte right away. It just says "OK", and figures it will grab it when you start using it. Thus the need for the infamous OOM killer.
Dynamic libraries are shared, and a single page of real memory can be mapped into the virtual address space of more than one process.
Furthermore, your pass over the proc filesystem is not atomic.
The upshot is that the output of free more accurately reflects the use of physical memory on your machine at a given moment.
By 'free' I assume you mean the standard Linux version, which is usually comes from the procps suite of command line tools. Different versions of free (such as the one from busybox) report different numbers.
The procps version of 'free' obtains information about the system memory by reading /proc/meminfo. There is a syscall (sysinfo) which is also available to get memory numbers from the kernel. This can be used if a system does not have the /proc filesystem, but that is rare outside of deeply embedded systems, and procps free does not use that syscall to my knowledge.
The calculation for "used" memory is derived by taking the total memory, and subtracting free memory, cached memory, reclaimable slab memory and buffer memory. The formula, using the names from /proc/meminfo, is:
used = MemTotal - MemFree - Cached - SReclaimable - Buffers
Note that free does not reference the Vm* values for any individual processes. These are numbers for virtual memory usage, which likely do not match the physical memory usage for a process. The numbers that free reports are for physical memory.
The result from 'free' is more likely accurate than adding up the Virtual Memory size of each process (which is just virtual memory afterall, might even add up to more memory than is physically present!)
/proc/meminfo will give you some more of the details than 'free'.