memory utilization by a linux process - linux

I have a lot of instances of a process running on my host, each seemingly consuming large amount of memory.
ps aux on the processes give me the following information
blah1 18634 0.0 0.4 131852 31188 pts/15 Ssl+ 00:27 0:00 myPgm
blah2 18859 0.0 0.3 131292 30656 pts/32 Sl+ 01:17 0:00 myPgm
blah3 19813 0.0 0.4 131960 31220 pts/44 Ssl+ 01:17 0:00 myPgm
blah4 20228 0.1 0.3 131728 31036 pts/54 Ssl+ 01:41 0:00 myPgm
blah5 20238 0.0 0.3 131688 30932 pts/20 Sl+ Nov15 0:00 myPgm
blah6 21181 0.0 0.3 131304 30632 pts/25 Sl+ Nov15 0:00 myPgm
blah7 21278 0.0 0.3 131824 31096 pts/61 Ssl+ Nov15 0:00 myPgm
blah8 21821 0.0 0.3 131444 30808 pts/7 Sl+ 00:54 0:00 myPgm
So VSZ is always around 130 MB and RSS around 30 MB. pmap for the processes have the following data:
For 18634:
mapped: 131852K writeable/private: 59692K shared: 28K
For 21181:
mapped: 131304K writeable/private: 59144K shared: 28K
and similar values for other processes as well. The host has 7GB of physical memory. At times I have around 700 to 800 instances of the same process running on the host. I am trying to understand how much memory each process consumes in reality. If i take "writeable/private" as the actual memory usage in each process then 58MB for each process would lead to 45 GB (for 800 process) which is crazy. Can anyone explain if im doing it wrong and how should be calculation be done?
Also free -k gives
total used free shared buffers cached
Mem: 7782580 4802104 2980476 0 380192 1931708
-/+ buffers/cache: 2490204 5292376
Swap: 1048568 32 1048536
Looks like not much swap is being used, now where does the memory for each process come from? Thanks.

You don't know what VSZ is. You think you know, but evidence suggests otherwise, therefore you need to find out what it is.
VZS is Virtual Memory Size and that's all memory required by the process, including shared memory. That's why you can't just sum(VSZ) and expect to get less than the physical amount of memory + swap.
The mapped memory in pmap probably corresponds to VSZ and writable/private, I guess, is memory that is shared by some processes, where each of them has write access to that memory (like allocated by their parent process or so).
To understand this, you need to understand how memory allocation and access works, which is difficult. http://emilics.com/blog/article/mconsumption.html this article seems to explain it in some detail (but I read that only in cursory way)

Related

Does linux process VSZ equal 0 mean kernel space application?

I notice some process always have VSZ as 0
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 19356 1400 ? Ss Jun13 0:00 /sbin/init
root 2 0.0 0.0 0 0 ? S Jun13 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Jun13 0:00 [migration/0]
root 4 0.0 0.0 0 0 ? S Jun13 0:01 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S Jun13 0:00 [stopper/0]
root 6 0.0 0.0 0 0 ? S Jun13 0:03 [watchdog/0]
root 7 0.0 0.0 0 0 ? S Jun13 0:00 [migration/1]
how to understand why they have 0 VSZ?
VSZ is the Virtual Memory Size. It includes all memory that the process can access, including memory that is swapped out, memory that is allocated, but not used, and memory that is from shared libraries.
So, the top command screenshot you shared showing VSZ values equaling 0, means that those processes are not using VSZ.
NOTE: They are kernel threads and memory statistics are irrelevant for them as they use kernel memory. Just to visualize kernel processes, press c when top command is running and it will show you all [bracketed] entries in last column named COMMAND.
You can get more details on VSZ and learn about its counterpart RSS (Resident Set Size) from here.

linux used memory by unknown (not Slab)

Memory occupied by unknown (VMware/CentOS)
Hello.
We have a server that has memory full used issue, but can not find what is eating memory.
Usage of memory has increased few days ago 40% -> neary 100% and stayed there since then.
We’d like to kill whatever eating memory.
[Env]
cat /etc/redhat-release
CentOS release 6.5 (Final)
# arch
x86_64
[status]
#free
total used free shared buffers cached
Mem: 16334148 15682368 651780 0 10168 398956
-/+ buffers/cache: 15273244 1060904
Swap: 8388600 129948 8258652
Result of top (some info are masked with ???)
#top -a
top - 10:19:14 up 49 days, 11:13, 1 user, load average: 1.05, 1.05, 1.10
Tasks: 145 total, 1 running, 143 sleeping, 0 stopped, 1 zombie
Cpu(s): 11.1%us, 18.4%sy, 0.0%ni, 69.5%id, 0.8%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 16334148k total, 15684824k used, 649324k free, 9988k buffers
Swap: 8388600k total, 129948k used, 8258652k free, 387824k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17940 ??? 20 0 7461m 6.5g 6364 S 16.6 41.5 1113:27 java
4982 ??? 20 0 941m 531m 5756 S 2.7 3.3 611:22.48 java
3213 root 20 0 2057m 354m 2084 S 99.8 2.2 988:43.79 python
28270 ??? 20 0 835m 157m 5464 S 0.0 1.0 106:48.55 java
1648 root 20 0 197m 10m 1452 S 0.0 0.1 42:35.95 python
1200 root 20 0 246m 7452 808 S 0.0 0.0 2:37.42 rsyslogd
Processes that are using memory (some info are masked with ???)
# ps aux --sort rss
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1200 0.0 0.0 251968 7452 ? Sl Sep12 2:37 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
root 1648 0.0 0.0 202268 10604 ? Ss Sep12 42:36 /usr/lib64/???
??? 28270 0.1 0.9 855932 161092 ? Sl Sep14 106:49 /usr/java/???
root 3213 96.1 2.0 2107704 332932 ? Ssl Oct31 992:25 /usr/lib64/???
??? 4982 0.8 3.3 964096 544328 ? Sl Sep12 611:25 /usr/java/???
??? 17940 6.6 41.5 7649356 6781076 ? Sl Oct20 1113:49 /usr/java/???
Memory is almost 100% used, but with ps and top, we can only find processes that uses half of it.
We have checked slab cache, but it was not the cause.
Slab is only 90444 kB.
Nothing is found in syslog too.
Anyone has any idea how to detect what is eating memory?
Thank you in advance.
Run free -m and see the difference. Column available shows real free memory.
And take a look at the https://www.linuxatemyram.com/
we have restarted server and solved this case.

Why the CPU time is different in other threads

I ran the top -H -p for a process which gave me the few threads with LWPs.
But when I sort the results with smallest PID first, I noticed the time in first thread is constant but the other threads time is changing. Why TIME+ is different?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16989 root 20 0 106m 28m 2448 S 0.0 0.2 0:22.31 glusterfs
16990 root 20 0 106m 28m 2448 S 0.0 0.2 0:00.00 glusterfs
16992 root 20 0 106m 28m 2448 S 0.0 0.2 0:00.00 glusterfs
16993 root 20 0 106m 28m 2448 S 0.0 0.2 0:00.00 glusterfs
16997 root 20 0 106m 28m 2448 S 0.0 0.2 0:11.71 glusterfs
17010 root 20 0 106m 28m 2448 S 0.0 0.2 0:21.07 glusterfs
17061 root 20 0 106m 28m 2448 S 0.0 0.2 0:00.00 glusterfs
Why TIME+ is different?
Because different threads are doing different percentages of the work. There could be a number of reasons for this1, but the most likely is that the application (glusterfs) is not attempting to distribute work evenly across the worker threads.
It is not something to worry about. It doesn't matter which thread does the work if the work level (see the %CPU) is negligible.
1 - If someone had the time and inclination, they could look at the source code of glusterfs to try to understand its behavior. However, I don't think the effort is warranted.
Because the time column referes to the time consumed by a process, so when a process time does not change it probably means that this process is "sleeping" or simply waiting for an other process to finish, but there could be many more reasons.
http://linux.about.com/od/commands/l/blcmdl1_top.htm
TIME:
Total CPU time the task has used since it started. If cumulative mode
is on, this also includes the CPU time used by the process's children
which have died. You can set cumulative mode with the S command line
option or toggle it with the interactive command S. The header line
will then be changed to CTIME.

OPENSHIFT app running slow. Low memory even when app is stopped

My app is running slow after a few days. Using the unix "top" command it seems there is not a lot of free memory. See below. Even if I stop the application about the same memory shows used. Any ideas why? Does this amount of memory look normal with no app running on a small gear application? How can I reboot the virtual machine?
Below is the output of the "top" command with no app running. Shows
7513700k total, 7327484k used, 186216k free
top - 22:06:26 up 14 days, 5:42, 0 users, load average: 1.83, 2.82, 3.21
Tasks: 3 total, 1 running, 2 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.2%us, 26.6%sy, 1.6%ni, 57.4%id, 4.0%wa, 0.0%hi, 0.0%si, 0.2%st
Mem: 7513700k total, 7327484k used, 186216k free, 170244k buffers
Swap: 6249464k total, 4210036k used, 2039428k free, 925320k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
48736 3558 20 0 14908 1176 944 R 0.7 0.0 0:00.04 top
48374 3558 20 0 102m 2684 848 S 0.0 0.0 0:00.00 sshd
48383 3558 20 0 106m 2072 1436 S 0.0 0.0 0:00.19 bash
What type of app are you running? Also, since openshift uses cgroups you'll want to see what your usage is within your cgroup (top output shows the whole system). Try including the output from for i in $(oo-cgroup-read all);do echo “oo-cgroup-read $i” && oo-cgroup-read $i; done and pay close attention to your memory limits.

Memory usage with high precision

How can I get high precision memory usage per proccess with "ps aux"?
$ ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 3672 1984 ? Ss Dec11 0:07 /sbin/init
root 2 0.0 0.0 0 0 ? S Dec11 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Dec11 0:23 [ksoftirqd/0]
root 6 0.0 0.0 0 0 ? S Dec11 0:00 [migration/0]
...
I need more than 1 digit after point.
Maybe I can format column with %MEM?
Look into the proc filesystem /proc/[pid]/status, /proc/[pid]/statm, /proc/[pid]/smaps.
To get fully detailed memory map /proc/[pid]/maps
Read the proc(5) manual page for all the details.
the ps command have that. you can type man ps for detail.
when the termianl show info after you typed it, you can type /memory , then heighlight contain 'memory''s string. you can type n show next palace with memory.

Resources