I am running apache2 on my RaspberryPi, mainly to interface with an mpd php client for streaming audio. After a month or so, I see the following:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1496 www-data 20 0 55900 17m 2112 S 0.0 9.3 0:30.32 apache2
7198 www-data 20 0 54868 15m 2188 S 0.0 8.4 0:10.57 apache2
7182 www-data 20 0 54868 15m 2168 S 0.0 8.3 0:11.67 apache2
1497 www-data 20 0 53844 15m 2132 S 0.0 8.2 0:07.58 apache2
2609 mysql 20 0 314m 15m 280 S 0.7 8.1 71:58.52 mysqld
7185 www-data 20 0 54868 14m 2180 S 0.0 8.1 0:08.71 apache2
7183 www-data 20 0 54868 14m 2120 S 0.0 8.1 0:14.36 apache2
1499 www-data 20 0 53844 14m 2144 S 0.0 8.0 0:07.73 apache2
1932 mpd 20 0 81204 8152 584 S 0.0 4.3 145:46.25 mpd
7211 www-data 20 0 45652 8004 2204 S 0.0 4.2 0:01.65 apache2
3318 www-data 20 0 45652 7944 2140 S 0.0 4.2 0:03.43 apache2
7210 www-data 20 0 45652 7784 2176 S 0.0 4.1 0:01.28 apache2
1965 root 20 0 44532 5268 216 S 0.0 2.8 1:53.06 apache2
7168 www-data 20 0 45652 7956 2140 S 0.0 4.2 0:02.42 apache2
Along with mpd and mysql, and the root apache2 process, 11 apache2 process running as www-data. On reboot, I see 5 apache2 processes under www-data.
Why are more processes spawnded, and not closed down? I continue to see this grow until there are 20+ processes, which slows down the something this small with limited resources.
Why are new processes spawned, then persist? Can I control this in conf.d (I have tried, but the feedback process takes a few days or week, so it's hard to tell).
Apache - when running in pre-fork mode - spawns a pool of worker processes in order to keep the response time low. Every worker will be responsible for a certain request. So, if there are 11 workers running, apache could serve 11 requests in "parallel" without spawning a new process (what would take a significant amount of time)
Apache spawns that workers / and keeps them alive intelligently, but you can set the maximum and minimum amount of workers in the apache2.conf
Related
Memory occupied by unknown (VMware/CentOS)
Hello.
We have a server that has memory full used issue, but can not find what is eating memory.
Usage of memory has increased few days ago 40% -> neary 100% and stayed there since then.
We’d like to kill whatever eating memory.
[Env]
cat /etc/redhat-release
CentOS release 6.5 (Final)
# arch
x86_64
[status]
#free
total used free shared buffers cached
Mem: 16334148 15682368 651780 0 10168 398956
-/+ buffers/cache: 15273244 1060904
Swap: 8388600 129948 8258652
Result of top (some info are masked with ???)
#top -a
top - 10:19:14 up 49 days, 11:13, 1 user, load average: 1.05, 1.05, 1.10
Tasks: 145 total, 1 running, 143 sleeping, 0 stopped, 1 zombie
Cpu(s): 11.1%us, 18.4%sy, 0.0%ni, 69.5%id, 0.8%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 16334148k total, 15684824k used, 649324k free, 9988k buffers
Swap: 8388600k total, 129948k used, 8258652k free, 387824k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17940 ??? 20 0 7461m 6.5g 6364 S 16.6 41.5 1113:27 java
4982 ??? 20 0 941m 531m 5756 S 2.7 3.3 611:22.48 java
3213 root 20 0 2057m 354m 2084 S 99.8 2.2 988:43.79 python
28270 ??? 20 0 835m 157m 5464 S 0.0 1.0 106:48.55 java
1648 root 20 0 197m 10m 1452 S 0.0 0.1 42:35.95 python
1200 root 20 0 246m 7452 808 S 0.0 0.0 2:37.42 rsyslogd
Processes that are using memory (some info are masked with ???)
# ps aux --sort rss
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1200 0.0 0.0 251968 7452 ? Sl Sep12 2:37 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
root 1648 0.0 0.0 202268 10604 ? Ss Sep12 42:36 /usr/lib64/???
??? 28270 0.1 0.9 855932 161092 ? Sl Sep14 106:49 /usr/java/???
root 3213 96.1 2.0 2107704 332932 ? Ssl Oct31 992:25 /usr/lib64/???
??? 4982 0.8 3.3 964096 544328 ? Sl Sep12 611:25 /usr/java/???
??? 17940 6.6 41.5 7649356 6781076 ? Sl Oct20 1113:49 /usr/java/???
Memory is almost 100% used, but with ps and top, we can only find processes that uses half of it.
We have checked slab cache, but it was not the cause.
Slab is only 90444 kB.
Nothing is found in syslog too.
Anyone has any idea how to detect what is eating memory?
Thank you in advance.
Run free -m and see the difference. Column available shows real free memory.
And take a look at the https://www.linuxatemyram.com/
we have restarted server and solved this case.
An important topic in software deveopment / programming is to assess the size of the product, and to match the application footprint to the system where it is running. One may need to optimize the product, and/or one may need to add more memory, use a faster processor, etc. In the case of virtual machines, it is important to make sure the application will work effectively by perhaps making the VM memory size larger, or allow a product to get more resources from the hypervisor when needed and available.
The linux top(1) command is great, with its ability to sort by different fields, add optional fields, highlight sort criteria on-screen, and switch sort field with < and >. On most systems though, there are very many processes running, making "at-a-glance" examination a little difficult. Consider:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ PPID SWAP nFLT COMMAND
2181 root 20 0 7565m 3.2g 7028 S 2.7 58.3 86:41.17 1 317m 10k java
1751 root 20 0 137m 2492 1056 S 0.0 0.0 0:02.57 1 5104 76 munin-node
11598 postgres 20 0 146m 23m 11m S 0.0 0.4 7:51.63 2143 3600 28 postmaster
1470 root 20 0 243m 1792 820 S 0.0 0.0 0:01.89 1 2396 23 rsyslogd
3107 postgres 20 0 146m 26m 11m S 0.0 0.5 7:40.61 2143 936 58 postmaster
3168 postgres 20 0 132m 14m 11m S 0.0 0.2 8:27.27 2143 904 53 postmaster
3057 postgres 20 0 138m 19m 11m S 0.0 0.3 6:55.63 2143 856 36 postmaster
3128 root 20 0 85376 900 896 S 0.0 0.0 0:00.11 1636 852 2 sshd
1728 root 20 0 80860 1080 952 S 0.0 0.0 0:00.61 1 776 0 master
3130 manager 20 0 85532 844 672 S 0.0 0.0 0:01.03 3128 712 36 sshd
436 root 16 -4 11052 264 260 S 0.0 0.0 0:00.01 1 688 0 udevd
2211 root 18 -2 11048 220 216 S 0.0 0.0 0:00.00 436 684 0 udevd
2212 root 18 -2 11048 220 216 S 0.0 0.0 0:00.00 436 684 0 udevd
1636 root 20 0 66176 524 436 S 0.0 0.0 0:00.12 1 620 25 sshd
1486 root 20 0 229m 2000 1648 S 0.0 0.0 0:00.79 1485 596 116 sssd_be
2306 postgres 20 0 131m 11m 9m S 0.0 0.2 0:01.21 2143 572 64 postmaster
3055 postgres 20 0 135m 16m 11m S 0.0 0.3 10:18.88 2143 560 36 postmaster
...etc... This is for about 20 processes, but there are well over 100 processes.
In this example I was sorting by SWAP field.
I would like to be able to aggregate related processes based on the "process group" of which they are a part, or based on the USER running the process, or based on the COMMAND being run. Essentially I want to:
Aggregate by PPID, or
Aggregate by USER, or
Aggregate by COMMAND, or
Turn off aggregation
This would allow me to see more quickly what is going on. The expectation is that all the postgres processes would show up together, as a single line, with process group leader (2143, not captured in the snippet) displaying aggegated metrics. Generally the aggregation would be a sum (VIRT, RES, SHR, %CPU, %MEM, TIME+, SWAP, nFLT), but sometimes not (as for PR and NI, which might be shown as just --).
For processes whose PPID is 1, it would be nice to have an option of toggling between aggregating them all together, or of leaving them listed individually.
Aggegation by the name of the process (java vs. munin-node, vs. postmaster, vs. chrome) would also be a nice option. The COMMAND arguments would not be used when aggregating by command name.
This would be very valuable when tuning an application. How can I do this, aggregating top data for at-a-glance viewing in larger scale systems? Has anyone written an app, perhaps that uses top in batch mode, to create a summary view like I'm discussing?
FYI, I'm specifically interest in something for CentOS, but this would be helpful on any OS variant.
Thanks!
...Alan
I am using RHEL and i have configured isolcups= in /boot/grub/grub.conf file so that I can isolate some cpu from os scheduling processes. Now I want to check if those cpu's are isolated or they are still using OS scheduling algorithm.
The machine config is twin 5690 processor in hyper threaded mode.
So a total of 24 cores.
I want to isolate 6 cores for an application.
However when i do "top", I find that there are some system processes running on those cores. I am pasting the supposed to be isolated 12th core.
100 root rt 0 0 0 0 S 0.0 0.0 0:00.01 migration/11 11
101 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/11 11
102 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/11:0 11
103 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/11:0H 11
What is sure shot way of checking isolated cpu in linux?
I am able to resolve it , It using only those cpu that are not isolated.
I did stress test on it, and it is only taking non-isolated cpu.The only chang i made is config file "/boot/grub/grub.cfg" and reboot the system.
You can use stress test and check if it is using isolated core or not.
I ran the top -H -p for a process which gave me the few threads with LWPs.
But when I sort the results with smallest PID first, I noticed the time in first thread is constant but the other threads time is changing. Why TIME+ is different?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16989 root 20 0 106m 28m 2448 S 0.0 0.2 0:22.31 glusterfs
16990 root 20 0 106m 28m 2448 S 0.0 0.2 0:00.00 glusterfs
16992 root 20 0 106m 28m 2448 S 0.0 0.2 0:00.00 glusterfs
16993 root 20 0 106m 28m 2448 S 0.0 0.2 0:00.00 glusterfs
16997 root 20 0 106m 28m 2448 S 0.0 0.2 0:11.71 glusterfs
17010 root 20 0 106m 28m 2448 S 0.0 0.2 0:21.07 glusterfs
17061 root 20 0 106m 28m 2448 S 0.0 0.2 0:00.00 glusterfs
Why TIME+ is different?
Because different threads are doing different percentages of the work. There could be a number of reasons for this1, but the most likely is that the application (glusterfs) is not attempting to distribute work evenly across the worker threads.
It is not something to worry about. It doesn't matter which thread does the work if the work level (see the %CPU) is negligible.
1 - If someone had the time and inclination, they could look at the source code of glusterfs to try to understand its behavior. However, I don't think the effort is warranted.
Because the time column referes to the time consumed by a process, so when a process time does not change it probably means that this process is "sleeping" or simply waiting for an other process to finish, but there could be many more reasons.
http://linux.about.com/od/commands/l/blcmdl1_top.htm
TIME:
Total CPU time the task has used since it started. If cumulative mode
is on, this also includes the CPU time used by the process's children
which have died. You can set cumulative mode with the S command line
option or toggle it with the interactive command S. The header line
will then be changed to CTIME.
I want to make a Linux application that send a message to a remote host when a local system (where the application runs) is going to suspend, hibernate or shutdown. I googled how to do this and found acpi_listen but it doesn't work.
Here is the result of ps aux | grep acpi on the system I tested (Ubuntu 10.04 LTS):
root 35 0.0 0.0 0 0 ? S Dec03 0:00 [kacpid]
root 36 0.0 0.0 0 0 ? S Dec03 0:00 [kacpi_notify]
root 37 0.0 0.0 0 0 ? S Dec03 0:00 [kacpi_hotplug]
root 934 0.0 0.0 2048 872 ? Ss Dec03 0:00 acpid -c /etc/acpi/events -s /var/run/acpid.socket
Communicating with acpid directly via a UNIX domain socket wouldn't work since that's the way acpi_listen works, although I didn't try it. Is there any other way?
Listen for them over D-Bus.