Three hours ago the server memory usage blowed up to 105% from around 60%.I am using a dedicated MediaTemple server with 512mb RAM.Should I be worried?Why would something like this happen?
Any help would be greatly appreciated.
Tasks: 38 total, 2 running, 36 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 946344k total, 550344k used, 396000k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 15 0 10364 740 620 S 0.0 0.1 0:38.54 init
3212 root 18 0 96620 4068 3200 R 0.0 0.4 0:00.21 sshd
3214 root 15 0 12080 1728 1316 S 0.0 0.2 0:00.05 bash
3267 apache 15 0 412m 43m 4396 S 0.0 4.7 0:03.88 httpd
3290 apache 15 0 412m 43m 4340 S 0.0 4.7 0:02.98 httpd
3348 root 15 0 114m 52m 2112 S 0.0 5.6 0:48.94 spamd
3349 popuser 15 0 114m 50m 972 S 0.0 5.5 0:00.06 spamd
3455 sw-cp-se 18 0 60116 3216 1408 S 0.0 0.3 0:00.12 sw-cp-serverd
3525 admin 18 0 81572 4604 2912 S 0.0 0.5 0:01.74 in.proftpd
3585 apache 18 0 379m 15m 3356 S 0.0 1.7 0:00.01 httpd
3589 root 15 0 12624 1224 956 R 0.0 0.1 0:00.00 top
7397 root 15 0 21660 944 712 S 0.0 0.1 0:00.58 xinetd
9500 named 16 0 301m 5284 1968 S 0.0 0.6 0:00.43 named
9575 root 15 -4 12632 680 356 S 0.0 0.1 0:00.00 udevd
9788 root 25 0 13184 608 472 S 0.0 0.1 0:00.00 couriertcpd
9790 root 25 0 3672 380 312 S 0.0 0.0 0:00.00 courierlogger
9798 root 25 0 13184 608 472 S 0.0 0.1 0:00.00 couriertcpd
First analyze the process which was taking that much of CPU by the same top command. If the process was a multi-threaded program use the following top command:
top -H -p "pid of that process"
It will help you find the thread whichever is taking a lot of CPU for further diagnosis.
Related
I run the command
for cpu in `cat /proc/cpuinfo | grep processor | wc -l`; do
top -b -c -n$cpu | egrep -v 'Mem|Swap|{top -b -c}' | grep load -A10 | grep -v grep
done
however shell prints extra 10 lines and I would like the last 10 lines after each invocation removed.
Here is the complete output and I would like the lines after '--' removed from each paragraph
]# for cpu in `cat /proc/cpuinfo |grep processor |wc -l`;do top -b -c -n$cpu |egrep -v 'Mem|Swap|{top -b -c}' |grep load -A10 |grep -v grep; done
top - 07:34:27 up 17 days, 9:04, 1 user, load average: 0.00, 0.02, 0.68
Tasks: 268 total, 1 running, 267 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.9%us, 0.1%sy, 0.0%ni, 98.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4193 root 20 0 28164 1652 1136 R 2.0 0.0 0:00.01 top -b -c -n8
14303 appuser 20 0 12.6g 3.2g 56m S 2.0 10.2 180:45.23 /apps/jdk1.8.0_151/bin/java -D[Standalone] -XX:+UseCompressedOops -server -Xms2048m -Xmx6144m -XX:PermSize=256m -XX:MaxPermSize=512m -Djava.ne
1 root 20 0 30068 1664 1444 S 0.0 0.0 0:01.54 /sbin/init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 [kthreadd]
3 root RT 0 0 0 0 S 0.0 0.0 0:00.39 [migration/0]
4 root 20 0 0 0 0 S 0.0 0.0 0:02.43 [ksoftirqd/0]
--
8629 daemon 20 0 56504 8412 5756 S 0.0 0.0 0:05.57 /opt/quest/sbin/.vasd -p /var/opt/quest/vas/vasd/.vasd.pid
8630 daemon 20 0 56476 7724 5148 S 0.0 0.0 0:00.04 /opt/quest/sbin/.vasd -p /var/opt/quest/vas/vasd/.vasd.pid
8631 daemon 20 0 54404 7436 4840 S 0.0 0.0 0:00.22 /opt/quest/sbin/.vasd -p /var/opt/quest/vas/vasd/.vasd.pid
13089 root 18 -2 10764 392 304 S 0.0 0.0 0:00.00 /sbin/udevd -d
13090 root 18 -2 10764 416 288 S 0.0 0.0 0:00.00 /sbin/udevd -d
13203 root 20 0 254m 8120 4944 S 0.0 0.0 12:02.50 /usr/sbin/vmtoolsd
13227 root 20 0 60060 9m 7280 S 0.0 0.0 0:00.06 /usr/lib/vmware-vgauth/VGAuthService -s
14259 root 20 0 0 0 0 S 0.0 0.0 0:13.53 [flush-253:6]
14262 appuser 20 0 103m 1456 1196 S 0.0 0.0 0:00.00 /bin/sh ./standalone-mdm-hub-server.sh -c standalone-full.xml -b 0.0.0.0 -bmanagement 0.0.0.0 -u 230.0.0.4 -Djboss.server.base.dir=../mdm-hub-
--
top - 07:34:30 up 17 days, 9:04, 1 user, load average: 0.00, 0.02, 0.68
Tasks: 268 total, 1 running, 267 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.1%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3581 root 20 0 2055m 51m 17m S 1.0 0.2 237:05.21 /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml -path.home /usr/share/metricbeat -path.config /etc/metricbeat -path.dat
4193 root 20 0 28172 1760 1236 R 0.7 0.0 0:00.03 top -b -c -n8
14303 appuser 20 0 12.6g 3.2g 56m S 0.7 10.2 180:45.25 /apps/jdk1.8.0_151/bin/java -D[Standalone] -XX:+UseCompressedOops -server -Xms2048m -Xmx6144m -XX:PermSize=256m -XX:MaxPermSize=512m -Djava.ne
42 root 20 0 0 0 0 S 0.3 0.0 3:05.13 [events/7]
13203 root 20 0 254m 8120 4944 S 0.3 0.0 12:02.51 /usr/sbin/vmtoolsd
14553 appuser 20 0 22.5g 18g 54m S 0.3 57.3 1467:43 /apps/jdk1.8.0_151/bin/java -D[Standalone] -XX:+UseCompressedOops -server -Xms2048m -Xmx16000m -XX:PermSize=512m -XX:MaxPermSize=1048m -Djava.
--
8629 daemon 20 0 56504 8412 5756 S 0.0 0.0 0:05.57 /opt/quest/sbin/.vasd -p /var/opt/quest/vas/vasd/.vasd.pid
8630 daemon 20 0 56476 7724 5148 S 0.0 0.0 0:00.04 /opt/quest/sbin/.vasd -p /var/opt/quest/vas/vasd/.vasd.pid
8631 daemon 20 0 54404 7436 4840 S 0.0 0.0 0:00.22 /opt/quest/sbin/.vasd -p /var/opt/quest/vas/vasd/.vasd.pid
13089 root 18 -2 10764 392 304 S 0.0 0.0 0:00.00 /sbin/udevd -d
13090 root 18 -2 10764 416 288 S 0.0 0.0 0:00.00 /sbin/udevd -d
13227 root 20 0 60060 9m 7280 S 0.0 0.0 0:00.06 /usr/lib/vmware-vgauth/VGAuthService -s
14259 root 20 0 0 0 0 S 0.0 0.0 0:13.53 [flush-253:6]
14262 appuser 20 0 103m 1456 1196 S 0.0 0.0 0:00.00 /bin/sh ./standalone-mdm-hub-server.sh -c standalone-full.xml -b 0.0.0.0 -bmanagement 0.0.0.0 -u 230.0.0.4 -Djboss.server.base.dir=../mdm-hub-
14512 appuser 20 0 103m 1452 1196 S 0.0 0.0 0:00.01 /bin/sh ./standalone-mdm-process-server.sh -c standalone-full.xml -b 0.0.0.0 -bmanagement 0.0.0.0 -Djboss.server.base.dir=../mdm-process-serve
Because it finds "grep load". Always take a moment to break down what you are doing, and look at the intermediates. Try:
top -b -c -n1 |egrep -v 'Mem|Swap|{top -b -c}' |grep load -A10
The quick fix would be to add -m1 to only shopw the first match.
So I have access to my shared hosting server via SSH. The hosting company warned me about too much resource consuming. When I logged to the server and run top to view the resource usage information on the Linux shared hosting server I get the following log:
top - 09:31:20 up 4 days, 19:59, 5 users, load average: 8.30, 8.84, 9.30
Tasks: 39 total, 1 running, 38 sleeping, 0 stopped, 0 zombie
Cpu(s): 26.8%us, 4.0%sy, 1.1%ni, 64.9%id, 2.9%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 65709912k total, 63435736k used, 2274176k free, 6961904k buffers
top - 09:38:42 up 4 days, 20:07, 5 users, load average: 7.06, 8.13, 8.81
Tasks: 40 total, 1 running, 39 sleeping, 0 stopped, 0 zombie
Cpu(s): 16.8%us, 3.4%sy, 0.2%ni, 77.7%id, 1.7%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 65709912k total, 64754844k used, 955068k free, 7025016k buffers
Swap: 4190204k total, 1044576k used, 3145628k free, 43596764k cached
And this is the table with processes:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
81448 my_username 20 0 105m 1920 1504 S 0.0 0.0 0:00.02 bash
107642 my_username 20 0 11708 1800 1388 S 0.0 0.0 0:00.00 bash
232879 my_username 20 0 105m 1200 1196 S 0.0 0.0 0:00.00 bash
232892 my_username 20 0 11804 508 504 S 0.0 0.0 0:00.00 git
232895 my_username 20 0 105m 1236 1232 S 0.0 0.0 0:00.00 git-pull
232911 my_username 20 0 11808 616 612 S 0.0 0.0 0:00.00 git
232912 my_username 20 0 76316 1980 1972 S 0.0 0.0 0:00.00 git-remote-http
389588 my_username 20 0 15048 1216 972 R 0.0 0.0 0:00.33 top
391829 my_username 20 0 105m 1912 1504 S 0.0 0.0 0:00.01 bash
693783 my_username 20 0 105m 1216 1212 S 0.0 0.0 0:00.00 bash
693792 my_username 20 0 11804 508 504 S 0.0 0.0 0:00.00 git
693793 my_username 20 0 105m 1280 1248 S 0.0 0.0 0:00.00 git-pull
693809 my_username 20 0 11808 732 612 S 0.0 0.0 0:00.00 git
693810 my_username 20 0 76316 2832 1972 S 0.0 0.0 0:00.00 git-remote-http
724630 my_username 20 0 105m 1216 1212 S 0.0 0.0 0:00.00 bash
724639 my_username 20 0 11804 508 504 S 0.0 0.0 0:00.00 git
724642 my_username 20 0 105m 1252 1248 S 0.0 0.0 0:00.00 git-pull
724695 my_username 20 0 11808 616 612 S 0.0 0.0 0:00.00 git
724696 my_username 20 0 76316 1976 1972 S 0.0 0.0 0:00.00 git-remote-http
880773 my_username 20 0 105m 1216 1212 S 0.0 0.0 0:00.00 bash
880782 my_username 20 0 11804 508 504 S 0.0 0.0 0:00.00 git
880783 my_username 20 0 105m 1252 1248 S 0.0 0.0 0:00.00 git-pull
880799 my_username 20 0 11808 616 612 S 0.0 0.0 0:00.00 git
880800 my_username 20 0 76316 1976 1972 S 0.0 0.0 0:00.00 git-remote-http
894182 my_username 20 0 105m 1216 1212 S 0.0 0.0 0:00.00 bash
894191 my_username 20 0 11804 508 504 S 0.0 0.0 0:00.00 git
894193 my_username 20 0 105m 1252 1248 S 0.0 0.0 0:00.00 git-pull
894209 my_username 20 0 11808 620 612 S 0.0 0.0 0:00.00 git
894210 my_username 20 0 76316 1976 1972 S 0.0 0.0 0:00.00 git-remote-http
994122 my_username 20 0 105m 1216 1212 S 0.0 0.0 0:00.00 bash
994131 my_username 20 0 11804 508 504 S 0.0 0.0 0:00.00 git
994132 my_username 20 0 105m 1252 1248 S 0.0 0.0 0:00.00 git-pull
994148 my_username 20 0 11808 616 612 S 0.0 0.0 0:00.00 git
994149 my_username 20 0 76316 1976 1972 S 0.0 0.0 0:00.00 git-remote-http
1025934 my_username 20 0 339m 21m 17m S 0.0 0.0 0:00.75 lsphp 1044176 my_username 20 0 105m 1200 1196 S 0.0 0.0 0:00.00 bash
So as you can see most of the commands that are issuing this problem are the git commands that I am running on the server. Why are they not killed after some git operation? How to deal with this.
BTW, I have a limit of 50 processes and their number is constantly around 40 .
Thanks!
On RedHat Linux 6.2 I'm running free -m and it shows nearly all 8GB used
total used free shared buffers cached
Mem: 7989 7734 254 0 28 7128
-/+ buffers/cache: 578 7411
Swap: 4150 0 4150
But at the same time in top -M I cannot see any processes using all this memory:
top - 16:03:34 up 4:10, 2 users, load average: 0.08, 0.04, 0.01
Tasks: 169 total, 1 running, 163 sleeping, 5 stopped, 0 zombie
Cpu(s): 0.7%us, 0.3%sy, 0.0%ni, 98.6%id, 0.4%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 7989.539M total, 7721.570M used, 267.969M free, 28.633M buffers
Swap: 4150.992M total, 0.000k used, 4150.992M free, 7115.312M cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1863 sroot 20 0 398m 24m 9.8m S 0.3 0.3 3:12.87 App1
1 sroot 20 0 2864 1392 1180 S 0.0 0.0 0:00.91 init
2 sroot 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 sroot RT 0 0 0 0 S 0.0 0.0 0:00.07 migration/0
4 sroot 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
5 sroot RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
6 sroot RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
7 sroot RT 0 0 0 0 S 0.0 0.0 0:00.08 migration/1
8 sroot RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
I also tried this ps mem script but it onlt shows about 400MB memory being used.
Don't look at the "Mem" line, look at the one below it.
The Linux kernel consumes as much memory as it can to provide the I/O cache (and other non-critical buffers, but the cache is going to be most of this usage). This memory is relinquished to processes when they request it. The "-/+ buffers/cache" line is showing you the adjusted values after the I/O cache is accounted for, that is, the amount of memory used by processes and the amount available to processes (in this case, 578MB used and 7411MB free).
The difference of used memory between the "Mem" and "-/+ buffers/cache" line shows you how much is in use by the kernel for the purposes of caching: 7734MB - 578MB = 7156MB in the I/O cache. If processes need this memory, the kernel will simply shrink the size of the I/O cache.
Also, as the first line shows
total used free shared buffers cached
Mem: 7989 7734 254 0 28 7128
-/+ buffers/cache: 578 7411
If we add (cached[7128] + buffers[28] + free[254]), we will get approximately the second line's free[7411] value
7128 + 28 + 254 = 7410
If the cached is small, try this command:
ps aux --sort -rss
How to check the process status if process is running and not listed in ps or top command output.
I have started httpd(Apache) service and its working perfectly, i am able to see the webpage. But top or ps command doesn't display the httpd process.
What is the issue? I am logged in as root user.
Can we check the process status by any command if process id is not listed?
[root#ip-xx-xxx-xx-xxx /]# service httpd restart
Stopping httpd: [ OK ]
Starting httpd: [ OK ]
top - 19:54:08 up 10 days, 5:04, 1 user, load average: 0.00, 0.01, 0.05
Tasks: 70 total, 1 running, 69 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 617044k total, 307312k used, 309732k free, 30660k buffers
Swap: 0k total, 0k used, 0k free, 218968k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 2892 1360 1164 S 0.0 0.2 0:00.34 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:01.82 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
6 root 20 0 0 0 0 S 0.0 0.0 0:26.84 events/0
7 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuset
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 netns
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 async/mgr
17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 xenwatch
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 xenbus
64 root 20 0 0 0 0 S 0.0 0.0 0:02.37 sync_supers
66 root 20 0 0 0 0 S 0.0 0.0 0:02.58 bdi-default
67 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/0
69 root 20 0 0 0 0 S 0.0 0.0 0:00.03 kblockd/0
76 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kseriod
184 root 20 0 0 0 0 S 0.0 0.0 0:00.16 khungtaskd
185 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd0
186 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
238 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/0
241 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/0
252 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khvcd
332 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kstriped
519 root 20 0 0 0 0 S 0.0 0.0 0:00.37 jbd2/xvda1-8
520 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ext4-dio-unwrit
548 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khubd
599 root 16 -4 2504 644 352 S 0.0 0.1 0:00.02 udevd
824 root 20 0 0 0 0 S 0.0 0.0 0:00.09 kauditd
858 root 18 -2 2500 640 352 S 0.0 0.1 0:00.00 udevd
859 root 18 -2 2500 636 348 S 0.0 0.1 0:00.00 udevd
983 root 20 0 2840 760 488 S 0.0 0.1 0:00.01 dhclient
1020 root 16 -4 10896 580 428 S 0.0 0.1 0:00.85 auditd
1035 root 20 0 29628 1436 964 S 0.0 0.2 0:01.37 rsyslogd
1056 dbus 20 0 2980 884 700 S 0.0 0.1 0:00.58 dbus-daemon
1151 root 20 0 8192 888 468 S 0.0 0.1 0:00.43 sshd
1171 ntp 20 0 5072 1368 1036 S 0.0 0.2 0:01.34 ntpd
I prefer to use:
pgrep -l httpd
Example:
[root#mywebserver ~]$ pgrep -l httpd
3661 httpd
3665 httpd
3673 httpd
3678 httpd
3683 httpd
3688 httpd
3694 httpd
3701 httpd
: .. more....
Counting those lines also helps me to see if the server is getting overloaded.
$ ps -efw |grep -i httpd
also
$ top
press: u == display user
'u'
followed by the username, in this case 'apache' enter:
apache
Problem: my mod_perl leaks and I cannot control it.
I run mod_perl script under Ubuntu (production code).
Usually there are 8-10 script instances running concurrently.
According to Unix "top" utilty each instance takes 55M of memory.
55M is a lot, but I was told here that most of this memory is shared.
The memory is leaking.
There are 512M on the server.
There is a significant decrease of free memory in 24 hours after reboot.
Test: free memory on the system at the moment 10 scripts are running:
-after reboot: 270M
-in 24 hours since reboot: 50M
In 24 hours memory taken by each script is roughly the same - 55M (according to "top" utility).
I don't understand where the memory leakes out.
And don't know how can I find the leaks.
I share memory, I preload all the modules required by the script in startup.pl.
One more test.
A very simple mod_perl script ("Hello world!") takes 52M (according to "top")
According to "Practical mod_perl" I can use GTop utility to measure the real memory taken by mod_perl.
I have made a very simple script that measures the memory with GTop.
It shows there are 54M real memory taken by a very simple perl script!
54 Megabytes by "Hello world"?!!!
proc-mem-size: 59,707392
proc-mem-share: 52,59264
diff: 54,448128
There must be something wrong in the way I measure mod_perl memory.
Help please!
This problem is driving me mad for several days.
These are the snapshots of "top" output after reboot and in 24 hours after reboot.
The processes are sorted by Memory.
---- RIGHT AFTER REBOOT ----
top - 10:25:24 up 55 min, 2 users, load average: 0.10, 0.07, 0.07
Tasks: 59 total, 3 running, 56 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 97.3%id, 0.7%wa, 0.0%hi, 0.0%si, 2.0%st
Mem: 524456k total, 269300k used, 255156k free, 12024k buffers
Swap: 0k total, 0k used, 0k free, 71276k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2307 www-data 15 0 58500 27m 5144 S 0.0 5.3 0:02.02 apache2
2301 www-data 15 0 58492 27m 4992 S 0.0 5.3 0:02.09 apache2
2302 www-data 15 0 57936 26m 4960 R 0.0 5.2 0:01.74 apache2
2895 www-data 15 0 57812 26m 5048 S 0.0 5.2 0:00.98 apache2
2903 www-data 15 0 56944 26m 4792 S 0.0 5.1 0:01.12 apache2
2886 www-data 15 0 56860 26m 4784 S 0.0 5.1 0:01.20 apache2
2896 www-data 15 0 56520 26m 4804 S 0.0 5.1 0:00.85 apache2
2911 www-data 15 0 56404 25m 4768 S 0.0 5.1 0:00.87 apache2
2901 www-data 15 0 56520 25m 4744 S 0.0 5.1 0:00.84 apache2
2893 www-data 15 0 56608 25m 4740 S 0.0 5.1 0:00.73 apache2
2277 root 15 0 51504 22m 6332 S 0.0 4.5 0:01.02 apache2
2056 mysql 18 0 98628 21m 5164 S 0.0 4.2 0:00.64 mysqld
3162 root 15 0 6356 3660 1276 S 0.0 0.7 0:00.00 vi
2622 root 15 0 8584 2980 2392 R 0.0 0.6 0:00.07 sshd
3083 root 15 0 8448 2968 2392 S 0.0 0.6 0:00.06 sshd
3164 par 15 0 5964 2828 1868 S 0.0 0.5 0:00.05 proftpd
1 root 18 0 3060 1900 576 S 0.0 0.4 0:00.00 init
2690 root 17 0 4272 1844 1416 S 0.0 0.4 0:00.00 bash
3151 root 15 0 4272 1844 1416 S 0.0 0.4 0:00.00 bash
2177 root 15 0 8772 1640 520 S 0.0 0.3 0:00.00 sendmail-mta
2220 proftpd 15 0 5276 1448 628 S 0.0 0.3 0:00.00 proftpd
2701 root 15 0 2420 1120 876 R 0.0 0.2 0:00.09 top
1966 root 18 0 5396 1084 692 S 0.0 0.2 0:00.00 sshd
---- ROUGHLY IN 24 HOURS AFTER REBOOT
top - 17:45:38 up 23:39, 1 user, load average: 0.02, 0.09, 0.11
Tasks: 55 total, 2 running, 53 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 524456k total, 457660k used, 66796k free, 127780k buffers
Swap: 0k total, 0k used, 0k free, 114620k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16248 www-data 15 0 63712 35m 6668 S 0.0 6.8 0:23.79 apache2
19417 www-data 15 0 60396 31m 6472 S 0.0 6.2 0:10.95 apache2
19419 www-data 15 0 60276 31m 6376 S 0.0 6.1 0:11.71 apache2
19321 www-data 15 0 60480 29m 4888 S 0.0 5.8 0:11.51 apache2
21241 www-data 15 0 58632 29m 6260 S 0.0 5.8 0:05.18 apache2
22063 www-data 15 0 57400 28m 6396 S 0.0 5.6 0:02.05 apache2
21240 www-data 15 0 58520 27m 4856 S 0.0 5.5 0:04.60 apache2
21236 www-data 15 0 58244 27m 4868 S 0.0 5.4 0:05.24 apache2
22499 www-data 15 0 56736 26m 4776 S 0.0 5.1 0:00.70 apache2
2055 mysql 15 0 100m 25m 5656 S 0.0 5.0 0:20.95 mysqld
2277 root 18 0 51500 22m 6332 S 0.0 4.5 0:01.07 apache2
22686 www-data 15 0 53004 21m 4092 S 0.0 4.3 0:00.21 apache2
22689 root 15 0 8584 2980 2392 R 0.0 0.6 0:00.06 sshd
2176 root 15 0 8768 1928 736 S 0.0 0.4 0:00.00 sendmail-
+mta
1 root 18 0 3064 1900 576 S 0.0 0.4 0:00.02 init
22757 root 15 0 4268 1844 1416 S 0.0 0.4 0:00.00 bash
2220 proftpd 18 0 5276 1448 628 S 0.0 0.3 0:00.00 proftpd
22768 root 15 0 2424 1100 876 R 0.0 0.2 0:00.00 top
1965 root 15 0 5400 1088 692 S 0.0 0.2 0:00.00 sshd
2258 root 18 0 3416 1036 820 S 0.0 0.2 0:00.01 cron
1928 klog 25 0 2248 1008 420 S 0.0 0.2 0:00.04 klogd
1946 messageb 19 0 2648 804 596 S 0.0 0.2 0:01.63 dbus-daem
+on
1908 syslog 18 0 2016 716 556 S 0.0 0.1 0:00.17 syslogd
It doesn't actually look like the number of apache/mod_perl processes in existence or the memory they use has changed much between the two reports you post. I note you did not post the header for the second report. It would be interesting to see the "cached" figure after 24 hours. I am going to go out on a limb and guess that this is where your memory is going - Linux is using it for caching file I/O. You can think of the file I/O cache as essentially free memory, since Linux will make that memory available if processes need it.
You can also check that this is what's going on by performing
sync; echo 3 > /proc/sys/vm/drop_caches
as root to cause the memory in use by the caches to be released, and confirming that this causes the amount of free memory reported to revert to initial values.