is there a way to get peak memory of a process on Mac and Linux with the same shell command?
Right now I have such logic working on Linix:
/usr/bin/time -f "%M" myProc 2>$1 | tee memory | tail -n 1 "$log" | int($1)
where time -f "%M" changes the output of time to only show peak memory and the rest is getting the value as an integer.
Now I need to do the same for Mac but it does not understand -f, I have tried to use
TIMEFMT='%M'
time myProc 2>$1 | tee memory
but it stopped working on Linux and had bunch of extra ouput on Mac.
Is there a way to do it better without installing additional dependencies?
Thanks.
Hello I have a memory leak on my server which I finding it difficult to trace, apparently so is support. They told me I to try and write a cron to detect when my server is low on memory but I have no idea how to do this.
I use PHP to build my apps on a VPS server with CentOS6 installed..
Quoting from https://cookbook.wdt.io/memory.html:
free is a standard unix command that displays used and available memory. Used with the options -m it will output the values in megabytes. The last value in the line labeled "-/+ buffers/cache:" shows the total available memory. So we can use grep and awk to get this value and turn it into a number.
free -m | grep cache: | awk '{ print int($NF) }'
*/5 * * * * ((`free -m | grep cache: | awk '{ print int($NF) }'` >= 50)) && curl -sm 30 http://any_monitoring_url
The "curl ... any_monitoring_url" in the above example is pinging an external monitoring system like the one we built (wdt.io) to catch memory leaks and then email / sms / slack you. This step is not strictly necessary. You could do something as simple as touch file_to_check_timestamp or echo "Low Memory!" >> file_to_check_for_low_memory_alerts. The problem is that if memory (or CPU or disk space) get pinned, you could hit deadlock and the scheduled cron task may not run. Hence the value of a third-party monitor.
Also see our articles on cron monitoring CPU and Disk Space and other recipes, in case they're of value as well.
I have a program that parses file dumps generated by dumping top into a text file. For example, I use top -n 1 -b > dump1. The problem is when my system is under load for example 60% cpu utilization top always returns 3-4% usage of cpu. When I run top manually, the cpu starts at 3-4% usage, then after 1-2 seconds it will jump to the expected load. The question is how do I capture top a couple of seconds after it has been executed?
Also you can try bash script which only capture cpu usage and put in log file like
#!/bin/bash
while :; do
top -bn 1 | sed -n '3p' >> log.txt
sleep 2
done
Here top command fire every 2 seconds.
I have an extremely long and complicated shell pipeline set up to grab 2.2Gb of data and process it. It currently takes 45 minutes to process. The pipeline is a number of cut, grep, sort, uniq, grep and awk commands tied together. I have my suspicion that it's the grep portion that is causing it to take so much time but I have no way of confirming it.
Is there anyway to "profile" the entire pipeline from end to end to determine which component is the slowest and if it is CPU or IO bound so it can be optimised?
I cannot post the entire command here unfortunately as it would require posting proprietary information but I suspect it is the following bit checking it out with htop:
grep -v ^[0-9]
One way to do this is to gradually build up the pipeline, timing each addition, and taking as much out of the equation as possible (such as outputting to a terminal or file). A very simple example is shown below:
pax:~$ time ( cat bigfile >/dev/null )
real 0m4.364s
user 0m0.004s
sys 0m0.300s
pax:~$ time ( cat bigfile | tr 'a' 'b' >/dev/null )
real 0m0.446s
user 0m0.312s
sys 0m0.428s
pax:~$ time ( cat bigfile | tr 'a' 'b' | tail -1000l >/dev/null )
real 0m0.796s
user 0m0.516s
sys 0m0.688s
pax:~$ time ( cat bigfile | tr 'a' 'b' | tail -1000l | sort -u >/dev/null )
real 0m0.892s
user 0m0.556s
sys 0m0.756s
If you add up the user and system times above, you'll see that the incremental increases are:
0.304 (0.004 + 0.300) seconds for the cat;
0.436 (0.312 + 0.428 - 0.304) seconds for the tr;
0.464 (0.516 + 0.688 - 0.436 - 0.304) seconds for the tail; and
0.108 (0.556 + 0.756 - 0.464 - 0.436 - 0.304) seconds for the sort.
This tells me that the main things to look into are the tail and the tr.
Now obviously, that's for CPU only, and I probably should have done multiple runs at each stage for averaging purposes, but that's the basic first approach I would take.
If it turns out it really is your grep, there are a few other options available to you. There are numerous other commands that can strip lines not starting with a digit but you may find that a custom-built command for doing this may be faster still, pseudo-code like (untested, but you should get the idea):
state = echo
lastchar = newline
while not end of file:
read big chunk from file
for every char in chunk:
if lastchar is newline:
if state is echo and char is non-digit:
state = skip
else if state is skip and and char is digit:
state = echo
if state is echo:
output char
lastchar = char
Custom, targeted code like this can sometimes be made more efficient than a general-purpose regex processing engine, simply because it can be optimised to the specific case. Whether that's true is this case, or any case for that matter, is something you should test. My number one optimisation mantra is measure, don't guess!
I found the problem myself after some further experimentation. It appears to be due to the encoding support in grep. Using the following hung the pipeline:
grep -v ^[0-9]
I replaced it with sed as follows and it finished in under 45 seconds!
sed '/^[0-9]/d'
This is straightforward with zsh:
zsh-4.3.12[sysadmin]% time sleep 3 | sleep 5 | sleep 2
sleep 3 0.01s user 0.03s system 1% cpu 3.182 total
sleep 5 0.01s user 0.01s system 0% cpu 5.105 total
sleep 2 0.00s user 0.05s system 2% cpu 2.121 total
Is there any ready-to-use solution to log the memory consumption from the start of the system? I'd like to log the data to simple text file or some database so I can analyze it later.
I'm working on Linux 2.4-based embedded system. I need to debug the problem related to memory consumption. My application automatically start on every system start. I need the way to get the data with timestamps from regular intervals (as often as possible), so I can track down problem.
The symptoms of my problem: when system starts it launched my main application and GUI to visualize the main parameters of the system. GUI based on GTK+ (X server). If I disable GUI and X server then my application works OK. If I enable GUI and X server it does not work when I have 256 MiB or 512 MiB of physical memory installed on the motherboard. If I have 1 GiB of memory installed then everything is OK.
The following script prints time stamps and a header.
#!/bin/bash -e
echo " date time $(free -m | grep total | sed -E 's/^ (.*)/\1/g')"
while true; do
echo "$(date '+%Y-%m-%d %H:%M:%S') $(free -m | grep Mem: | sed 's/Mem://g')"
sleep 1
done
The output looks like this (tested on Ubuntu 15.04, 64-bit).
date time total used free shared buffers cached
2015-08-01 13:57:27 24002 13283 10718 522 693 2308
2015-08-01 13:57:28 24002 13321 10680 522 693 2308
2015-08-01 13:57:29 24002 13355 10646 522 693 2308
2015-08-01 13:57:30 24002 13353 10648 522 693 2308
A small script like
rm memory.log
while true; do free >> memory.log; sleep 1; done
I am a big fan of logging everything and I find it useful to know which processes are using the memory and how much each process is using (as well as sumary statistics). The following command records a top printout ordered by memory consumption every 0.5 seconds:
top -bd0.5 -o +%MEM > memory.log
Just note that the log file will grow a lot faster than if you only store the total memory utilization statistics so be sure you don't run out of disk space.
There's a program called
sar
on *nix systems. You could try to use that to monitor memory usage. It takes measurements at regular intervals. Do a
man sar
for more details. I think the option is -r for taking memory measurements, -i to specify the interval you'd like.
I think adding a crontab entry will be enough
*/5 * * * * free -m >> some_output_file
There are other tools like SeaLion, New Relic, Server Density etc which will almost do the same but are much easier to install and configure. My favorite is SeaLion, as it being free and also it gives a awesome timeline view of raw outputs of common linux commands.
You could put something like
vmstat X >> mylogfile
into a startup script. Since your application is already in startup you could just add this line to the end of the initialization script your application is already using.
(where X is # of seconds between log messages)
To periodically log the memory usage efficiently, I combined another answer here with a method to only retain the top-K memory-using processes.
top -bd 1.5 -o +%MEM | grep "load average" -A 9 > memory_usage.log
This command will record, every 1.5s, the top header information and the 3 highest memory-consuming processes (there's a 6-line offset for top's header information). This saves lots of disk space over recording top's information for every process.
So I know that I am late to this game, but I just came up with this answer, as I needed to do this, and really didn't want the extra fields that vmstat, free, etc... all will seem to output without excess filtering. So here is the answer that I came up with:
top -bd 0.1 | grep 'KiB Mem' | cut -d' ' -f10 > memory.txt
OR:
top -bd 0.1 | grep 'KiB Mem' | cut -d' ' -f10 | tee memory.txt
the standard output from top when grep ing with Kib Mem is:
KiB Mem : 16047368 total, 8708172 free, 6015720 used, 1323476 buff/cache
By running this through cut, we filter down to literally just the number prior to used
The user can indeed modify the 0.1 to another number in order to run different capture sample rates. In my case I wanted to use top also because you can run memory stats faster than 1 second per capture, as you can see here I wanted to capture a stat every 1/10th of a second.
NOTES:
It does turn out that piping through cut cause MASSIVE delay in getting anything out to file. As we later found out, it is much faster to leave out the cut command during data acquisition, then perform the cut command on the output file later.
Also, we had no need for timestamps in our tests.
This thus looks as follows:
Begin Logging:
top -bd 0.1 | grep 'KiB Mem' | tee memory_raw.txt
Exit Logging:
ctrl-z (to exit logging)
Filter:
2 levels of cut (filtering), first by comma, then by space. This is due to the alignment of top and provides much cleaner output:
cut memory_raw -d',' -f3 | tee memory_used_withlabel.txt
cut memory_used_withlabel.txt -d' ' -f3 | tee memory_used.txt