How can I grab the percentage of cpu usage on a per process basis? So, for example, I'd like to run my program prog and get the cpu usage it incurred in, for example:
prog name cpu0 cpu1 cpu2 cpu3 total
prog 15 20 45 47 127%
Is there any tool for this?
Thanks.
I think that you can make use of the information in /proc/[pid]/stat and /proc/stat to estimate this.
Check out the great answers to How to calculate the CPU usage of a process by PID in Linux from C? which explain how to calculate CPU usage % for a single processor.
The 6th from last number you get from /proc/[pid]/stat is "processor %d, CPU number last executed on" (on Ubuntu 12.04 at least).
To extend to multiple processors, you could sample the CPU usage over a period and (very roughly!) estimate the proportion of time on each processor. Then use these proportions to split the CPU usage between the processors. Based on the info in /proc/stat you can also sample the total time for each processor and then you have all the variables you need!
See http://linux.die.net/man/5/proc for more info about proc.
For firefox:
while [ 1 ]; do ps --no-heading -C firefox -L -o command,psr,pcpu|sort -k 2 -n; echo; sleep 1; done
You'd have to sum the third column (which I see no ridiculously easy way to do) because it's actually showing you every thread. First column is name, second processor, third, %cpu.
linux process explorer project provides this functionality, you can see a graph for the CPU/Memory/IO for each process in the properties dialog.
Here is a simple python i've made:
import re,time,sys
cpuNum=0
if len(sys.argv)==1:
print "use pidcpu <pid1,pid2,..,pidn>"
sys.exit(0)
pids=sys.argv.pop()
def getCpuTot():
global cpuNum
f=open("/proc/stat","r")
ln=f.read()
f.close()
#cpu 858286704 148088 54216880 117129864 2806189 5046 16997674 0 0 0
r=re.findall("cpu[\d\s]{1}\s+(\d+)\s(\d+)\s(\d+)\s(\d+)\s.*?",ln,re.DOTALL)
cpuNum=len(r)-1
return int(r[0][0])+int(r[0][1])+int(r[0][2])+int(r[0][3])
def getPidCPU(pid):
f=open("/proc/"+ str(pid) +"/stat","r")
ln=f.readline()
f.close()
a=ln.split(" ")
return int(a[13])+int(a[14])
cpu1=getCpuTot()
cpupid1=[]
for pid in pids.split(","):
cpupid1.append(getPidCPU(pid))
time.sleep(1)
cpu2=getCpuTot()
cpupid2=[]
for pid in pids.split(","):
cpupid2.append(getPidCPU(pid))
i=0
for pid in pids.split(","):
perc=int(cpuNum*(cpupid2[i]-cpupid1[i])*100/float(cpu2-cpu1))
i+=1
print pid,perc
Related
I'm trying to find out how I can get \ calculate the CPU utilization of a specific process over X amount of time ( I write my code in python over a Linux based system ).
What I want to get for example is the average CPU of a process in the last hour\day\10 minutes...
Is there a command or a calculation I can run?
*I can't run a command like "top" in the background for X time and calculate the CPU, I need it to be in one set of commands or calculation.
I tried top research on top command but I didn't found useful info for my case.
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu - give the average consumption on the process lifetime
Can there be a way to use uptime or proc[pid]\stat to calculate this?
Thanks,
What about using pidstat?
$ pidstat -p 12345 10
Will output the stats for pid 12345 every 10 seconds. This include CPU%
From there you can run it on background, and redirect output to a file:
$ pidstat -p 12345 10 > my_pid_stats.txt &
Here is a Link with several examples. There is a lot of flexibility in the output, so you can probably customize it to better suit your needs:
https://www.thegeekstuff.com/2014/11/pidstat-examples/
pidstat is part of the sysstat package on ubuntu, in case you decide to install it.
I have a bash script that I plan to run every 5 or 15 mins using crontab based on the load it puts on server.
I can find time of running the script, but load, memory usage and CPU usage I am not sure how to find.
Can someone help me?
Also any suggestions of rough benchmark that will help me decide if the script puts too much load and should be run every 15 mins and not 5 mins.
Thanks in Advance!
You can use "top -b", top gives the CPU usage, memory usage etc,
Insert these lines in your script, this will process in background and will terminate the process as soon as your testing overs.
ssh server_name "nohup top -b -d 0.5 >> file_name &"
\top process will run in background because of &, -d 0.5 will give you the cpu status at every 0.5 secs, redirect the output in file_name for latter analysis.
for killing the process after your test, insert following in your script,
ssh server_name "kill \`ps -elf | grep 'top -b' | grep -v grep | sed 's/ */ /g' |cut -d ' ' -f4\`"
Your main testing script should be between top command and command for killing top.
I presumed you are running the script from client side, if not ignore "ssh server_name".
If you are running it from client side, because of "ssh", you will be asked for the password, for avoiding this follow these 3 simple steps
This will definitely solve the issue.
You can check following utilities
pidstat for CPU load, man page
pmap for memory load, man page
Although you might need to make measurements also for the child processes of your executable, in order to collect summarizing information
For memory, use free -m. Your actual memory available is the second number next to +/- buffers/cache (in megabytes with -m) (source).
For CPU, it's a bit more complicated. Start by looking at cat /proc/stat | grep 'cpu ' (note the space). You'll see something like this:
cpu 2255 34 2290 22625563 6290 127 456
The columns are from left to right, "user, nice, system, idle". CPU usage is usually calculated as (user+nice+system) / (user+nice+system+idle). However, these numbers show the number of "time units" that the CPU has spent doing that thing since boot, and thus are always increasing. If you were to do the aforementioned calculation, you'd get the CPU usage average since boot. To get a point-in-time usage, you have to take 2 samples, find their difference, and calculate the usage from that. To be clear, that will be the average CPU usage between your samples. (source)
It's crucial to measure not user time for some program/script, but the cpu time and kill it when this time limit will be breached.
What's the best way to do it?
One of the most obvious solutions is to check with some time step process tree to see if the requested program/script hasn't breached it's limits. It's implemented in a perl script (pshved/timeout). I'm looking for other aproaches
You can use ulimit(1) or setrlimit(2) to limit the cpu time. The process will be automatically killed if it uses more cpu time. It is also possible to specify a soft limit that can be ignored.
Simple example:
#! /bin/bash
(
ulimit -t 5
python -c '
a, b = 0, 1
while True:
a += b
b += a
'
echo $?
)
echo "..."
I'm running some JMeter tests against a Java process to determine how responsive a web application is under load (500+ users). JMeter will give the response time for each web request, and I've written a script to ping the Tomcat Manager every X seconds which will get me the current size of the JVM heap.
I'd like to collect stats on the server of the % of CPU being used by Tomcat. I tried to do it in a shell script using ps like this:
PS_RESULTS=`ps -o pcpu,pmem,nlwp -p $PID`
...running the command every X seconds and appending the results to a text file. (for anyone wondering, pmem = % mem usage and nlwp is number of threads)
However I've found that this gives a different definition of "% of CPU Utilization" than I'd like - according to the manpages for ps, pcpu is defined as:
cpu utilization of the process in "##.#" format. It is the CPU time used divided by the time the process has been running (cputime/realtime ratio), expressed as a percentage.
In other words, pcpu gives me the % CPU utilization for the process for the lifetime of the process.
Since I want to take a sample every X seconds, I'd like to be collecting the CPU utilization of the process at the current time only - similar to what top would give me
(CPU utilization of the process since the last update).
How can I collect this from within a shell script?
Use top -b (and other switches if you want different outputs). It will just dump to stdout instead of jumping into a curses window.
The most useful tool I've found for monitoring a server while performing a test such as JMeter on it is dstat. It not only gives you a range of stats from the server, it outputs to csv for easy import into a spreadsheet and lets you extend the tool with modules written in Python.
User load: top -b -n 2 |grep Cpu |tail -n 1 |awk '{print $2}' |sed 's/.[^.]*$//'
System load: top -b -n 2 |grep Cpu |tail -n 1 |awk '{print $3}' |sed 's/.[^.]*$//'
Idle load: top -b -n 1 |grep Cpu |tail -n 1 |awk '{print $5}' |sed 's/.[^.]*$//'
Every outcome is a round decimal.
Off the top of my head, I'd use the /proc filesystem view of the system state - Look at man 5 proc to see the format of the entry for /proc/PID/stat, which contains total CPU usage information, and use /proc/stat to get global system information. To obtain "current time" usage, you probably really mean "CPU used in the last N seconds"; take two samples a short distance apart to see the current rate of CPU consumption. You can then munge these values into something useful. Really though, this is probably more a Perl/Ruby/Python job than a pure shell script.
You might be able to get the rough data you're after with /proc/PID/status, which gives a Sleep average for the process. Pretty coarse data though.
also use 1 as iteration count, so you will get current snapshot without waiting to get another one in $delay time.
top -b -n 1
This will not give you a per-process metric, but the Stress Terminal UI is super useful to know how badly you're punishing your boxes. Add -c flag to make it dump the data to a CSV file.
I have a couple variants of a program that I want to compare on performance. Both perform essentially the same task.
One does it all in C and memory. The other calls an external utility and does file IO.
How do I reliably compare them?
1) Getting "time on CPU" using "time" favors the second variant for calling system() and doing IO. Even if I add "system" time to "user" time, it'll still not count for time spent blocked on wait().
2) I can't just clock them for they run on a server and can be pushed off the CPU any time. Averaging across 1000s of experiments is a soft option, since I have no idea how my server is utilized - it's a VM on a cluster, it's kind of complicated.
3) profilers do not help since they'll give me time spent in the code, which again favors the version that does system()
I need to add up all CPU time that these programs consume, including user, kernel, IO, and children's recursively.
I expected this to be a common problem, but still don't seem to find a solution.
(Solved with times() - see below. Thanks everybody)
If I've understood, typing "time myapplication" on a bash command line is not what you are looking for.
If you want accuracy, you must use a profiler... You have the source, yes?
Try something like Oprofile or Valgrind, or take a look at this for a more extended list.
If you haven't the source, honestly I don't know...
/usr/bin/time (not built-in "time" in bash) can give some interesting stats.
$ /usr/bin/time -v xeyes
Command being timed: "xeyes"
User time (seconds): 0.00
System time (seconds): 0.01
Percent of CPU this job got: 0%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:04.57
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 0
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 9
Minor (reclaiming a frame) page faults: 517
Voluntary context switches: 243
Involuntary context switches: 0
Swaps: 0
File system inputs: 1072
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Run them a thousand times, measure actual time taken, then average the results. That should smooth out any variances due to other applications running on your server.
I seem to have found it at last.
NAME
times - get process times
SYNOPSIS
#include
clock_t times(struct tms *buf);
DESCRIPTION
times() stores the current process times in the struct tms that buf
points to. The struct tms is as defined in :
struct tms {
clock_t tms_utime; /* user time */
clock_t tms_stime; /* system time */
clock_t tms_cutime; /* user time of children */
clock_t tms_cstime; /* system time of children */
};
The children's times are a recursive sum of all waited-for children.
I wonder why it hasn't been made a standard CLI utility yet. Or may be I'm just ignorant.
I'd probably lean towards adding "time -o somefile" to the front of the system command, and then adding it to the time given by time'ing your main program to get a total. Unless I had to do this lots of times, then I'd find a way to take two time outputs and add them up to the screen (using awk or shell or perl or something).