We know how to count the instructions of a process dynamically over some time using 'perf' :
perf stat -p <pid> -- sleep <period>
I want to be able to do the same thing on a thread. Assuming a Process with PID 1500 has threads with SPID(TID) 1501, 1502, 1503, and 1504. I want to count each threads instructions using
perf stat -p <spid> -- sleep <period>
However, when I do so, I get:
Performance counter stats for process id '1502':
<not counted> instructions
0.504 seconds time elapsed.
Some events weren't counted. Try disabling NMI watchdog:
echo 0 > /proc/sys/kernel/nmi_watchdog
perf stat ...
echo 1 > /proc/sys/kernel/nmi_watchdog
I am using perf on Raspberry Pi, Raspbian Jessie. There is no '/proc/sys/kernel/nmi_watchdog'. The problem I suspect is that I am trying to profile a Thread ID. I would appreciate if there is a workaround or attribute to the 'perf stat' command to check out thread stats.
Thanks..
Related
I have a bash script where I would like to get know how much percentage of CPU time a process uses until I kill it in the last line of the script.
I know that I could normally do this via the time -v command, however my process is Erlang/OTP based and by using the time command it only measures the startup process statistics.
Therefore I'd like to use the process PID I can get easily and use that one to get the CPU time percentage until the end of the script.
Currently I'm using pidstat but it is only giving me statistics for linear time intervals.
I want to measure the exact timeinterval from when the process started until it gets killed.
Peak RAM statistics woul also be nice.
Could you recommend me any command I could use in this case?
This is my bash script:
sudo emqx start
sleep 10
mypid=$(sudo emqx pid)
echo $mypid
sudo pidstat -h -r -u -v -p "$mypid" 5 > $local/server_results/test1/emqxstats_$b.txt &
# process for load testing
# jthreads = amount of publishing users
sleep 5
until sudo ~/Downloads/apache-jmeter-5.2.1/bin/jmeter -n -t $local/testplans/csv.jmx -Jport=$a | grep -m 1 "... end of run"; do : ; done
sudo emqx stop
kill %!
So I want to measure the CPU percentage from the interval between starting mosquitto and until the Apache Jmeter test finished when it reaches the last 2 lines.
Kind regards
I found the command I was looking after some more research.
ps -o pcpu -p $pid
this was exactly the command I needed, because it calculates the percentage over the lifetime of the process.
I have some intermittent performance issues that I want to capture via perf top. The issue is intermittent, so I want to write a script that runs perf top when the issue is occurring so that I can save the data and view it at a later time.
I can't seem to figure out how to get perf top to put its output in a file, it seems to demand to be run interactively. Here is what I've tried so far:
# timeout 10 perf top --stdio -E 20 > 'perf-top'
This does not kill perf, just leaves it running in the background forever until I create another console session, find the PID, and kill it.
# timeout --signal=9 10 perf top --stdio -E 20 > 'perf-top'
This kills perf in the expected 10 seconds, but the output is not written to the file that I specified.
Is there some special way that this command needs to be run? It works if I run it from an interactive ssh session, but I'd really like to be able to run it from a script. I'm trying to put it into an ansible task with a few other metrics-gathering programs.
SIGKILL (9) isn't catchable, so it's impossible for perf to flush any buffered output.
Use any another signal so perf can clean up after receiving the signal and write output.
If the default SIGTERM doesn't work, then maybe try SIGHUP, SIGINT, or others.
timeout --signal=INT
timeout 10 perf top --stdio -E 20 > perf-top (with the default SIGTERM) works for me (as non-root) on my Arch Linux desktop. perf version 5.0.g1c163f4
For killing a process after a given timeout in Bash, there is a nice command called timeout. However, I'm running my program on a multi-user server, and I don't want the performance of my program to be influenced by others. Is there a way to kill a process in Bash, after a given time that the program is really running?
On Bash+Linux, you can use ulimit -t. Here's from help ulimit:
-t the maximum amount of cpu time in seconds
Here's an example:
$ time bash -c 'ulimit -t 5; while true; do true; done'
Killed
real 0m8.549s
user 0m4.983s
sys 0m0.008s
The infinite loop process was scheduled (i.e. actually ran) for a total of 5 seconds before it was killed. Due to other processes competing for the CPU at the same time, this took 8.5 seconds of wall time.
A command like sleep 3600 would never be killed, since it doesn't use any CPU time.
Is there a way to limit the absolute CPU time (in CPU seconds) spend in a process group?
ulimit -t 10; ./my-process looks like a good option but if my-process forks then each process in the process group gets its own limit. The whole process group can use an arbitrary amount of time by forking every 9 seconds.
The accepted answer on a similar question is to use cgroups but doesn't explain how. However, there are other answers (Limit total CPU usage with cgroups) saying that this is not possible in cgroups and only relative cpu usage can be limited (for example, 0.2 seconds out of every 1 second).
Liran Funaro suggested using a long period for cpu.cfs_period_us (https://stackoverflow.com/a/43660834/892961) but the parameter for the quota can be at most 1 second. So even with a long period I don't see how to set a CPU time limit of 10 seconds or an hour.
If ulimit and cgroups cannot do this, is there another way?
you can do it with cgroups. Do as root:
# Create cgroup
cgcreate -g cpu:/limited
# set shares (cpu limit)
cgset -r cpu.shares=256 limited
# run your program
cgexec -g cpu:limited /my/hungry/program
Alternatively you can use the cpulimit program which can freeze your code periodically. cgroups is the most advanced method though.
to set fixed cpu share :
cgcreate -g cpu:/fixedlimit
# allow fix 25% cpu usage (1 cpu)
cgset -r cpu.cfs_quota_us=25000,cpu.cfs_period_us=100000 fixedlimit
cgexec -g cpu:fixedlimit /my/hungry/program
It turned out, the goal is to limit runtime to certain seconds while measuring it. After setting the desired cgroup limits (in order to get a fair sandbox) you can achieve this goal by running:
((time -p timeout 20 cgexec -g cpu:fixedlimit /program/to/test ) 2>&1) | grep user
After 20 seconds the program will be stopped no matter what, and we can parse for user time (or system or real time) to evaluate it's performance.
This not directly answer the question but refers to the discussion on the actual need of the OP.
If your competition ignores everything except CPU time, it may be fundamentally flawed. One can simply, for example, cache results in the primary storage device. Since you do not count storage access time, it may have the least CPU cycles, but the worse actual performance.
A perfect crime would be to simply send the data via the Internet to another computer, which calculate the task then return the answer. This would finish the task with what appear to be zero cycles.
You actually want to measure "real" time and give this process the highest priority in your system (or actually running it secludedly).
When checking students' homework, we simply used an unrealistic time limit (e.g., 5 minutes for what should be a 10 seconds program), then killing the process if it has not finished in time and failing this submission.
If you want to pick a winner, then simply re-run the best competitors multiple times to ensure the validity of their results.
I found a solution that works for me. It is still far from perfect (read the caveats before using it). I'm somewhat new to bash scripting so any comments about this are welcome.
#!/bin/bash
#
# This script tries to limit the CPU time of a process group similar to
# ulimit but counting the time spent in spawned processes against the
# limit. It works by creating a temporary cgroup to run the process in
# and checking on the used CPU time of that process group. Instead of
# polling in regular intervals, the monitoring process assumes that no
# time is lost to I/O (i.e., wall clock time = CPU time) and checks in
# after the time limit. It then updates its assumption by comparing the
# actual CPU usage to the time limit and waiting again. This is repeated
# until the CPU usage exceeds its limit or the monitored process
# terminates. Once the main process terminates, all remaining processes
# in the temporary cgroup are killed.
#
# NOTE: this script still has some major limitations.
# 1) The monitored process can exceed the limit by up to one second
# since every iteration of the monitoring process takes at least that
# long. It can exceed the limit by an additional second by ignoring
# the SIGXCPU signal sent when hitting the (soft) limit but this is
# configurable below.
# 2) It assumes there is only one CPU core. On a system with n cores
# waiting for t seconds gives the process n*t seconds on the CPU.
# This could be fixed by figuring out how many CPUs the process is
# allowed to use (using the cpuset cgroup) and dividing the remaining
# time by that. Since sleep has a resolution of 1 second, this would
# still introduce an error of up to n seconds.
set -e
if [ "$#" -lt 2 ]; then
echo "Usage: $(basename "$0") TIME_LIMIT_IN_S COMMAND [ ARG ... ]"
exit 1
fi
TIME_LIMIT=$1
shift
# To simulate a hard time limit, set KILL_WAIT to 0. If KILL_WAIT is
# non-zero, TIME_LIMIT is the soft limit and TIME_LIMIT + KILL_WAIT is
# the hard limit.
KILL_WAIT=1
# Update as necessary. The script needs permissions to create cgroups
# in the cpuacct hierarchy in a subgroup "timelimit". To create it use:
# sudo cgcreate -a $USER -t $USER -g cpuacct:timelimit
CGROUPS_ROOT=/sys/fs/cgroup
LOCAL_CPUACCT_GROUP=timelimit/timelimited_$$
LOCAL_CGROUP_TASKS=$CGROUPS_ROOT/cpuacct/$LOCAL_CPUACCT_GROUP/tasks
kill_monitored_cgroup() {
SIGNAL=$1
kill -$SIGNAL $(cat $LOCAL_CGROUP_TASKS) 2> /dev/null
}
get_cpu_usage() {
cgget -nv -r cpuacct.usage $LOCAL_CPUACCT_GROUP
}
# Create a cgroup to measure the CPU time of the monitored process.
cgcreate -a $USER -t $USER -g cpuacct:$LOCAL_CPUACCT_GROUP
# Start the monitored process. In case it fails, we still have to clean
# up, so we disable exiting on errors.
set +e
(
set -e
# In case the process doesn't fork a ulimit is more exact. If the
# process forks, the ulimit still applies to each child process.
ulimit -t $(($TIME_LIMIT + $KILL_WAIT))
ulimit -S -t $TIME_LIMIT
cgexec -g cpuacct:$LOCAL_CPUACCT_GROUP --sticky $#
)&
MONITORED_PID=$!
# Start the monitoring process
(
REMAINING_TIME=$TIME_LIMIT
while [ "$REMAINING_TIME" -gt "0" ]; do
# Wait $REMAINING_TIME seconds for the monitored process to
# terminate. On a single CPU the CPU time cannot exceed the
# wall clock time. It might be less, though. In that case, we
# will go through the loop again.
sleep $REMAINING_TIME
CPU_USAGE=$(get_cpu_usage)
REMAINING_TIME=$(($TIME_LIMIT - $CPU_USAGE / 1000000000))
done
# Time limit exceeded. Kill the monitored cgroup.
if [ "$KILL_WAIT" -gt "0" ]; then
kill_monitored_cgroup XCPU
sleep $KILL_WAIT
fi
kill_monitored_cgroup KILL
)&
MONITOR_PID=$!
# Wait for the monitored job to exit (either on its own or because it
# was killed by the monitor).
wait $MONITORED_PID
EXIT_CODE=$?
# Kill all remaining tasks in the monitored cgroup and the monitor.
kill_monitored_cgroup KILL
kill -KILL $MONITOR_PID 2> /dev/null
wait $MONITOR_PID 2>/dev/null
# Report actual CPU usage.
set -e
CPU_USAGE=$(get_cpu_usage)
echo "Total CPU usage: $(($CPU_USAGE / 1000000))ms"
# Clean up and exit with the return code of the monitored process.
cgdelete cpuacct:$LOCAL_CPUACCT_GROUP
exit $EXIT_CODE
I have a Linux (CentOS) server on which I run a game server on which recently started leaking memory after an update. How can I find out what is causing the memory leak in the server?
Memory profiling
Use Perf tool to check the leaks.
Run the last command for all the processes running in the application and tally the results to find out what is causing memory leak.
A sample usage of probes with perf could be to check libc's malloc() and free() calls:
$ perf probe -x /lib64/libc.so.6 malloc
$ perf probe -x /lib64/libc.so.6 free
Added new event:
probe_libc:malloc (on 0x7eac0)
A probe has been created. Now, let's record the global usage of malloc and free across all the system during 4 second:
$ perf record -e probe_libc:malloc -agR sleep 4
$ perf record -e probe_libc:free -agR sleep 4
Let's record the usage of malloc and free across all any process during 4 second:
$ perf stat -e probe_libc:free -e probe_libc:malloc -ag -p $(pgrep $process_name$) sleep 4
Output:
Performance counter stats for process id '1153':
11,312 probe_libc:free
11,644 probe_libc:malloc
4.001091828 seconds time elapsed
If there is increase in difference between malloc and free count for every time perf command is run, it is a hint of memory leak.
$ perf record -e probe_libc:free -e probe_libc:malloc -agR sleep 2
Run the above command to check for the whole application.
Later run,
$ perf report
to get the report of the above run.