I ran the following command in the shell:
sudo perf record -F 60000 -e LLC-misses:u mplayer video.mp4
After the program finished execution, I ran this command:
sudo perf report
The report shows the locations of code (in MPlayer and the associated shared libraries) in the userspace with the percentage of LLC misses. Is it possible to determine the origin of the misses? i.e., if they are instruction misses or data misses?
One more question: when I change LLC-misses:u to LLC-misses:up the generated report will be empty. What is wrong with the PEBS switch addition?
Related
I have some intermittent performance issues that I want to capture via perf top. The issue is intermittent, so I want to write a script that runs perf top when the issue is occurring so that I can save the data and view it at a later time.
I can't seem to figure out how to get perf top to put its output in a file, it seems to demand to be run interactively. Here is what I've tried so far:
# timeout 10 perf top --stdio -E 20 > 'perf-top'
This does not kill perf, just leaves it running in the background forever until I create another console session, find the PID, and kill it.
# timeout --signal=9 10 perf top --stdio -E 20 > 'perf-top'
This kills perf in the expected 10 seconds, but the output is not written to the file that I specified.
Is there some special way that this command needs to be run? It works if I run it from an interactive ssh session, but I'd really like to be able to run it from a script. I'm trying to put it into an ansible task with a few other metrics-gathering programs.
SIGKILL (9) isn't catchable, so it's impossible for perf to flush any buffered output.
Use any another signal so perf can clean up after receiving the signal and write output.
If the default SIGTERM doesn't work, then maybe try SIGHUP, SIGINT, or others.
timeout --signal=INT
timeout 10 perf top --stdio -E 20 > perf-top (with the default SIGTERM) works for me (as non-root) on my Arch Linux desktop. perf version 5.0.g1c163f4
There are apparently some counters in Linux perf like syscall:sys_enter_select, but on my system perf list does not show any of them
Evidence that other people do have these counters is here: http://www.brendangregg.com/blog/2014-07-03/perf-counting.html
If I run perf top -e 'syscalls:sys_enter_*' it says:
Can't open event dir: Permission denied
invalid or unsupported event: 'syscalls:sys_enter_*'
Other event types (the ones in perf list) work fine.
What do I need to do to access syscall counters in perf? I'm using Linux kernel and perf version 3.10 on x86_64.
Some perf counters, including all the syscall ones, are only available to the root user. sudo perf list will show all the counters, including syscall ones assuming the kernel is built with CONFIG_HAVE_SYSCALL_TRACEPOINTS (see Grisha Levit's answer regarding that).
So, to make perf top -e 'syscalls:sys_enter_*' work, run it under sudo--even if you do not need sudo for other counters like cycles.
You must have an old version of perf to go with your crusty old 3.10 kernel.
On a modern system (x86-64 Arch Linux with Linux 4.15.8-1-ARCH, and a matching version of perf), perf answers this question for you:
$ perf stat -e 'syscalls:sys_enter_*stat*' ls -l
event syntax error: 'syscalls:sys_enter_*stat*'
\___ can't access trace events
Error: No permissions to read /sys/kernel/debug/tracing/events/syscalls/sys_enter_*stat*
Hint: Try 'sudo mount -o remount,mode=755 /sys/kernel/debug/tracing'
Run 'perf list' for a list of valid events
...
$ ll /sys/kernel/debug/ -d
drwx------ 33 root root 0 Mar 14 00:02 /sys/kernel/debug/
Interesting that you could make it world-readable, similar to how you can put kernel.perf_event_paranoid = 0 and kernel.yama.ptrace_scope = 0 in /etc/sysctl.d/99-local.conf for convenient debugging / tracing / profiling on a single-user desktop without using root all the time.
These will be missing if the kernel was not built with CONFIG_HAVE_SYSCALL_TRACEPOINTS.
You can check like so:
# grep TRACEPOINTS "/boot/config-$(uname -r)"
CONFIG_TRACEPOINTS=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
I am using perf tool to bench mark one of my projects. The issue I am facing is that wo get automatihen I run perf tool on my machine, everything works fine.
However, I am trying to run perf in automation servers to make it part of my check in process but I am getting the following error from automation servers
WARNING: Kernel address maps (/proc/{kallsyms,modules}) are restricted,
check /proc/sys/kernel/kptr_restrict.
Samples in kernel functions may not be resolved if a suitable vmlinux
file is not found in the buildid cache or in the vmlinux path.
Samples in kernel modules won't be resolved at all.
If some relocation was applied (e.g. kexec) symbols may be misresolved
even with a suitable vmlinux or kallsyms file.
Error:
Permission error - are you root?
Consider tweaking /proc/sys/kernel/perf_event_paranoid:
-1 - Not paranoid at all
0 - Disallow raw tracepoint access for unpriv
1 - Disallow cpu events for unpriv
2 - Disallow kernel profiling for unpriv
fp: Terminated
I tried changing /proc/sys/kernel/perf_event_paranoid to -1 and 0 but still see the same issue.
Anybody seen this before? Why would I need to run the command as root? I am able to run it on my machine without sudo.
by the way, the command is like this:
perf record -m 32 -F 99 -p xxxx -a -g --call-graph fp
You can't use -a (full system profiling) and sample kernel from non-root user: http://man7.org/linux/man-pages/man1/perf-record.1.html
Try running it without -a option and with event limited to userspace events by :u suffix:
perf record -m 32 -F 99 -p $PID -g --call-graph fp -e cycles:u
Or use software event for virtualized platforms without PMU passthrough
perf record -m 32 -F 99 -p $PID -g --call-graph fp -e cpu-clock:u
I'm stumped, I read the perf tutorial and am trying to do a simple test beyond "perf stat" which works. However perf record either doesnt work ,or perf annotate shows no samples recorded. Running perf
For example(im running with sudo because without it i get a bunch of errors which i will post at the end):
sudo perf record -e cycles,instructions,cache-misses -a -c 1 ./FooExe
[ perf record: Woken up 4 times to write data ]
[ perf record: Captured and wrote 1.794 MB perf.data (~78393 samples) ]
.
sudo perf report -D -i perf.data |grep RECORD_SAMPLE |wc -l
Failed to open /tmp/perf-23796.map, continuing without symbols
20486
.
sudo perf annotate -d ./FooExe
the perf.data file has no samples! Press any key
So thats as far as i get. I tried to rebuild perf for my ssystem from source but that didnt seem to help either.
Im using Ubuntu 14.04 kernel 3.19.0-49-generic. This is on intel i7 I4510U cpu . I made sure to compile my program with symbols , but i get the same results regardless of which application i try to profile.
-- if i run without sudo :
WARNING: Kernel address maps (/proc/{kallsyms,modules}) are restricted,
check /proc/sys/kernel/kptr_restrict.
Samples in kernel functions may not be resolved if a suitable vmlinux
file is not found in the buildid cache or in the vmlinux path.
Samples in kernel modules won't be resolved at all.
If some relocation was applied (e.g. kexec) symbols may be misresolved
even with a suitable vmlinux or kallsyms file.
Cannot read kernel map
Error:
You may not have permission to collect system-wide stats.
Consider tweaking /proc/sys/kernel/perf_event_paranoid:
-1 - Not paranoid at all
0 - Disallow raw tracepoint access for unpriv
1 - Disallow cpu events for unpriv
2 - Disallow kernel profiling for unpriv
I just tried your command. The problem was that you used -a to profile all processes system-wide, so it never ran ./FooExe. You can confirm this with strace -f perf ... ./FooExe, and note the lack of any execve system call. And also the fact that it returns instantly, even if FooExe should have taken several seconds.
Here's an example of recording samples for a busy-loop awk command:
perf record -e cycles,instructions,cache-misses awk 'BEGIN{for(i=0;i<40000000;i++){}}'
Now perf report works. You don't need to specify the executable for the report command, because perf.data only has data for the one executable.
This works the same way with the ocperf.py wrapper, but you could record events for more uarch-specific events using symbolic names (instead of looking up codes and numeric arguments in -e):
$ ocperf.py record -e cycles,cache-misses,uops_dispatched_port.port_0 awk 'BEGIN{for(i=0;i<40000000;i++){}}'
perf record -e cycles,cache-misses,cpu/event=0xa1,umask=0x1,name=uops_dispatched_port_port_0,period=2000003/ awk 'BEGIN{for(i=0;i<40000000;i++){}}'
(warning lines about kernel symbols)
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.352 MB perf.data (7819 samples)
$ ocperf.py report
Summary:
I want to be able to profile the event sched:sched_stat_sleep without running as root. Is this possible?
Details:
When I run the command: sudo perf stat -e sched:sched_stat_sleep sleep 1 I get output that looks correct:
Performance counter stats for 'sleep 1':
1,001,729,231 sched:sched_stat_sleep
1.002880455 seconds time elapsed
But when I run without sudo I get the error
invalid or unsupported event: 'sched:sched_stat_sleep'
Run 'perf list' for a list of valid events
usage: perf stat [<options>] [<command>]
-e, --event <event> event selector. use 'perf list' to list available events
And when I run perf list without sudo part of the output is:
[ Tracepoints not available: Permission denied ]
I found anther question which suggests setting kernel.perf_event_paranoid to -1. I tried doing this but it doesn't appear to make a difference. I did not reboot after making the change (is that necessary?). Viewing the setting seems to indicate that it's set to -1 now.
cat /proc/sys/kernel/perf_event_paranoid
-1
I'm running Arch Linux with Linux kernel version 3.19.2-1-ARCH #1 SMP PREEMPT