I am recording a performance counters frm linux using the command perf record.
I want to use the result perf.data as an input to other programming apps. Do you know how shall I read and parse the data in perf.data? Is there a way to transform it to .text file or .csv?
There is builtin perf.data parser and printer in perf tool of linux tools with subcommand "script".
To convert perf.data file
perf script > perf.data.txt
To convert output of perf record in other file (perf record -o filename.data) use -i option:
perf script -i filename.data > filename.data.txt
perf script is documented at man perf-script, available online at http://man7.org/linux/man-pages/man1/perf-script.1.html
perf-script - Read perf.data (created by perf record) and display
trace output
This command reads the input file and displays the trace recorded.
'perf script' to see a detailed trace of the workload that was
recorded.
perf data convert --to-json landed in April.
https://man7.org/linux/man-pages/man1/perf-data.1.html
The quipper sub-project of https://github.com/google/perf_data_converter can parse perf.data files.
An example command definition that redirects service check performance data to a text file for later processing by another application is shown below:
define command{
command_name store-service-perfdata
command_line /bin/echo -e "$LASTSERVICECHECK$\t$HOSTNAME$\t$SERVICEDESC$\t$SERVICESTATE$\t$SERVICEATTEMPT$\t$SERVICESTATETYPE$\t$SERVICEEXECUTIONTIME$\t$SERVICELATENCY$\t$SERVICEOUTPUT$\t$SERVICEPERFDATA$" >> /usr/local/nagios/var/service-perfdata.dat
}
Related
I have recently started working with ftrace. I able to get function graph trace using
trace-cmd record -p function_graph -F <bash command>
trace-cmd report
With the above commands I can see time taken by function to finish before ' }
'
But I want to script for listing functions taking more time than 10ms.
Can anyone help me to achieve this?
TLDR
it can be achieved by
trace-cmd report -O fgraph:tailprint=yes
explanation
as per help
-O plugin option -O [plugin:]var[=val]
but when we try but plugin name function_graph what we use for record doesn't work
as well if we dig in ftrace docs
we have option funcgraph-tail but even this won't work
finally when you dig in source code of trace-cmd
trace-cmd/lib/trace-cmd/trace-ftrace.c
has these options.
When using perf report, I don't see any symbols for my program, instead I get output like this:
$ perf record /path/to/racket ints.rkt 10000
$ perf report --stdio
# Overhead Command Shared Object Symbol
# ........ ........ ................. ......
#
70.06% ints.rkt [unknown] [.] 0x5f99b8
26.28% ints.rkt [kernel.kallsyms] [k] 0xffffffff8103d0ca
3.66% ints.rkt perf-32046.map [.] 0x7f1d9be46650
Which is fairly uninformative.
The relevant program is built with debugging symbols, and the sysprof tool shows the appropriate symbols, as does Zoom, which I think is using perf under the hood.
Note that this is on x86-64, so the binary is compiled with -fomit-frame-pointer, but that's the case when running under the other tools as well.
This post is already over a year old, but since it came out at the top of my Google search results when I had the same problem, I thought I'd answer it here. After some more searching around, I found the answer given in this related StackOverflow question very helpful. On my Ubuntu Raring system, I then ended up doing the following:
Compile my C++ sources with -g (fairly obvious, you need debug symbols)
Run perf as
record -g dwarf -F 97 /path/to/my/program
This way perf is able to handle the DWARF 2 debug format, which is the standard format gcc uses on Linux. The -F 97 parameter reduces the sampling rate to 97 Hz. The default sampling rate was apparently too large for my system and resulted in messages like this:
Warning:
Processed 172390 events and lost 126 chunks!
Check IO/CPU overload!
and the perf report call afterwards would fail with a segmentation fault. With the reduced sampling rate everything worked out fine.
Once the perf.data file has been generated without any errors in the previous step, you can run perf report etc. I personally like the FlameGraph tools to generate SVG visualizations.
Other people reported that running
echo 0 > /proc/sys/kernel/kptr_restrict
as root can help as well, if kernel symbols are required.
In my case the solution was to delete the elf files which contained cached symbols from previous builds and were messing things up.
They are in ~/.debug/ folder
You can always use the '$ nm ' command.
here is some sample output:
Ethans-MacBook-Pro:~ phyrrus9$ nm a.out
0000000100000000 T __mh_execute_header
0000000100000f30 T _main
U _printf
0000000100000f00 T _sigint
U _signal
U dyld_stub_binder
I had this problem too, I couldn't see any userspace symbol, but I saw some kernel symbols. I thought this was a symbol loading issue. After tried all the possible solutions I could find, I still couldn't get it work.
Then I faintly remember that
ulimit -u unlimited
is needed. I tried and it magically worked.
I found from this wiki that this command is needed when you use too many file descriptors.
https://perf.wiki.kernel.org/index.php/Tutorial#Troubleshooting_and_Tips
my final command was
perf record -F 999 -g ./my_program
didn't need --call-graph
Make sure that you compile the program using -g option along with gcc(cc) so that debugging information is produced in the operating system's native format.
Try to do the following and check if there are debug symbols present in the symbol table.
$objdump -t your-elf
$readelf -a your-elf
$nm -a your-elf
How about your dev host machine? Is it also running the x86_64 OS?
If not, please make sure the perf is cross-compiled, because the perf depends on the objdump and other tools in toolchain.
I got the same problem with perf after overriding the name of my program via prctl(PR_SET_NAME)
As I can see your case is pretty similar:
70.06% ints.rkt [unknown]
Command you have executed (racket) is different from the one perf have seen.
you can check the value of kptr_restrict by cat /proc/kallsyms. If the addresses of the symbols in the result are all 0x000000, you can fix it by command echo 0 > sys/kernel/kptr_restrict . After this , you may get a wanted result of the perf report
There is a nice example of HTML output from criterion at http://bos.github.com/criterion/.
Which command line option is used to generate this output?
An answer to a related question asserts that this output exits, but it does not seem to show up in the command line options when using --help.
Sorry I didn't get around to your comment-question.
The answer Jedai gives is right - just use -o. For example, here is a line from one of my Makefiles for running benchmarks using defaultMain from Criterion:
./Bench -g -u Bench.csv -o Bench.html -s $(SAMPLES)
Breaking that down, it says:
-g run GC between each sample
-u output CSV data to the given file
-o output HTML data to the given file
-s collect this many samples
Well if you just want html output, then yourBench -o yourReport.html will generate some perfectly reasonable output. If you want to use your own template, look at the templates/report.tpl example in the distribution and use the -t option.
It seems to me that you just pass the template as a command line option, and then it populates it. If the template happens to be an html template, then you've generated html.
See the source here: https://github.com/bos/criterion
I would like to convert some PCAP traces to Netflow format for further analysis with netflow tools. Is there any way to do that?
Specifically, I want to use "flow-export" tool in order to extract some fields of interest from a netflow trace as follows:
$ flow-export -f2 -mUNIX_SECS,SYSUPTIME,DPKTS,DOCTETS < mynetflow.trace
In this case, the mynetflow.trace file is taken by converting a PCAP file using the following commands:
$ nfcapd -p 12345 -l ./
$ softflowd -n localhost:12345 -r mytrace.pcap
This, generates a netflow trace but it cannot be used by flow-export correctly, since it is not in the right format. I tried also to pipe the output of the following command to flow-export as follows:
$ flow-import -V1 -z0 -f0 <mynetflow.trace | flow-export -f2 -mUNIX_SECS,SYSUPTIME,DPKTS,DOCTETS
but the output of the first command generated zero timestamps.
Any ideas?
I took at look at the flow-export documentation and there are some acknowledged bugs with the pcap implementation. Not sure if they are fixed yet.
Depending on the content of your capture, you have a couple of other options:
If you captured straight-up traffic from a link and you want to turn that into NetFlow format you can download a free netflow exporter tool that reads PCAP here:
FlowTraq Free Exporter
or here:
NProbe
If you captured NetFlow traffic in transit (say UDP/2055), then you can replay it with a tool like 'tcpreplay', available in any linux distribution.
If you are using a Linux environment, you can use the argus Linux package. Just install argus using apt or your distribution's package manager, and then you can use this with Argus' ra client to get the binetflow format.
Here is the command:
argus -F /mnt/argus.conf -r " +f+" -w - | ra -F /mnt/ra.conf -Z b -n >"+f.split(".")[0]+".binetflow
Is there way to compress the core files during core dump generation?
If the storage space is limited in the system, is there a way of conserving it in case of need for core dump generation with immediate compression?
Ideally the method would work on older versions of linux such as 2.6.x.
The Linux kernel /proc/sys/kernel/core_pattern file will do what you want: http://www.mjmwired.net/kernel/Documentation/sysctl/kernel.txt#191
Set the filename to something like |/bin/gzip -1 > /var/crash/core-%t-%p-%u.gz and your core files should be saved compressed for you.
For an embedded Linux systems, following script change perfectly works to generate compressed core files in 2 steps
step 1: create a script
touch /bin/gen_compress_core.sh
chmod +x /bin/gen_compress_core.sh
cat > /bin/gen_compress_core.sh #!/bin/sh exec /bin/gzip -f - >"/var/core/core-$1.$2.gz"
ctrl +d
step 2: update the core pattern file
cat > /proc/sys/kernel/core_pattern |/bin/gen_compress_core.sh %e %p ctrl+d
As suggested by other answer, the Linux kernel /proc/sys/kernel/core_pattern file is good place to start: http://www.mjmwired.net/kernel/Documentation/sysctl/kernel.txt#141
As documentation says you can specify the special character "|" which will tell kernel to output the file to script. As suggested you could use |/bin/gzip -1 > /var/crash/core-%t-%p-%u.gz as name, however it doesn't seem to work for me. I expect that the reason is that on my system kernel doesn't treat the > character as a output, rather it probably passes it as a parameter to gzip.
In order to avoid this problem, like other suggested you can create your file in some location I am using /home//crash/core.sh, create it using the following command, replacing with your user. Alternatively you can also obviously change the entire path.
echo -e '#!/bin/bash\nexec /bin/gzip -f - >"/home/<username>/crashes/core-$1-$2-$3-$4-$5.gz"' > ~/crashes/core.sh
Now this script will take 5 input parameters and concatenate them and add to core-path. The full paths must be specified in the ~/crashes/core.sh. Also the location of this script can be specified. Now lets tell kernel to use tour executable with parameters when generating file:
sudo sysctl -w kernel.core_pattern="|/home/<username>/crashes/core.sh %e %p %h %t"
Again should be replaced (or entire path to match location and name of core.sh script). Next step is to crash some program, lets create example crashing cpp file:
int main (){
int * a = nullptr;
int b = *a;
}
After compiling and running there are 2 options, either we will see:
Segmentation fault (core dumped)
Or
Segmentation fault
In case we see the latter, there are few possible reasons.
ulimit is not set, ulimit -c should specify what is limit for cores
apport or your distro core dump collector is not running, this should be investigated further
there is an error in script we wrote, I suggest than checking some basic dump path to check if the other things aren't reason the below should create /tmp/core.dump:
sudo sysctl -w kernel.core_pattern="/tmp/core.dump"
I know there is already an answer for this question however it wasn't obvious for me why it isn't working "out of the box" so I wanted to summarize my findings, hope it helps someone.