I'm using perf to do some optimization work. By using perf script to show my perf records and get results like this:
MyServer 13631 [015] 2611179.755027: probe_app:stat_timer: (52bbe0)
......
I write a script to analysis the results and need to change timestamp to epoch timestamp (at least accurate in second).
To to this, I first get the host start up epoch timestamp
$ date -d "`uptime -s`" +%s
1562557105
And verified by adding the uptime
$ cat /proc/uptime
2612552.50 36615651.34
The the start epoch timestamp is correct.
But I found the perf timestamp 211179.755027 is not the elapsed seconds since system startup as document said, there are some error about 100s+. How to get accurate timestamp while exec "perf record"? I tried "-T" but it seems not work.
The default clock used by perf (or the underlying perf_event_open) is not actually documented - so I would not necessarily rely on it to be any specific clock. Fortunately, perf allows you to chose a specific clock with the -k option. In general CLOCK_MONOTONIC and CLOCK_MONOTONIC_RAW are supported. Other clocks may be available depending on the events and version.
If you choose a specific clock, you can then use clock_gettime to get matching timestamps. I'm not aware of a simple way to do this from command line, but a C program will do.
Related
I've been trying to analyze the output of perf sched record but I don't understand with what frame of reference do I try to understand the "20624.983302 secs". It isn't Unix time for sure, so what is it? How would I go about converting this into Unix time?
*A0 20624.983302 secs A0 => migration/0:12
*. 20624.983311 secs . => swapper:0
*B0 20624.983318 secs B0 => IPC I/O Child:33924
*. 20624.983355 secs
*C0 20624.983485 secs C0 => WRScene~lder#15:39974
*. 20624.983581 secs
*D0 20624.983972 secs D0 => IPC I/O Parent:33780
These timestamps are captured using the kernel scheduler clock, which counts in nanoseconds since boot. The exact details depend on the compile-time parameters chosen to build a particular Linux distribution and the target architecture.
In general, the timestamp of a sample is captured around the same time when it's recorded. Timestamps on the same core are guaranteed to be monotonically increasing as long the core remains in an active state. The samples you've shown were all captured on the same core and the core remained active from the first sample to the last sample. So the timestamps are guaranteed to be monotonic in this case irrespective of the platform and distribution. When profiling on multiple cores, there is no guarantee that the clocks on all cores are in sync.
All perf tools use the same clock to capture timestamps, but they may differ in the way timestamps are printed and it may happen that two tools print timestamps from the same sample file differently. This depends on the kernel version.
It's possible to specify a clock source when calling perf_event_open() by setting use_clockid to 1 and setting clockid to one of the clock sources defined in linux/time.h, such as CLOCK_MONOTONIC. perf record provides the -k or --clockid option to specify the clock source for capturing timestamps.
Modern distributions on x86 typically use TSC as the source for the scheduler clock (check /sys/devices/system/clocksource/clocksource0/current_clocksource). So if you're on an x86 processor, most probably the TSC of the profiled core was used to capture the current value of TSC cycles, which internally gets converted into nanoseconds. When a timestamp is printed, it may get converted to a different unit. In this case, timestamps are printed in the format "seconds.microseconds". A summary of the behavior of TSC on Intel processors can be found at: Can constant non-invariant tsc change frequency across cpu states?.
I'd like to use 'perf' to measure real execution time of a function. 'perf script' command gives timestamp when the function is called.
Xorg 1523 [001] 25712.423702: probe:sock_write_iter: (ffffffff95cd8b80)
The timestamp field's format is X.Y. How can I understand this value? Is it X.Y seconds?
X.Y is the timestamp in units of seconds.microseconds.
How this value is displayed can be looked at here. You can pass the switch --ns to perf script to display the timestamps in seconds.nanoseconds format too.
To understand this value, you need to understand how the perf module calculates timestamps. You can associate each event with a different clock function to compute the timestamps. By default, perf uses sched_clock function to compute timestamps for an event, more details here.
event->clock = &local_clock;
But you can use the -k switch along with perf record command to associate an event with various clockids.
-k, --clockid
Sets the clock id to use for the various time fields in the
perf_event_type records. See clock_gettime(). In particular
CLOCK_MONOTONIC and CLOCK_MONOTONIC_RAW are supported, some
events might also allow CLOCK_BOOTTIME, CLOCK_REALTIME and
CLOCK_TAI.
Adding the -k switch to perf record command will enable various clock functions depending on which clockid you use, as can be seen here.
sched_clock function shall return the number of nanoseconds since the system was started. A particular architecture may or may not provide an implementation of sched_clock() on its own. The system jiffy counter will be used as sched_clock(), if a local implementation is not provided.
Note that, all of the above code snippets are for Linux kernel 5.6.7.
X.Y is the raw format of the date. The perf's time function, which operates in nanoseconds, needs to be converted to a human readable format. Use this website to convert it to date, http://www.timestamp.fr/ , or using bash
date -d #25712
I am using the iostat utility on my RedHat Linux server to monitor the performance of a disk. When I use "iostat -xd sdh 1", I get the perf result printed every one second. When I use "iostat -xd sdh 5", I get the perf result printed every five second. My feeling is the latter command is printing a snapshot of the perf every five second, rather than averaging over the past 5 seconds. Am I correct in my understanding?
If so, is there a way I can make iostat print the perf. number averaged over n seconds, or is there some other utility that will do that.
Currently, the perf number is fluctuating within a range, and I want to get a somewhat "stable" number. I am hoping that averaging over a period of time will give me such a number.
Thank you,
Ahmed.
i still want to check my Bootloader + Linux Startupcode for an embedded device. Therefore i want to catch the time for every command printed to the serial port.
I know there are programs like putty (which i can dearly recommend), getty, cutecom, picocom, screen etc. But none of these add timestamps to the incomming messages on the host screen (I'm not really talking about the date, more like how many ms have gone since the first output). It actually sounds not like a big deal.
I found out there is one script doing what i wanted to have, called grabserial but it's not working properly, since it's to slow to process the whole output. I discussed this problem in a different forum (if you want to know: grabserial problem but it's not part of the topic). So i can't use that script.
Now again: can you tell me a terminal for Linux which adds timestamps to every line, which was received from a Serial Port?
Thank you
[Edit:] I've found a pretty rough workaround with cereal, which wants to have some settings, since it locks the port everytime you use it. In the end, it adds the actual date and time, not the startup time and difftime between each step, so as you can see I'm still looking for an adequate solution.
This might come 3 years too late, but minicom (https://en.wikipedia.org/wiki/Minicom) supports timestamps for every line printed on the terminal. In Ubuntu it's directly available in the default repos.
You might want to look in to using strace to monitor the serial port. See How can I monitor data on a serial port in Linux?
If you are willing to build the binary by yourself, you can try a branched picocom (https://github.com/codepox/picocom). This is based on picocom 1.7 which is a little old.
I have forked and enhanced this picocom and made it be able to show either delta-time or wall-clock timestamp. You can find it here (https://github.com/tdwong/picocom-with-timestamp). You still have to build the binary by yourself.
Here is how I use it. Note, N is the command to enable/toggle timestamp.
$ picocom -b 115200 /dev/ttyUSB0
...
<ctrl-a> N # enable delta-time timestamp
<ctrl-a> N # toggle wall-clock timestamp
<ctrl-a> N # disable timestamp
tio found at https://tio.github.io provides various timestamp options:
-t, --timestamp
Enable line timestamp.
--timestamp-format <format>
Set timestamp format to any of the following timestamp formats:
24hour 24-hour format ("hh:mm:ss.sss")
24hour-start 24-hour format relative to start time
24hour-delta 24-hour format relative to previous timestamp
iso8601 ISO8601 format ("YYYY-MM-DDThh:mm:ss.sss")
Default format is 24hour
In your case, showing how much time has passed since start, that would be something like this:
tio -t --timestamp-format 24hour-start /dev/ttyUSB0
Settings can also be enabled in tio configuration file ~/.tioconfig
I believe that ExtraPuTTY is the solution you are looking for.
However, I wasn't clear if you wanted something to run ON Linux or just to be able to monitor it (SSH to Linux). If you didn't want a Windows solution, then I apologize.
On a Linux box, I need to display the average CPU utilisation per hour for the last week. Is that information logged somewhere? Or do I need to write a script that wakes up every 15 minutes to copy /proc/loadavg to a logfile?
EDIT: I'm not allowed to use any tools other than those that come with Linux.
You might want to check out sar (man page), it fits your use case nicely.
System Activity Reporter (SAR) - capture important system performance metrics at
periodic intervals.
Example from IBM Developer Works Article:
Add an entry to your root crontab
# Collect measurements at 10-minute intervals
0,10,20,30,40,50 * * * * /usr/lib/sa/sa1
# Create daily reports and purge old files
0 0 * * * /usr/lib/sa/sa2 -A
Then you can simply query this information using a sar command (display all of today's info):
root ~ # sar -A
Or just for a certain days log file:
root ~ # sar -f /var/log/sa/sa16
You can usually find it in the sysstat package for your linux distro
As far as I know it's not stored anywhere... It's a trivial thing to write, anyway. Just add something like
cat /proc/loadavg >> /var/log/loads
to your crontab.
Note that there are monitoring tools (like Munin) which can do this kind of thing for you, and generate pretty graphs of it to boot... they might be overkill for your situation though.
I would recommend looking at Multi Router Traffic Grapher (MRTG).
Using snmpd to read the load average, it will automatically calculate averages at any time interval and length, along with nice charts for analysis.
Someone has already posted a CPU usage example.