Save "perf stat" output per second to a file - linux

Question

I want to get perf output and analyse it. I used
while (true) {
system("sudo perf kvm stat -e r10f4 -a sleep 1 2>&1 | sed '1,3d' | sed '2,$d' > 'system.log' 2>&1");
sleep(0.5);
}
The code above frequently uses perf, which is costly. Instead, I'm running perf stat like: perf kvm stat -a -I 1000 > system.log 2>&1.
This command will keep writting data to "system.log", but I only need the data in a second.
I'm wondering how to let the new data overwrites the older data every second.
Does anybody know if my thoughts are feasible? Or other ways which can solve my problem.

Answer 1

You mean have perf keep overwriting, instead of appending, so you only have the final second?
One way would be to have your own program pipe perf stat -I1000 into itself (e.g. via C stdio popen or with POSIX pipe / dup2 / fork/exec). Then you get to implement the logic that chooses where and how to write the lines to a file. Instead of just writing them normally, you can lseek to the start of the output file before every write. Or pwrite instead of write to always write at file-position 0. You can append some spaces for padding out to a fixed width to make sure a longer line didn't leave some characters in the file that you're not overwriting. (Or ftruncate the output file after writes with lines shorter than the previous line).
Or: redirect perf stat -I 1000 >> file.log with O_APPEND, and periodically truncate the length.
A file opened for append will automatically write at wherever the current end is, so you can leave perf ... -I 1000 running and truncate the file every second or every 5 seconds or so. So at most you have 5 lines to read through to get to the final one you actually want. If using system to run it through a shell, use >>. Or use open(O_APPEND)/dup2 if using fork / execve.
To do the actual truncation, truncate() by path, or ftruncate() by open fd, in a sleep loop. Ideally you'd truncate right before a new line was about to be written, so most of the time there'd be a line. But unless you make extra system calls like fstat or inotify you won't know when exactly to expect another one, although dead reckoning and assuming that sleep sleeps for the minimum time can work ok.
Or: redirect perf stat -I 1000 (without append), and lseek its fd from the parent process, to create the same effect as piping through a process / thread that seeks before write.
Why does forking my process cause the file to be read infinitely shows that parent/child processes with file descriptors that share the same open file description can influence each other's file position with lseek.
This only works if you do the redirect yourself with open / dup2, not if you leave it to a shell via system. You need your own descriptor that refers to the same open file description
Not sure if you could play tricks like lseek on the same fd that a child process had open; in that case avoiding O_APPEND could let you set the write position to 0 without truncating, so the next write will overwrite the contents. But that only works if you open / dup2 / fork/exec and the fd in the parent has its file position tied to the fd in the child process.
Totally untested example
This might not actually compile even with the right headers, but hopefully illustrates the idea.
// must *not* use O_APPEND, otherwise file position is ignored for write.
int fd = open("system.log", O_WRONLY|O_CREAT|O_TRUNC, 0666);
if ((pid = fork()) != 0){
// parent
while(1) {
sleep(1);
lseek(fd, 0, SEEK_SET); // reset the position at which perf will do its next write
}
} else {
// child
dup2(fd, 1); // redirect stdout to the open file
// TODO: look up best practices for fork/exec, and error checking
execlp("sudo", "perf", "kvm", "stat", "-er10f4", "-a", "sleep", "1", NULL);
perror("execlp sudo perf failed");
}
Note the total lack of error checking of return values. This is more like pseudocode, even though it's written in (hopefully) valid C syntax.
And BTW, you can reduce that pipeline to one sed command, just sed -n 4p to print line 4 but not other lines. But perf ... stat -I 1000 doesn't waste lines so there'd be no need to filter on line numbers that way.