From a great answer on here, my understanding is that perf samples every time an event counter exceeds a threshold. When asked to sample at a certain frequency, it tries to set the event counter threshold such that the overflow occurs at the specified interval, by adjusting its estimate of the rate of events per time.
I'm asking myself how this works with multiple threads, for instance with the default "cycles" event.
Is there a global "cycles" counter that samples whatever threads are running at the time when the overflow occurs? Or does each CPU have its own "cycles" counter that samples the thread that it is currently running, and if yes, does "each CPU" mean logical or physical cores? Or is it a counter per thread?
Are only cycles counted that were spent running the program?
I hope to be able to explain patterns like these – obviously the rate of events is being recalibrated, but it's not clear to me why:
tid timestamp event counter
5881 187296.210979: 15736902 cycles:
5881 187296.215945: 15664720 cycles:
5881 187296.221356: 15586918 cycles:
5881 187296.227022: 1 cycles:
5881 187296.227032: 1 cycles:
5881 187296.227037: 62 cycles:
5881 187296.227043: 6902 cycles:
5881 187296.227048: 822728 cycles:
5881 187296.231842: 90947120 cycles:
I'd expect it to calculate the next counter threshold at each sample, so after the sample at 187296.215945, it sets the next sample to occur after 15586918 cycles, right? But it doesn't make sense to set the threshold after 187296.221356 to 1 cycle, when the target frequency of 200Hz has been stable before. Is this interference from another thread?
Disclamer
I am not a expert on this topic, but I found this question very interesting, so I tried to come up with an answer. Take this answer with a grain of salt. Corrections are welcome -- and maybe Cunningham's law will get us better answers.
What cycles maps to
According to the perf wiki, on Intel, perf uses the UNHALTED_CORE_CYCLES event.
From the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 4, 18.2.1 Architectural Performance Monitoring Version 1
Configuration facilities and counters are not shared between logical processors sharing a processor core.
For AMD, the perf wiki, states that the CPU_CLK_UNHALTED hardware event is used. I couldn't find this event in the current Preliminary Processor Programming Reference (PPR) for AMD Family 19h Model 01h, Revision B1 Processors Volume 2 of 2, but I found this in section 2.1.17.1:
There are six core performance event counters per thread, six performance events counters per L3 complex and four Data
Fabric performance events counters
I would conclude that the processors support tracking the cycles event per logical core, and I would assume it to be similar on ARM and other architectures (otherwise, I think the performance counters would be a lot less useful)
What perf does
Now perf has different sampling modes:
The perf tool can be used to count events on a per-thread, per-process, per-cpu or system-wide basis. In per-thread mode, the counter only monitors the execution of a designated thread. When the thread is scheduled out, monitoring stops. When a thread migrated from one processor to another, counters are saved on the current processor and are restored on the new one.
and
By default, perf record operates in per-thread mode, with inherit mode enabled.
From these sources, I would expect the following behavior from perf:
When a thread starts executing on a core, the performance counter is reset
As the thread runs, whenever the counter overflows, a sample is taken
If the thread stops executing, the monitoring stops
Your questions
So, I would conclude that
Is there a global "cycles" counter that samples whatever threads are running at the time when the overflow occurs? Or does each CPU have its own "cycles" counter that samples the thread that it is currently running, and if yes, does "each CPU" mean logical or physical cores?
Each logical core has its own counter.
Or is it a counter per thread?
It is a hardware counter on the cpu core, but perf allows you to use it as if it were per thread -- if a thread of a different program gets scheduled, this should not have any impact on you. By default, perf does not annotate thread information to the samples stored in perf.data. According to the man page, you can use -s or --stat to store this information. Then, perf report will allow you to analyze individual threads of your application.
Are only cycles counted that were spent running the program?
Yes, unless specified otherwise.
Your output
tid timestamp event counter
5881 187296.210979: 15736902 cycles:
5881 187296.215945: 15664720 cycles:
5881 187296.221356: 15586918 cycles:
5881 187296.227022: 1 cycles:
5881 187296.227032: 1 cycles:
5881 187296.227037: 62 cycles:
5881 187296.227043: 6902 cycles:
5881 187296.227048: 822728 cycles:
5881 187296.231842: 90947120 cycles:
What did you execute to get this output? Maybe I'm misinterpreting, but I would guess that the following happened:
The points here are partly invalidated by the experiment below
You recorded with a specified target-frequency. That means perf tries to optimize the current overflow value of the hardware counter such that you get as many cycles overflows per second as you specified.
For the first three timestamps, threads of your program were executed on the CPU, this triggered high cycles counts. perf took samples approximately every 0.005s.
Then, it looks like your threads were not executed for that many cpu cycles per second anymore. Maybe it was waiting for IO operations most of its time?* Thus, the next sample was taken after 0.006s and the cycles count dropped to one. perf noticed that the actual sampling frequency had dropped, so it decremented the overflow threshold with the idea to keep the sampling rate stable.
Then, maybe the IO operation was finished and your threads started running for more cpu cycles per second again. This caused lots of cycles events, but with the lower overflow threshold, perf now took a sample after fewer events (after 0.00001s and 0.000005s for the next 3 samples). Perf incremented the overflow threshold back up during this period.
For the last sample, it seems to have arrived back at around 0.005s distance between samples
Experiment
I think the following might give more insights. Let's create a small, easy to understand workload:
int main() {
volatile unsigned int i = 0;
while(1) {
i++;
}
}
gcc compiles the loop to four instructions: memory load, increment, memory store, jump. This utilizes one logical core, according to htop, just as you'd expect. I can simulate that it stopped executing as if it was waiting for IO or not scheduled, by simply using ctrl+Z in the shell to suspend it.
Now, we run
perf record -F 10 -p $(pidof my_workload)
let it run for a moment. Then, use ctrl+Z to suspend execution. Wait for a moment and then use fg to resume execution. After a few seconds, end the program.
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0,021 MB perf.data (65 samples) ]
In my case, perf record wrote 65 samples. We can use perf script to inspect the sample data written and get (full output, because I think I might accidentally remove something important. This was on an Intel i5-6200U, Ubuntu 20.04, kernel 5.4.0-72-generic):
my_workload 831622 344935.025844: 1 cycles: ffffffffa0673594 native_write_msr+0x4 ([kernel.kallsyms])
my_workload 831622 344935.025847: 1 cycles: ffffffffa0673594 native_write_msr+0x4 ([kernel.kallsyms])
my_workload 831622 344935.025849: 2865 cycles: ffffffffa0673594 native_write_msr+0x4 ([kernel.kallsyms])
my_workload 831622 344935.025851: 16863383 cycles: ffffffffa12016f2 nmi_restore+0x25 ([kernel.kallsyms])
my_workload 831622 344935.031948: 101431200645 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344935.693910: 269342015 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344935.794233: 268586235 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344935.893934: 269806654 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344935.994321: 269410272 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344936.094938: 271951524 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344936.195815: 269543174 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344936.295978: 269967653 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344936.397041: 266160499 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344936.497782: 265215251 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344936.596074: 269863048 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344936.696280: 269857624 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344936.796730: 269274440 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344936.897487: 269115742 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344936.997988: 266867300 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344937.097088: 269734778 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344937.196955: 270487956 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344937.297281: 270136625 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344937.397516: 269741183 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344943.438671: 173595673 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344943.512800: 251467821 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344943.604016: 274913168 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344943.703440: 276448269 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344943.803753: 275059394 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344943.903362: 276318281 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344944.005543: 266874454 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344944.105663: 268220344 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344944.205213: 269369912 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344944.305541: 267387036 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344944.405660: 266906130 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344944.506126: 266194513 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344944.604879: 268882693 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344944.705588: 266555089 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344944.804896: 268419478 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344944.905269: 267700541 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344945.004885: 267365839 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344945.103970: 269655126 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344945.203823: 269330033 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344945.304258: 267423569 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344945.403472: 269773962 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344945.504194: 275795565 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344945.606329: 271013012 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344945.703866: 277537908 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344945.803821: 277559915 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344945.904299: 277242574 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344946.005167: 272430392 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344946.104424: 275291909 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344946.204402: 275331204 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344946.304334: 273818645 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
my_workload 831622 344946.403674: 275723986 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344946.456065: 1 cycles: ffffffffa0673594 native_write_msr+0x4 ([kernel.kallsyms])
my_workload 831622 344946.456069: 1 cycles: ffffffffa0673594 native_write_msr+0x4 ([kernel.kallsyms])
my_workload 831622 344946.456071: 2822 cycles: ffffffffa0673594 native_write_msr+0x4 ([kernel.kallsyms])
my_workload 831622 344946.456072: 17944993 cycles: ffffffffa0673596 native_write_msr+0x6 ([kernel.kallsyms])
my_workload 831622 344946.462714: 107477037825 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344946.804126: 281880047 cycles: 558f3623317e main+0x15 (/tmp/my_workload)
my_workload 831622 344946.907508: 274093449 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344947.007473: 270795847 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344947.106277: 275006801 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344947.205757: 274972888 cycles: 558f36233178 main+0xf (/tmp/my_workload)
my_workload 831622 344947.305405: 274436774 cycles: 558f3623317b main+0x12 (/tmp/my_workload)
I think we can see two main things in this output
The sample at 344937.397516 seems to be the last sample before I suspended the program and 344943.438671 seems to be the first sample after it resumed. We a have a little lower cycles count here. Apart from that, it looks just like the other lines.
However, your pattern can be found directly after starting -- this is expected I'd say -- and at timestamp 344946.456065. With the annotation native_write_msr I think what we observe here is perf doing internal work. There was this question regarding what native_write_msr does. According to the comment of Peter to that question, this is the kernel programming hardware performance counters. It's still a bit strange. Maybe, after writing out the current buffer to perf.data, perf behaves just as if it was just started?
* As Jérôme pointed out in the comments, there can be many reasons why less cycles events happened. I'd guess your program was not executed because it was sleeping or waiting for IO or memory access. It's also possible that your program simply wasn't scheduled to run by the OS during this time.
If you're not measuring a specific workload, but your general system, it may also happen that the CPU reduces clock rate or goes into a sleep state because it has no work to do. I assumed that you probably did something like perf record ./my_program with my_program being a CPU intensive workload, so it think it was was unlikely that the cpu decided to sleep.
Related
I am attempting to set up a basic pipe that'll transfer all data written to ttyS3 to ttyUSB0. I found a few solutions to the problem such as this, but they don't seem to help much. The issue seems to be that anytime I do anything with ttyS3, I get this:
stty: /dev/ttyS3: Input/output error
Doing ls -l /dev/ttyS* and the same for /dev/ttyUSB* I get the following:
root#arm-64:~# ls -l /dev/ttyS*
crw-rw---- 1 root dialout 4, 64 Feb 9 13:08 /dev/ttyS0
crw-rw---- 1 root dialout 4, 65 Feb 9 13:08 /dev/ttyS1
crw--w---- 1 root tty 4, 66 Feb 9 13:08 /dev/ttyS2
crw-rw---- 1 root dialout 4, 67 Feb 9 13:08 /dev/ttyS3
crw-rw---- 1 root dialout 4, 68 Feb 9 13:08 /dev/ttyS4
root#arm-64:~# ls -l /dev/ttyUSB*
crw-rw---- 1 root dialout 188, 0 Feb 9 13:08 /dev/ttyUSB0
I've created the following script to do the job for me at startup. I changed the major/minor values to match that of USB0 after reading somewhere that this could work as a pipe. Although it does execute without throwing an Input/output error, it doesn't seem to work as intended.
#!/bin/bash
rm /dev/ttyS3
mknod -m 666 /dev/ttyS3 c 188 0
chown root.dialout /dev/ttyS3
chmod 666 /dev/ttyS3
stty -F /dev/ttyUSB0 speed 115200 cs8
stty -F /dev/ttyS3 speed 115200 cs8
cat /dev/ttyS3 > /dev/ttyUSB0 &
I just need to create a basic pipe that'll take all data written to ttyS3 and pass it on to ttyUSB0. Although I don't think it's relevant, I'm running Armbian bullseye on a TV box (Tx3 Mini)
I just need to create a basic pipe that'll take all data written to ttyS3 and pass it on to ttyUSB0
Don't see a problem so long as each serial terminal is properly setup and functional/operational. Before you create the "pipe", did you verify that each serial terminal is operating properly?
On a SBC I have the console on a serial terminal, and established two more serial terminals using a SoC USART and a USB adapter:
# ls -l /dev/tty*S*
crw-rw---- 1 root dialout 246, 0 Jan 1 2012 /dev/ttyGS0
crw------- 1 root tty 4, 64 Jul 31 22:46 /dev/ttyS0
crw-rw---- 1 root dialout 4, 65 Jul 31 22:25 /dev/ttyS1
crw-rw---- 1 root dialout 188, 0 Jul 31 22:28 /dev/ttyUSB0
#
Note that the udev daemon created these device nodes, and no funny business (i.e. manual re-creating device nodes) was necessary to accomplish the "pipe".
To remove canonical processing, each serial terminal is put in raw mode and with matching baudrates:
# stty raw 115200 -F /dev/ttyUSB0
# stty raw 115200 -F /dev/ttyS1
A report of all termios settings:
# stty -aF /dev/ttyUSB0
speed 115200 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>;
eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;
werase = ^W; lnext = ^V; discard = ^O; min = 1; time = 0;
-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff
-iuclc -ixany -imaxbel -iutf8
-opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt
echoctl echoke -flusho -extproc
#
# stty -aF /dev/ttyS1
speed 115200 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>;
eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;
werase = ^W; lnext = ^V; discard = ^O; min = 1; time = 0;
-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff
-iuclc -ixany -imaxbel -iutf8
-opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt
echoctl echoke -flusho -extproc
#
Then when the command
# cat /dev/ttyS1 > /dev/ttyUSB0 &
is issued, whatever is typed on the remote terminal-emulator program connected to /dev/ttyS1 shows up on the remote terminal-emulator program connected to /dev/ttyUSB0.
This seems to behave like the desired "basic pipe that'll take all data written to ttyS? and pass it on to ttyUSB0".
Bottom line:
Unable to duplicate problems, and can create "pipe" of two serial links.
# uname -a
Linux sama5d2-xplained 5.4.81-linux4sam-2020.10 #1 Thu Jan 14 12:54:56 UTC 2021
armv7l armv7l armv7l GNU/Linux
#
The issue seems to be that anytime I do anything with ttyS3, I get this:
stty: /dev/ttyS3: Input/output error
... I'm running Armbian bullseye on a TV box (Tx3 Mini)
As previously mentioned, you need to verify that each serial terminal is operating properly.
Since a "TV box" doesn't really need five (!) serial terminals, you might be seeing/creating bogus device nodes that don't have any hardware to access.
Search the system log for the actual hardware that was initialized, e.g. 'dmesg | grep tty'. One of those UARTs might be used to interface to an IR receiver.
While running time command, one of the programs gives following output:
real 1m33.523s
user 0m15.156s
sys 0m1.312s
Here the real and user+sys time have a lot of difference. This is most likely due to time spent on IO wait/calls. I want to measure total time spend by program in IO wait or IO calls. Is there any way to do that?
I tried using iotop. However, it doesnot report total time spent by the program performing IO.
Yes, strace - which can provide per-system-call statistics.
Example 1
I want to measure time spent on I/O while accessing stackoverflow.com:
$ time curl stackoverflow.com >/dev/null 2>&1
curl stackoverflow.com > /dev/null 2>&1 0.00s user 0.01s system 2% cpu 0.392 total
OK, 2% CPU and 0.01 s in system. Let's find out:
$ strace -c curl stackoverflow.com >/dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 240k 0 240k 0 0 127k 0 --:--:-- 0:00:01 --:--:-- 130k
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
54.12 0.005497 11 506 write
18.16 0.001845 43 43 fstat
11.95 0.001214 30 41 poll
5.75 0.000584 32 18 recvfrom
3.40 0.000345 3 101 mmap
2.51 0.000255 4 62 mprotect
1.98 0.000201 4 50 close
1.84 0.000187 31 6 getsockname
0.29 0.000029 1 42 1 open
Especially useful compare this results with results measured for runing curl without args.
Anyway. strace shows that curl mostly spends time in write, fstat and poll.
Another example
The first approach seems show incorrect results for sleep. If you are not satisfied with the first approach you can just print get times of each syscall (strace -T). Get this data and process them to find summary time of each syscall.
$ strace 2>&1 -T curl stackoverflow.com >/dev/null | head -n 20
execve("/usr/bin/curl", ["curl", "stackoverflow.com"], [/* 62 vars */]) = 0 <0.000219>
brk(0) = 0x186e000 <0.000175>
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc04c9e6000 <0.000166>
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) <0.000238>
open("/etc/ld.so.cache", O_RDONLY) = 3 <0.000144>
fstat(3, {st_mode=S_IFREG|0644, st_size=96498, ...}) = 0 <0.000175>
mmap(NULL, 96498, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc04c9ce000 <0.000164>
close(3) = 0 <0.000160>
open("/usr/lib64/libcurl.so.4", O_RDONLY) = 3 <0.000047>
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\333\300\">\0\0\0"..., 832) = 832 <0.000160>
fstat(3, {st_mode=S_IFREG|0755, st_size=346008, ...}) = 0 <0.000216>
mmap(0x3e22c00000, 2438600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3e22c00000 <0.000189>
mprotect(0x3e22c51000, 2097152, PROT_NONE) = 0 <0.000032>
mmap(0x3e22e51000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x51000) = 0x3e22e51000 <0.000119>
close(3) = 0 <0.000110>
open("/lib64/libidn.so.11", O_RDONLY) = 3 <0.000257>
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0/#U1\0\0\0"..., 832) = 832 <0.000051>
fstat(3, {st_mode=S_IFREG|0755, st_size=209088, ...}) = 0 <0.000041>
mmap(0x3155400000, 2301736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3155400000 <0.000037>
mprotect(0x3155432000, 2093056, PROT_NONE) = 0 <0.000037>
When a Qt app using QSerialPort experiences a non-clean shutdown (e.g. due to receiving and not handling SIGINT), how is the file descriptor of the serial port affected?
After running an app that opens a QSerialPort on /dev/ttyS0, then quitting with Ctl-C, I am finding that cat < /dev/ttyS0 returns instantly (without printing anything) rather than waiting for data (as it usually does).
I would expect that if this is due to an open file handle left hanging around, it would show up in the output of lsof, but lsof | grep ttyS0 returns nothing. (I'm not really sure how else to search for handles on a particular file descriptor.)
I realize this is a bit of an XY problem, since I could avoid the problem entirely by rewriting my app to properly handle SIGINT, but I'd like to have a deeper understanding of what's going on here and if there's a way to recover the serial port when it's in this state.
EDIT: As requested, here is the output of strace cat /dev/ttyS0:
execve("/bin/cat", ["cat", "/dev/ttyS0"], [/* 17 vars */]) = 0
brk(0) = 0x91ce000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb76fb000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=72063, ...}) = 0
mmap2(NULL, 72063, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb76e9000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/i686/cmov/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\240o\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1446056, ...}) = 0
mmap2(NULL, 1460600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7584000
mmap2(0xb76e3000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15e) = 0xb76e3000
mmap2(0xb76e6000, 10616, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb76e6000
close(3) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7583000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb75838d0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0xb76e3000, 8192, PROT_READ) = 0
mprotect(0x8054000, 4096, PROT_READ) = 0
mprotect(0xb771a000, 4096, PROT_READ) = 0
munmap(0xb76e9000, 72063) = 0
brk(0) = 0x91ce000
brk(0x91ef000) = 0x91ef000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=1534672, ...}) = 0
mmap2(NULL, 1534672, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb740c000
close(3) = 0
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 3), ...}) = 0
open("/dev/ttyS0", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFCHR|S_ISVTX|0660, st_rdev=makedev(4, 64), ...}) = 0
fadvise64_64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
read(3, "", 32768) = 0
close(3) = 0
close(1) = 0
close(2) = 0
exit_group(0) = ?
And here is the output of stty -a -F /dev/ttyS0:
speed 57600 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;
werase = ^W; lnext = ^V; flush = ^O; min = 0; time = 0;
-parenb -parodd cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany -imaxbel -iutf8
-opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon -iexten -echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke
Under POSIX, terminal devices (that is, serial ports and pseudoterminals) have a whole bunch of settings which enable the computer to speak the multitude of variations on the basic RS-232 protocol that exist or have existed. A great deal of this API was designed back in the days when dinosaurs and teletypewriters (hence "tty") ruled the earth, and we wouldn't do it over again the same way, but we're stuck with it now.
The terminal settings are persistent; once one program sets them, they stay that way until another program changes them. The command-line utility stty can print or change these settings; stty sane resets them all to "reasonable" defaults; stty -a prints them all out.
Here are all the terminal settings that differ between what stty sane applies on my computer, and what QSerialPort did to your serial port. (The ones that are just a cryptic label, possibly with a dash in front, are boolean flags; the leading dash means "off", no leading dash means "on".)
QSerialPort stty sane
---------------- ----------------
speed 57600 baud speed 38400 baud
min = 0 min = 1
clocal -clocal
-brkint brkint
ignpar -ignpar
-icrnl icrnl
-ixon ixon
-imaxbel imaxbel
-opost opost
-isig isig
-icanon icanon
-iexten iexten
-echo echo
Many of the QSerialPort settings are abnormal in the sense that a line- or file-oriented program hooked up to a serial port in this state will misbehave. (However, they are perfectly appropriate for a program that knows it is talking to a serial port and is prepared to deal with the consequences of having turned these particular knobs; presumably the authors of QSerialPort knew what they were doing.) The one that is causing cat to quit immediately is min = 0, which (together with the default time = 0) means "read() should return zero bytes if there is no input pending." Under normal circumstances, zero bytes returned from read() means end of file, so cat quits immediately because it thinks it's been handed an empty file. (This mode may well have been invented years before O_NONBLOCK.)
stty sane is the "way to recover the serial port" that you were looking for. The documentation does not say either way, but if QSerialPort::close() called from your SIGINT handler does not restore the terminal to its original state, I would consider that a bug in Qt. You should also do this upon receipt of SIGHUP, SIGQUIT, SIGABRT, SIGTERM, and arguably SIGTSTP, SIGTTIN, SIGTTOU as well (but that's more complicated, because those aren't fatal). Make sure to restore the default handler and re-raise the signal afterward, so that the exit status is correct.
I've implemented CCNET protocol to communicate with CashCode bill acceptor from my software on Linux.
Initially I spent lots of time trying to figure out why the device does not respond to the commands I'm sending. Using trial-and-error method I found the solution to set the following options for the serial port:
stty -F /dev/ttyS0 9600 cs8 -cstopb -parenb clocal -crtscts -ixon -ixoff ignpar -icrnl -opost -isig -icanon -iexten -echo
It was working on the development machine and on two another testing machines (all of them had different motherboards). However, on the third testing machine (having yet another mobo) it seems that the device is not responding again.
It is, however, working on Windows with different software on the same machine.
Under "does not respond" I mean that nothing can be read from the serial port during 10-second timeout after sending the command. The whole code is tested and working on another motherboards.
The port itself is detected by the kernel correctly as stated in dmesg:
[ 1.099382] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 1.127531] 00:0b: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
There are no warnings in dmesg regarding this serial port.
Any ideas on how to debug this issue?
The full output of "stty -F /dev/ttyS0":
speed 9600 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V;
flush = ^O; min = 1; time = 0;
-parenb -parodd cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany -imaxbel -iutf8
-opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon -iexten -echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke
Problem is solved.
This model of bill acceptor is working at 19200 baud.
Switching the port to 19200 solves the problem.
How does the vncpasswd program produce output on the console even though both standard out and standard error have been redirected to /dev/null?
$ vncpasswd > /dev/null 2> /dev/null
Password:
Edit: Here's a partial strace:
open("/dev/tty", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(3, SNDCTL_TMR_CONTINUE or TCSETSF, {B38400 opost -isig icanon -echo ...}) = 0
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost -isig icanon -echo ...}) = 0
fstat(3, {st_mode=S_IFCHR|0666, st_rdev=makedev(5, 0), ...}) = 0
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost -isig icanon -echo ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb7c3eda000
write(3, "Password: ", 10Password: ) = 10
read(3,
It's opening the underlying terminal directly (something like open("/dev/tty", ...)). No amount of redirecting will get rid of that. If you don't want to see it, you'll have to run it not attached to a tty (e.g. through cron or something).
It opens your terminal and writes there:
open("/dev/tty", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3
...
write(3, "Password: ", 10Password: ) = 10
Run it under setsid if you want to make that impossible.
You're just redirecting stderr to /dev/null try vncpasswd &> /dev/null instead.