Identify the cause of \r artifacts when reading from a serial terminal? - linux

Question

I'm trying to diagnose what is the cause of \r artifacts when reading from a serial terminal.
The following code can provoke the issue. There is an embedded Linux device connected to the physical end of the uart cable
import serial
ser = serial.Serial(
port='/dev/ttyUSB0',
baudrate=115200,
timeout=5)
ser.reset_input_buffer()
ser.reset_output_buffer()
b_NEW_LINE_WRITTEN = b'\n'
b_alphabet = b'A a B b C c D d E e F f G g H h I i J j K k L l M m N n O o P p Q q R r S s T t U u V v W w X x Y y Z z'
ser.write(b_alphabet + b_NEW_LINE_WRITTEN)
raw_line = ser.readline()
print(raw_line)
# b'A a B b C c D d E e F f G g H h I i J j K k L l M m N n O o P p Q q\rq R r S s T t U u V v W w X x Y y Z z\r\n'
The print the read data has an \r around the letter q.
When running a terminal emulator on top of /dev/ttyUSB0 the line starts wrapping around itself after 80 characters, similarly to
this question. This line wrapping seems to be a manifestation of the /r that I get when directly reading with the example python code.
When connected with a terminal emulator (picocom) this problem gets solved by running:
shopt -s checkwinsize
resize
there are no line wraps after that. I'm trying to understand why this is happening or where (is the host where /dev/tttyUSB0 what adds them or are they added by the embedded device being accessed), and how to workaround it without having to run external commands.
The following candidates could be causing the behavior:
driver settings for /dev/ttyUSB0 on the shot ps
driver settings for /dev/S0 on the target embedded device
shell on the target device
Setting /dev/ttyUSB0
Trying to modify /dev/ttyUSB0 in separate terminal while the device is consumed by the python script doesn't show any changes.
# Separate terminal on the host
stty -F /dev/ttyUSB0 cols 100
stty -F /dev/ttyUSB0 raw
stty -F /dev/ttyUSB0 -a
speed 115200 baud; rows 80; columns 100; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^A; eol = <undef>; eol2 = <undef>; swtch = <undef>;
start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; discard = ^O;
min = 1; time = 0;
-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany
-imaxbel -iutf8
-opost -olcuc -ocrnl -onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon -iexten -echo -echoe -echok -echonl -noflsh -xcase -tostop -echoprt -echoctl -echoke
-flusho -extproc
#ipython
...
ser.write(b_alphabet + b_NEW_LINE_WRITTEN)
raw_line = ser.readline()
print(raw_line)
# b'A a B b C c D d E e F f G g H h I i J j K k L l M m N n O o P p Q q\rq R r S s T t U u V v W w X x Y y Z z\r\n'
Setting /dev/S0
Setting the target tty device also doesn't influence the artifacts existence.
stty_cmd_set = b'stty cols 200'
ser.write(stty_cmd_set + b_NEW_LINE_WRITTEN)
ser.reset_input_buffer()
ser.reset_output_buffer()
stty_cmd_confirm = b'stty -a'
ser.write(stty_cmd_confirm + b_NEW_LINE_WRITTEN)
# After reading a few lines there is a confirmation that the tty device on the target has indeed been set to 200
print(ser.readline())
b'speed 115200 baud; rows 56; columns 200; line = 0;\r\n'
ser.reset_input_buffer()
ser.reset_output_buffer()
ser.write(b_alphabet + b_NEW_LINE_WRITTEN)
raw_line = ser.readline()
print(raw_line)
# b'A a B b C c D d E e F f G g H h I i J j K k L l M m N n O o P p Q q\rq R r S s T t U u V v W w X x Y y Z z\r\n'
For example a workaround would be to somehow set a fixed amount of 200 columns before reading so that the serial terminal stops trying to be smart.

Answer 1

Reading between the lines, you are running something like a Linux shell on your embedded device.
This will have the same set of Linux app/TTY/driver hierarchy as your host and is defaulting to cooked mode processing of the input that it receives from your application. This is why running the command to change the number of columns (on your embedded application) works. It is telling the line discipline in that device to treat the screen as being 200 columns wide (and so the line-editing logic doesn't need to split the line).
Switching to raw input on both your host and embedded shell should fix it.
If you'd like to know more details of how Linux terminals handle input (and echoing of input back to the output stream), see https://www.linusakesson.net/programming/tty/

Answer 2

It is not clear what is actually happening but this is solution that seems to work.
It depends on existence of shopt -s checkwinsize and resize on the target board so it isn't a generic enough solution to be the accepted answer.
Also it doesn't provide an insight on how to apply a not runtime fix (by setting the driver defaults or some config in bash).
import serial
import time
ser = serial.Serial(
port='/dev/ttyUSB0',
baudrate=115200,
timeout=5)
b_NEW_LINE_WRITTEN = b'\n'
b_NEW_LINE_READ = b'\r\n'
b_alphabet = b'A a B b C c D d E e F f G g H h I i J j K k L l M m N n O o P p Q q R r S s T t U u V v W w X x Y y Z z'
ser.write(b_alphabet + b_NEW_LINE_WRITTEN)
print(ser.readline())
# b'A a B b C c D d E e F f G g H h I i J j K k L l M m N n O o P p Q q\rq R r S s T t U u V v W w X x Y y Z z\r\n'
shopt_cmd = b'shopt -s checkwinsize'
ser.write(shopt_cmd + b_NEW_LINE_WRITTEN)
ser.readline()
ser.readline()
resize_cmd = b'resize'
ser.write(resize_cmd + b_NEW_LINE_WRITTEN)
ser.readline()
ser.readline()
stty_cmd_set = b'stty cols 200'
ser.write(stty_cmd_set + b_NEW_LINE_WRITTEN)
ser.readline()
ser.readline()
ser.write(b_alphabet + b_NEW_LINE_WRITTEN)
print(ser.readline())
# b'A a B b C c D d E e F f G g H h I i J j K k L l M m N n O o P p Q q R r S s T t U u V v W w X x Y y Z z\r\n'
ser.reset_output_buffer()
stty_cmd_set = b'stty cols 5'
ser.write(stty_cmd_set + b_NEW_LINE_WRITTEN)
ser.readline()
ser.readline()
ser.write(b_alphabet + b_NEW_LINE_WRITTEN)
print(ser.readline())
# A a B \r b C c\rc D d \r E e F\rF f G \r g H h\rh I i \r J j K\rK k L \r l M m\rm N n \r O o P\rP p Q \r q R r\rr S s \r T t U\rU u V \r v W w\rw X x \r Y y Z\rZ z\r\n'

Answer 3

\r q is happening because your screen ends near q so it is moved to next line its carriage return of Unix. unix carriage return can be used as new line ,go to next line end of line both . Replace \r with empty.
Try auto wrap off
\r (Carriage Return) → moves the cursor to the beginning of the line without advancing to the next line
\n (Line Feed) → moves the cursor down to the next line without returning to the beginning of the line — In a *nix environment \n moves to the beginning of the line.
\r\n (End Of Line) → a combination of \r and \n