I'm using screen to read the text from a serial console. The problem is the output seems to only have newline \n but not carriage return \r, so the display looks like this...
Line1
Line2
Line3
I wonder if there is any patch to fix this issue?
onlcr is for translating outgoing newlines to carriage returns.
stty -F /dev/ttyS0 inlcr will translate incoming newlines to carriage returns. You can run that from another terminal after starting screen to avoid any resetting that screen may do on startup. Unfortunately however, this will only change the problem. You'll then get only returns and no newlines.
What is needed is an option to append a return to an incoming newline so that the terminal receives \n\r, which is what the serial device should have output in the first place. There seems to be an onlret option to do this for outgoing data, but no inlret option as we would seem to need in this case.
I have the exact same problem (using picocom though) and I've been googling off and on for days trying to find the standard fix, but no one seems to have one. There are a number of serial devices out there which only output \n and simply can't be made to output \r\n and I refuse to believe that all of them belong to only two linux users. What gives!?
If you use the miniterm.py program that comes with pyserial it will interpret newlines as crlf. It is not the most fully-featured terminal emulator but for interacting with simple serial devices it gets the job done.
Usage syntax (on OSX):
miniterm.py /dev/tty.usbserial-XXXXXX 115200
Replace XXXXXX with whatever the device comes up on your system as.
Try stty onlcr.
The man page says it will translate newlines to carriage return / newline pairs on output, which seems to be what you need.
In my case worked: stty -F /dev/ttyACM0 -icrnl
Because the serial was implicitly set to translate CR to NL. This command set it back. Notice the minus character preceding icrnl.
Related
In GCP compute Linux Accidentally did cat filebeat instead of filebeat.yaml
after that my bashrc contains below chars and if I type '~' bash is printing 'ü'
Need help in fixing this
if Ä -f ü/.bash_aliases Å; then
. ü/.bash_aliases
fi
This looks like your terminal was accidentally configured for legacy ISO-646-SE or a variant. Your file is probably fine; it's just that your terminal remaps the display characters according to a scheme from the 1980s.
A quick hex dump should verify that the characters in the file are actually correct. Here's an example of what you should see.
bash$ echo '[\]' | xxd
00000000: 5b5c 5d0a [\].
Even if the characters are displayed as ÄÖÅ, they are correct if you see the hex codes 5B, 5C, and 5D. (If you don't have xxd, try hexdump or od -t x1.)
Probably
bash$ tput reset
can set your terminal back to sane settings. Maybe stty sane might work too (but less likely, in my experience). Else, try logging out and back in.
Back when ASCII was the only game in town, but American (or really any) hardware was exported to places where the character repertoire was insufficient, the local vendor would replace the ROM chips in terminals to remap some slightly less common character codes to be displayed as the missing local glyphs. Over time, this became standardized; the ISO-646 standard was updated to document these local overrides. (The linked Wikipedia page has a number of tables with details.)
Eventually, 8-bit character sets became the norm, and then most locales switched to Latin-1 or some other suitable character set which no longer needed this hack. However, it was still rather prevalent even in the early 1990s. In the early 2000s, Unicode started taking over, and so now this seems like an absurd arrangement.
I'm guessing the file you happened to cat contained some control characters which instructed your terminal to switch to this legacy character set. It's not entirely uncommon (though usually when it happens to me, it switches to some "graphical" character set where some characters display box-drawing characters or mathematical symbols).
This question already has answers here:
What does the ^M character mean in Vim?
(15 answers)
Closed 2 years ago.
I'm new to Linux sorry if my question sounds dumb.
We know that Linux and Mac OS X use \n (0xa), which is the ASCII line feed (LF) character. MS Windows and Internet protocols such as HTTP use the sequence \r\n (0xd 0xa). If you create a file foo.txt in Windows and then view it in a Linux text editor, you’ll see an annoying ^M at the end of each line, which is how Linux tools display the CR character.
Bu why Linux tools display the CR character as ^M? as my understanding is, \r (carriage return) is to move the cursor in the beginning of the current line, so the sensible approach to display it is like, when you open the file, you see the cursor is in the beginning of the line(that contains \r), so ^M shouldn't be displayed?
PS: some people post answers that how to remove ^M, but I wnat to know why eventually^M is displayed rather than moving the cursor in the beginning, which is the definition of carriage return.
The ASCII control characters like TAB, CR, NL and others are intended to control the printing position of a teletypewriter-like display device.
A text editor isn't such a device. It is not appropriate for a text editor to treat a CR character literally as meaning "go to the first column"; it would make a confusing gibberish out of the editing experience.
A text editor works by parsing a text file's representation, to create an internal representation which is presented to the user. On Unix-like operating systems, a file is represented by zero or more lines, which are terminated by the ASCII NL character. Any CR characters that occur just look like part of the data, and not part of the line separation.
Not all editors behave the same way. For instance, the Vim editor will detect that a file uses CR-LF line endings, and load it properly using that representation. A flag is set for that buffer which indicates that it's a "DOS" file, so that when you save it, the same representation is reproduced.
That said, there is a feature in the actual Linux kernel for representing control characters like CR using the ^M notation. The TTY line discipline for any given TTY device can be configured to print characters in this notation, but only when echoing back the characters received.
Demo:
$ stty echoctl # turn on notational echo of control characters
$ cat # run some non-interactive program with rudimentary line input
^F^F^F^F^F^F
^C
$
Above, the Ctrl-F that I entered was echoed back as ^F. So, in fact there is a "Linux editor" which uses this notation: the rudimentary line editor of the "canonical input mode" line discipline.
I have to run a third-party program in background and capture its output to file. I'm doing this simply using the_program > output.txt. However, the coders of said program decided to be flashy and show processed lines in real-time, using \b characters to erase the previous value. So, one of the lines in output.txt ends up like Lines: 1(b)2(b)3(b)4(b)5, (b) being an unprintable character with ASCII code 08. I want that line to end up as Lines: 5.
I'm aware that I can write it as-is and post-process the file using AWK, but I wonder if it's possible to somehow process the control characters in-place, by using some kind of shell option or by piping some commands together, so that line would become Lines: 5 without having to run any additional commands after the program is done?
Edit:
Just a clarification: what I wrote here is a simplified version, actual line count processed by the program is a hundred thousands, so that string ends up quite long.
Thanks for your comments! I ended up piping the output of that program to AWK Script I linked in the question. I get a well-formed file in the end.
the_program | ./awk_crush.sh > output.txt
The only downside is that I get the output only once the program itself is finished, even though the initial output exceeds 5M and should be passed in the lesser chunks. I don't know the exact reason, perhaps AWK script waits for EOF on stdin. Either way, on more modern system I would use
stdbuf -oL the_program | ./awk_crush.sh > output.txt
to process the output line-by-line. I'm stuck on RHEL4 with expired support though, so I'm unable to use neither stdbuf nor unbuffer. I'll leave it as-is, it's fine too.
The contents of awk_crush.sh are based on this answer, except with ^H sequences (which are supposed to be ASCII 08 characters entered via VIM commands) replaced with escape sequence \b:
#!/usr/bin/awk -f
function crushify(data) {
while (data ~ /[^\b]\b/) {
gsub(/[^\b]\b/, "", data)
}
print data
}
crushify($0)
Basically, it replaces character before \b and \b itself with empty string, and repeats it while there are \b in the string - just what I needed. It doesn't care for other escape sequences though, but if it's necessary, there's a more complete SED solution by Thomas Dickey.
Pipe it to col -b, from util-linux:
the_program | col -b
Or, if the input is a file, not a program:
col -b < input > output
Mentioned in Unix & Linux: Evaluate large file with ^H and ^M characters.
How do some programs edit whats being displayed on the terminal (to pick a random example, the program 'sl')? I'm thinking of the Linux terminal here, it may happen in other OS's too, I don't know. I've always thought once some text was displayed, it stayed there. How do you change it without redrawing the entire screen?
Depending on the terminal you send control seuqences. Common sequences are for example esc[;H to send the cursor to a specific position (e.g. on Ansi, Xterm, Linux, VT100). However, this will vary with the type or terminal the user has ... curses (in conjunction with the terminfo files) will wrap that information for you.
Many applications make use of the curses library, or some language binding to it.
For rewriting on a single line, such as updating progress information, the special character "carriage return", often specified by the escape sequence "\r", can return the cursor to the start of the current line allowing subsequent output to overwrite what was previously written there.
try this shellscript
#!/bin/bash
i=1
while [ true ]
do
echo -e -n "\r $i"
i=$((i+1))
done
the -n options prevents the newline ... and the \r does the carriage return ... you write again and again into the same line - no scroling or what so ever
If you terminate a line sent to the terminal with a carriage return ('\r') instead of a linefeed ('\n'), it will move the cursor to the beginning of the current line, allowing the program to print more text over top of what it printed before. I use this occasionally for progress messages for long tasks.
If you ever need to do more terminal editing than that, use ncurses or a variant thereof.
There are characters that can be sent to the terminal that move the cursor back. Then text can be overwritten.
There is a list here. Note the "move cursor something" lines.
NCurses is a cross-platform library that lets you draw user interfaces on smart terminals.
Corporal Touchy has answered how this is done at the lowest level. For easier development the curses library gives a higher level of control than simply sending characters to the terminal.
To build on #Corporal Touchy's answer, there are libraries available that will handle some of this functionality for you such as curses/ncurses
I agree with danio, ncurses is the way to go. Here's a good tutorial:
http://tldp.org/HOWTO/NCURSES-Programming-HOWTO/
If I do something like:
$ cat /bin/ls
into my terminal, I understand why I see a bunch of binary data, representing the ls executable. But afterwards, when I get my prompt back, my own keystrokes look crazy. I type "a" and I get a weird diagonal line. I type "b" and I get a degree symbol.
Why does this happen?
Because somewhere in your binary data were some control sequences that your terminal interpreted as requests to, for example, change the character set used to draw. You can restore everything to normal like so:
reset
Just do a copy-paste:
echo -e '\017'
to your bash and characters will return to normal. If you don't run bash, try the following keystrokes:
<Ctrl-V><Ctrl-O><Enter>
and hopefully your terminal's status will return to normal when it complains that it can't find either a <Ctrl-V><Ctrl-O> or a <Ctrl-O> command to run.
<Ctrl-N>, or character 14 —when sent to your terminal— orders to switch to a special graphics mode, where letters and numbers are replaced with symbols. <Ctrl-O>, or character 15, restores things back to normal.
The terminal will try to interpret the binary data thrown at it as control codes, and garble itself up in the process, so you need to sanitize your tty.
Run:
stty sane
And things should be back to normal. Even if the command looks garbled as you type it, the actual characters are being stored correctly, and when you press return the command will be invoked.
You can find more information about the stty command here.
You're getting some control characters piped into the shell that are telling the shell to alter its behavior and print things differently.
VT100 is pretty much the standard command set used for terminal windows, but there are a lot of extensions. Some control character set used, keyboard mapping, etc.
When you send a lot of binary characters to such a terminal, a lot of settings change. Some terminals have options to 'clear' the settings back to default, but in general they simply weren't made for binary data.
VT100 and its successors are what allow Linux to print in color text (such as colored ls listings) in a simple terminal program.
-Adam
If you really must dump binary data to your terminal, you'd have much better luck if you pipe it to a pager like less, which will display it in a slightly more readable format. (You may also be interested in strings and od, both can be useful if you're fiddling around with binary files.)