How to set output buffer width/columns when outputting to file? - linux

I want to execute yum list installed > file.txt and output results to a txt file which is NOT wrapped by X characters. At least, I want to be able to control "width" of this output buffer.
I know that stty columns 250 will set column width of my console window to 250 characters but how do I accomplish this when I redirect output to a file?
This has certainly been asked before but I just could not find an answer...
Edit:
This seems to be a yum thing since ps aux > ps.txt works just fine. With yum, file is limited to only 80 characters so I'm adding yum tag. I have no idea how can yum give different output on screen and on the file while other programs work just fine (also note that I'm a beginner in bash).

I think you have two options.
Edit the yum source (http://yum.baseurl.org/download/3.4/yum-3.4.3.tar.gz). The 80 is hard coded in output.py, line 53.
It's probably doable to make yum believe it is writing to a terminal. Whether -- and if, how -- it is possible to set the number of columns for that fake terminal I am not sure of. One thing that pops up is Linux' unbuffer (cf. Piping data to Linux program which expects a TTY (terminal)). Perhaps a little self written unbuffer-like C wrapper may use a pty and have more control over it; even bash may have an esoteric feature for that.

I stumbled across a hackish way to do this using screen:
screen -L yum --color=never list installed
This will result in a file called screenlog.0 containing the output, and yum will think it is outputting to a TTY with the same width as your current screen.

Related

Cli colors disappear when piping into a text file [duplicate]

This question already has answers here:
How to trick an application into thinking its stdout is a terminal, not a pipe
(9 answers)
Closed 5 years ago.
Various bash commands I use -- fancy diffs, build scripts, etc, produce lots of color output.
When I redirect this output to a file, and then cat or less the file later, the colorization is gone -- presumably b/c the act of redirecting the output stripped out the color codes that tell the terminal to change colors.
Is there a way to capture colorized output, including the colorization?
One way to capture colorized output is with the script command. Running script will start a bash session where all of the raw output is captured to a file (named typescript by default).
Redirecting doesn't strip colors, but many commands will detect when they are sending output to a terminal, and will not produce colors by default if not. For example, on Linux ls --color=auto (which is aliased to plain ls in a lot of places) will not produce color codes if outputting to a pipe or file, but ls --color will. Many other tools have similar override flags to get them to save colorized output to a file, but it's all specific to the individual tool.
Even once you have the color codes in a file, to see them you need to use a tool that leaves them intact. less has a -r flag to show file data in "raw" mode; this displays color codes. edit: Slightly newer versions also have a -R flag which is specifically aware of color codes and displays them properly, with better support for things like line wrapping/trimming than raw mode because less can tell which things are control codes and which are actually characters going to the screen.
Inspired by the other answers, I started using script. I had to use -c to get it working though. All other answers, including tee, different script examples did not work for me.
Context:
Ubuntu 16.04
running behavior tests with behave and starting shell command during the test with python's subprocess.check_call()
Solution:
script --flush --quiet --return /tmp/ansible-output.txt --command "my-ansible-command"
Explanation for the switches:
--flush was needed, because otherwise the output is not well live-observable, coming in big chunks
--quiet supresses the own output of the script tool
-c, --command directly provides the command to execute, piping from my command to script did not work for me (no colors)
--return to make script propagate the exit code of my command so I know if my command has failed
I found that using script to preserve colors when piping to less doesn't really work (less is all messed up and on exit, bash is all messed up) because less is interactive. script seems to really mess up input coming from stdin even after exiting.
So instead of running:
script -q /dev/null cargo build | less -R
I redirect /dev/null to it before piping to less:
script -q /dev/null cargo build < /dev/null | less -R
So now script doesn't mess with stdin and gets me exactly what I want. It's the equivalent of command | less but it preserves colors while also continuing to read new content appended to the file (other methods I tried wouldn't do that).
some programs remove colorization when they realize the output is not a TTY (i.e. when you redirect them into another program). You can tell some of those to use color forcefully, and tell the pager to turn on colorization, for example use less -R
This question over on superuser helped me when my other answer (involving tee) didn't work. It involves using unbuffer to make the command think it's running from a shell.
I installed it using sudo apt install expect tcl rather than sudo apt-get install expect-dev.
I needed to use this method when redirecting the output of apt, ironically.
I use tee: pipe the command's output to teefilename and it'll keep the colour. And if you don't want to see the output on the screen (which is what tee is for: showing and redirecting output at the same time) then just send the output of tee to /dev/null:
command| teefilename> /dev/null

Why when I cat the node directory in `/usr/local/bin` does it unleash cthulhu?

So I was installing node and somehow I ended up in the /user/local/bin folder checking what was there.
I wanted to so a ls | grep node but did a ls | cat node. The node directory then started to show itself in all its glory. I chuckled, laughed, then stopped the cat but that left my terminal looking something like this:
It is easily fixable by exiting the window.
I don't recommend you try it but i've managed to recreate it reliably. Just cat the node directory leave it for a few seconds then cancel it.
My question is what on earth is going on here? AFAIK using cat shouldn't be able to make this stuff happen.
Terminals interpret certain combinations of bytes as commands. This is how command line programs can output colors, or change the cursor position. What you're seeing is the output after one of these commands activated a box drawing character set by accident. You can activate it yourself with echo or printf:
my cool prompt$ printf '\033(0'
└≤ ␌⎺⎺┌ ⎻⎼⎺└⎻├$
To get out of this mess, type reset and press Enter - it will clear the screen and set all settings back to default.
└≤ ␌⎺⎺┌ ⎻⎼⎺└⎻├$ ⎼␊⎽␊├
(..screen clears..)
my cool prompt$
You managed to output the contents of the node executable to the terminal. What happens when you output random octets to the terminal depends on what those characters are and on the terminal. It often leaves the terminal in a state that the user did not really expect.

execute last output line of previous command in linux

How to execute last output line of previous command in linux bash terminal by shortcut?
For example:
$ tree
The 'tree' is currently not installed. You can install it by running:
sudo apt-get install tree
I want to copy for execution sudo apt-get install tree
If the last line is safe for running, a simple command, and contains no spaces or quotes or anything else the shell would need to parse/understand then this should work (though you need to re-run the first command for this).
$(!! 2>&1 | tail -n 1)
That being said I don't recommend this at all. It is much simpler (and safer and supports more complex commands) if you copy and paste or type it out again.
You cannot get the last line of displayed output though so you do need to re-run the command.
Your terminal might support shortcuts to let you copy and paste text though. GNU screen can do this. I believe rxvt-unicode might support this with perl scripts.

How to follow the progress of a linux command?

I am currently working with a large data set where even the file format conversion takes at least an hour per subject and as a result I am often unsure whether my command has been executed or the program has frozen. I was wondering whether anyone has a tip to how to follow the progress of the commands/scripts I am trying to run in linux?
Your help will be much appreciated.
In addition to #basile-starynkevitch answer,
I have a bash script that can measure how much file did you processed in percents.
It watch into procfs get current position from fd information (/proc/pid/fdinfo), and count this in percents, relative to total file size.
See https://gist.github.com/azat/2830255
curl -s https://gist.github.com/azat/2830255/raw >| progress_fds.sh \
&& chmod +x progress_fds.sh
Usage:
./progress_fds.sh /path/to/file [ PID]
Сan be useful to someone
If the long-lasting command produces some output in a file foo.out, you could do watch ls -l foo.out or tail -f foo.out
You could also list /proc/$(pidof prog)/fd to find out the opened files of some prog
You can follow the syscalls of a program by using strace, which will enable you to follow the open calls.
You can use verbose output, but it will slow things down even more.
I guess there can't be a general answer to that, it just depends on the type of program (that doesn't even has to do anything with Linux, see the "halting problem").
If you happen to use a pipe during the conversion I find the pv(1) tool pretty helpful. Even if pv can't know the total size of the data it helps to see if there is actual progress and how good the datarate is. It isn't part of most standard installations though and probably has to be installed explicitly.

Limit output of all Linux commands

I'm looking for a way to limit the amount of output produced by all command line programs in Linux, and preferably tell me when it is limited.
I'm working over a server which has a lag on the display. Occasionally I will accidentally run a command which outputs a large amount of text to the terminal, such as cat on a large file or ls on a directory with many files. I then have to wait a while for all the output to be printed to the terminal.
So is there a way to automatically pipe all output into a command like head or wc to prevent too much output having to be printed to terminal?
I don't know about the general case, but for each well-known command (cat, ls, find?)
you could do the following:
hardlink a copy to the existing utility
write a tiny bash function that calls the utility and pipes to head (or wc, or whatever)
alias the name of the utility to call your function.
So along these lines (utterly untested):
$ ln `which cat` ~/bin/old_cat
function trunc_cat () {
`old_cat $# | head -n 100`
}
alias cat=trunc_cat
Making aliases of all your commands would be a good start. Something like
alias lm="ls -al | more"
alias cam="cat $# | more"
Perhaps using screen could help?
this makes me think of bash-completion.
As complete command in bash enables you to specify handler when a program is not found,
what about write your own handler and clear $PATH, in order to execute every command with redirection to a filtering pipe?
#Did not try it myself.
Assuming you're working over a network connection, like ssh, into a remote server then try piping the output of the command to less. That way you can manage and navigate the output from the program on the server better. Use 'j' and 'k' to move up and down per line and 'ctrl-u' and 'ctrl-d' to move 1/2 a page up and down. When you do this only the relevant text (i.e. what fits on the screen) will be transmitted over the network.

Resources