Some command pipelines that I expect to write to the terminal or to a file don't write anything until a lot of time has passed.
It looks like the data is not being line-buffered but if I use Ctrl-C to send SIGINT none of the expected output appears. What causes that?
I want to modify and collect each line of vmstat output as it comes without losing data when I finally decide to stop collecting.
Here are some closely related examples:
output to terminal: vmstat -n 1 | sed ''
output to terminal: echo foo | sed '' | tee /dev/null
output to file: vmstat -n 1 > somefile
output to file: echo foo | sed '' > somefile
no output to terminal: vmstat -n 1 | sed '' | tee /dev/null
no output to file: vmstat -n 1 | sed '' > somefile
You can try stdbuf with option --output=L or --output=0 to make the output line buffered or unbuffered respectively
stdbuf --output=L vmstat -n 1 >somefile
Related
I need to run a script and output the full contents to the terminal.
I then want to get the last 10 lines from the terminal output and put them in a log file.
I have tried using ./script.sh 2>&1 | tail -10 > log.log
but this stops the output to the terminal.
Leverage process substitution of bash with tee:
./script.sh |& tee >(tail -10 >file.txt)
|& is a shortcut for sending both STDOUT and STDERR over the pipe.
tee redirects it's STDIN to both STDOUT and to the file(s) given as argument(s) -- we have used process substitution to get a file descriptor and used tail -10 >file.txt inside process substitution to save the desired content.
For that you have to use the tee command. Then you can pipe to a file and your console.
ls -a | tee output.file
main 2>&1 is giving INFO, ERROR and Running status lines.
But when I use sed on it, it is giving INFO and Running status lines along with ERROR lines but they are not in the order where they have to.
Jumbled status are at the end after job executed succefully.
main 2>&1 | sed "/ERROR.*[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}-[0-9]\{3\}.*20[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}_[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}.*(stderr)/ s/ERROR/NOTE/" | tee -a $log_file
These are the lines which are missing after I use sed command:
if you append /\(Running\|INFO\ERROR... in sed as below, this can also print Running and info along with ERROR lines
main 2>&1 | sed "/\(Running\|INFO\|ERROR.*[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}.[0-9]\{1\}-[0-9]\{3\}.*20[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}_[0-9]\{2\}-[0-9]\{2\}-[0-9]\{2\}.*(stderr)\)/ s/ERROR/NOTE/" | tee -a $log_file
I want to send a program the most recent lines from a text file using tail as stdin.
First, I echo to the program some input that will be the same every time, then send in tail input from an inputfile which should first be processed through sed. The following is the command line that I expect to work. But when the program runs it only receives the echo input, not the tail input.
(echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex' && cat) | ./program
However, the following works exactly as expected, printing everything out to the terminal:
echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex' && cat
So I tried with another type of output, and again while the echoed text posted, the tail text does not appear anywhere:
(echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex') | tee out.txt
This made me think it is a problem with buffering, but I tried the unbuffer program and all other advice here (https://superuser.com/questions/59497/writing-tail-f-output-to-another-file) without results. Where is the tail output going and how can I get it to go into my program as expected?
The buffering problem was resolved when I prefixed the sed command with the following:
stdbuf -i0 -o0 -e0
Much more preferable to using unbuffer, which didn't even work for me. Dave M's suggestion of using sed's relatively new -u also seems to do the trick.
One thing you may be getting confused by -- | (pipeline) is higher precedence than && (consecutive execution). So when you say
(echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex' && cat) | ./program
that is equivalent to
(echo "new" && (tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex') && cat) | ./program
So the cat isn't really doing anything, and the sed output is probably buffered a bit. You can try using the -u option to sed to get it to use unbuffered output:
(echo "new" && (tail -f ~/inputfile 2> /dev/null | sed -n -u -r 'some regex')) | ./program
I believe some versions of sed default to -u when the output is a terminal and not when it is a pipe, so that may be the source of the difference you're seeing.
You can use the i command in sed (see the command list in the manpage for details) to do the inserting at the beginning:
tail -f inputfile | sed -e '1inew file' -e 's/this/that/' | ./program
I am trying to capture the output of "sort -c" in linux. I tried redirecting it to a file, used tee command but both did not helped. Any suggestions ?
For example:
roor>cat db | sort -c
sort: -:319: disorder: 1842251880: aa bb bc dd ee
Following failed to give me output
roor>cat db | sort -c > fileName
roor>cat db | sort -c |tee fileName
Sample file:
>cat file
111 aa as sdasd
222 sadf dzfasf af
333 sada gvsdgf h hgfhfghfg
444 asdfafasfa gsdgsdg sgsg
222 asdasd fasdfaf asdasdasd
root>cat file |sort -c
sort: -:5: disorder: 222 asdasd fasdfaf asdasdasd
8>sort -c db 2> fileName
sort: extra operand `2' not allowed with -c
0>sort -c < file 2> result1.txt
sort: open failed: 2: No such file or directory
ANY ALTERNATE TO SORT -C would ALSO WORK FOR ME!!
If sort -c is producing an error, it sends that error to "standard error" (stderr), not to "standard output" (stdout).
In shell, you need to use special redirects to capture standard error.
sort -c inputfile > /path/to/stdout.txt 2> /path/to/stderr.txt
These two output streams are called "file descriptors", and you can alternately redirect one of them to the other:
sort -c inputfile > /path/to/combined.txt 2>&1
You can read more about how these work at tldp.org, in the Bash reference manual, the bash-hackers wiki and the Bash FAQ. Happy reading! :-D
other good alternative for '2>' is STDERR pipe
|&
cat db | sort -c -h |& tee >fileName
Some time it is very suitable when present STDIN, for example:
TIMEFORMAT=%R;for i in `seq 1 20` ; do time kubectl get pods -l app=pod >/dev/null ; done |& sort -u -h
or
TIMEFORMAT=%R;for i in `seq 1 20` ; do time kubectl get pods >>log1 ; done |& sort -u -h
sort -c has no output as you might expect:
[root#home:~] cat /etc/services | sort -c
sort: -:2: disorder: #
As described by the manpage, the -c argument simply checks whether a given file or input is sorted or not.
If you're trying to catch the message from the command, try redirecting the error stream (2), not the standard output (1):
cat file | sort -c 2>result.txt
sort -conly checks if the input is sorted. It does not performs any sorting.
See: man sort
Remove -c to sort the lines.
PS: It gives the "disorder" error because the file of yours isn't already sorted. On line 5, "222" appears after "444" on the previous line.
EDIT: I think I misunderstood.
To redirect the error to a file you must use 2>.
So, the command would become: roor>cat db | sort -c 2> fileName
EDIT2: You can simply use: sort -c db 2> fileName
I have a shell script that greps some data.. I want to print the result into a file, but doing that prevents the result being displayed on the terminal. Is there a way that can both print the result on the screen and also write into a file.
Thanks in advance.
Pipe your output to the tee command.
Example:
[me#home]$ echo hello | tee out.txt
hello
[me#home]$ cat out.txt
hello
Note that the stdout of echo is printed out as well as written to the file specified by thr tee command.
Note you can add the -a flag to tee to append to the output file
[me#home]$ echo hello | tee out.txt
hello
[me#home]$ echo hello again | tee -a out.txt
hello again
[me#home]$ cat out.txt
hello
hello again
Does exactly your thing
http://linux.die.net/man/1/tee