I have a question from the book "unix/linux your ultimate guide" it asks
Suppose there is a program named "prog" that outputs on both stderr and stdout. Give a single command to run "prog" with the 'o' option and the string 'arg' passed as its only argument, where it takes its stdin from the output of the program "progBefore", where "prog"s stdout is ignored, and "prog"s stderr is given to the program "progAfter" through "progAfter"s stdin. Do not use any temporary files.
Here is what i tried:
prog -o 'arg' < `progBefore` 1>/dev/null 2> progAfter
Any help would be appreciated thank you
What is this doing?
prog -o 'arg' < progBefore 1>/dev/null 2> progAfter
It is calling the program prog, taking input from the file progBefore, passing stdout to /dev/null (which ignores it) and passing stderr to the file progAfter. You are using file redirection when you should be using pipes:
progBefore | prog -o 'arg' 2>&1 1>/dev/null | progAfter
A pipe (more correctly, an anonymous pipe) indicated by | takes the stdout from the program on the left and sends it to the stdin of the program on the right.
2>&1 redirects stderr to whatever stdout is pointing at, note that the order is important.
I'm trying to write a perl one liner for listing the content of a directory like with the "ls -l" Unix command however I only want to output directories and not files
I have managed to do it using a perl script but I want to try and see if it can be reduced to a single line using a pipe like
ls -l | grep "something"
or
ls -l | perl -...
the way I have done it so far is this
#!/usr/bin/perl
open(LS_D, "ls -l |");
while(<LS_D>) {
print if /^d.*/; #print line if line starts with d
}
Also could you tell me why this works with the pipe in "ls -l |" but not with "ls -l"?
Thanks
The long listing of directories can be directly obtained with
ls -ld */
without a need to filter the output.
As for the |, that is how you tell open to open a process and replace its STDIN or STDOUT
For three or more arguments if MODE is |-, the filename is interpreted as a command to which output is to be piped, and if MODE is -|, the filename is interpreted as a command that pipes output to us. In the two-argument (and one-argument) form, one should replace dash (-) with the command. See Using open() for IPC in perlipc for more examples of this.
Without |, so with open(LS_D, "ls -l"), you are trying to open a file, with the name 'ls -l'
The three-argument open, which is recommended with its lexical filehandle, would be
open my $read_fh, '-|', 'ls', '-l';
where the command with arguments is supplied as a list.
Finally, what you have in your Perl script can be done with a one-liner for example as
perl -we'print grep { /^d/ } `ls -l`'
but, as shown above, you can get that directly with ls on the command line.
I should add
Always have use warnings; and use strict; at the top of your programs
Always check calls like open and you'll know about, and see the reason for, the failure
open my $fh ... or die "Can't open ... : $!";
See Error Variables in perlvar for $!
I have a program running on linux
This program take its input from the stdin
So I can launch it with input file in this way
myprogram < file
in order to avoid typing input to the program
Now I want that the program take the input from a command output. something like that
myprogram < anycommand
but this does not work because it's expecting a file and not a command.
How I can make it work? Are there a shell syntax to make it work?
Note: I can not use pipe like anycommand | myprogram
normally (IMHO) myprogram does not know anything about file. The bash starts myprogram and reads the file, and writes the content of file to the stdin of myprogram.
So myprogram should not know that his stdin is a file.
So, anycommand | myprogram must work.
If it doesn't work with ash, maybe you can make a named pipe (mkfifo /tmp/testpipe)
Now you can start your program with myprogram < /tmp/testpipe and you can write your input to /tmp/testpipe
On my Linux system, ash is a symbolic link to dash and that handles pipes just fine:
pax> ls -ld $(which ash)
lrwxrwxrwx 1 root root 4 Mar 1 2012 /bin/ash -> dash
pax> ash
$ echo hello | tr '[a-z]' '[A-Z]'
HELLO
So I'd give the anycommand | myprogram another shot just in case.
If your ash has no piping capability, you can always revert to using temporary files, provided anycommand isn't a long-lived process that you need to handle the output of in an incremental fashion:
anycommand >/tmp/tempfile
myprogram </tmp/tempfile
You need to use it like this:
myprogram < <(anycommand)
This is called process substitution
Is it possible to redirect the output of a command to one file, but still keep the output echoing on the terminal?
I want the output to go to a file, but I still want to see it on the terminal (just so I know something is going on)
Yes, tee does what you want:
command | tee output.log
You can use "tee" i.e.
ls -ltr | tee -a mylog.log
In bash, calling foo would display any output from that command on the stdout.
Calling foo > output would redirect any output from that command to the file specified (in this case 'output').
Is there a way to redirect output to a file and have it display on stdout?
The command you want is named tee:
foo | tee output.file
For example, if you only care about stdout:
ls -a | tee output.file
If you want to include stderr, do:
program [arguments...] 2>&1 | tee outfile
2>&1 redirects channel 2 (stderr/standard error) into channel 1 (stdout/standard output), such that both is written as stdout. It is also directed to the given output file as of the tee command.
Furthermore, if you want to append to the log file, use tee -a as:
program [arguments...] 2>&1 | tee -a outfile
$ program [arguments...] 2>&1 | tee outfile
2>&1 dumps the stderr and stdout streams.
tee outfile takes the stream it gets and writes it to the screen and to the file "outfile".
This is probably what most people are looking for. The likely situation is some program or script is working hard for a long time and producing a lot of output. The user wants to check it periodically for progress, but also wants the output written to a file.
The problem (especially when mixing stdout and stderr streams) is that there is reliance on the streams being flushed by the program. If, for example, all the writes to stdout are not flushed, but all the writes to stderr are flushed, then they'll end up out of chronological order in the output file and on the screen.
It's also bad if the program only outputs 1 or 2 lines every few minutes to report progress. In such a case, if the output was not flushed by the program, the user wouldn't even see any output on the screen for hours, because none of it would get pushed through the pipe for hours.
Update: The program unbuffer, part of the expect package, will solve the buffering problem. This will cause stdout and stderr to write to the screen and file immediately and keep them in sync when being combined and redirected to tee. E.g.:
$ unbuffer program [arguments...] 2>&1 | tee outfile
Another way that works for me is,
<command> |& tee <outputFile>
as shown in gnu bash manual
Example:
ls |& tee files.txt
If ‘|&’ is used, command1’s standard error, in addition to its standard output, is connected to command2’s standard input through the pipe; it is shorthand for 2>&1 |. This implicit redirection of the standard error to the standard output is performed after any redirections specified by the command.
For more information, refer redirection
You can primarily use Zoredache solution, but If you don't want to overwrite the output file you should write tee with -a option as follow :
ls -lR / | tee -a output.file
Something to add ...
The package unbuffer has support issues with some packages under fedora and redhat unix releases.
Setting aside the troubles
Following worked for me
bash myscript.sh 2>&1 | tee output.log
Thank you ScDF & matthew your inputs saved me lot of time..
Using tail -f output should work.
In my case I had the Java process with output logs. The simplest solution to display output logs and redirect them into the file(named logfile here) was:
my_java_process_run_script.sh |& tee logfile
Result was Java process running with output logs displaying and
putting them into the file with name logfile
You can do that for your entire script by using something like that at the beginning of your script :
#!/usr/bin/env bash
test x$1 = x$'\x00' && shift || { set -o pipefail ; ( exec 2>&1 ; $0 $'\x00' "$#" ) | tee mylogfile ; exit $? ; }
# do whaetever you want
This redirect both stderr and stdout outputs to the file called mylogfile and let everything goes to stdout at the same time.
It is used some stupid tricks :
use exec without command to setup redirections,
use tee to duplicates outputs,
restart the script with the wanted redirections,
use a special first parameter (a simple NUL character specified by the $'string' special bash notation) to specify that the script is restarted (no equivalent parameter may be used by your original work),
try to preserve the original exit status when restarting the script using the pipefail option.
Ugly but useful for me in certain situations.
Bonus answer since this use-case brought me here:
In the case where you need to do this as some other user
echo "some output" | sudo -u some_user tee /some/path/some_file
Note that the echo will happen as you and the file write will happen as "some_user" what will NOT work is if you were to run the echo as "some_user" and redirect the output with >> "some_file" because the file redirect will happen as you.
Hint: tee also supports append with the -a flag, if you need to replace a line in a file as another user you could execute sed as the desired user.
< command > |& tee filename # this will create a file "filename" with command status as a content, If a file already exists it will remove existed content and writes the command status.
< command > | tee >> filename # this will append status to the file but it doesn't print the command status on standard_output (screen).
I want to print something by using "echo" on screen and append that echoed data to a file
echo "hi there, Have to print this on screen and append to a file"
tee is perfect for this, but this will also do the job
ls -lr / > output | cat output