Hi I was wondering how to read the argument after "|" pipe from shell script.
For example, when I run ./tmp.sh ls -la | sort
I could only get 2 arguments, which is "ls" and "-la".
Is there any way to read "| sort" without modifying the command, and realize only with shell script?
Thanks a lot!!
One way would be to pass the entire command as a string to your script.
./tmp.sh -c "ls -la | sort"
...or without a flag...
./tmp.sh "ls -la | sort"
Afterward, you can split the string into an array in your script.
I guess you could check ps axf, but part of the beauty of pipes is the loose coupling they give, because bash knows what is in your pipeline, not the individual pieces of the pipeline. This makes writing filters simple.
I have a tool (written in C) that takes in output file parameter to which the tool writes some output string.
tool -o output-file-name
I would like to invoke the tool from a shell script and have the output string assigned to a variable.
I tried:
var=$(tool -o a.txt 1>/dev/null;cat a.txt && rm a.txt)
The above works, but I would like a more elegant solution.
P.S: I am far from a scripting guru.
You can start the tool without the option -o
tool
or like William Pursell wrote with /dev/stdout as file
tool -o /dev/stdout
or in short form
tool -o -
I'm assuming not possible to change to code to support output to stdout. Probably, because the code is sending additional information (logging) to stdout.
You can use IO redirection or process substitution to eliminate the need to save/recall the data from the file. The construct '>(cat)' argument tell bash the generate a pipe, between 'tool' and 'cat' (which will simply print the file).
# PREFERRED
# IO Redirection, using extra fd (#3).
var=$(./f.sh -o /dev/fd/3 3>&1 1>/dev/null)
# Use process substitution, use extra 'cat' process.
var=$(tool -o >(cat) >/dev/null)
1st solution more efficient (no extra cat running ...). 2nd solution is slightly more compact.
Since you're on linux, you can use tool -o /dev/stdout
I want to use the function like the command under gdb.
$ cat arg.txt | xargs ./binary
Is there any way to make it?
When gdb is invoked via xargs it's stdin by default is redirected from /dev/null. Obviously gdb needs stdin to read and perform user input but it can't because stdin is /dev/null.
One way to solve this issue is to use xargs with --arg-file:
xargs --arg-file arg.txt gdb --args ./binary
See man xargs:
-a file, --arg-file=file
Read items from file instead of standard input. If you use
this option, stdin remains unchanged when commands are run.
Otherwise, stdin is redirected from /dev/null.
Thanks and I got an simple solution.
(gdb) run $( cat arg.txt )
It is also possible to pass the output of a command to be the arguments.
(gdb) run $( ruby -e 'print( "text as arguments" )' )
I have a bunch of executable files and I want to store the output of each one of them in a separate file.
For that purpose i am using following command but "2>&1" does not work every time. And sometimes the output files remain empty even though the script does print stuff in shell when run from shell.
What should I use instead of 2>&1?
./$file 2>&1 | tee "$outputFile"
Some executables don't just write to stdout and stderr, but instead open /dev/tty and write to that.
So to redirect those it is necessary to do more complicated procedures involving a psuedo-tty. See the command script, for something that can do this.
If you want to capture both stdout and stderr use:
./$file > $outputfile 2>&1
However, some programs are smart and detect to what type of terminal their output is redirected. They might generate different output if you send it to a file...
Just to elaborate on Darron's answer, you can use the script command to capture all of the output. Here's an example that writes to stdout, stderr, and /dev/tty using a python script, and captures all three outputs:
brent#battlecruiser:~$ cat test.py
import sys
sys.stdout.write('o\n')
sys.stderr.write('e\n')
with open('/dev/tty', 'w') as tty:
tty.write('t\n')
brent#battlecruiser:~$ script testout
Script started, file is testout
brent#battlecruiser:~$ python test.py
o
e
t
brent#battlecruiser:~$ exit
Script done, file is testout
brent#battlecruiser:~$ head -n -3 testout | tail -n +3
o
e
t
Contrast this with what happens when you use 2>1& and tee:
brent#battlecruiser:~$ python test.py 2>&1 | tee testout
e
t
o
brent#battlecruiser:~$ cat testout
e
o
As you can see, the write to /dev/tty is not captured. Try executing your command within the script command as shown and see if it captures all the results.
In bash, calling foo would display any output from that command on the stdout.
Calling foo > output would redirect any output from that command to the file specified (in this case 'output').
Is there a way to redirect output to a file and have it display on stdout?
The command you want is named tee:
foo | tee output.file
For example, if you only care about stdout:
ls -a | tee output.file
If you want to include stderr, do:
program [arguments...] 2>&1 | tee outfile
2>&1 redirects channel 2 (stderr/standard error) into channel 1 (stdout/standard output), such that both is written as stdout. It is also directed to the given output file as of the tee command.
Furthermore, if you want to append to the log file, use tee -a as:
program [arguments...] 2>&1 | tee -a outfile
$ program [arguments...] 2>&1 | tee outfile
2>&1 dumps the stderr and stdout streams.
tee outfile takes the stream it gets and writes it to the screen and to the file "outfile".
This is probably what most people are looking for. The likely situation is some program or script is working hard for a long time and producing a lot of output. The user wants to check it periodically for progress, but also wants the output written to a file.
The problem (especially when mixing stdout and stderr streams) is that there is reliance on the streams being flushed by the program. If, for example, all the writes to stdout are not flushed, but all the writes to stderr are flushed, then they'll end up out of chronological order in the output file and on the screen.
It's also bad if the program only outputs 1 or 2 lines every few minutes to report progress. In such a case, if the output was not flushed by the program, the user wouldn't even see any output on the screen for hours, because none of it would get pushed through the pipe for hours.
Update: The program unbuffer, part of the expect package, will solve the buffering problem. This will cause stdout and stderr to write to the screen and file immediately and keep them in sync when being combined and redirected to tee. E.g.:
$ unbuffer program [arguments...] 2>&1 | tee outfile
Another way that works for me is,
<command> |& tee <outputFile>
as shown in gnu bash manual
Example:
ls |& tee files.txt
If ‘|&’ is used, command1’s standard error, in addition to its standard output, is connected to command2’s standard input through the pipe; it is shorthand for 2>&1 |. This implicit redirection of the standard error to the standard output is performed after any redirections specified by the command.
For more information, refer redirection
You can primarily use Zoredache solution, but If you don't want to overwrite the output file you should write tee with -a option as follow :
ls -lR / | tee -a output.file
Something to add ...
The package unbuffer has support issues with some packages under fedora and redhat unix releases.
Setting aside the troubles
Following worked for me
bash myscript.sh 2>&1 | tee output.log
Thank you ScDF & matthew your inputs saved me lot of time..
Using tail -f output should work.
In my case I had the Java process with output logs. The simplest solution to display output logs and redirect them into the file(named logfile here) was:
my_java_process_run_script.sh |& tee logfile
Result was Java process running with output logs displaying and
putting them into the file with name logfile
You can do that for your entire script by using something like that at the beginning of your script :
#!/usr/bin/env bash
test x$1 = x$'\x00' && shift || { set -o pipefail ; ( exec 2>&1 ; $0 $'\x00' "$#" ) | tee mylogfile ; exit $? ; }
# do whaetever you want
This redirect both stderr and stdout outputs to the file called mylogfile and let everything goes to stdout at the same time.
It is used some stupid tricks :
use exec without command to setup redirections,
use tee to duplicates outputs,
restart the script with the wanted redirections,
use a special first parameter (a simple NUL character specified by the $'string' special bash notation) to specify that the script is restarted (no equivalent parameter may be used by your original work),
try to preserve the original exit status when restarting the script using the pipefail option.
Ugly but useful for me in certain situations.
Bonus answer since this use-case brought me here:
In the case where you need to do this as some other user
echo "some output" | sudo -u some_user tee /some/path/some_file
Note that the echo will happen as you and the file write will happen as "some_user" what will NOT work is if you were to run the echo as "some_user" and redirect the output with >> "some_file" because the file redirect will happen as you.
Hint: tee also supports append with the -a flag, if you need to replace a line in a file as another user you could execute sed as the desired user.
< command > |& tee filename # this will create a file "filename" with command status as a content, If a file already exists it will remove existed content and writes the command status.
< command > | tee >> filename # this will append status to the file but it doesn't print the command status on standard_output (screen).
I want to print something by using "echo" on screen and append that echoed data to a file
echo "hi there, Have to print this on screen and append to a file"
tee is perfect for this, but this will also do the job
ls -lr / > output | cat output