I want to redirect stdop of bzip command to logfile using tee command but its not working and giving error for '-a' in tee command. Please see error below,
> bzip2 file -c 1> tee -a logfile
bzip2: Bad flag `-a'
bzip2, a block-sorting file compressor. Version 1.0.5, 10-Dec-2007.
usage: bzip2 [flags and input files in any order]
-h --help print this message
-d --decompress force decompression
-z --compress force compression
-k --keep keep (don't delete) input files
-f --force overwrite existing output files
-t --test test compressed file integrity
-c --stdout output to standard out
-q --quiet suppress noncritical error messages
-v --verbose be verbose (a 2nd -v gives more)
-L --license display software version & license
-V --version display software version & license
-s --small use less memory (at most 2500k)
-1 .. -9 set block size to 100k .. 900k
--fast alias for -1
--best alias for -9
If invoked as `bzip2', default action is to compress.
as `bunzip2', default action is to decompress.
as `bzcat', default action is to decompress to stdout.
If no file names are given, bzip2 compresses or decompresses
from standard input to standard output. You can combine
short flags, so `-v -4' means the same as -v4 or -4v, &c.
What is the issue? why bzip is considering the '-a' flag of tee command.
Try:
bzip2 -c file | tee -a logfile
The | (pipe) is redirecting the stdout of the left command to the stdin of the right command.
-c is is an option from bzip2 that says Compress or decompress to standard output.. see man bzip2
Your problem is that 1>does not pipe output of the bzip2 command to the tee command, but instead redirects the output to a file which will be named tee. Furthermore you probably don't want to use -c. You should be using the pipe | instead, as follows:
bzip2 file | tee -a logfile
Also, the reason why bzip2 is complaining is because the command as you mentioned above will be interpreted exactly as this one:
bzip2 file -a logfile 1> tee
And hence all arguments after the teeare actually added to the bzip2 command.
As others have pointed out, you want a pipe, not output redirection:
bzip2 file | tee -a logfile
However, bzip2 doesn't produce any output; it simply replaces the given file with a compressed version of the file. You might want to pipe standard error to the log file:
bzip2 file 2>&1 | tee -a logfile
(2>&1 copies standard error to standard output, which can then be piped.)
Related
I'm trying to extract large .tar file using pv.
pv large_file.tar.gz | tar -xcf /../MyFolder.
The pv command works like expected,showing the progress in the console.
I'm trying to split the stdout, to show the progress both in the console and save the same standout, to a file.
I tried doing so with tee, but couldn't make it work.
pv large_file.tar.gz | tee /tmp/strout.log | tar -xcf /../MyFolder
Any suggestions how can i display the progress to the console an in the same time save it to a file?
Thanks!
Not sure that your original command works, as there are several errors in the options given to tar.
Given that ../MyFolder exists, your first command need to be
pv large_file.tar.gz | tar -xz -C ../MyFolder
If you insert tee call between pv and tar calls, then the whole chain works.
pv large_file.tar.gz | tee /tmp/strout.log | tar -xz -C ../MyFolder
However i'm not sure it does what you expect. If you pipe pv output to tee, tee will pipe it to tar, and dump the same contents as the original tar to /tmp/strout.log, resulting in your tar extracted to ../MyFolder and copied to /tmp/strout.log.
EDIT
As suggested by #DownloadPizza, you can use process substitution (see How do I write stderr to a file while using "tee" with a pipe?). By using -f flag with pv, your command will become
pv -f large_file.tar.gz 2> >(tee /tmp/strout.log) > >(tar -xz -C ../MyFolder)
and will produce expected output.
PV progress is sent to stderr, can you try this?:
pv large_file.tar.gz > >(tar -xz -C ./MyFolder/) | echo you might need to edit the tar command as i couldnt get yours to work for me
I am relatively new to informatics, and have just discovered the virtues of the parallel command. However, I am having trouble using this in conjunction with piping and output.
I am using this command:
parallel -j 2 echo ./hisat2 --dta -p 32 -x path/to/index -U {} | ./samtools view -b - > /path/to/storage/folder/{/.}.bam :::: fs1 > executable.sh
fs1 contains a list of all the files I want to run. executable.sh is the executable command list. I wish for each file listed in fs1 to be individually processed by a program (called hisat2) and the ouput sam file to be converted into bam format with samtools. However, it does not seem to like the piping - it complains with the following:
bash: /path/to/storage/folder/{/.}.bam: No such file or directory
parallel: Warning: Input is read from the terminal. Only experts do this on purpose. Press CTRL-D to exit.
How can I overcome this? Is the only way around this to first process all files to sam, and then parallel bam convert?
You need to quote the pipe and redirection:
parallel -j 2 "./hisat2 --dta -p 32 -x path/to/index -U {} | ./samtools view -b - > /path/to/storage/folder/{/.}.bam" :::: fs1
Use --dry-run to see what would be run:
parallel --dry-run -j 2 "./hisat2 --dta -p 32 -x path/to/index -U {} | ./samtools view -b - > /path/to/storage/folder/{/.}.bam" :::: fs1
(Are you sure samtools is in current dir? Usually that is installed for a wider audience.)
May I suggest you spend an hour walking through man parallel_tutorial? Your command line will love you for it.
Is it possible to direct that input from this command to a file and also to standard output?
tar -t all.tar
Also Is there a way to kill all processes running lets say an xclock?
you can use pipe the output of tar -t all.tar to the command tee
tar -t all.tar | tee output_file_name
New to bash programming. I am not sure what is meant by 'output to stdout'. Does it mean print out to the command line?
If I have a simple bash script:
#!/bin/bash
wget -q http://192.168.0.1/test -O - | grep -m 1 'Hello'
it outputs a string to the terminal. Does this mean it's 'outputting to stdout' ?
Thanks
Yes, stdout is the terminal (unless it's redirected to a file using the > operator or into the stdin of another process using |)
In your specific example, you're actually redirecting using | grep ... through grep then to the terminal.
Every process on a Linux system (and most others) has at least 3 open file descriptors:
stdin (0)
stdout (1)
stderr (2)
Regualary every of this file descriptors will point to the terminal from where the process was started. Like this:
cat file.txt # all file descriptors are pointing to the terminal where you type the command
However, bash allows to modify this behaviour using input / output redirection:
cat < file.txt # will use file.txt as stdin
cat file.txt > output.txt # redirects stdout to a file (will not appear on terminal anymore)
cat file.txt 2> /dev/null # redirects stderr to /dev/null (will not appear on terminal anymore
The same is happening when you are using the pipe symbol like:
wget -q http://192.168.0.1/test -O - | grep -m 1 'Hello'
What is actually happening is that the stdout of the wget process (the process before the | ) is redirected to the stdin of the grep process. So wget's stdout isn't a terminal anymore while grep's output is the current terminal. If you want to redirect grep's output to a file for example, then use this:
wget -q http://192.168.0.1/test -O - | grep -m 1 'Hello' > output.txt
Unless redirected, standard output is the text terminal which initiated the program.
Here's a wikipedia article: http://en.wikipedia.org/wiki/Standard_streams#Standard_output_.28stdout.29
I'm running the following command (on Ubuntu)
time wget 'http://localhost:8080/upLoading.jsp' --timeout=0
and get a result in the command line
real 0m0.042s
user 0m0.000s
sys 0m0.000s
I've tried the following:
time -a o.txt wget 'http://localhost:8080/upLoading.jsp' --timeout=0
and get the following error
-a: command not found
I want to get the result to be redirected to some file. How can I do that?
-a is only understood by the time binary (/usr/bin/time), When just using time you're using the bash built-in version which does not process the -a option, and hence tries to run it as a command.
/usr/bin/time -o foo.txt -a wget 'http://localhost:8080/upLoading.jsp' --timeout=0
Checking man time, I guess what you need is
time -o o.txt -a ...
(Note you need both -a and -o).
[EDIT:] If you are in bash, you must also take care to write
/usr/bin/time
(check manpage for explanation)
You can direct the stdout output of any commmand to a file using the > character.
To append the output to a file use >>
Note that unless done explicitly, output to stderr will still go to the console. To direct both stderr and stdout to the same output stream use
command 2>&1 outfile.txt (with bash)
or
command >& outfile.txt (with t/csh)
If you are working with bash All about redirection will give you more details and control about redirection.
\time 2> time.out.text command
\time -o time.out.text command
This answer based on earlier comments. It is tested it works. The advantage of the \ over /usr/bin/ is that you don't have to know the install directory of time.
These answers also only capture the time, not other output.
Exactly the time from GNU writes it's output to stderr and if you want to redirect it to file, you can use --output=PATH parameter of time
See this http://unixhelp.ed.ac.uk/CGI/man-cgi?time
And if you want to redirect stdout to some file, you can use > filename to create file and fill it or >> filename to append to some file after the initial command.
If you want to redirect stderr by yourself, you can use $ command >&2 your_stderr_output
Try to use /usr/bin/time since many shells have their own implementation of time which may or may not support the same flags as /usr/bin/time
so change your command to
/usr/bin/time -a -o foo.txt wget ....
How about your LANG ?
$ time -ao o.txt echo 1
bash: -ao: コマンドが見つかりません
real 0m0.001s
user 0m0.000s
sys 0m0.000s
$ export|grep LANG
declare -x LANG="ja_JP.utf8"
$ LANG=C time -ao o.txt echo 1
1
$ cat o.txt
0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 1984maxresident)k
0inputs+0outputs (0major+158minor)pagefaults 0swaps
Try:
command 2> log.txt
and the real-time output from "command" can be seen in another console window with:
tail -f log.txt
This worked for me:
( time command ) |& tee output.txt
https://unix.stackexchange.com/questions/115980/how-can-i-redirect-time-output-and-command-output-to-the-same-pipe
You can do that with > if you want to redirect the output.
For example:
time wget 'http://localhost:8080/upLoading.jsp' --timeout=0 > output.txt 2>&1
2>&1 says to redirect STDERR to the same file.
This command will erase any output.txt files and creates a new one with your output. If you use >> it will append the output at the end of any existing output.txt file. If it doesn't exist, it will create it.