I'm trying to extract large .tar file using pv.
pv large_file.tar.gz | tar -xcf /../MyFolder.
The pv command works like expected,showing the progress in the console.
I'm trying to split the stdout, to show the progress both in the console and save the same standout, to a file.
I tried doing so with tee, but couldn't make it work.
pv large_file.tar.gz | tee /tmp/strout.log | tar -xcf /../MyFolder
Any suggestions how can i display the progress to the console an in the same time save it to a file?
Thanks!
Not sure that your original command works, as there are several errors in the options given to tar.
Given that ../MyFolder exists, your first command need to be
pv large_file.tar.gz | tar -xz -C ../MyFolder
If you insert tee call between pv and tar calls, then the whole chain works.
pv large_file.tar.gz | tee /tmp/strout.log | tar -xz -C ../MyFolder
However i'm not sure it does what you expect. If you pipe pv output to tee, tee will pipe it to tar, and dump the same contents as the original tar to /tmp/strout.log, resulting in your tar extracted to ../MyFolder and copied to /tmp/strout.log.
EDIT
As suggested by #DownloadPizza, you can use process substitution (see How do I write stderr to a file while using "tee" with a pipe?). By using -f flag with pv, your command will become
pv -f large_file.tar.gz 2> >(tee /tmp/strout.log) > >(tar -xz -C ../MyFolder)
and will produce expected output.
PV progress is sent to stderr, can you try this?:
pv large_file.tar.gz > >(tar -xz -C ./MyFolder/) | echo you might need to edit the tar command as i couldnt get yours to work for me
Related
I run some command that generates ip address as output. But I am developing a workflow where I need the ip address to be written twice. Below is the sample command and its output.
$ some command >> out.txt
$ cat out.txt
10.241.1.85
hdfs://10.241.1.236/
hdfs://10.241.1.237/
What i want is to duplicate the output and it should look like this.
10.241.1.85
hdfs://10.241.1.236/
hdfs://10.241.1.237/
10.241.1.85
hdfs://10.241.1.236/
hdfs://10.241.1.237/
Any help please?
The solution given by #ott in a comment seems fine:
var=$(some cmd); echo -e "$var\n$var".
This is not assigning the command o a varable but it is assigning the output of the command to a variable.
When you do not want this, you can use tee (perhaps this will give some ordering problems) or duplicate it differently:
some_command > out.txt.tmp
cat out.txt.tmp out.txt.tmp > out.txt
rm out.txt.tmp
This way you first get the lines of the copy after al the lines of first entries. When you want to double the output directly, you can use
some_command | sed 'p' > out.txt
some command | tee -a out.txt out.txt
Or
some command | tee -a out.txt >> out.txt
Or
some command | tee -a out.txt out.txt >/dev/null
Run command
Pipe to tee
Enable append mode
Append to same file twice
Generate the output to some temporary, then duplicate the temporary to destination and remove temporary :
some command > /tmp/out.txt; cat /tmp/out.txt /tmp/out.txt > out.txt; rm /tmp/out.txt
Here are some more options you could play around with. Seeing as the output is too large to store in a variable, I'd probably go with tee, a temp file, and gzip if the disk write speed is a bottleneck.
someCommand > tmp.txt && cat tmp.txt tmp.txt > out.txt && rm tmp.txt
Now, if the disk read/write speed is a bottleneck, you can tee the output of someCommand and redirect one of the pipelines through gzip initially.
someCommand | tee >(gzip > tmp.gz) > out.txt && gunzip -c tmp.gz >> out.txt && rm tmp.gz
Additionally, if you don't need random access abilities for out.txt and plan on processing it through some other pipeline, you could always keep it stored gzipped until you need it.
someCommand | gzip > tmp.gz && cat tmp.gz tmp.gz > out.txt.gz && rm tmp.gz
I would suggest this:
(someCommand | tee tmp.txt; cat tmp.txt) > out.txt; rm tmp.txt
Not sure there's a way to safely do this without resorting to a temporary file. You could capture it to a variable, as some have suggested, but you have to be careful about quoting then to make sure whitespace doesn't get mangled, and you also might run into problems if the output is particularly large.
Is it possible to direct that input from this command to a file and also to standard output?
tar -t all.tar
Also Is there a way to kill all processes running lets say an xclock?
you can use pipe the output of tar -t all.tar to the command tee
tar -t all.tar | tee output_file_name
I want to redirect stdop of bzip command to logfile using tee command but its not working and giving error for '-a' in tee command. Please see error below,
> bzip2 file -c 1> tee -a logfile
bzip2: Bad flag `-a'
bzip2, a block-sorting file compressor. Version 1.0.5, 10-Dec-2007.
usage: bzip2 [flags and input files in any order]
-h --help print this message
-d --decompress force decompression
-z --compress force compression
-k --keep keep (don't delete) input files
-f --force overwrite existing output files
-t --test test compressed file integrity
-c --stdout output to standard out
-q --quiet suppress noncritical error messages
-v --verbose be verbose (a 2nd -v gives more)
-L --license display software version & license
-V --version display software version & license
-s --small use less memory (at most 2500k)
-1 .. -9 set block size to 100k .. 900k
--fast alias for -1
--best alias for -9
If invoked as `bzip2', default action is to compress.
as `bunzip2', default action is to decompress.
as `bzcat', default action is to decompress to stdout.
If no file names are given, bzip2 compresses or decompresses
from standard input to standard output. You can combine
short flags, so `-v -4' means the same as -v4 or -4v, &c.
What is the issue? why bzip is considering the '-a' flag of tee command.
Try:
bzip2 -c file | tee -a logfile
The | (pipe) is redirecting the stdout of the left command to the stdin of the right command.
-c is is an option from bzip2 that says Compress or decompress to standard output.. see man bzip2
Your problem is that 1>does not pipe output of the bzip2 command to the tee command, but instead redirects the output to a file which will be named tee. Furthermore you probably don't want to use -c. You should be using the pipe | instead, as follows:
bzip2 file | tee -a logfile
Also, the reason why bzip2 is complaining is because the command as you mentioned above will be interpreted exactly as this one:
bzip2 file -a logfile 1> tee
And hence all arguments after the teeare actually added to the bzip2 command.
As others have pointed out, you want a pipe, not output redirection:
bzip2 file | tee -a logfile
However, bzip2 doesn't produce any output; it simply replaces the given file with a compressed version of the file. You might want to pipe standard error to the log file:
bzip2 file 2>&1 | tee -a logfile
(2>&1 copies standard error to standard output, which can then be piped.)
I have tried redirecting the terminal output to a file using tee and > as in the examples here and the question. It worked for echo test | tee log.txt or ls -l | tee log.txt
But It does not work (does not add anything to the log.txt) when I run a command like divine verify file.dve | tee log.txt
where divine is an installed tool. Any ideas or alternatives?
Try divine verify file.dve 2>&1 | tee log.txt. If the program is outputting to stderr instead of stdout, this redirects stderr to stdout.
works on ffmpeg output too
{ echo ffmpeg -i [rest of command]; ffmpeg -i [rest of command]; } 2>&1 | tee ffmpeg.txt
and tee -a to append if file already exists
======
also if you want to see mediainfo on all files in a folder and make sure command is also visible in mediainfo.txt
{ echo mediainfo *; mediainfo *; } 2>&1 | tee mediainfo.txt
NB: { echo cmd; cmd; } means the command is kept in the txt file ; without this it is not printed
I'm making a script and every time something is done I would like to write into my custom .log file. How do I do that?
And in the end.. I'd just like to read it with Bash,.. do I just use cat?
Thanks.
The simplest syntax I always use is 2>&1 | tee -a file_name.log.
The syntax can be used after a command or execution of a file. e.g.
find . -type f 2>&1 | tee -a file_name.log
or
./test.sh 2>&1 | tee -a file_name.log
Just cat <log message here> >> custom.log.
The >> means add on to the bottom of the file rather than > which would delete the contents of the file and then write the message.