I'm trying to silent zcat warning through the option -q or 2>/dev/null
so far nothing is working. I keep getting the same warning when a file name is missing.
I'm looping through 100s of compressed files to extract a specific data. The idea is if zcat encounter a bad name or a missing file name, zcat will just stay quite and wait for the next cycle, but currently this is what I'm getting when using both options
zcat -q $ram | head -n1 or zcat $ram | head -n1 2>/dev/null
gzip: compressed data not read from a terminal. Use -f to force decompression.
For help, type: gzip -h
Any idea how to solve that or a faster way to read a .gz file with a silent feature that works?
Thanks
At present, you're redirecting only stderr from head; you're not redirecting from zcat at all. If you want to redirect stderr from zcat, then you need to put the redirection before the pipe symbol, like so:
zcat $ram 2>/dev/null | head -n1
Related
I try to use script for find messages in mail server. I need to see file location and output. then i do
grep -r KTsiPtCf0YDQvtC00LDQstC10YYt0LrQvtC90YHRg9C70YzRgtCw0L3RgiDRgdCw0LvQ ./aaa/Maildir/cur/
output is
./aaa/Maildir/cur/1506410272.M30769P16754.ml.exmp.com,S=134882,W=136851:2,S:KTsiPtCf0YDQvtC00LDQstC10YYt0LrQvtC90YHRg9C70YzRgtCw0L3RgiDRgdCw0LvQ
And then, i need to cut all before ":" and make search result readable.
grep -r KTsiPtCf0YDQvtC00LDQstC10YYt0LrQvtC90YHRg9C70YzRgtCw0L3RgiDRgdCw0LvQ ./aaa/Maildir/cur/ | sed 's/.*://' |base64 -d| enca -L ru -x utf-8
But if i do it by pipe i miss file location. How to output location of file and then do pipe?
I have "gz" files that I am downloading using "rsync". Then, as these files are compressed, I need to extract them using gunzip (I am open to any other alternative for gunzip). I want to put all these commands together into a pipe to have something like that rsync file | gunzip
My original command is the following:
awk -F "\t" '$5~/^(reference genome|representative genome)$/ {sub("ftp", "rsync", $20); b=$20"/*genomic.fna.gz"; print b" viral/." }' assembly_summary_viral.txt | xargs -l1 rsync --copy-links --times --recursive --verbose --exclude="*rna*" --exclude="*cds*"
It looks a little bit complicated, but it's downloading the files that I need, and there is no problem with it. I added | gunzip However the extraction of the compressed files is not working, and it's only downloading them.
Any suggestion?
A pipe takes the stdout of the left command and sends it to the stdin of the right command. Here we have to take the stdout of rsync and pipe to the stdin of gunzip.
rsync doesn't really output much without the -v flag so you'll have to add that. It will now spit out to stdout something like the following:
>rsync -rv ./ ../viral
sending incremental file list
file1
file2
file3
test1_2/
test1_2/file1
test1_2/file2
sent 393 bytes received 123 bytes 1,032.00 bytes/sec
total size is 0 speedup is 0.00
We can pipe that to awk first to grab only the file path/name and prepend viral/ to the front of it so that it gunzips the files that you just rsync'd TO (instead of the ones FROM which you rsync'd):
rsync -rv ./ ../viral | awk '!NF{endFileList=1} NR>1 && endFileList!=1{print "../viral/"$0}'
Now we have rsync and awk spitting out a list of filenames that are being sent to the TO directory. Now we need to get gunzip to process that list. Unfortunately, gunzip can't take in a list of files. If you send gunzip something to it's stdin it will assume that the stream is a gzipped stream and will attempt to gunzip it.
Instead we'll employ that xargs method you have above take the stdin and feed it into gunzip as the parameter (filename) that it needs:
rsync -rv ./ ../viral | awk '!NF{endFileList=1} NR>1 && endFileList!=1{print "../viral/"$0}' | xargs -l1 gunzip
Most likely you will have to tweak this a bit to insure you are gunzipping the right files (either your FROM location files or your TO location files). This gets trickier if you are rsyncing to a remote computer of SSH, obviously. Not sure if that can just be piped.
I am trying to recursively download several files using wget -m, and I intend to grep all of the downloaded files to find specific text. Currently, I can wait for wget to fully complete, and then run grep. However, the wget process is time consuming as there are many files and instead I would like to show progress by grep-ing each file as it downloads and printing to stdout, all before the next file downloads.
Example:
download file1
grep file1 >> output.txt
download file2
grep file2 >> output.txt
...
Thanks for any advice on how this could be achieved.
As c4f4t0r pointed out
wget -m -O - <wesbites>|grep --color 'pattern'
using grep's color function to highlight the patterns may seem helpful especially when dealing with bulky data output to terminal.
EDIT:
Below is a command line you can use. it creates a file called file and save the output messages from wget.Afterwards it tails the message file.
Using awk to find any lines with "saved" and extract filename, then use grep to pattern from filename.
wget -m websites &> file & tail -f -n1 file|awk -F "\'|\`" '/saved/{system( ("grep --colour pattern ") $2)}'
Based on Xorg's solution I was able to achieve my desired effect with some minor adjustments:
wget -m -O file.txt http://google.com 2> /dev/null & sleep 1 && tail -f -n1 file.txt | grep pattern
This will print out all lines that contain pattern to stdout, and wget itself will produce no output visible from the terminal. The sleep is included because otherwise file.txt would not be created by the time the tail command executed.
As a note, this command will miss any results that wget downloads within the first second.
I am trying to evaluate the disk usage of a number of Unix user accounts.
Simply, I am using the following command:
du -cBM --max-depth=1 | sort -n
But I’ve seen many error message like below. How can I exclude all such “Permission denied” messages from display?
du: `./james/.gnome2': Permission denied
My request could be very similar to the following list, by replacing “find” to “du”.
How can I exclude all "permission denied" messages from "find"?
The following thread does not work. I guess I am using bash.
Excluding hidden files from du command output with --exclude, grep -v or sed
du -cBM --max-depth=1 2>/dev/null | sort -n
or better in bash (just filter out this particular error, not all like last snippet)
du -cBM --max-depth=1 2> >(grep -v 'Permission denied') | sort -n
2> /dev/nul hides only error messages.
the command du always try run over directory. Imagine that you have thousands of dirs?
du needs eval, if you have persmission run if not, follow with the next dir...
I'd use something concise that excludes only the lines you don't want to see. Redirect stderr to stdout, and grep to exclude all "denied"s:
du -cBM --max-depth=1 2>&1 | grep -v 'denied' | sort -n
To remove all errors coming from the du command i used this:
du -sh 2>&1 | grep -v '^du:'
If 2>/dev/null does not work, probably the shell you are using is not bash.
To check what shell you are using, you may try ps -p $$ (see https://askubuntu.com/a/590903/130162 )
you can pipe it to a temporary file, like -
du ... > temp_file
Errors get printed on the terminal and only disk usage information gets printed into the temp_file.
time curl http://www.google.com | tee | wc | gzip > google.gz
Why doesn't this command work? It creates the file, and times the operation, but does not print the number of lines, words, and characters (wc).
time curl http://www.google.com | tee | wc
This will print print the words characters and lines, but obviously, the tee portion is pointless.
Is it because I'm sending the word count of the url to google.gz?
I have to use tee, gzip, time, curl to download google web page to a gziped file, print the word count, how long it took.
It is an assignment, so I'm not looking for someone to do it for me. I just am having a problem in that I can't tee to utility, and I cant to and gzip at the same time.
Maybe there is a way to use gzip with curl?
Well, wc outputs the number of characters, words and lines, but then you send it to gzip which compresses it. Eventually, compressed information ends up in google.gz. If you decompress the file, e.g. with
gunzip google.gz
you'll see the three numbers.
Also, normally when one uses tee, they specify a file where the tee'ed data is supposed to be stored.
time curl http://www.google.com | tee /dev/tty | gzip > google.gz
I'm going to guess that something like this is what you want:
time curl http://www.google.com | tee /tmp/z | gzip > google.gz; wc /tmp/z; rm /tmp/z