I try to use script for find messages in mail server. I need to see file location and output. then i do
grep -r KTsiPtCf0YDQvtC00LDQstC10YYt0LrQvtC90YHRg9C70YzRgtCw0L3RgiDRgdCw0LvQ ./aaa/Maildir/cur/
output is
./aaa/Maildir/cur/1506410272.M30769P16754.ml.exmp.com,S=134882,W=136851:2,S:KTsiPtCf0YDQvtC00LDQstC10YYt0LrQvtC90YHRg9C70YzRgtCw0L3RgiDRgdCw0LvQ
And then, i need to cut all before ":" and make search result readable.
grep -r KTsiPtCf0YDQvtC00LDQstC10YYt0LrQvtC90YHRg9C70YzRgtCw0L3RgiDRgdCw0LvQ ./aaa/Maildir/cur/ | sed 's/.*://' |base64 -d| enca -L ru -x utf-8
But if i do it by pipe i miss file location. How to output location of file and then do pipe?
Related
I am running the following command:
for file in 2017120[1-9]/54100_*.zip; do unzip -c "$file" | grep "3613825" | grep '3418665' ; done
This does a grep job of pulling the data that matches my grep parameters, but I can't figure out how to capture which file the results came from.
I have tried adding grep -H but the result comes back with (standard input).
How can I capture the file name?
When I need to do something like this I just add an echo of the file name to the for loop like this:
for file in 2017120[1-9]/54100_*.zip; do echo $file; unzip -c "$file" | grep "3613825" | grep '3418665' ; done
This prints out the list of files, and the grep line that matches will print immediately after the file that the match is in. like this:
file_1
file_2
file_3
matching line
file_4
file_5
another matching line
file_6
...
Thus I know the matching lines occurred in file_3 and file_5.
Currently I'm trying to tail a log but only showing the lines that has some keywords. Currently I'm using
tail -F file.log | grep -ie 'error\|fatal\|exception\|shutdown\|started'
and I'm getting the expected results: (for example)
10:22 This is an error
10:23 RuntimeException: uncaught problem
I also want to exclude lines that contain a <DATATAG>, even if the keywords slipped into it, because it contains a lot of binary data that clutters my log. I'm then trying to add to the pipe another grep that excludes the tag:
tail -F file.log | grep -ie 'error\|fatal\|exception\|shutdown\|started' | grep -vF '<DATATAG>'
However, this time no lines appear, not even the previous ones that has 'error'/'exception' but not <DATATAG>. When I tried the excluding grep alone:
tail -F file.log | grep -vF '<DATATAG>'
all lines appear, including those that have 'error'/'exception'.
Am I doing something wrong?
Your problem is one of buffering. grep is a tricky tool when it comes to that. From the man page:
By default, output is line buffered when standard output is a terminal and block buffered otherwise.
In your example, the first grep is buffering at the block level, so it will not turn an output to the 2nd grep for a while. The solution is to use the --line-buffered option to look like:
tail -F file.log | grep --line-buffered -ie 'error\|fatal\|exception\|shutdown\|started' | grep -vF '<DATATAG>'
I have practiced in using command line in CentOS6.
I tried to create file, which content would be the output of command man grep. Also I used command man with col -b option to save file as Text-Only. All of this must be in one command.
I tried to do like this:
grep man grep | col -b > output.txt
But it didn't work.
What is the proper way to save output of command man grep as Text-Only file with using option col -b?
Don't you really need this:
man grep | col -b > output.txt
Why do you need to call grep in the first place?
Other, hacky way using grep:
man grep | grep -v somephrasethatwontoccur | col -b > output.txt
But, truly, it makes no sense. grep -v looks for lines without the specified phrase.
I am trying to recursively download several files using wget -m, and I intend to grep all of the downloaded files to find specific text. Currently, I can wait for wget to fully complete, and then run grep. However, the wget process is time consuming as there are many files and instead I would like to show progress by grep-ing each file as it downloads and printing to stdout, all before the next file downloads.
Example:
download file1
grep file1 >> output.txt
download file2
grep file2 >> output.txt
...
Thanks for any advice on how this could be achieved.
As c4f4t0r pointed out
wget -m -O - <wesbites>|grep --color 'pattern'
using grep's color function to highlight the patterns may seem helpful especially when dealing with bulky data output to terminal.
EDIT:
Below is a command line you can use. it creates a file called file and save the output messages from wget.Afterwards it tails the message file.
Using awk to find any lines with "saved" and extract filename, then use grep to pattern from filename.
wget -m websites &> file & tail -f -n1 file|awk -F "\'|\`" '/saved/{system( ("grep --colour pattern ") $2)}'
Based on Xorg's solution I was able to achieve my desired effect with some minor adjustments:
wget -m -O file.txt http://google.com 2> /dev/null & sleep 1 && tail -f -n1 file.txt | grep pattern
This will print out all lines that contain pattern to stdout, and wget itself will produce no output visible from the terminal. The sleep is included because otherwise file.txt would not be created by the time the tail command executed.
As a note, this command will miss any results that wget downloads within the first second.
I was trying to create a little shell script that allowed me to check the transfer progress when copying large files from my laptop's hdd to an external drive.
From the command line this is a simple feat using pv and simple redirection, although the line is rather long and you must know the file size (which is why I wanted the script):
console: du filename (to get the exact file size)
console: cat filename | pv -s FILE_SIZE -e -r -p > dest_path/filename
On my shell script I added egrep "[0-9]{1,}" -o to strip the filename and keep just the size numbers from the return value of du, and the rest should be straightforward.
#!/bin/bash
du $1 | egrep "[0-9]{1,}" -o
sudo cat $1 | pv -s $? -e -r -p > $2/$1
The problem is when I try to copy file12345.mp3 using this I get an ambiguous redirection error because egrep is getting the 12345 from the filename, but I just want the size.
Which means the return value from the first line is actually:
FILE_SIZE
12345
which bugs it.
How should I modify this script to parse just the first numbers until the first " " (space)?
Thanks in advance.
If I understand you correctly:
To retain only the filesize from the du command output:
du $1 | awk '{print $1}'
(assuming the 1st field is the size of the file)
Add double quotes to your redirection to avoid the error:
sudo cat $1 | pv -s $? -e -r -p > "$2/$1"
This quoting is done since your $2 contains spaces.