I have a stream of logs from a specific tool on my server with persistent output.
I've say it's like
task 3 done
all is ok
added another task
something made
I cannot grep from output file by a reason, but I have an opportunity of make something through pipes or output streams:
./myexecutablefile | grep something
In this case I receive highlighted "something" if it appeared in output.
I want to make something if first occurrence is met, but i don't know how, because of continuous stream. For example my goal is to kill process if I will see some error.
Surely this doesn't work, && kill will begin after executable ends
./myexecutable | grep 'Error occured' && kill -9 myexecutable
You need to use -q option in grep:
./myexecutable | grep -q 'Error occured' && kill -9 myexecutable
If you are interested in the first occurrence only, you can use the -m switch of grep:
./myexecutablefile | grep -m1 something && do_something
This will still output the matching line. To suppress that, you can turn on -q, but then, you don't need -m1 anymore:
./myexecutablefile | grep -q something && do_something
Related
I've got a long running process and I want to capture a tiny bit of data from the big swath of output.
I can do this by piping it through grep, but then I can't watch it spew out all the other info.
I basically want grep to save what it finds into a variable and leave stdout alone. How can I do that?
With tee, process substitution, and I/O redirection:
{ var=$(cmd | tee >(grep regexp) >&3); } 3>&1
There isn't any way to use a variable like that. It sounds like you want to write your own filter:
./some_long_program | tee >(my_do_stuff.sh)
Then, my_do_stuff.sh is:
#!/bin/bash
while read line; do
echo "$line" | grep -q 'pattern' || continue
VARIABLE=$line # Now it's in a variable
done
If you have the storage space, this is probably more like what you want:
./some_long_program | tee /tmp/slp.log
Then, simply:
grep 'pattern' /tmp/slp.log && VARIABLE=true
or:
VARIABLE=$(grep 'pattern' /tmp/slp.log)
This will let you run the grep at any time. I don't think the variable really adds anything though.
EDIT:
#mpen Based on your last answer above, it sounds like you want to use xargs. Try:
(echo 1 ; sleep 5 ; echo 2) | xargs -L1 echo got:
The -L1 will run the command for every instance found, otherwise it grabs lots of stdin and passes them all (up to some maximum) to the command at once. You'll still want to use this via tee if you want to see all the command output as well:
./some_long_program | tee >(grep 'pattern' | xargs -L1 ./my_do_stuff.sh)
I think this can be done a little more simply if we tee to /dev/tty:
❯ BAR=$(echo foobar | tee /dev/tty | grep -Pom1 b..); echo "[$BAR]"
foobar
[bar]
I have the following commands saved in a .sh file
prog=$1
ps axf | grep $prog | grep -v grep | awk '{print "kill -9 " $1}'
I get the following output when I execute it
kill -9 3184
kill -9 20359
But I just need the first line of it as that is the only valid pid. How can I remove the 2nd line from the output.
There are a few issues with what you want to do:
You're building a chain of 4 commands for something relatively simple
You're going to get as a result only the first line of a list of processes containing $prog (excluding the grep $prog which you filtered out); how can you be sure that's the process you want?
The correct command to use is
pkill $prog`
as suggested in the comments, which probably will do what you want.
Just for information, and to answer your question, you can pipe an output to head -n 1 to return only the first line:
<list of commands> | head -n 1
However, in your case this would add a fifth command to the chain, so I recommend you don't do it this way.
Why read -t doesn't time out when reading from pipe on RHEL5 or RHEL6?
Here is my example which doesn't timeout on my RHEL boxes wile reading from the pipe:
tail -f logfile.log | grep 'something' | read -t 3 variable
If I'm correct read -t 3 should timeout after 3 seconds?
Many thanks in advance.
Chris
GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
The solution given by chepner should work.
An explanation why your version doesn't is simple: When you construct a pipe like yours, the data flows through the pipe from the left to the right. When your read times out however, the programs on the left side will keep running until they notice that the pipe is broken, and that happens only when they try to write to the pipe.
A simple example is this:
cat | sleep 5
After five seconds the pipe will be broken because sleep will exit, but cat will nevertheless keep running until you press return.
In your case that means, until grep produces a result, your command will keep running despite the timeout.
While not a direct answer to your specific question, you will need to run something like
read -t 3 variable < <( tail -f logfile.log | grep "something" )
in order for the newly set value of variable to be visible after the pipeline completes. See if this times out as expected.
Since you are simply using read as a way of exiting the pipeline after a fixed amount of time, you don't have to worry about the scope of variable. However, grep may find a match without printing it within your timeout due to its own internal buffering. You can disable that (with GNU grep, at least), using the --line-buffered option:
tail -f logfile.log | grep --line-buffered "something" | read -t 3
Another option, if available, is the timeout command as a replacement for the read:
timeout 3 tail -f logfile.log | grep -q --line-buffered "something"
Here, we kill tail after 3 seconds, and use the exit status of grep in the usual way.
I dont have a RHEL server to test your script right now but I could bet than read is exiting on timeout and working as it should. Try run:
grep 'something' | strace bash -c "read -t 3 variable"
and you can confirm that.
I have a Linux driver running in the background that is able to return the current system data/stats. I view the data by running a console utility (let's call it dump-data) in a console. All data is dumped every time I run dump-data. The output of the utility is like below
Output:
- A=reading1
- B=reading2
- C=reading3
- D=reading4
- E=reading5
...
- variableX=readingX
...
The list of readings returned by the utility can be really long. Depending on the scenario, certain readings would be useful while everything else would be useless.
I need a way to grep only the useful readings whose names might have have nothing in common (via a bash script). I.e. Sometimes I'll need to collect A,D,E; and other times I'll need C,D,E.
I'm attempting to graph the readings over time to look for trends, so I can't run something like this:
# forgive my pseudocode
Loop
dump-data | grep A
dump-data | grep D
dump-data | grep E
End Loop
to collect A,D,E as that would actually give me readings from 3 separate calls of dump-data as that would not be accurate.
If you want to save all result of grep in the same file, you can just join all expressions in one:
grep -E 'expr1|expr2|expr3'
But if you want to have results (for expr1, expr2 and expr3) in separate files, things are getting more interesting.
You can do this using tee >(command).
For example, here I process the same pipe with thre different commands:
$ echo abc | tee >(sed s/a/_a_/ > file1) | tee >(sed s/b/_b_/ > file2) | sed s/c/_c_/ > file3
$ grep "" file[123]
file1:_a_bc
file2:a_b_c
file3:ab_c_
But the command seems to be too complex.
I would better save dump-data results to a file and then grep it.
TEMP=$(mktemp /tmp/dump-data-XXXXXXXX)
dump-data > ${TEMP}
grep A ${TEMP}
grep B ${TEMP}
grep C ${TEMP}
You can use dump-data | grep -E "A|D|E". Note the -E option of grep. Alternatively you could use egrep without the -E option.
you can simply use:
dump-data | grep -E 'A|D|E'
awk '/MY PATTERN/{print > "matches-"FILENAME;}' myfile{1,3}
thx Guru at Stack Exchange
Is there a cheap way to monitor a log file like tail -f log.txt, then if something like [error] appears, execute a command?
Thank you.
tail -fn0 logfile | \
while read line ; do
echo "$line" | grep "pattern"
if [ $? = 0 ]
then
... do something ...
fi
done
I also found that you can use awk to monitor for pattern and perform some action when pattern is found:
tail -fn0 logfile | awk '/pattern/ { print | "command" }'
This will execute command when pattern is found in the log. Command can be any unix command including shell scripts or anything else.
An even more robust approach is monit. This tool can monitor very many things, but one of them is that it will easily tail one or more logs, match against regex and then trigger a script. This is particularly useful if you have a collection of log files to watch or more than one event to trigger.
Better and simple:
tail -f log.txt | egrep -m 1 "error"
echo "Found error, do sth."
...