i have this piece of code
for f in ${CURRENT_IMAGE_FILES[#]};
do
echo "Checking if too many process are running"
while :
do
RUNNING_PROCESS=`ps aux | grep [f]fmpeg -c`
if [[ $RUNNING_PROCESS -gt 22 ]]
then
echo "Too many process running now, sleep for 1 second"
sleep 1
else
echo "Total ffmpeg process running is less then 20, continue"
echo "current process count is " $RUNNING_PROCESS
break
fi
done
convert $f -resize 1280x720 -gravity center $f
composite -gravity center $f /var/bash/b.jpg $f
ffmpeg -loop 1 -f image2 -i $f -c:v libx264 -pix_fmt yuvj420p -preset ultrafast -t 3 $DIR/$COUNT.mp4 &
((++COUNT))
done
The thing is that when place inside the for loop, grep command is getting an error. Something along the lines of Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
But when place outside the for loop, i do not get such error. Is there something wrong with my loop that is interfering with grep?
You are using grep with the options in the wrong order and the [f] character class is unnecessary. From man grep
grep [OPTIONS] PATTERN [FILE...]
So you should change your RUNNING_PROCESS command to
RUNNING_PROCESS=`ps aux | grep -c ffmpeg`
Additionally, why are you using [f]fmpeg? The character class [f] will only match/expand to f, so get rid of it to prevent grep from interpreting it as part of the pattern. Further, it does not help as a wildcard as fmpeg will match anything ffmpeg will match.
You are using
RUNNING_PROCESS=`ps aux | grep [f]fmpeg -c`
to count the process instances of ffmpeg currently running.
The [f] is used so that grep [f]fmpeg does not match it's own command line, that's good.
But there's a problem: The [f] is unquoted, so the shell may try to interpret it, depending on options.
Quoting it should make the line work:
RUNNING_PROCESS=`ps aux | grep '[f]fmpeg' -c`
There is another problem in this: it matches not only ffmpeg -a foo -b bar ..., but also less ffmpeg-manual.txt or vim notes-on-ffmpeg-commands.txt.
But there is a better way to do that - using the right tool for the job.
ps is the right to list processes, but for matching processes, pgrep is better suited:
RUNNING_PROCESS=$(pgrep -c ffmpeg)
does the what you need, getting the count of processes of ffmpeg into RUNNING_PROCESS.
The above works for pgrep of the package procps, common in Linux. There are versions that do not support -c for count, or use -c for something else. If it does not work, use:
RUNNING_PROCESS=$(pgrep ffmpeg | wc -l)
pgrep takes care of not matching itself on its own, so there is no need for the [f] workaround.
I also replaced the back quotes ` ... ` by $( ... ), which has some advantages if you need to nest it.
If you do need to match not only command name, but command line too, use pgrep -f.
Problem is that you are using [ and]asgrep` pattern but pattern is unquoted. Try this in BASH:
RUNNING_PROCESS=$(ps aux | grep '[f]fmpeg' -c)
Alternatively if you have pgrep then you can use:
RUNNING_PROCESS=$(pgrep -f 'ffmpeg' | wc -l)
Related
So i'm currently trying to grep a single result from a random file in a specific directory. The grepping works just fine and the expected output file is populated as expected, but for some reason, even after the output file has already been filled, the process won't stop. This is the grep command where the program seems to be getting stuck.
searchFILE(){
case $2 in
pref)
echo "Populating output file: $3-$1.data.out"
dataOutputFile="$3-$1.data.out"
zgrep -a "\"someParameter\"\:\"$1\"" /folder/anotherFolder/filetemplate.log.* | zgrep -a "\"parameter2\"\:\"$3\"" | head -1 > $dataOutputFile
;;
*)
echo "Unrecognized command"
;;
esac
echo "Query finished"
}
What is currently happening is that the output file is being populated as expected with the head pipe, but for some reason I'm not getting the "Query finished" message, and the process seems not to stop at all.
grep does not know that head -n1 is no longer reading from the pipe until it attempts to write to the pipe, which it will only do if another match is found. There is no direct communication between the processes. It will eventually stop, but only once all the data is read, a second match is found and write fails with EPIPE, or some other error occurs.
You can watch this happen in a simple pipeline like this:
cat /dev/urandom | grep -ao "12[0-9]" | head -n1
With a sufficiently rare pattern, you will observe a delay between output and exit.
One solution is to change your stop condition. Instead of waiting for SIGPIPE as your pipeline does, wait for grep to match once using the -m1 option:
cat /dev/urandom | grep -ao -m1 "12[0-9]"
I saw better performance results with zcat myZippedFile | grep whatever paradigm...
The first difference you need to try is pipe with | head -z --lines=1
The reason is null terminated lines instead of newlines (just in case).
My example script below worked (drop the case statement to make it more simple). If I hold onto $1 $2 inside functions things go wrong. I use parameter $names and only use the $1 $2 $# once, because it also goes wrong for me if I don't and in any case you can then shift over $# and catch arguments. The $# in the script itself are not the same as arguments in bash functions.
grep searching for 2 or multiple parameters in any order means using grep twice; in your case zgrep | grep. The second grep is a normal grep! You only need the first grep to be zgrep to do the unzip. Your question is simpler if you drop the case statement as bash case scares off people: bash was always an ugly lady that works good for short scripts.
zgrep searches text or compressed text, but newlines in LINUX style vs WINDOWS are not the same. So use dos2unix to convert files so that newlines work. I use compressed file simply because it is strange and rare to see zgrep, so it is demonstrated in a shell script with a compressed file! It works for me. I changed a few things, like >> and "sort -u" but you can obviously change them back.
#!/usr/bin/env bash
# Search for egA AND egB using option go
# COMMAND LINE: ./zgrp egA go egB
A="$1"
cOPT="$2" # expecting case go
B="$3"
LOG="./filetemplate.log" # use parameters for long names.
# Generate some data with gzip and delete the temporary file.
echo "\"pramA\":\"$A\" \"pramB\":\"$B\"" >> $B$A.tmp
rm -f ${LOG}.A; tar czf ${LOG}.A $B$A.tmp
rm -f $B$A.tmp
# Use paramaterise $names not $1 etc because you may want to do shift etc
searchFILE()
{
outFile="$B-$A.data.out"
case $cOPT in
go) # This is zgrep | grep NOT zgrep | zgrep
zgrep -a "\"pramA\":\"$A\"" ${LOG}.* | grep -a "\"pramB\":\"$B\"" | head -z --lines=1 >> $outFile
sort -u $outFile > ${outFile}.sorted # sort unique on your output.
;;
*) echo -e "ERROR second argument must be go.\n Usage: ./zgrp egA go egB"
exit 9
;;
esac
echo -e "\n ============ Done: $0 $# Fin. ============="
}
searchFILE "$#"
cat ${outFile}.sorted
I was trying to extract a process size using the command:
size=`ps -eo vsz,pid | grep $pid | cut -'d' -f1`
However, this appeared to only work on some computers but not all. So on the ones where it wasn't working, I tried:
size=`ps -eo vsz,pid | grep $pid | awk '{print $1}'`
however, now this didn't work on the computers where the first command worked.
What I mean by "working" and "not working" is that sometimes:
echo "|$size|"
Will return something like:
|8762348
9835|
And thus the following returns an arithmetic error:
(( $size > $threshold ))
because of the newline or carriage return characters stored in $size. Is there a way to reliable extract simply the first field across different versions of linux?
First you ask ps to displat info for all processes, next you try to select 1 of them.
Your command had problems on some computers, and you tagged the question with both ksh and linux, so I am not sure what command can be used best in your case:
size=$(ps -q ${pid} -o vsz --no-headers)
# or
size=$(ps -p ${pid} -o vsz | grep -v "VSZ")
Is that possible to use grep on a continuous stream?
What I mean is sort of a tail -f <file> command, but with grep on the output in order to keep only the lines that interest me.
I've tried tail -f <file> | grep pattern but it seems that grep can only be executed once tail finishes, that is to say never.
Turn on grep's line buffering mode when using BSD grep (FreeBSD, Mac OS X etc.)
tail -f file | grep --line-buffered my_pattern
It looks like a while ago --line-buffered didn't matter for GNU grep (used on pretty much any Linux) as it flushed by default (YMMV for other Unix-likes such as SmartOS, AIX or QNX). However, as of November 2020, --line-buffered is needed (at least with GNU grep 3.5 in openSUSE, but it seems generally needed based on comments below).
I use the tail -f <file> | grep <pattern> all the time.
It will wait till grep flushes, not till it finishes (I'm using Ubuntu).
I think that your problem is that grep uses some output buffering. Try
tail -f file | stdbuf -o0 grep my_pattern
it will set output buffering mode of grep to unbuffered.
If you want to find matches in the entire file (not just the tail), and you want it to sit and wait for any new matches, this works nicely:
tail -c +0 -f <file> | grep --line-buffered <pattern>
The -c +0 flag says that the output should start 0 bytes (-c) from the beginning (+) of the file.
In most cases, you can tail -f /var/log/some.log |grep foo and it will work just fine.
If you need to use multiple greps on a running log file and you find that you get no output, you may need to stick the --line-buffered switch into your middle grep(s), like so:
tail -f /var/log/some.log | grep --line-buffered foo | grep bar
you may consider this answer as enhancement .. usually I am using
tail -F <fileName> | grep --line-buffered <pattern> -A 3 -B 5
-F is better in case of file rotate (-f will not work properly if file rotated)
-A and -B is useful to get lines just before and after the pattern occurrence .. these blocks will appeared between dashed line separators
But For me I prefer doing the following
tail -F <file> | less
this is very useful if you want to search inside streamed logs. I mean go back and forward and look deeply
Didn't see anyone offer my usual go-to for this:
less +F <file>
ctrl + c
/<search term>
<enter>
shift + f
I prefer this, because you can use ctrl + c to stop and navigate through the file whenever, and then just hit shift + f to return to the live, streaming search.
sed would be a better choice (stream editor)
tail -n0 -f <file> | sed -n '/search string/p'
and then if you wanted the tail command to exit once you found a particular string:
tail --pid=$(($BASHPID+1)) -n0 -f <file> | sed -n '/search string/{p; q}'
Obviously a bashism: $BASHPID will be the process id of the tail command. The sed command is next after tail in the pipe, so the sed process id will be $BASHPID+1.
Yes, this will actually work just fine. Grep and most Unix commands operate on streams one line at a time. Each line that comes out of tail will be analyzed and passed on if it matches.
This one command workes for me (Suse):
mail-srv:/var/log # tail -f /var/log/mail.info |grep --line-buffered LOGIN >> logins_to_mail
collecting logins to mail service
Coming some late on this question, considering this kind of work as an important part of monitoring job, here is my (not so short) answer...
Following logs using bash
1. Command tail
This command is a little more porewfull than read on already published answer
Difference between follow option tail -f and tail -F, from manpage:
-f, --follow[={name|descriptor}]
output appended data as the file grows;
...
-F same as --follow=name --retry
...
--retry
keep trying to open a file if it is inaccessible
This mean: by using -F instead of -f, tail will re-open file(s) when removed (on log rotation, for sample).
This is usefull for watching logfile over many days.
Ability of following more than one file simultaneously
I've already used:
tail -F /var/www/clients/client*/web*/log/{error,access}.log /var/log/{mail,auth}.log \
/var/log/apache2/{,ssl_,other_vhosts_}access.log \
/var/log/pure-ftpd/transfer.log
For following events through hundreds of files... (consider rest of this answer to understand how to make it readable... ;)
Using switches -n (Don't use -c for line buffering!).By default tail will show 10 last lines. This can be tunned:
tail -n 0 -F file
Will follow file, but only new lines will be printed
tail -n +0 -F file
Will print whole file before following his progression.
2. Buffer issues when piping:
If you plan to filter ouptuts, consider buffering! See -u option for sed, --line-buffered for grep, or stdbuf command:
tail -F /some/files | sed -une '/Regular Expression/p'
Is (a lot more efficient than using grep) a lot more reactive than if you does'nt use -u switch in sed command.
tail -F /some/files |
sed -une '/Regular Expression/p' |
stdbuf -i0 -o0 tee /some/resultfile
3. Recent journaling system
On recent system, instead of tail -f /var/log/syslog you have to run journalctl -xf, in near same way...
journalctl -axf | sed -une '/Regular Expression/p'
But read man page, this tool was built for log analyses!
4. Integrating this in a bash script
Colored output of two files (or more)
Here is a sample of script watching for many files, coloring ouptut differently for 1st file than others:
#!/bin/bash
tail -F "$#" |
sed -une "
/^==> /{h;};
//!{
G;
s/^\\(.*\\)\\n==>.*${1//\//\\\/}.*<==/\\o33[47m\\1\\o33[0m/;
s/^\\(.*\\)\\n==> .* <==/\\o33[47;31m\\1\\o33[0m/;
p;}"
They work fine on my host, running:
sudo ./myColoredTail /var/log/{kern.,sys}log
Interactive script
You may be watching logs for reacting on events?
Here is a little script playing some sound when some USB device appear or disappear, but same script could send mail, or any other interaction, like powering on coffe machine...
#!/bin/bash
exec {tailF}< <(tail -F /var/log/kern.log)
tailPid=$!
while :;do
read -rsn 1 -t .3 keyboard
[ "${keyboard,}" = "q" ] && break
if read -ru $tailF -t 0 _ ;then
read -ru $tailF line
case $line in
*New\ USB\ device\ found* ) play /some/sound.ogg ;;
*USB\ disconnect* ) play /some/othersound.ogg ;;
esac
printf "\r%s\e[K" "$line"
fi
done
echo
exec {tailF}<&-
kill $tailPid
You could quit by pressing Q key.
you certainly won't succeed with
tail -f /var/log/foo.log |grep --line-buffered string2search
when you use "colortail" as an alias for tail, eg. in bash
alias tail='colortail -n 30'
you can check by
type alias
if this outputs something like
tail isan alias of colortail -n 30.
then you have your culprit :)
Solution:
remove the alias with
unalias tail
ensure that you're using the 'real' tail binary by this command
type tail
which should output something like:
tail is /usr/bin/tail
and then you can run your command
tail -f foo.log |grep --line-buffered something
Good luck.
Use awk(another great bash utility) instead of grep where you dont have the line buffered option! It will continuously stream your data from tail.
this is how you use grep
tail -f <file> | grep pattern
This is how you would use awk
tail -f <file> | awk '/pattern/{print $0}'
an Unix shell script with only purpose - count the number of running processes of qmail (could be anything else). Easy thing, but there must be some bug in code:
#!/bin/bash
rows=`ps aux | grep qmail | wc -l`
echo $rows
Because
echo $rows
always shows greater number of rows (11) than if I just count rows in
ps aux | grep qmail
There are just 8 rows. Does it work this way on your system too?
Nowadays with linux, there is pgrep. If you have it on your system, you can skip grep -v grep
$ var=$(pgrep bash) # or `pgrep bash | wc -l`
$ echo $var
2110 2127 2144 2161 2178 2195 2212 2229
$ set -- $var; echo ${#}
8
also, if your ps command has -C option, another way
$ ps -C bash -o pid= | wc -l
if not, you can set a character class in your grep pattern
$ ps aux|grep [q]mail | wc -l
It appears that you're counting the grep process itself and the header line that ps normally prints before its output.
I'd suggest something more like:
qprocs=$(ps auxwww | grep -c "[q]mail")
... note that GNU grep has a "-c" switch to have it print a "count" of matches rather than the lines themselves. The trick with the regular expression here is to match qmail without matching the literal string that's on the grep command line. So we take any single character in the string and wrap it in square brackets such that it is a single character "class." The regexp: [q]mail matches the string qmail without matching the string [q]mail.
Note that even with this regex you may still find some false positive matches. If you really want to be more precise then you should supply a custom output format string to your ps command (see the man pages) or you should feed it through a pipemill or you should parse the output of the ps command based on fields (using awk or cut or a while read loop). (The -o option to ps is by far the easiest among these).
No, since I'm not running qmail. However, you will want to, at a bare minimum, exclude the process running your grep:
ps aux | grep qmail | grep -v grep
For debugging, you may want to do:
rows=`ps aux | grep qmail`
echo $rows >debug.input
od -xcb debug.input
(to see your input to the script in great detail) and then rewrite your script temporarily as:
#!/bin/bash
rows=`cat debug.input | wc -l`
echo $rows
That way, you can see the input and figure out what effect it's having on your code, even as you debug it.
A good debugger will eventually learn to only change one variable at a time. If your changing your code to get it working, that's the variable - don't let the input to your code change as well.
Use
$ /sbin/pidof qmail
A few ways...
ps -e | grep ' [q]mail' | wc -l
ps -C qmail -opid= | wc -l
pidof qmail | tr ' ' '\n' | wc -l
pgrep is on many Linux distributions, and I imagine available for other Unices.
[dan#khorium ~]$ whatis pgrep
pgrep (1) - look up or signal processes based on name and other attributes
[dan#khorium ~]$ pgrep mingetty
1920
1921
1922
1923
1924
In your case, pgrep qmail | wc -l should do the trick.
How do I use grep to perform a search which, when a match is found, will print the file name as well as the first n characters in that file? Note that n is a parameter that can be specified and it is irrelevant whether the first n characters actually contains the matching string.
grep -l pattern *.txt |
while read line; do
echo -n "$line: ";
head -c $n "$line";
echo;
done
Change -c to -n if you want to see the first n lines instead of bytes.
You need to pipe the output of grep to sed to accomplish what you want. Here is an example:
grep mypattern *.txt | sed 's/^\([^:]*:.......\).*/\1/'
The number of dots is the number of characters you want to print. Many versions of sed often provide an option, like -r (GNU/Linux) and -E (FreeBSD), that allows you to use modern-style regular expressions. This makes it possible to specify numerically the number of characters you want to print.
N=7
grep mypattern *.txt /dev/null | sed -r "s/^([^:]*:.{$N}).*/\1/"
Note that this solution is a lot more efficient that others propsoed, which invoke multiple processes.
There are few tools that print 'n characters' rather than 'n lines'. Are you sure you really want characters and not lines? The whole thing can perhaps be best done in Perl. As specified (using grep), we can do:
pattern="$1"
shift
n="$2"
shift
grep -l "$pattern" "$#" |
while read file
do
echo "$file:" $(dd if="$file" count=${n}c)
done
The quotes around $file preserve multiple spaces in file names correctly. We can debate the command line usage, currently (assuming the command name is 'ngrep'):
ngrep pattern n [file ...]
I note that #litb used 'head -c $n'; that's neater than the dd command I used. There might be some systems without head (but they'd pretty archaic). I note that the POSIX version of head only supports -n and the number of lines; the -c option is probably a GNU extension.
Two thoughts here:
1) If efficiency was not a concern (like that would ever happen), you could check $status [csh] after running grep on each file. E.g.: (For N characters = 25.)
foreach FILE ( file1 file2 ... fileN )
grep targetToMatch ${FILE} > /dev/null
if ( $status == 0 ) then
echo -n "${FILE}: "
head -c25 ${FILE}
endif
end
2) GNU [FSF] head contains a --verbose [-v] switch. It also offers --null, to accomodate filenames with spaces. And there's '--', to handle filenames like "-c". So you could do:
grep --null -l targetToMatch -- file1 file2 ... fileN |
xargs --null head -v -c25 --