I'm testing some netcat udp shell tools and was trying to take output and send it through standard pipe stuff. In this example, I have a netcat client which sends 'foo' then 'bar' then 'foo' with newlines for each attempt reading from the listener:
[root#localhost ~ 05:40:20 ]# exec 5< <(nc -d -w 0 -6 -l -k -u 9801)
[root#localhost ~ 05:40:25 ]# awk '{print}' <&5
foo
bar
foo
^C
[root#localhost ~ 05:40:48 ]# awk '{print}' <&5 | grep foo
^C
[root#localhost ~ 05:41:12 ]# awk '{print}' <&5 | grep --line-buffered foo
^C
[root#localhost ~ 05:41:37 ]#
[root#localhost ~ 05:44:38 ]# grep foo <&5
foo
foo
^C
[root#localhost ~ 05:44:57 ]#
I've checked the --line-buffered... and I also get the same behavior from 'od -bc', ie, nothing at all. Grep works on the fd... but if I pipe that grep to anything (like od -bc, cut), I get the same (nothing). Tried prefixing the last grep with stdbuf -oL to no avail. Could > to file and then tail -f it, but feel like I'm missing something.
Update:
Appears to be something descriptor/order/timing related. I created a file 'tailSource' and used this instead, which produced the same issue (no output) when I ran echo -e "foo\nfoo\nbar\nfoo" >> tailSource
[root#localhost shm 07:16:18 ]# exec 5< <(tail -n 0 -f tailSource)
[root#localhost shm 07:16:32 ]# awk '{print}' <&5 | grep foo
... and when I run without the '| grep foo', I get the output I'd expect.
(GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu))
awk is buffering when its output is not going to a terminal. If you have GNU awk, you can use its fflush() function to flush after every print
gawk '{print; fflush()}' <&5 | grep foo
In this particular case though, you don't need awk and grep, either will do.
awk /foo/ <&5
grep foo <&5
See BashFAQ 9 for more on buffering and how to work around it.
For what it's worth
$ exec 5< <(echo -e "foo\nbar\nfoo")
$ awk '{print}' <&5 | grep foo
prints
foo
foo
where as
$ exec 5< <(echo -e "foo\nbar\nfoo")
$ awk '{print}' <&5
foo
bar
foo
$ awk '{print}' <&5
$
the second call does not output anything.
I think this is because the file descriptor is at its end and you need to rewind it, see also this answer to this question question.
In order to use it, you could alternatively either capture it in a temporary file or perform the transformations in one line.
Related
Doing the following:
First console
touch /tmp/test
Second console
tail -f /tmp/test |grep propo |grep -v miles
Third console
echo propo >> /tmp/test
Second console must show "propo" but it doesn't shows anything, if you run in second console instead:
tail -f /tmp/test |grep propo
And do echo propo >> /tmp/test it will show propo, but the grep -v is for miles not for propo
Why?
Test into your own environment if you want, it's pretty obvious but not working.
Why?
Most probably because the output of a command when piped to another command is fully buffered, not line buffered. The output could be buffered in the first pipe or by grep.
Use stdbuf -oL to force line buffering and grep --line-buffered for line buffered grep.
the problem is that grep does not use line buffering by default; so the output will be buffered. You could use grep --line-buffered:
tail -f /tmp/test | grep --line-buffered propo | grep -v miles
In one of my tools is needed the PID of specyfic process in system. I try do this by following command:
parasit#host:~/# ps -ef | grep beam.smp |grep -v grep |awk '{ print $2 }' |head -n1
11982
Works fine, but when i try use the same command in script in the vast majority of cases got PID of grep instead of target process (beam.smp in this case) despite of 'grep -v grep`.
parasit#host:~/# cat getPid.sh
#!/bin/bash
PROC=$1
#GET PID
CMD="ps -ef | grep $PROC |grep -v grep |awk '{ print \$2 }' |head -n1"
P=`eval $CMD`
parasit#host:~/# bash -x ./getPid.sh beam.smp
+ PROC=beam.smp
+ CMD='ps -ef |grep beam.smp |grep -v grep |awk '\''{ print $2 }'\'' |head -n1'
++ eval ps -ef '|grep' beam.smp '|grep' -v grep '|awk' ''\''{' print '$2' '}'\''' '|head' -n1
+++ head -n1
+++ awk '{ print $2 }'
+++ grep -v grep
+++ grep beam.smp
+++ ps -ef
+ P=2189
Interestingly, it is not deterministic, I know it sounds strange, but sometimes it works OK, and sometimes no, I have no idea what it depends on.
How it is possibile? Is there any better method to get rid of "grep" from results?
BR
Parasit
pidof -s is made for that (-s: single ID is returned):
pidof -s "beam.smp"
However, pidof also returns defunct (zombie, dead) processes. So here's a way to get PID of the first alive-and-running process of a specified command:
# function in bash
function _get_first_pid() {
ps -o pid=,comm= -C "$1" | \
sed -n '/'"$1"' *$/{s:^ *\([0-9]*\).*$:\1:;p;q}'
}
# example
_get_first_pid "beam.smp"
-o pid=,comm=: list only PID and COMMAND columns; ie. only list what we need to check; if all are listed then it is more difficult to process later on
-C "$1": of the command specified in -C; ie. only find the process of that specific command, not everything
sed: print only PID for first line that do not have "defunct" or anything after the base command name
I want to send a program the most recent lines from a text file using tail as stdin.
First, I echo to the program some input that will be the same every time, then send in tail input from an inputfile which should first be processed through sed. The following is the command line that I expect to work. But when the program runs it only receives the echo input, not the tail input.
(echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex' && cat) | ./program
However, the following works exactly as expected, printing everything out to the terminal:
echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex' && cat
So I tried with another type of output, and again while the echoed text posted, the tail text does not appear anywhere:
(echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex') | tee out.txt
This made me think it is a problem with buffering, but I tried the unbuffer program and all other advice here (https://superuser.com/questions/59497/writing-tail-f-output-to-another-file) without results. Where is the tail output going and how can I get it to go into my program as expected?
The buffering problem was resolved when I prefixed the sed command with the following:
stdbuf -i0 -o0 -e0
Much more preferable to using unbuffer, which didn't even work for me. Dave M's suggestion of using sed's relatively new -u also seems to do the trick.
One thing you may be getting confused by -- | (pipeline) is higher precedence than && (consecutive execution). So when you say
(echo "new" && tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex' && cat) | ./program
that is equivalent to
(echo "new" && (tail -f ~/inputfile 2> /dev/null | sed -n -r 'some regex') && cat) | ./program
So the cat isn't really doing anything, and the sed output is probably buffered a bit. You can try using the -u option to sed to get it to use unbuffered output:
(echo "new" && (tail -f ~/inputfile 2> /dev/null | sed -n -u -r 'some regex')) | ./program
I believe some versions of sed default to -u when the output is a terminal and not when it is a pipe, so that may be the source of the difference you're seeing.
You can use the i command in sed (see the command list in the manpage for details) to do the inserting at the beginning:
tail -f inputfile | sed -e '1inew file' -e 's/this/that/' | ./program
mySample.sh
pid=$(ps -Af | grep $1 | grep -v grep | awk ' { print $2 } ');
echo $pid
The above command is printing and killing the temporary process that was created for grep
Even though i do not have any process running with Abcd,
This is printing pid
Any ways to ignore it,
iam actually ignoring it using grep -v, still...
./mySample.sh Abcd
6251 6378 6379
Any Issue in fetching the process id.?
Basic command line output is below,After running a process with name Acc_Application_One
[root#localhost Desktop]# ps -Af | grep Acc
root 6251 2758 0 16:16 pts/1 00:00:00 ./Acc_Application_One
root 7288 2758 0 16:57 pts/1 00:00:00 grep Acc
Changed mySample.sh
pgrep -fl "$1"
And the output is
[root#localhost Desktop]# mySample.sh Acc_Application_One
6251 7289
To kill a process with the pattern anywhere in command line use pkill -f:
pkill -f "$1"
As per man pkill:
-f Match the pattern anywhere in the full argument string of the process instead of just the executable name.
Similarly you can use pgrep -f "$1" to list the process id of the matching process.
Try something much simpler:
pid=$(pgrep "$1")
And if you want to kill it:
pkill "$1"
The problem will become clear when you remove the awk: mySample.sh will have Abcd as well.
ps -Af | grep " $1" | grep -Ev "grep|$0" | awk ' { print $2 } '
Changed mySample.sh script with below code
And This is just fetching the processId using the parameter sent
and killing it
pid=$(pgrep -fl $1 | grep -v '[k]ill_script'| awk ' { print $1 } ')
echo $pid
if [[ -n ${pid} ]]; then
echo "Stopping Acc Application $1 with pid=${pid}"
kill -9 ${pid}
fi
Thanks
I've got two test files, namely, ttt.txt and ttt2.txt, the Content of which is shown as below:
#ttt.txt
(132) 123-2131
543-732-3123
238-3102-312
#ttt2.txt
1
2
3
I've already tried the following commands in bash and it works fine:
if grep -oE "(\(\d{3}\)[ ]?\d{3}-\d{4})|(\d{3}-\d{3}-\d{4})" ttt1.txt ; then echo "found"; fi
# with output 'found'
if grep -oE "(\(\d{3}\)[ ]?\d{3}-\d{4})|(\d{3}-\d{3}-\d{4})" ttt2.txt ; then echo "found"; fi
But when I combine the above command with xargs, it complains error '-bash: syntax error near unexpected token `then''. Could anyone give me some explanation? Thanks in advance!
ll | awk '{print $9}' | grep ttt | xargs -I $ if grep --quiet -oE "(\(\d{3}\)[ ]?\d{3}-\d{4})|(\d{3}-\d{3}-\d{4})" $; then echo "found"; fi
$ is a special character in bash (it marks variables) so don't use it as your xargs marker, you'll only get confused.
The real problem here though is that you are passing if grep --quiet -oE "(\(\d{3}\)[ ]?\d{3}-\d{4})|(\d{3}-\d{3}-\d{4})" $ as the argument to xargs, and then the remainder of the line is being treated as a new command, because it breaks at the ;.
You can wrap the whole thing in a sub-invocation of bash, so that xargs sees the whole command:
$ ll | awk '{print $9}' | grep ttt | xargs -I xx bash -c 'if grep --quiet -oE "(\(\d{3}\)[ ]?\d{3}-\d{4})|(\d{3}-\d{3}-\d{4})" xx; then echo "found"; fi'
found
Finally, ll | awk '{print $9}' | grep ttt is a needlessly complicated way of listing the files that you're looking for. You actually you don't need any of the code above, just do this:
$ if grep --quiet -oE "(\(\d{3}\)[ ]?\d{3}-\d{4})|(\d{3}-\d{3}-\d{4})" ttt*; then echo "found"; fi
found
Alternatively, if you want to process each file in turn (which you don't need here, but you might want when this gets more complicated):
for file in ttt*
do
if grep --quiet -oE "(\(\d{3}\)[ ]?\d{3}-\d{4})|(\d{3}-\d{3}-\d{4})" "$file"
then
echo "found"
fi
done