Why doesn't xargs in parallel mode work with grep? - linux

Question

cat gg
gives
192
and
cat tmpfilelist
gives
android/app/src/main/res/drawable-mdpi/src_assets_images_alerts_iconnotificationsmartpluggreen.png
android/app/src/main/res/drawable-mdpi/src_assets_images_ic_biggames.png
android/app/src/main/res/drawable-xhdpi/src_assets_images_alerts_iconnotificationsmartpluggreen.png
android/app/src/main/res/drawable-xhdpi/src_assets_images_ic_biggames.png
android/app/src/main/res/drawable-xxhdpi /src_assets_images_alerts_iconnotificationsmartpluggreen.png
android/app/src/main/res/drawable-xxhdpi/src_assets_images_ic_biggames.png
gg
ios/WebRTC.framework/Headers/RTCCallbackLogger.h
ios/WebRTC.framework/Headers/RTCFileLogger.h
ios/WebRTC.framework/Headers/RTCLogging.h
I run xargs with parralel mode on - this does not find the required text "192":
cat tmpfilelist | \xargs -P0 -t -I {} \bash -c "\grep -C 2 -H -I -r 192 {}" |& \grep -C 3 "192 gg"
bash -c '\grep -C 2 -H -I -r 192 android/app/src/main/res/drawable-xhdpi/src_assets_images_ic_biggames.png'
bash -c '\grep -C 2 -H -I -r 192 android/app/src/main/res/drawable-xxhdpi/src_assets_images_alerts_iconnotificationsmartpluggreen.png'
bash -c '\grep -C 2 -H -I -r 192 android/app/src/main/res/drawable-xxhdp/src_assets_images_ic_biggames.png'
bash -c '\grep -C 2 -H -I -r 192 gg'
bash -c '\grep -C 2 -H -I -r 192 ios/WebRTC.framework/Headers/RTCCallbackLogger.h'
bash -c '\grep -C 2 -H -I -r 192 ios/WebRTC.framework/Headers/RTCFileLogger.h'
bash -c '\grep -C 2 -H -I -r 192 ios/WebRTC.framework/Headers/RTCLogging.h'
When I disable parralel mode it succesfully finds the text "192" in file "gg":
cat tmpfilelist | \xargs -t -I {} \bash -c "\grep -C 2 -H -I -r 192 {}" |& \grep -C 3 "192 gg"
bash -c '\grep -C 2 -H -I -r 192 android/app/src/main/res/drawable-xhdpi/src_assets_images_ic_biggames.png'
bash -c '\grep -C 2 -H -I -r 192 android/app/src/main/res/drawable-xxhdpi/src_assets_images_alerts_iconnotificationsmartpluggreen.png'
bash -c '\grep -C 2 -H -I -r 192 android/app/src/main/res/drawable-xxhdp/src_assets_images_ic_biggames.png'
bash -c '\grep -C 2 -H -I -r 192 gg'
gg:192
bash -c '\grep -C 2 -H -I -r 192 ios/WebRTC.framework/Headers/RTCCallbackLogger.h'
bash -c '\grep -C 2 -H -I -r 192 ios/WebRTC.framework/Headers/RTCFileLogger.h'
Any reason why parallel mode would break grep? Or have I made a mistake somwhere?
Many thanks

Answer 1

The issue is indeed due to the parallel execution. The issue is that the output is not deterministic when the you run it in parallel. I create a similar test
bash-5.0# ls -alh
total 32K
drwxr-xr-x 9 root root 288 May 20 17:00 .
drwxr-xr-x 1 root root 4.0K May 20 17:11 ..
-rw-r--r-- 1 root root 3 May 20 17:00 a
-rw-r--r-- 1 root root 3 May 20 17:00 b
-rw-r--r-- 1 root root 3 May 20 17:00 c
-rw-r--r-- 1 root root 4 May 20 17:00 d
-rw-r--r-- 1 root root 3 May 20 17:00 e
-rw-r--r-- 1 root root 3 May 20 17:00 f
-rw-r--r-- 1 root root 11 May 20 17:00 files
bash-5.0# tail -n +1 *
==> a <==
192
==> b <==
193
==> c <==
192
==> d <==
195
==> e <==
200
==> f <==
198
==> files <==
a
b
c
d
e
f
Now if we run your two commands without the last grep the outputs are like below
Without parallel
bash-5.0# cat files | \xargs -t -I {} \bash -c "\grep -C 2 -H -I -r 192 {}"
bash -c \grep -C 2 -H -I -r 192 a
a:192
bash -c \grep -C 2 -H -I -r 192 b
bash -c \grep -C 2 -H -I -r 192 c
c:192
bash -c \grep -C 2 -H -I -r 192 d
bash -c \grep -C 2 -H -I -r 192 e
bash -c \grep -C 2 -H -I -r 192 f
With Parallel
bash-5.0# cat files | \xargs -P4 -t -I {} \bash -c "\grep -C 2 -H -I -r 192 {}"
bash -c \grep -C 2 -H -I -r 192 a
bash -c \grep -C 2 -H -I -r 192 b
bash -c \grep -C 2 -H -I -r 192 c
bash -c \grep -C 2 -H -I -r 192 d
bash -c \grep -C 2 -H -I -r 192 e
a:192
bash -c \grep -C 2 -H -I -r 192 f
c:192
Hope you can see how it is affecting the output and orders of your line. So your issue is that when you do grep -C 3 192 gg, you are supposed to get 3 lines before printing of 192 gg and 3 after, which you actually do get.
But gg:192 is printed sometime later because of each command sending their output in parallel on the same output terminal

Answer 2

Not an answer - but pointing to possible problem.
The code runs two filters in pipeline
grep -C 2 -H -I -r 192 FILENAMe
grep -C 3 "192 gg"
The output from the first line will follow the format 'FILENAME:DATA'. In the example above
gg:192
The second filter is not going to find the pattern '192 gg'.
So the question may be how does the non-parallel produces any output ?

Answer 3

Short answer
Your final grep -C 3 requests 3 lines of context around the 192 gg. In the parallel case, this might or might not be enough to find the line containing gg:192.
Details
Before the final grep, your output will consist of a number of lines like:
bash -c \grep -C 2 -H -I -r 192 <filename>
which are echoed to stderr just before xargs launches each bash -c ... command, and the line
gg:192
is echoed to stdout when the relevant bash -c ... command (i.e. the one involving the file gg) finds the match.
In the parallel case, the whole output that is piped into your the final grep (once stderr and stdout are combined using |&) might look something like this, where I have replaced the other filenames for brevity:
bash -c \grep -C 2 -H -I -r 192 some_file
bash -c \grep -C 2 -H -I -r 192 some_other_file
bash -c \grep -C 2 -H -I -r 192 another_file
bash -c \grep -C 2 -H -I -r 192 gg
bash -c \grep -C 2 -H -I -r 192 and_another_file
bash -c \grep -C 2 -H -I -r 192 yet_another_file
gg:192
bash -c \grep -C 2 -H -I -r 192 and_yet_another_file
In this example, the bash -c ... commands involving and_another_file and yet_another_file were launched after the one involving gg was launched, but before the one involving gg wrote its output (or at least, before any stdio buffers associated with that output were flushed), so they appear between the lines containing 192 gg and gg:192.
The number of such intervening lines between the lines containing 192 gg and the gg:192 (in this example, 2) will depend on timings and the number of other parallel tasks being launched after the one involving gg. This will vary, and for example if you inserted a sleep statement (e.g. ... \bash -c "sleep 1; \grep -C ...) then there would tend to be more such lines. In any case, you are then piping it to a grep -C 3 to extract 3 lines of context. If it so happens that there are fewer than 3 intervening lines, then this grep -C will find the line containing gg:192, but if there are 3 or more, then it will be outside the requested amount of context and will not be in the final output.
In the serial case, however, the gg:192 line is guaranteed to always be immediately after the 192 gg line, as follows:
bash -c \grep -C 2 -H -I -r 192 some_file
bash -c \grep -C 2 -H -I -r 192 some_other_file
bash -c \grep -C 2 -H -I -r 192 another_file
bash -c \grep -C 2 -H -I -r 192 gg
gg:192
bash -c \grep -C 2 -H -I -r 192 and_another_file
bash -c \grep -C 2 -H -I -r 192 yet_another_file
bash -c \grep -C 2 -H -I -r 192 and_yet_another_file
and so it is always within the 3 lines of context.