Read from a endless pipe bash [duplicate] - linux

Question

This question already has answers here:
How do you pipe input through grep to another utility?
(3 answers)
Closed 7 years ago.
I want to create a script that runs another script for each line that the first one receives piping.
Like this:
journalctl -f | myScript1.sh
this myScript1.sh will run another one like this:
./myScript2.sh $line_in_pipe
Problem I found is every code I tested just runs well in a finite pipe (till EOF).
But when I pipe programs like tail -f or others it just won't execute. I think it just waits for EOF to execute the loop.
EDIT:
the endless pipe is like this:
tail -f /var/log/apache2/access.log | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | script_ip_check.sh
so the idea on script_ip_check.sh is doing something like this:
#!/bin/bash
for line in $(cat); do
echo "process:$line"
nmap -sV -p1234 --open -T4 $line | grep 'open' -B3 | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' >> list_of_ip_mapped &
done
for each line, this case IP, I will spawn a thread of nmap to scan something special on that host.
I will use it to scan IPs that tries to connect some "hidden" port on my server.
So my script must runs all the time till I cancel it or it receives an EOF.
EDIT2:
I just found out that grep flushes its buffer so that's why it's not working.
I I use --line-buffered to force grep to output each line as it's being processed.

Answer 1

We can't say definitively without knowing what's in your script.
For instance, if you're doing this:
# DON'T DO THIS: Violates http://mywiki.wooledge.org/DontReadLinesWithFor
for line in $(cat); do
: ...do something with "$line"...
done
...that'll wait until all stdin is available, resulting in the hang you describe.
However, if you're following best practices (per BashFAQ #1), your code will operate more like this:
while IFS= read -r line; do
: ...do something with "$line"
done
...and that'll actually behave properly, subject to any buffering performed by the writer. For hints on controlling buffering, see BashFAQ #9.
Finally, quoting from DontReadLinesWithFor:
The final issue with reading lines with for is inefficiency. A while read loop reads one line at a time from an input stream; $(<afile) slurps the entire file into memory all at once. For small files, this is not a problem, but if you're reading large files, the memory requirement will be enormous. (Bash will have to allocate one string to hold the file, and another set of strings to hold the word-split results... essentially, the memory allocated will be twice the size of the input file.)
Obviously, if the content is indefinite, the memory requirements and completion time are likewise.