This question already has answers here:
Piping tail output though grep twice
(2 answers)
Closed 4 years ago.
Here is my sample log file.http://pastebin.com/DwWeFhJk.
When I am doing
tail -f log | awk '{if (NF>3) {print $1}; }'
the result I am getting is correct
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
But when I am doing:
tail -f log |
awk '{if (NF>3) {print $1}; }' |
awk '{print $1}'
I am not getting any output. Even no output in case of
tail -f log | awk '{if (NF>3) {print $1}; }' | grep "64"
I am not getting the reason why the output of the first awk is not getting passed as the input of the second awk/grep after the pipe.
When the output of the first awk is going to the terminal, the output is line-buffered, so each line is printed as it is produced. When the output is going to the second awk or the grep, it is fully buffered. The output won't be sent until the buffer is full. When enough extra records are appended to the log, then the second awk will a buffer full of data to process. Until then, nothing will happen.
You start the command with tail -f, that keeps the output open and therefore does not send a needed newline to the other commands.
This works perfectly fine:
cat log | awk '{if (NF>3) {print $1}; }' | grep 64
So, the problem is buffering. The middle awk is doing normal buffering instead of interactive buffering. This works (non-portably) with mawk:
tail -f log | mawk -W interactive '{if (NF>3) {print $1}; }' | awk '{print}'
You could read GNU description of the issue.
In any case, just check that the awk used in the middle could be told to buffer interactively.
Added:
The command system("") seems to unblock the buffering. It is POSIX, but does not work with mawk.
tail -f log | awk '{if (NF>3) {print $1}; system("")}' | awk '{print}'
search for "parallel --pipe" in the link to avoid the buffering
https://www.gnu.org/software/parallel/parallel_tutorial.html
Related
I am running the following command in a bash script:
echo `netstat -plten | grep -i autossh | awk '{print $4}'` >> /root/logs/autossh.txt
The output displays in a single line:
127.0.0.1:25001 127.0.0.1:15501 127.0.0.1:10001 127.0.0.1:20501 127.0.0.1:15001 127.0.0.1:5501 127.0.0.1:20001
I would like each IP to display line by line. What do I need to do with the awk command to make the output display line by line
Just remove the echo and subshell:
netstat -plten | grep -i autossh | awk '{print $4}' >> /root/logs/autossh.txt
awk is already printing them one per line, but when you pass them to echo it parses its arguments and prints them each with a space between them. Every line of awk output then becomes a separate argument to echo so you lose your line endings.
Of course, awk can do pattern matching too, so no real need for grep:
netstat -plten | awk '/autossh/ {print $4}' >> /root/logs/autossh.txt
with gawk at least you can have it ignore case too
netstat -plten | awk 'BEGIN {IGNORECASE=1} /autossh/ {print $4}' >> /root/logs/autossh.txt
or as Ed Morton pointed out, with any awk you could do
netstat -plten | awk 'tolower($0) ~ /autossh/ {print $4}' >> /root/logs/autossh.txt
You can just quote the result of command substitution to prevent the shell from performing word splitting.
You can modify it as follows to achieve what you want.
echo "`netstat -plten | grep -i autossh | awk '{print $4}'`" >> /root/logs/autossh.txt
I have what I am assuming to be a very simple problem. I have used the following command in scripts many times before without issues.
display=/some/file/with/data
sed -i "s/$(more ${display} | grep 110 | gawk '{ print $3 }')/"replacestring"/g" ${display}
Never had an issue with this command.
I created a script that simplified everything that was needed and have the following code:
displayText=$(more ${display} | grep 110 | gawk '{ print $3 }')
I keep receiving
line 2: syntax error: unexpected end of file.
I am very familiar with Linux so the problem has been quite irritating as the answer has escaped me. Please assist.
Not sure why you are using more.
You also have backquotes around the awk script where you should have single quotes.
displayText=$(cat ${display} | grep 110 | gawk '{ print $3 }')
You will be told that this is a useless-use-of-cat (the cat command isn't needed since grep can read a file). You can just do:
displayText=$(grep 110 ${display} | gawk '{ print $3 }')
As awk can also search, this can be simplified further into:
displayText=$(gawk '/110/ {print $3}' ${display})
Having trouble to wrap my head around piping and potential buffering issue. I am trying to perform set of operations piped that seem to break at some piping level. To simplify , I narrowed it down to 3 piping operations that do not work correctly
tail -f | awk '{print $1}' > file
results in no data redirected to the file , however
tail -f | awk '{print $1}'
results are output to stdout fine
also
tail -10 | awk '{print $1}' > file
works fine as well.
thinking it might be buffering issue, tried
tail -f | unbuffer awk '{print $1}' > file
what produced no positive results
(note: in original request, i have more operation in between using grep --line-buffer, but the problem was narrowed down to 3 piped commands tail -f | awk > file
The following will tail -f on a given file and whenever new data is added will automatically execute the while loop:
tail -f file_to_watch | while read a; do echo "$a" |awk '{print $1}' >> file; done
or more simply if you really only need to print the first field you could read it directly to your variable like this:
tail -f file_to_watch | while read a b; do echo "$a" >> file; done
Here is how to handle log files:
tail --follow=name logfile | awk '{print $1 | "tee /var/log/file"}'
or for you this may be ok:
tail -f | awk '{print $1 | "tee /var/log/file"}'
--follow=name this prevents stop of command while log file are rolled.
| "tee /var/log/file" this is used to get the output to the file.
I am trying print the first field of the first row of an output. Here is the case. I just need to print only SUSE from this output.
# cat /etc/*release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 2
Tried with cat /etc/*release | awk {'print $1}' but that print the first string of every row
SUSE
VERSION
PATCHLEVEL
Specify NR if you want to capture output from selected rows:
awk 'NR==1{print $1}' /etc/*release
An alternative (ugly) way of achieving the same would be:
awk '{print $1; exit}'
An efficient way of getting the first string from a specific line, say line 42, in the output would be:
awk 'NR==42{print $1; exit}'
Specify the Line Number using NR built-in variable.
awk 'NR==1{print $1}' /etc/*release
try this:
head -1 /etc/*release | awk '{print $1}'
df -h | head -4 | tail -1 | awk '{ print $2 }'
Change the numbers to tweak it to your liking.
Or use a while loop but thats probably a bad way to do it.
You could use the head instead of cat:
head -n1 /etc/*release | awk '{print $1}'
sed -n 1p /etc/*release |cut -d " " -f1
if tab delimited:
sed -n 1p /etc/*release |cut -f1
Try
sed 'NUMq;d' /etc/*release | awk {'print $1}'
where NUM is line number
ex. sed '1q;d' /etc/*release | awk {'print $1}'
awk, sed, pipe, that's heavy
set `cat /etc/*release`; echo $1
the most code-golfy way i could think of to print first line only in awk :
awk '_{exit}--_' # skip the quotations and make it just
# awk _{exit}--_
#
# if u're feeling adventurous
first pass through exit block, "_" is undefined,
so it fails and skips over for row 1.
then the decrementing of the same counter will make
it "TRUE" in awk's eyes (anything not empty string
or numeric zero is considered "true" in their agile boolean sense). that same counter also triggers default action of print for row 1.
—- incrementing… decrementing… it's same thing,
merely direction and sign inverted.
then finally, at start of row 2, it hits criteria to
enter the action block, which instructs it to instantly
exit, thus performing essentially the same functionality as
awk '{ print; exit }'
… in a slightly less verbose manner. For a single line print, it's not even worth it to set FS to skip the field splitting part.
using that concept to print just 1st row 1st field :
awk '_{exit} NF=++_'
awk '_++{exit} NF=_'
awk 'NR==1&&NF=1' file
grep -om1 '^[^ ]\+' file
# multiple files
awk 'FNR==1&&NF=1' file1 file2
You can kill the process which is running the container.
With this command you can list the processes related with the docker container:
ps -aux | grep $(docker ps -a | grep container-name | awk '{print $1}')
Now you have the process ids to kill with kill or kill -9.
Something like this will hang:
tail -f textfile | awk '{print $0}'
while grep won't hang when used instead of awk.
My actual intention is to add color to some log output using merely standard commands; however it seems that piping tail -f into awk won't work. I don't know if it's a buffer problem, but I tried some approaches that haven't worked, like:
awk '{print $0;fflush()}'
and also How to pipe tail -f into awk
Any ideas?
I ran into almost exactly the same problem with mawk. I think it is due to the way mawk is flushing its buffer, the problem went away when I switched to gawk. Hope this helps (a bit late I know).
I tried this command :
tail -f test | awk '{print $0;}'
And it doesn't hang. Awk will print the new values each time I add something in the test file.
echo "test" >> test
I think you just forgot a quote in your command because you wrote (edit : well, before your post was edited) :
tail -f textfile | awk {print $0}'
Instead of :
tail -f textfile | awk '{print $0}'