"awk" is not processing shell variables as expected - linux

I am working on filtering Specific text from a log file. Problem here is that awk is not processing on shell variable. But working fine on filename.
I am storing a new log entry that comes in log file in a shell variable using "new_log=tail -n5 alerts.log" in a loop whenever a new log comes, then ,
Level_no=`awk '{FS="Rule: "}{print $2}' "$new_log" | sed '/^$/d' | awk '{FS=" "}{print $3}' | sed 's/)//g'
Output:
awk: fatal: cannot open file `** Alert 1564460779.1380: mail - ossec,syscheck
* New Log starts from ** Alert 1564460779.1380: mail - ossec,syscheck *`
Above mentioned command works well when I run it in terminal using filename instead of shell variable as follows:
awk '{FS="Rule: "}{print $2}' logs_mining | sed '/^$/d' | awk '{FS=" "}{print $3}' | sed 's/)//g'
But its performance issue if I store new log entry in another file and process from there.
So I researched more and more and came to know about awk variables...here is my shell script..
Level_no=`awk -v var="$new_log" '{FS="Rule: "}{print $2}' var | sed '/^$/d' | awk '{FS=" "}{print $3}' | sed 's/)//g'
Then output says
awk: fatal: cannot open file `var' for reading (No such file or directory)
Actual Result should be Successful execution of awk script.

If new_log contains the data you want to process, not a filename, you need to pipe it to awk. You can do this with a here-string.
Level_no=`awk '{FS="Rule: "}{print $2}' <<<"$new_log" | sed '/^$/d' | awk '{FS=" "}{print $3}' | sed 's/)//g'`
It's also not necessary to pipe the output to sed and another awk, you can do it all in the first script.
Level_no=$(awk -F'Rule: ' '$2 != "" {split($2, a, " "); gsub(/)/, "", a[3]); print a[3]}' <<<"$new_log")
You probably don't need the variable, though, just pipe the output of the command to awk:
Level_no=$(tail -n5 alerts.log | awk -F'Rule: ' '$2 != "" {split($2, a, " "); gsub(/)/, "", a[3]); print a[3]}')

Related

Why is my linux command with awk not working

i am trying to let this command work but it won't let me do anything
awk -F: ‘{if($3>'1000') print$1}’ passwd | sort > users.txt
I get an error which is saying:
bash: syntaxfout nabij onverwacht symbool '('
Can someone help me out?
You're using ‘ instead of '. And then, you should replace ' with " in the awk program (or just leave them out):
awk -F: '{if ($3 > 1000) print $1}' passw | ...
You're using backticks instead of single quotes. Try:
awk -F: '{if($3>1000) print $1} passwd | sort > users.txt
or just
awk -F: '$3>1000 {print $1}' passwd | sort > users.txt

Pipe to AWK ignore line containing Unknown host

How could I tell AWK to ignore line that contains "Unknown host":
user1#ubuntu:~$ gethostip -d blabla | awk '{print $1;exit}'
blabla: Unknown host
user1#ubuntu:~$
Essentially, I want it to return nothing if "Unknown hosts" is contained in the line.
Use a condition before the block to test the contents of the line:
gethostip -d blabla | awk '!/Unknown host/ {print $1;exit}'
The error message goes to standard error, not standard output. If you want to discard it, redirect it to nowhere:
gethostip -d blabla 2>/dev/null | awk '{print $1;exit}'
If you want to process it, you can redirect stderr to stdout so awk sees both:
gethostip -d blabla 2>&1 | | awk '(/Unknown host/){print "Error";exit}{print $1}'
You should be able to simply do:
awk '$0 !~ /Unknown host/'

awk - Delimiter as combination of number and | (pipe) not working

I have an input file with some records as below,
input.txt
Record|111|aaa|aaa|11|1-bb|bb|1111|cccc|cccc
Record|11|1-aaa|aaa|111|bb|bb|1111|cccc|cccc
Record|111|aaa|aaa|11|1-bb|bb|1111|cccc|cccc
Record|111|aaa|aaa|111|bb|bb|11|1-cccc|cccc
Record|22|aaa|aaa|222|bb|bb|2222|cccc|cccc|11|1-dddd|dd
Record|333|aaa|aaa|11|1-bb|bb|333|cccc|cccc
Record|11|1-aaa|aaa|102|bb|bb|1111|cccc|cccc
i want to use a delimiter |11| in awk and get the second field, i tried the most common way as below,
Command
awk -F'|11|' '{print $2}' input.txt
Output
1|aaa|aaa|
|1-aaa|aaa|
1|aaa|aaa|
1|aaa|aaa|
|1-dddd|dd
|1-bb|bb|333|cccc|cccc
|1-aaa|aaa|102|bb|bb|
Expected Output
1-bb|bb|1111|cccc|cccc
1-aaa|aaa|111|bb|bb|1111|cccc|cccc
1-bb|bb|1111|cccc|cccc
1-cccc|cccc
1-dddd|dd
1-bb|bb|333|cccc|cccc
1-aaa|aaa|102|bb|bb|1111|cccc|cccc
Basically its not considering the last | of the delimiter |11|, instead it is taking a delimiter |11.
i tried all below, none gave me the expected output,
awk -F"|11|" '{print $2}' input.txt # gives wrong output
awk -F\|11\| '{print $2}' input.txt # gives Wrong output
awk -v FS='|11|' '{print $2}' input.txt # gives Wrong output
Finally i had to write a for loop inside awk with delimiter as | to make it work, i would like to know why the simple solution doesn't work
Argument to -F is a regex.
awk -F "\\\|11\\\|" '{print $2}' file
or
awk -F '\\|11\\|' '{print $2}' file
or (Thanks to EdMorton)
awk -F'[|]11[|]' '{print $2}' input.txt
Output:
1-bb|bb|1111|cccc|cccc
1-aaa|aaa|111|bb|bb|1111|cccc|cccc
1-bb|bb|1111|cccc|cccc
1-cccc|cccc
1-dddd|dd
1-bb|bb|333|cccc|cccc
1-aaa|aaa|102|bb|bb|1111|cccc|cccc
Cyrus explained why your delimiter does not work as expected (a combination of regular expression quoting issues).
With sed, removing everything up to and including the |11| on each line:
$ sed 's/.*|11|//' input.txt
1-bb|bb|1111|cccc|cccc
1-aaa|aaa|111|bb|bb|1111|cccc|cccc
1-bb|bb|1111|cccc|cccc
1-cccc|cccc
1-dddd|dd
1-bb|bb|333|cccc|cccc
1-aaa|aaa|102|bb|bb|1111|cccc|cccc

How to save the output of this awk command to file?

I wanna save this command to another text:
awk '{print $2}'
it extract's from text.
now i wanna save output too another text.
thanks
awk '{ print $2 }' text.txt > outputfile.txt
> => This will redirect STDOUT to a file. If file not exists, it will create it. If file exists it will clear out (in effect) the content and will write new data to it
>> => This means same as above but if file exists, this will append new data to it.
Eg:
$ cat /etc/passwd | awk -F: '{ print $1 }' | tail -10 > output.txt
$ cat output.txt
_warmd
_dovenull
_netstatistics
_avbdeviced
_krb_krbtgt
_krb_kadmin
_krb_changepw
_krb_kerberos
_krb_anonymous
_assetcache
Alternatively you can use the command tee for redirection. The command tee will redirect STDOUT to a specified file as well as the terminal screen
For more about shell redirection goto following link:
http://www.techtrunch.com/scripting/redirections-and-file-descriptors
There is a way to do this from within awk itself (docs)
➜ cat text.txt
line 1
line 2
line three
line 4 4 4
➜ awk '{print $2}' text.txt
1
2
three
4
➜ awk '{print $2 >"text.out"}' text.txt
➜ cat text.out
1
2
three
4
try this command.
cat ORGFILENAME.TXT | awk '{print $2}' > SAVENAME.TXT
thx.

perl linux command not working

cat t.incopt.02.20110221 | awk -F, '{print $1}' | sort | uniq
got unque records
but if i inserted into perl,
#FETCH_REQ_DETAILS = `cat t.incopt.02.20110221 | awk -F\, '{print $1}' \| sort \| uniq`;
if i print the above array vari, i getting entire file content, i guess the linux command not working correctly when i use inside perl,
I think you just need to enclose the command in back tick and escape only the $
#FETCH_REQ_DETAILS = `cat t.incopt.02.20110221 | awk -F, '{print \$1}' | sort | uniq;`
Try the following:
my $cmd='cat t.incopt.02.20110221 | awk -F, \'{print $1}\' | sort | uniq';
#FETCH_REQ_DETAILS = `$cmd`;

Resources