Hi I want to know how to use user defined variables inside egrep with or condition.
If i use below command i am getting result as required.
grep -w "Connect" audit.log.13766735635311490 | egrep 'USER=\"root\"|DB=\"CUST_PRESTIGEWW_DB\"' | wc -l
24416
now i have done below and executed the same command but getting different result
DATABASE=CUST_PRESTIGEWW_DB
grep -w "Connect" audit.log.13766735635311490 | egrep 'USER=\"root\"|DB=\"$DATABASE\"' | wc -l
40
grep -w "Connect" audit.log.13766735635311490 | egrep 'USER=\"root\"|DB=\""$DATABASE"\"' | wc -l
40
echo $DATABASE
CUST_PRESTIGEWW_DB
Now tried changing the variable value but not worked
DATABASE=`echo "\"CUST_PRESTIGEWW_DB\""`
echo $DATABASE
"CUST_PRESTIGEWW_DB"
grep -w "Connect" audit.log.13766735635311490 | egrep 'USER=\"root\"|DB=$DATABASE' | wc -l
grep -w "Connect" audit.log.13766735635311490 | egrep 'USER=\"root\"|DB='"$DATABASE" | wc -l
119956
grep -w "Connect" audit.log.13766735635311490 | egrep 'USER=\"root\"|DB="'$DATABASE'"' | wc -l
114
Can any one help me how to do it ?
Use:
... egrep 'USER="root"|DB='"$DATABASE" ...
You need to close the single quoted section to make bash expand $DATABASE. Further note that you won't need to escape double quotes in single quoted string.
Related
$ grep HxH 20170213.csv | awk -F',' '{print $13}' | cut -b 25-27 | sort -u
868
881
896
904
913
914
918
919
920
Question> How to pipe the sorted results and feed into grep?
Now I have to do the following command manually.
grep 868 /tmp/aaa/*.csv
grep 881 /tmp/aaa/*.csv
...
grep 920 /tmp/aaa/*.csv
Since your output is numeric (output lines do not contain spaces), you can use a for loop with command substitution:
for id in $(grep HxH 20170213.csv | awk -F',' '{print $13}' \
| cut -b 25-27 | sort -u); do
grep $id /tmp/aaa/*.csv
done
Another option is to use xargs:
grep HxH 20170213.csv | awk -F',' '{print $13}' | cut -b 25-27 | sort -u \
| xargs -n1 grep /tmp/aaa/*.csv -e
The xargs variant requires one to jump through a couple hoops to get right:
by default xargs would stick more than one pattern to the same grep, which is prevented using -n1;
xargs specifies the stdin contents as the last argument in the command line, which is a problem because grep expects pattern then file name. Fortunately, grep PATTERN FILES... can be spelled as grep FILES... -e PATTERN, which is why grep must be followed by -e.
I have logs in redmine, about users, that have connected from ip of my server. I can do this via command:
tail -n 100000 /usr/share/redmine/log/production.log | grep -A 3 "192.168.110.50" | grep "Current user" | awk '{print $3}' | head -n 1 | tail -n 1
I need to write this parameter in variable.
No problem:
temp=$(tail -n 100000 /usr/share/redmine/log/production.log | grep -A 3 "192.168.110.50" | grep "Current user" | awk '{print $3}' | head -n 1 | tail -n 1)
It works. But it can return user name anonymous. To return other user name i should write head -n 2. If it still anonymous, i can change in my formula to head -n 3.
So ... of course i can do this work
#!/bin/bash
temp=$(tail -n 100000 /usr/share/redmine/log/production.log | grep -A 3 "192.168.110.50" | grep "Current user" | awk '{print $3}' | head -n 1 | tail -n 1)
if [[ $temp == "anonymous" ]]; then
temp=$(tail -n 100000 /usr/share/redmine/log/production.log | grep -A 3 "192.168.110.50" | grep "Current user" | awk '{print $3}' | head -n 2 | tail -n 1)
fi
But it will work for one iteration. I tried to:
while [ $tmp != "anonymous" ]; do
temp=$(tail -n 100000 /usr/share/redmine/log/production.log | grep -A 3 "192.168.110.50" | grep "Current user" | awk '{print $3}' | head -n ((var=var+1)) | tail -n 1)
done
But of course it does not work. I can't understand just logically, how can i do this ? Can you help me ? Thanks for your attention.
The immediate problem is that you're setting the variable temp, but checking tmp. You also need double-quotes around the variable in case it's blank, a wildcard, or something else troublesome.
But there's an easier way, because awk can handle most of the work by itself:
tail -n 100000 /usr/share/redmine/log/production.log |
grep -A 3 "192.168.110.50" |
awk '/Current user/ {if ($3 != "anonymous") {print $3; exit}}'
It'd actually be possible to have awk handle looking at the lines after "192.168.110.50" too, but that'd be a little more complicated. BTW, since I don't have any redmine logs (let alone yours) to test with, this has not been properly tested.
You can also use grep -v :
temp=$(tail -n 100000 /usr/share/redmine/log/production.log | grep -A 3 "192.168.110.50" | grep "Current user" | grep -v "anonymous" | awk '{print $3}' | head -n 1 )
Note you don't need final tail -n 1 after head -n 1
I'd like to grep for 1:N:0:CGATGT within a file and extract the line containing 1:N:0:CGATGT and 3 additional lines after (4 lines total for each match). I've tired to grep numerous ways, all yielding unsuccessful:
[ssabri#login2 data]$ history | tail -n 8
1028 zcat A1_S1_L008_R1_001.fastq.gz | grep -A4 "1[[:]][[N]][[:]]0[[:]]CGATGT" | wc -l
1029 zcat A1_S1_L008_R1_001.fastq.gz | grep -A4 "1[[:]][[N]][[:]]0[[:]]CGATGT$" | wc -l
1030 zcat A1_S1_L008_R1_001.fastq.gz | grep -A4 "1[[:]][[N]][[:]][[0]][[:]]CGATGT$" | wc -l
1031 zcat A1_S1_L008_R1_001.fastq.gz | grep -A4 -w "1[[:]][[N]][[:]][[0]][[:]]CGATGT$" | wc -l
1032 zcat A1_S1_L008_R1_001.fastq.gz | egrep -A4 -w "1[[:]][[N]][[:]][[0]][[:]]CGATGT$" | wc -l
1033 zcat A1_S1_L008_R1_001.fastq.gz | grep -x -A4 -w "1:N:0:CGATGT" | wc -l
1034 zcat A1_S1_L008_R1_001.fastq.gz | grep -E -A4 -w "1:N:0:CGATGT" | wc -l
1035 zcat A1_S1_L008_R1_001.fastq.gz | grep -A4 -w "1\:N\:0\:CGATGT$" | wc -l
EDIT: The input files looks something like this:
[ssabri#login2 data]$ zcat A1_S1_L008_R1_001.fastq.gz | head -n 12
#J00153:28:H7LNWBBXX:8:1101:28625:1191 1:N:0:CGAGGT
ACNTGCTCCATCCATAGCACCTAGAACAGAGCCTGGNACAGAANAAGNGC
+
A-#<-<<FJJAJFFFF-FJJJJJAJFJJJFF-A-FA#JJJJFJ#JJA#FJ
#J00153:28:H7LNWBBXX:8:1101:29457:1191 1:N:0:CGATGT
GTNGTGGTAGATCTGGACGCGGCTGAAGGCCTGGGGNCCCGTGNCAGN
+
-<#<FJJJJJJJJJJJJJJJJJJJJJJFJJJJJJJJ#JJJJJJ#JJJ#
#J00153:28:H7LNWBBXX:8:1101:31000:1191 1:N:0:CCATGT
TCNAATTATCACCATTACAGGAGGGTCAGTAGAACANGCGTTCTGGTNGG
+
<A#<AFFJJJFJJJFJJJJJJFFFJ7A<<JJFJJJJ#JJJAFJJJJF#-A
grep -A3 "1:N:0:CGATGT" file
#J00153:28:H7LNWBBXX:8:1101:29457:1191 1:N:0:CGATGT
GTNGTGGTAGATCTGGACGCGGCTGAAGGCCTGGGGNCCCGTGNCAGN
+
-<#<FJJJJJJJJJJJJJJJJJJJJJJFJJJJJJJJ#JJJJJJ#JJJ#
sometimes simpler thinking is better, here you don't need any regex extensions since you're string matching without any special regex chars that will need escaping. A(fter) context match should be 3 since you want 3 trailing lines (total will be 4 with matching line).
I understand that you are looking for a grep solution. However this is not the only option in text processing. If you use awk then this might be a solution:
awk ' BEGIN {line=4;} \
/1:N:0:CGATGT/ {line=0; print $0; next;} \
{ if (line<3) { print $0; line = line+1; } } ' your-file
Given the problem you seem to be having with using grep and pulling out a fixed 4 lines, try this:
$ awk 'NF>1{f=0} $NF=="1:N:0:CGATGT"{f=1} f' file
#J00153:28:H7LNWBBXX:8:1101:29457:1191 1:N:0:CGATGT
GTNGTGGTAGATCTGGACGCGGCTGAAGGCCTGGGGNCCCGTGNCAGN
+
-<#<FJJJJJJJJJJJJJJJJJJJJJJFJJJJJJJJ#JJJJJJ#JJJ#
Rather than printing a fixed number of lines after a match, it will print from the first line where the last field is your target string to just before the next line that COULD contain your target string.
To identify any blocks that have some number other than 4 lines including the target, use this:
$ awk 'f && NF>1{ if (f!=5) print NR, f-1 | "cat>&2"; f=0} $NF=="1:N:0:CGATGT"{f=1} f{print; ++f}' file
It will output to stderr the input file line number and the count of the number of lines in the unexpected block.
I'm trying to run a shell script using cron every 15 & 45 mins of the hour. But for some vague reasons it always produces empty strings while executed by cron whereas in when i run using terminals ./my_script.sh it produces expected results. I read many of answers relating to this questions, but none could solve my problem.
codes:
#!/bin/bash
PATH=/bin:/home/mpadmin/bin:/opt/ant/bin:/opt/cc/bin:/opt/cvsconsole:/opt/cvsconsole:/opt/cvsconsole:/sbin:/usr/bin:/usr/lib/qt-3.3/bin:/usr/local/bin:/usr/local/sbin:/usr/sbin:/var/mp/95930/scripts:/var/mp/95930/opt/bin:/opt/prgs/dlc101c/bin:/opt/cvsconsole
tail -n 1000000 conveyor2.log | grep -P 'curingresult OK' | sed 's/FT\ /FT/g' |awk '{print $5 $13}' |sed 's/\"//g' | uniq | sort -n |uniq > /var/www/html/95910/master_list.txt
tail -n 1000000 registration.log | grep -P 'TirePresent: 8' | sed 's/GT\ /GT/g' |awk '{print $7 $15}' |sed 's/\"//g' | uniq | sort -n |uniq > /var/www/html/95910/TBM_list.txt
my cron
# run 15 and 45 mins every hour, every day
15,47 * * * * sh /var/mp/95910/log/update_master_list.sh
permissions:
all files are having read write and execute permissions for all users
Hope I have given all the relevant & necessary infos
Probably you need to change to the /var/mp/95910/log/ directory first...
#!/bin/bash
cd /var/mp/95910/log/
PATH=/bin:/home/mpadmin/bin:/opt/ant/bin:/opt/cc/bin:/opt/cvsconsole:/opt/cvsconsole:/opt/cvsconsole:/sbin:/usr/bin:/usr/lib/qt-3.3/bin:/usr/local/bin:/usr/local/sbin:/usr/sbin:/var/mp/95930/scripts:/var/mp/95930/opt/bin:/opt/prgs/dlc101c/bin:/opt/cvsconsole
tail -n 1000000 conveyor2.log | grep -P 'curingresult OK' | sed 's/FT\ /FT/g' |awk '{print $5 $13}' |sed 's/\"//g' | uniq | sort -n |uniq > /var/www/html/95910/master_list.txt
tail -n 1000000 registration.log | grep -P 'TirePresent: 8' | sed 's/GT\ /GT/g' |awk '{print $7 $15}' |sed 's/\"//g' | uniq | sort -n |uniq > /var/www/html/95910/TBM_list.txt
or specify the file paths explicitly
#!/bin/bash
PATH=/bin:/home/mpadmin/bin:/opt/ant/bin:/opt/cc/bin:/opt/cvsconsole:/opt/cvsconsole:/opt/cvsconsole:/sbin:/usr/bin:/usr/lib/qt-3.3/bin:/usr/local/bin:/usr/local/sbin:/usr/sbin:/var/mp/95930/scripts:/var/mp/95930/opt/bin:/opt/prgs/dlc101c/bin:/opt/cvsconsole
tail -n 1000000 /var/mp/95910/log/conveyor2.log | grep -P 'curingresult OK' | sed 's/FT\ /FT/g' |awk '{print $5 $13}' |sed 's/\"//g' | uniq | sort -n |uniq > /var/www/html/95910/master_list.txt
tail -n 1000000 /var/mp/95910/log/registration.log | grep -P 'TirePresent: 8' | sed 's/GT\ /GT/g' |awk '{print $7 $15}' |sed 's/\"//g' | uniq | sort -n |uniq > /var/www/html/95910/TBM_list.txt
So i've been goofing with this since last night and I can get a lot of things to happen just not what I want.
I need a code to find the file with the most lines in a directory and then print the name of the file and the number of lines that file has.
I can get the entire directory's lines to print but can't seem to narrow the field so to speak.
Any help for a fool of a learner?
wc -l $1/* 2>/dev/null
| grep -v ' total$'
| sort -n -k1
| tail -1l
After some pro help in another question, this is where I got to, but it returns them all, and doesn't print their line counts.
Following awk command should do the job for you and you can avoid all redundant piped commands:
wc -l $1/* | awk '$2 != "total"{if($1>max){max=$1;fn=$2}} END{print max, fn}'
UPDATE: To avoid last line of wc's output this might be better awk command:
wc -l $1/* | awk '{arr[cnt++]=$0} END {for (i=0; i<length(arr)-1; i++)
{split(arr[i], a, " "); if(a[1]>max) {max=a[1]; fn=a[2]}} print max, fn}'
you can try:
wc -l $1/* | grep -v total | sort -g | tail -1
actually to avoid the grep that would also remove files containing "total":
for f in $1/*; do wc -l $f; done | sort -g | tail -1
or even better, as suggested in comments:
wc -l $1/* | sort -rg | sed -n '2p'
you can even make it a function:
function get_biggest_file() {
wc -l $* | sort -rg | sed -n '2p'
}
% ls -l
... 0 Jun 12 17:33 a
... 0 Jun 12 17:33 b
... 0 Jun 12 17:33 c
... 0 Jun 12 17:33 d
... 25 Jun 12 17:33 total
% get_biggest_file ./*
5 total
EDIT2: using the function I gave, you can simply output what you need as follows:
get_biggest $1/* | awk '{print "The file \"" $2 "\" has the maximum number of lines: " $1}'
EDIT: if you tried to write the function as you've written it in the question, you should add line continuation character at the end, as follows, or your shell will think you're trying to issue 4 commands:
wc -l $1/* 2>/dev/null \
| grep -v ' total$' \
| sort -n -k1 \
| tail -1l