I'm after some assistance in getting some stats from an nginx log file. Something is hammering our site and I can see the top ip from this awk command:
sudo awk '{ print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head -n 50
I need to be able to get a list of the urls from this top ip? Can anyone help with the best way to acheive this?
I've got the awk command to lisst the top urls here but need to put them together:
sudo awk '{ print $7}' /var/log/nginx/access.log| sort | uniq -c | sort -nr | head -n 20
Thanks
John
You can use this:
logfile="/var/log/nginx/access.log"
grep "^$(cat "${logfile}" | cut -d' ' -f1 | sort | uniq -c | sort -nr | head -n 1 | awk -F' ' '{print $2}') " "${logfile}" | cut -d' ' -f7 | sort | uniq -c | sort -nr | head -n 50
sudo awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr
Related
I'm trying to run a shell script using cron every 15 & 45 mins of the hour. But for some vague reasons it always produces empty strings while executed by cron whereas in when i run using terminals ./my_script.sh it produces expected results. I read many of answers relating to this questions, but none could solve my problem.
codes:
#!/bin/bash
PATH=/bin:/home/mpadmin/bin:/opt/ant/bin:/opt/cc/bin:/opt/cvsconsole:/opt/cvsconsole:/opt/cvsconsole:/sbin:/usr/bin:/usr/lib/qt-3.3/bin:/usr/local/bin:/usr/local/sbin:/usr/sbin:/var/mp/95930/scripts:/var/mp/95930/opt/bin:/opt/prgs/dlc101c/bin:/opt/cvsconsole
tail -n 1000000 conveyor2.log | grep -P 'curingresult OK' | sed 's/FT\ /FT/g' |awk '{print $5 $13}' |sed 's/\"//g' | uniq | sort -n |uniq > /var/www/html/95910/master_list.txt
tail -n 1000000 registration.log | grep -P 'TirePresent: 8' | sed 's/GT\ /GT/g' |awk '{print $7 $15}' |sed 's/\"//g' | uniq | sort -n |uniq > /var/www/html/95910/TBM_list.txt
my cron
# run 15 and 45 mins every hour, every day
15,47 * * * * sh /var/mp/95910/log/update_master_list.sh
permissions:
all files are having read write and execute permissions for all users
Hope I have given all the relevant & necessary infos
Probably you need to change to the /var/mp/95910/log/ directory first...
#!/bin/bash
cd /var/mp/95910/log/
PATH=/bin:/home/mpadmin/bin:/opt/ant/bin:/opt/cc/bin:/opt/cvsconsole:/opt/cvsconsole:/opt/cvsconsole:/sbin:/usr/bin:/usr/lib/qt-3.3/bin:/usr/local/bin:/usr/local/sbin:/usr/sbin:/var/mp/95930/scripts:/var/mp/95930/opt/bin:/opt/prgs/dlc101c/bin:/opt/cvsconsole
tail -n 1000000 conveyor2.log | grep -P 'curingresult OK' | sed 's/FT\ /FT/g' |awk '{print $5 $13}' |sed 's/\"//g' | uniq | sort -n |uniq > /var/www/html/95910/master_list.txt
tail -n 1000000 registration.log | grep -P 'TirePresent: 8' | sed 's/GT\ /GT/g' |awk '{print $7 $15}' |sed 's/\"//g' | uniq | sort -n |uniq > /var/www/html/95910/TBM_list.txt
or specify the file paths explicitly
#!/bin/bash
PATH=/bin:/home/mpadmin/bin:/opt/ant/bin:/opt/cc/bin:/opt/cvsconsole:/opt/cvsconsole:/opt/cvsconsole:/sbin:/usr/bin:/usr/lib/qt-3.3/bin:/usr/local/bin:/usr/local/sbin:/usr/sbin:/var/mp/95930/scripts:/var/mp/95930/opt/bin:/opt/prgs/dlc101c/bin:/opt/cvsconsole
tail -n 1000000 /var/mp/95910/log/conveyor2.log | grep -P 'curingresult OK' | sed 's/FT\ /FT/g' |awk '{print $5 $13}' |sed 's/\"//g' | uniq | sort -n |uniq > /var/www/html/95910/master_list.txt
tail -n 1000000 /var/mp/95910/log/registration.log | grep -P 'TirePresent: 8' | sed 's/GT\ /GT/g' |awk '{print $7 $15}' |sed 's/\"//g' | uniq | sort -n |uniq > /var/www/html/95910/TBM_list.txt
I would like to get the name of the person who has maximum age in a unix data file. How can I do this?
Rob,20
Tom,30
I tried this as below but it gives me only max age.
awk -F"," '{print $2}' age.txt | sort -r | head -1
$ cat file | awk -F, '{print $2,$1;}' | sort -n | tail -n1
30 Tom
$ cat file | awk -F, '{print $2,$1;}' | sort -n | tail -n1 | awk '{print $2;}'
Tom
Try perhaps
awk -F, '{if (maxage<$2) { maxage= $2; name=$1; };} END{print name}' \
age.txt
traditional:
sort -t, -nr +1 age.txt | head -1 | cut -d, -f1
POSIXy:
sort -t, -k2,2nr age.txt | head -n 1 | cut -d, -f1
i think you can easily do this using below command
echo -e "Rob,20\nTom,30\nMin,10\nMax,50" | sort -t ',' -rk 2 | head -n 1
Please comment in case of any issues.
I want to analyse some data from a named pipe, for example:
cat trace_pipe | awk '{print $6}' | head -100000 | sort | uniq -c |
sort -k 1 -nr
This command will collect first 100k lines of data from trace_pipe. Is there a way for me to collect data for 10 seconds rather than a limited number of lines?
You can use the
timeout
command, you might need to split the commands
timeout 10 cat trace_pipe > trace_pipe.cut
awk '{print $6}' trace_pipe.cut | sort | uniq -c | sort -k 1 -nr
Here is the output of my netstat command. I want to count total of first field number like 7+8+1+1+1+1+3+1+2..so on... How do i use bc or any other method command to total count them?
[root#example httpd]# netstat -natp | grep 7143 | grep EST | awk -F' ' '{print $5}' | awk -F: '{print $1}' | sort -nr | uniq -c
7 209.139.35.xxx
8 209.139.35.xxx
1 209.139.35.xxx
1 209.139.35.xxx
1 208.46.149.xxx
3 96.17.177.xxx
1 96.17.177.xxx
2 96.17.177.xxx
You need to get the first column with awk (You don't actually need this, but I'm leaving it as a monument to my eternal shame)
awk {'print $1'}
and then use awk again to sum the column of numbers and print the result
awk '{ sum+=$1} END {print sum}'
All together:
netstat -natp | grep 7143 | grep EST | awk -F' ' '{print $5}' | awk -F: '{print $1}' | sort -nr | uniq -c | awk {'print $1'} | awk '{ sum+=$1} END {print sum}'
I know this doesn't use bc, but it gets the job done, so hopefully that's enough.
I make few commands to parse values from free -m and now need to output all in one file. It looks like this:
free -m | grep 'Mem' | awk '{print $3'} && free -m | grep 'Mem' | awk '{print $6'} && free -m | grep 'Mem' | awk '{print $7'} && free -m | grep 'cache:' | awk '{print $3'}
if I add > /some/file on end, it write only last value, how to write output from all this commands to one file?
thanks for any help
Add parentheses to open a sub shell:
( free -m | grep 'Mem' | awk '{print $3'} && free -m | grep 'Mem' | awk '{print $6'} && free -m | grep 'Mem' | awk '{print $7'} && free -m | grep 'cache:' | awk '{print $3'} ) > result.txt