I want to use echo and grep statement together. I have tried most of the thing but couldn't get the exact output
as I want.
aa=$(grep -A100000 "2010-03-24" log.txt|grep "ORA")
echo "Ip-Address|Directory Name|${aa}" > output.txt
I am grepping date because I want all the lines after current date and then grep "ORA" from it. There are other ways but according to my log file this is most suitable way.
I am getting the output like this.
10.46.162.86|ASD----Exception|2010-03-24 07 ORA-00001 - 80 -
173.45.230.59
2010-03-24 07:00:47 ORA-00942 - 80 - 173.45.230.59
2010-03-24 07:01:15 ORA-00001 - 80 - 173.45.230.59
2010-03-24 07:02:17 ORA-12849 - 80 - 173.45.230.59
2010-03-24 07:05:09 ORA-00001 - 80 - 173.45.230.59
The ideal output should be like
10.46.162.86|ASD----Exception|2010-03-24 07 ORA-00001 - 80 -
173.45.230.59
10.46.162.86|ASD----Exception|2010-03-24 07:00:47 ORA-00942 - 80 -
173.45.230.59
10.46.162.86|ASD----Exception|2010-03-24 07:01:15 ORA-00001 - 80 -
173.45.230.59
10.46.162.86|ASD----Exception|2010-03-24 07:02:17 ORA-12849 - 80 -
173.45.230.59
10.46.162.86|ASD----Exception|2010-03-24 07:05:09 ORA-00001 - 80 -
173.45.230.59
I am fetching ORA from log files from different directories.
Input is like
2010-03-22 07:00:47 ZZZZC941948879 RUFFLES 222.222.222.222 ORA-00001 -
80 - 98.88.35.133 HTTP/1.1 Mozilla/5.0+(Windows;+U;+Windows+NT+9.0;+en-
US;+rv:1.9.2.2)
2010-03-22 07:00:47 ZZZZC941948879 RUFFLES 222.222.222.222 GET
/2009/10/yep-twitter-down.ht
2010-03-22 07:00:48 ZZZZC941948879 RUFFLES 222.222.222.222 GET
/img/input-bg.jpg - 80 - 98.88.35.133 HTTP/1.1 Mozilla/5.0+
(Windows;+U;+Windows+NT+9.0;+en-
US;+rv:1.9.2.2)+Gecko/20100319+Firefox/3.9.2
2010-03-23 07:00:48 ZZZZC941948879 RUFFLES 222.222.222.222 ORA-00001 -
80 - 98.88.35.133 HTTP/1.1 Mozilla/5.0+(Windows;+U;+Windows+NT+9.0;+en-
US;+rv:1.9.2.2)+Gecko/20100319+Firefox/3.9.2
2010-03-23 07:00:48 ZZZZC941948879 RUFFLES 222.222.222.222 GET
/img/topnav-about.jpg - 80 - 98.88.35.133 HTTP/1.1 Mozilla/5.0+
(Windows;+U;+Windows+NT+9.0;+en-US;+rv:1.9.2.2)+Gecko/20100319
2010-03-23 07:00:48 ZZZZC941948879 RUFFLES 222.222.222.222 GET
/img/entry-hr.jpg - 80 - 98.88.35.133 HTTP/1.1 Mozilla/5.0+
(Windows;+U;+Windows+NT+9.0;+en-US;+rv:1.9.2.2)+Gecko/20100319+Firefox
2010-03-23 07:00:48 ZZZZC941948879 RUFFLES 222.222.222.222 ORA-00001 -
80 - 98.88.35.133 HTTP/1.1 Mozilla/5.0+(Windows;+U;+Windows+NT+9.0;+en-
US;+rv:1.9.2.2)+Gecko/20100319+Firefox/3.9.2
2010-03-24 07:00:48 ZZZZC941948879 RUFFLES 222.222.222.222 GET
/img/header-bg.jpg - 80 - 98.88.35.133 HTTP/1.1 Mozilla/5.0+
(Windows;+U;+Windows+NT+9.0;+en-US;+rv:1.9.2.2)+Gecko/20100319
2010-03-24 07:00:48 ZZZZC941948879 RUFFLES 222.222.222.222 GET
/img/bullet.gif - 80 - 98.88.35.133 HTTP/1.1 Mozilla/5.0+
(Windows;+U;+Windows+NT+9.0;+en-US;+rv:1.9.2.2)+Gecko/20100319+Firefox
2010-03-24 07:00:49 ZZZZC941948879 RUFFLES 222.222.222.222 ORA-00001 -
80 - 98.88.35.133 HTTP/1.1 Mozilla/5.0+(Windows;+U;+Windows+NT+9.0;+en-
US;+rv:1.9.2.2)+Gecko/20100319+Firefox/3.9.2
2010-03-24 07:00:49 ZZZZC941948879 RUFFLES 222.222.222.222 GET /img/bg-
module.jpg - 80 - 98.88.35.133 HTTP/1.1 Mozilla/5.0+
(Windows;+U;+Windows+NT+9.0;+en-US;+rv:1.9.2.2)+Gecko/20100319
2010-03-24 07:00:50 ZZZZC941948879 RUFFLES 222.222.222.222 ORA-00942 -
80 - 98.88.35.133 HTTP/1.1 Mozilla/5.0+(Windows;+U;+Windows+NT+9.0;+en-
US;+rv:1.9.2.2)+Gecko/20100319+Firefox/3.9.2
2010-03-24 07:00:50 ZZZZC941948879 RUFFLES 222.222.222.222 GET /img/bg-
sidebarul.jpg - 80 - 98.88.35.133 HTTP/1.1 Mozilla/5.0+
(Windows;+U;+Windows+NT+9.0;+en-US;+rv:1.9.2.2)+Gecko/20100319
2010-03-24 07:00:50 ZZZZC941948879 RUFFLES 222.222.222.222 ORA-00001 -
80 - 98.88.35.133 HTTP/1.1 Mozilla/5.0+(Windows;+U;+Windows+NT+9.0;+en-
US;+rv:1.9.2.2)+Gecko/20100319+Firefox/3.9.2
2010-03-24 07:00:51 ZZZZC941948879 RUFFLES 222.222.222.222 ORA-00942 -
80 - 98.88.35.133 HTTP/1.1 Mozilla/5.0+(Windows;+U;+Windows+NT+9.0;+en-
US;+rv:1.9.2.2)+Gecko/20100319+Firefox/3.9.2
The problem here is when I am doing the grep operation it fetches 100 or more lines depending upon the exception and I am able to append the Ip-Address and node name to one line only.
Also, the IP-Address and node name are generated at run time.
Please do suggest a way to get the desired output.
Thanks.
Since I just know that special characters are going to show up in the directory names, I'd prefer awk over sed for this to avoid code injection problems:
grep -A100000 "2010-03-24" log.txt | awk -v prefix="IP-Address|Directory name|" '/ORA/ { print prefix $0 }' > output.txt
The relevant part is
awk -v prefix="IP-Address|Directory name|" '/ORA/ { print prefix $0 }'
With -v prefix=value, a variable named prefix with the given value is made known to awk, and /ORA/ { print prefix $0 } instructs awk to process all lines that match the regex ORA by printing prefix followed by the line (which is $0).
#etanreisner gave you the answer.
One way:
try:
grep -A100000 "2010-03-24" log.txt|grep "ORA" |
while read aa
do
echo "Ip-Address|Directory Name|${aa}"
done > output.txt
Related
How do I find what process/application is running an http server on a machine? All the Usual tools (netstat, lsof, fuser, ss aren't helping in this instance)
vinayb#carbon ~ $ sudo fuser 80/tcp
vinayb#carbon ~ $ sudo ss -pt state listening 'sport = :80'
Recv-Q Send-Q Local Address:Port Peer Address:Port Process
vinayb#carbon ~ $ curl http://localhost:80
404 page not found
vinayb#carbon ~ $ curl -vv http://localhost:80
* Trying 127.0.0.1:80...
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< Vary: Accept-Encoding
< X-Content-Type-Options: nosniff
< Date: Sat, 27 Feb 2021 12:45:05 GMT
< Content-Length: 19
<
Using netstat usually helps in this case, ie. netstat -tupan.
Best executed as root, that will give you a nice list, such as:
tom:~/ $ sudo netstat -tupan
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1450/master
tcp 0 0 0.0.0.0:445 0.0.0.0:* LISTEN 1764/smbd
tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN 1764/smbd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1106/sshd
...
I have a list of banners which are at this format:
Hostname: []
IP: xxx.xxx.xxx.xxx
Port: xx
HTTP/1.0 301 Moved Permanently
Location: /login.html
Content-Type: text/html
Device-Access-Level: 255
Content-Length: 3066
Cache-Control: max-age=7200, must-revalidate
I have used the following grep statement in order to grab the ip:
grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}"
What do I have to add to the statement in order to grab the port also? (while still getting the IP.).
Thank you for the answers..!
Why not use awk
awk '/IP:/ {ip=$2} /Port:/ {print ip,$2}' file
When it find line with IP: it stores the IP in variable ip
When it find port, print ip and port number.
Example
cat file
Hostname: []
IP: 163.248.1.20
Port: 843
HTTP/1.0 301 Moved Permanently
Location: /login.html
Content-Type: text/html
Device-Access-Level: 255
Content-Length: 3066
Cache-Control: max-age=7200, must-revalidat
awk '/IP:/ {ip=$2} /Port:/ {print ip,$2}' file
163.248.1.20 843
this is the code I am using to save files from a camera and name them from 0001 onward. The camera is running Busybox, and it has an ash shell inside.
The code is based on a previous answer by Charles Duffy here.
#!/bin/sh
# Snapshot script
cd /mnt/0/foto
sleep 1
set -- *.jpg # put the sorted list of picture namefiles on argv ( the number of files on the list can be requested by echo $# )
while [ $# -gt 1 ]; do # as long as there's more than one...
shift # ...some rows are shifted until only one remains
done
if [ "$1" = "*.jpg" ]; then # If cycle to determine if argv is empty because there is no jpg file present in the dir. #argv is set so that following cmds can start the sequence from 0 on.
set -- snapfull0000.jpg
else
echo "Piu' di un file jpg trovato."
fi
num=${1#*snapfull} # $1 is the first row of $#. The alphabetical part of the filename is removed.
num=${num%.*} # removes the suffix after the name.
num=$(printf "%04d" "$(($num + 1))") # the variable is updated to the next digit and the number is padded (zeroes are added)
# echoes for debug
echo "variabile num="$num # shows the number recognized in the latest filename
echo "\$#="$# # displays num of argv variables
echo "\$1="$1 # displays the first arg variable
wget http://127.0.0.1/snapfull.php -O "snapfull${num}.jpg" # the snapshot is requested to the camera, with the sequential naming of the jpeg file.
This is what I get on the cmd line during the script operation. I manually ran the script nine times, but after the saving of file snapfull0008.jpg, as you can see in the last lines, files are named only snapfull0000.jpg.
# ./snap4.sh
variable num=0001
$#=1
$1=snapfull0000.jpg
Connecting to 127.0.0.1 (127.0.0.1:80)
127.0.0.1 127.0.0.1 - [05/Dec/2014:20:22:22 +0000] "GET /snapfull.php HTTP/1.1" 302 0 "-" "Wget"
snapfull0001.jpg 100% |*******************************| 246k --:--:-- ETA
# ./snap4.sh
More than a jpg file found.
variable num=0002
$#=1
$1=snapfull0001.jpg
Connecting to 127.0.0.1 (127.0.0.1:80)
127.0.0.1 127.0.0.1 - [05/Dec/2014:20:22:32 +0000] "GET /snapfull.php HTTP/1.1" 302 0 "-" "Wget"
snapfull0002.jpg 100% |*******************************| 249k --:--:-- ETA
# ./snap4.sh
More than a jpg file found.
variable num=0003
$#=1
$1=snapfull0002.jpg
Connecting to 127.0.0.1 (127.0.0.1:80)
127.0.0.1 127.0.0.1 - [05/Dec/2014:20:22:38 +0000] "GET /snapfull.php HTTP/1.1" 302 0 "-" "Wget"
snapfull0003.jpg 100% |*******************************| 248k --:--:-- ETA
# ./snap4.sh
More than a jpg file found.
variable num=0004
$#=1
$1=snapfull0003.jpg
Connecting to 127.0.0.1 (127.0.0.1:80)
127.0.0.1 127.0.0.1 - [05/Dec/2014:20:22:43 +0000] "GET /snapfull.php HTTP/1.1" 302 0 "-" "Wget"
snapfull0004.jpg 100% |*******************************| 330k --:--:-- ETA
# ./snap4.sh
More than a jpg file found.
variable num=0005
$#=1
$1=snapfull0004.jpg
Connecting to 127.0.0.1 (127.0.0.1:80)
127.0.0.1 127.0.0.1 - [05/Dec/2014:20:22:51 +0000] "GET /snapfull.php HTTP/1.1" 302 0 "-" "Wget"
snapfull0005.jpg 100% |*******************************| 308k --:--:-- ETA
# ./snap4.sh
More than a jpg file found.
variable num=0006
$#=1
$1=snapfull0005.jpg
Connecting to 127.0.0.1 (127.0.0.1:80)
127.0.0.1 127.0.0.1 - [05/Dec/2014:20:22:55 +0000] "GET /snapfull.php HTTP/1.1" 302 0 "-" "Wget"
snapfull0006.jpg 100% |*******************************| 315k --:--:-- ETA
# ./snap4.sh
More than a jpg file found.
variable num=0007
$#=1
$1=snapfull0006.jpg
Connecting to 127.0.0.1 (127.0.0.1:80)
127.0.0.1 127.0.0.1 - [05/Dec/2014:20:22:59 +0000] "GET /snapfull.php HTTP/1.1" 302 0 "-" "Wget"
snapfull0007.jpg 100% |*******************************| 316k --:--:-- ETA
# ./snap4.sh
More than a jpg file found.
variable num=0008
$#=1
$1=snapfull0007.jpg
Connecting to 127.0.0.1 (127.0.0.1:80)
127.0.0.1 127.0.0.1 - [05/Dec/2014:20:23:04 +0000] "GET /snapfull.php HTTP/1.1" 302 0 "-" "Wget"
snapfull0008.jpg 100% |*******************************| 317k --:--:-- ETA
# ./snap4.sh
More than a jpg file found.
variable num=0000
$#=1
$1=snapfull0008.jpg
Connecting to 127.0.0.1 (127.0.0.1:80)
127.0.0.1 127.0.0.1 - [05/Dec/2014:20:23:10 +0000] "GET /snapfull.php HTTP/1.1" 302 0 "-" "Wget"
snapfull0000.jpg 100% |*******************************| 318k --:--:-- ETA
What could be the cause of the sequence stopping after file number 8?
The problem is that leading 0s cause a number to be read as octal.
In bash, using $((10#$num)) will force decimal. Thus:
num=$(printf "%04d" "$((10#$num + 1))")
To work with busybox ash, you'll need to strip the 0s. One way to do this which will work even in busybox ash:
while [ "${num:0:1}" = 0 ]; do
num=${num:1}
done
num=$(printf '%04d' "$((num + 1))")
See the below transcript showing use (tested with ash from busybox v1.22.1):
$ num=0008
$ while [ "${num:0:1}" = 0 ]; do
> num=${num:1}
> done
$ num=$(printf '%04d' "$((num + 1))")
$ echo "$num"
0009
If your shell doesn't support even the baseline set of parameter expansions required by POSIX, you could instead end up using:
num=$(echo "$num" | sed -e 's/^0*//')
num=$(printf '%04d' "$(($num + 1))")
...though this would imply that your busybox was built with a shell other than ash, a decision I would strongly suggest reconsidering.
I'm trying to use this project to integrate WebDAV into my .NET MVC2 application.
I've traced the traffic from Office to my WebDAV server, and compared it to this example on how office determines if the document should be readonly or edit.
After Office successfully authenticates with the server I see these requests as the document is opening.
2014-07-22 18:41:36 127.0.0.1 OPTIONS / - 80 username#mydomain.com 127.0.0.1 Microsoft+Office+Protocol+Discovery 200 0 0 23
2014-07-22 18:41:36 127.0.0.1 OPTIONS /wordstorage - 80 username#mydomain.com 127.0.0.1 Microsoft-WebDAV-MiniRedir/6.1.7601 200 0 0 5
2014-07-22 18:41:36 127.0.0.1 PROPFIND /wordstorage - 80 username#mydomain.com 127.0.0.1 Microsoft-WebDAV-MiniRedir/6.1.7601 200 0 0 29
2014-07-22 18:41:36 127.0.0.1 PROPFIND /wordstorage - 80 username#mydomain.com 127.0.0.1 Microsoft-WebDAV-MiniRedir/6.1.7601 200 0 0 10
2014-07-22 18:41:36 127.0.0.1 OPTIONS / - 80 - 127.0.0.1 Microsoft-WebDAV-MiniRedir/6.1.7601 403 0 0 7
2014-07-22 18:41:36 127.0.0.1 PROPFIND /wordstorage - 80 - 127.0.0.1 Microsoft-WebDAV-MiniRedir/6.1.7601 302 0 0 9
2014-07-22 18:41:36 127.0.0.1 PROPFIND /Account/LogOn ReturnUrl=%2fwordstorage 80 - 127.0.0.1 Microsoft-WebDAV-MiniRedir/6.1.7601 200 0 0 29
2014-07-22 18:42:25 127.0.0.1 PROPFIND /wordstorage - 80 username#mydomain.com 127.0.0.1 Microsoft-WebDAV-MiniRedir/6.1.7601 200 0 0 33
2014-07-22 18:42:25 127.0.0.1 PROPFIND /wordstorage - 80 username#mydomain.com 127.0.0.1 Microsoft-WebDAV-MiniRedir/6.1.7601 200 0 0 6
2014-07-22 18:42:59 127.0.0.1 GET /wordstorage/Test-2.docx - 80 username#mydomain.com 127.0.0.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/7.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+.NET4.0C;+.NET4.0E;+InfoPath.2;+IPH+1.1.21.4019;+MSOffice+12) 200 0 0 37
2014-07-22 18:42:59 127.0.0.1 HEAD /wordstorage/Test-2.docx - 80 username#mydomain.com 127.0.0.1 Microsoft+Office+Existence+Discovery 200 0 0 186
The first two OPTIONS and PROPFIND requests return a 200 OK, but the third OPTIONS request is denied with a 403 - forbidden code.
If authentication is successful why would MiniRedir not send authentication with the OPTIONS request?
Here's my environment:
Win 7
Office 2007
IIS 7.5
Have you checked if the IIS Webdav module is disabled ?
It seems that it may cause problems if not disabled.
I want to sort and calculate how much clients downloaded files (3 types) from my server.
I installed tshark and ran followed command that should capture GET requests:
`./tshark 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' -R'http.request.method == "GET"'`
so sniffer starts to work and every second I get new row, here is a result:
0.000000 144.137.136.253 -> 192.168.4.7 HTTP GET /pids/QE13_593706_0.bin HTTP/1.1
8.330354 1.1.1.1 -> 2.2.2.2 HTTP GET /pids/QE13_302506_0.bin HTTP/1.1
17.231572 1.1.1.2 -> 2.2.2.2 HTTP GET /pids/QE13_382506_0.bin HTTP/1.0
18.906712 1.1.1.3 -> 2.2.2.2 HTTP GET /pids/QE13_182406_0.bin HTTP/1.1
19.485199 1.1.1.4 -> 2.2.2.2 HTTP GET /pids/QE13_302006_0.bin HTTP/1.1
21.618113 1.1.1.5 -> 2.2.2.2 HTTP GET /pids/QE13_312106_0.bin HTTP/1.1
30.951197 1.1.1.6 -> 2.2.2.2 HTTP GET /nginx_status HTTP/1.1
31.056364 1.1.1.7 -> 2.2.2.2 HTTP GET /nginx_status HTTP/1.1
37.578005 1.1.1.8 -> 2.2.2.2 HTTP GET /pids/QE13_332006_0.bin HTTP/1.1
40.132006 1.1.1.9 -> 2.2.2.2 HTTP GET /pids/PE_332006.bin HTTP/1.1
40.407742 1.1.2.1 -> 2.2.2.2 HTTP GET /pids/QE13_452906_0.bin HTTP/1.1
what I need to do to store results type and count like /pids/*****.bin in to other file.
Im not strong in linux but sure it can be done with 1-3 rows of script.
Maybe with awk but I don't know what is the technique to read result of sniffer.
Thank you,
Can't you just grep the log file of your webserver?
Anyway, to extract the lines of captured http traffic relative to your server files, just try with
./tshark 'tcp port 80 and \
(((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' \
-R'http.request.method == "GET"' | \
egrep "HTTP GET /pids/.*.bin"