How can I extract IPs from a PCAP file whose packet has any of these hex strings.
--hex-string '|AF0DFT5F14|'
--hex-string '|AF0DFT5F25|'
--hex-string '|AF0DFT5F45|'
I just need the list of IPs.
I tried to search a lot on web but couldn't get any specific about this. I found out to extract all the IPs from a pcap file using the below command
( tcpdump -n -r m.pcap | grep IP | cut -f 3 "-d " | cut -f1-4 -d.; tcpdump -n -r m.pcap | grep IP | cut -f 5 "-d " | cut -f1-4 -d. ) | sort -u
Grep, awk, anything can do
Related
I'm having issues with piping the translate command into the word count command using the shell for an assignment. Using Debian 9 Linux distro.
I need to remove colons from the passwd file in the /etc directory and pipe the results into "word count" or wc -w. I have read the man pages, google searched and tried youtube videos, but could not find anything that would point me in the right direction. Things I have tried include:
tr -d ":" | wc -w /etc/passwd
tr -d [:punct:] | wc -w /etc/passwd
tr -- delete [:punct:] | wc -w /etc/passwd
tr -s [:punct:] [:space:] | wc -w /etc/passwd
tr -t [:] [" "] | wc -w /etc/passwd
The piped command is supposed to delete colons, replace them with spaces, and change the word count/"wc" commands output.
Before using translate and piping to wc, passwd's word count is equal to 37 lines, 60 words and 2054 bytes. I believe the number is supposed to increase when you remove the colons.
You have to send the content of the file to tr first.
< /etc/passwd tr ":" " " | wc -w
Or with cat, even when it's a useless use of cat.
cat /etc/passwd | tr ":" " " | wc -w
you mean something like this?
tr ":" " " < /etc/passwd | wc -w
I have log file where I look for strings like:
tail -n 1000 -f logfile.log | grep -i "host"
and then I receive strings like these:
host2 %host-DEREG: host c459.cf00.1105 is deregistered on E0/1:60.
Could I choose mac addresses from these strings and look for strings with these mac addresses inside of another file?
There is no macaddress in your example
grep `tail -n 1000 -f logfile.log | grep -i "host" | grep -o "[a-f0-9][a-f0-9][a-f0-9][a-f0-9]\.[a-f0-9][a-f0-9][a-f0-9][a-f0-9]\.[a-f0-9][a-f0-9][a-f0-9][a-f0-9]"` anotherfile
I have output my nmap result to a file called test.txt and it looks like this:
Nmap scan report for 192.168.1.5
Host is up (0.13s latency).
PORT STATE SERVICE VERSION
23/tcp open telnet Linux telnetd
--
Nmap scan report for 192.168.1.7
Host is up (0.13s latency).
PORT STATE SERVICE VERSION
80/tcp open http Popper
--
Nmap scan report for 192.168.1.20
Host is up (0.13s latency).
PORT STATE SERVICE VERSION
110/tcp open pop3 Dove
I want to output the results from the file in my terminal using grep and pipe that displays the result in format: IP address followed by the open ports like this:
192.168.10.2
80
223
53
192.168.10.7
80
223
You probably want:
cat test.txt | sed 's/Nmap scan report for //' | sed '/Host is/d' | sed '/Not shown/d' | sed '/All/d' | sed '/PORT /d' | cut -f1 -d"/" | sed '/^$/d' | sed '/--/d'
For nmap -F 192.168.0.1/24 > test.txt:
cat test.txt | tail -n+3 | sed '/Nmap done/d' | awk 'NR>1{print l}{l=$0}' | sed 's/Nmap scan report for //' | sed '/Host is/d' | sed '/Not shown/d' | sed '/All/d' | sed '/PORT /d' | cut -f1 -d"/" | sed '/^$/d'
Joke. It does not make sense... Use nmap with XML Output -oX.
Read: https://nmap.org/book/output-formats-xml-output.html
I'd do the following:
Get a regular expression to match ip's and ports for the nmap output. You can take advantage of the positions of both, i.e ports are at the beginning of the line(^) and end with a '/' character, and ip's are the last item on the line($).
Use grep -o to look for any of that regular expressions (look at the OR operator on the regex (|)). This will reduce the output to just the matches.
I have this bash script that i wrote to analyse the html of any given web page. What its actually supposed to do is to return the domains on that page. Currently its returning the number of URL's on that web page.
#!/bin/sh
echo "Enter a url eg www.bbc.com:"
read url
content=$(wget "$url" -q -O -)
echo "Enter file name to store URL output"
read file
echo $content > $file
echo "Enter file name to store filtered links:"
read links
found=$(cat $file | grep -o -E 'href="([^"#]+)"' | cut -d '"' -f2 | sort | uniq | awk '/http/' > $links)
output=$(egrep -o '^http://[^/]+/' $links | sort | uniq -c > out)
cat out
How can i get it to return the domains instead of the URL's. From my programming knowledge I know its supposed to do parsing from the right but i am a newbie at bash scripting. Can someone please help me. This is as far as I have gone.
I know there's a better way to do this in awk but you can do this with sed, by appending this after your awk '/http/':
| sed -e 's;https\?://;;' | sed -e 's;/.*$;;'
Then you want to move your sort and uniq to the end of that.
So that the whole line will look like:
found=$(cat $file | grep -o -E 'href="([^"#]+)"' | cut -d '"' -f2 | awk '/http/' | sed -e 's;https\?://;;' | sed -e 's;/.*$;;' | sort | uniq -c > out)
You can get rid of this line:
output=$(egrep -o '^http://[^/]+/' $links | sort | uniq -c > out)
EDIT 2:
Please note, that you might want to adapt the search patterns in the sed expressions to your needs. This solution considers only http[s]?://-protocol and www.-servers...
EDIT:
If you want count and domains:
lynx -dump -listonly http://zelleke.com | \
sed -n '4,$ s#^.*http[s]?://\([^/]*\).*$#\1#p' | \
sort | \
uniq -c | \
sed 's/www.//'
gives
2 wordpress.org
10 zelleke.com
Original Answer:
You might want to use lynx for extracting links from URL
lynx -dump -listonly http://zelleke.com
gives
# blank line at the top of the output
References
1. http://www.zelleke.com/feed/
2. http://www.zelleke.com/comments/feed/
3. http://www.zelleke.com/
4. http://www.zelleke.com/#content
5. http://www.zelleke.com/#secondary
6. http://www.zelleke.com/
7. http://www.zelleke.com/wp-login.php
8. http://www.zelleke.com/feed/
9. http://www.zelleke.com/comments/feed/
10. http://wordpress.org/
11. http://www.zelleke.com/
12. http://wordpress.org/
Based on this output you achieve desired result with:
lynx -dump -listonly http://zelleke.com | \
sed -n '4,$ s#^.*http://\([^/]*\).*$#\1#p' | \
sort -u | \
sed 's/www.//'
gives
wordpress.org
zelleke.com
You can remove path from url with sed:
sed s#http://##; s#/.*##
I want to say you also, that these two lines are wrong:
found=$(cat $file | grep -o -E 'href="([^"#]+)"' | cut -d '"' -f2 | sort | uniq | awk '/http/' > $links)
output=$(egrep -o '^http://[^/]+/' $links | sort | uniq -c > out)
You must make either redirection ( > out ), or command substitution $(), but not two thing at the same time. Because the variables will be empty in this case.
This part
content=$(wget "$url" -q -O -)
echo $content > $file
would be also better to write this way:
wget "$url" -q -O - > $file
you may be interested by it:
https://www.rfc-editor.org/rfc/rfc3986#appendix-B
explain the way to parse uri using regex.
so you can parse an uri from the left this way, and extract the "authority" that contains domain and subdomain names.
sed -r 's_^([^:/?#]+:)?(//([^/?#]*))?.*_\3_g';
grep -Eo '[^\.]+\.[^\.]+$' # pipe with first line, give what you need
this is interesting to:
http://www.scribd.com/doc/78502575/124/Extracting-the-Host-from-a-URL
assuming that url always begin this way
https?://(www\.)?
is really hazardous.
I have been trying to get the unique values in each column of a tab delimited file in bash. So, I used the following command.
cut -f <column_number> <filename> | sort | uniq -c
It works fine and I can get the unique values in a column and its count like
105 Linux
55 MacOS
500 Windows
What I want to do is instead of sorting by the column value names (which in this example are OS names) I want to sort them by count and possibly have the count in the second column in this output format. So It will have to look like:
Windows 500
MacOS 105
Linux 55
How do I do this?
Use:
cut -f <col_num> <filename>
| sort
| uniq -c
| sort -r -k1 -n
| awk '{print $2" "$1}'
The sort -r -k1 -n sorts in reverse order, using the first field as a numeric value. The awk simply reverses the order of the columns. You can test the added pipeline commands thus (with nicer formatting):
pax> echo '105 Linux
55 MacOS
500 Windows' | sort -r -k1 -n | awk '{printf "%-10s %5d\n",$2,$1}'
Windows 500
Linux 105
MacOS 55
Mine:
cut -f <column_number> <filename> | sort | uniq -c | awk '{ print $2" "$1}' | sort
This will alter the column order (awk) and then just sort the output.
Hope this will help you
Using sed based on Tagged RE:
cut -f <column_number> <filename> | sort | uniq -c | sort -r -k1 -n | sed 's/\([0-9]*\)[ ]*\(.*\)/\2 \1/'
Doesn't produce output in a neat format though.