I used wireshark to collect data from some sites, and then used tcpdump to get it as a text file. For the project I'm working on, I want to count how many DNS resolutions are involved in accessing a particular website, and what the nature of the DNS responses was. The problem is I don't really understand the output from wireshark or how to interpret it to find what I'm looking for. For instance, here is a line:
21:08:05.454852 IP 10.0.0.2.57512 > ord08s09-in-f21.1e100.net.https:
Flags [.], seq 1:1419, ack 55, win 65535, options [nop,nop,TS val
1348891674 ecr 2473250009], length 1418
What do the different parts of this mean, and what will the data I'm looking for look like? I'm worried I might be using Wireshark incorrectly without knowing it.
I used wireshark to collect data from some sites, and then used tcpdump to get it as a text file.
Most people who use both tools use them for the opposite purposes. :-) I.e., they use tcpdump to capture traffic into a file and then read the file with Wireshark. If you're only using Wireshark to capture traffic, that's probably overkill - you can do the same thing with dumpcap or possibly even tcpdump.
The output you're showing is text output, so, if you "used tcpdump to get it as a text file", it's output from tcpdump, not from Wireshark; text output from Wireshark would look different. If you "used wireshark to collect data from some sites, and then used tcpdump to get it as a text file", the output from Wireshark is either a pcap file or a pcap-ng file, which is a binary file, and is completely uninterpreted raw data. The interpretation of the data in your example is being done by tcpdump, not Wireshark.
What the output is saying is:
"21:08:05.454852": the packet arrived at 21:08:05 and a fraction of a second, local time.
"IP": the packet is an IPv4 packet.
"10.0.0.2.57512 > ord08s09-in-f21.1e100.net.https": the packet is from IP address 10.0.0.2, port 57512, to the IP address whose for which the host name is "ord08s09-in-f21.1e100.net", and the port for "https", which is port 443.
See the tcpdump man page, and a description of TCP, for details on the rest of the line.
The key point here is that this is NOT DNS traffic! It's probably "HTTP-over-SSL", or "https", traffic.
In tcpdump, DNS traffic would look like
11:06:25.247272 IP 10.0.1.3.50953 > 10.0.1.1.domain: 7088+ A? www.kernel.org. (32)
11:06:25.282723 IP 10.0.1.1.domain > 10.0.1.3.50953: 7088 3/0/0 CNAME pub.us.kernel.org., A 149.20.4.69, A 198.145.20.140 (85)
or
11:06:30.622744 IP 10.0.1.3.62767 > 10.0.1.1.domain: 2439+ A? e3191.c.akamaiedge.net.0.1.cn.akamaiedge.net. (62)
11:06:30.639279 IP 10.0.1.1.domain > 10.0.1.3.62767: 2439 1/0/0 A 184.85.109.15 (78)
"A?" means that a query is being done for an A record; "CNAME" means that a CNAME record is being returned (i.e., "www.kernel.org" is an alias for "pub.us.kernel.org", and "A" means that an A record is being returned, giving an IPv4 address.
In Wireshark or TShark, it would look like:
12.316361 10.0.1.3 -> 10.0.1.1 DNS Standard query 0xc2fa A 1.courier-sandbox-push-apple.com.akadns.net
12.332894 10.0.1.1 -> 10.0.1.3 DNS Standard query response 0xc2fa A 17.149.34.59 A 17.149.34.61 A 17.149.34.62 A 17.149.34.63 A 17.149.34.57
or
15.163941 10.0.1.3 -> 10.0.1.1 DNS Standard query 0x168c A www.gnu.org
15.176266 10.0.1.1 -> 10.0.1.3 DNS Standard query response 0x168c CNAME wildebeest.gnu.org A 208.118.235.148
If you're only trying to capture DNS packet, you should use a capture filter such as "port 53" or "port domain", so that non-DNS traffic will be discarded. That filter will work with Wireshark, TShark, or tcpdump (as they use the same libpcap code for packet capture).
Related
I know I got quite rusty when it comes to bash coding, especially the more elaborate needed trickery handling awk or sed parts.
I do have a script that logs the IP address currently in use for the interwebs.
It gets that by either using wget -q0 URL or lynx -dump URL.
The most easy one was a site that only returned the IP address in plain text and nothing else. Unfortunately that site no longer exists.
The code was simple as can be:
IP=$(wget -qO - http://cfaj.freeshell.org/ipaddr.cgi)
But alas! using the code returns nothing cause the site is gone, as lynx can tell us:
$ lynx -dump http://cfaj.freeshell.org/ipaddr.cgi
Looking up cfaj.freeshell.org
Unable to locate remote host cfaj.freeshell.org.
Alert!: Unable to connect to remote host.
lynx: Can't access startfile http://cfaj.freeshell.org/ipaddr.cgi
Some other sites I used to retrieve for the same purpose no longer work either.
And the one I want to use is a German speaking one, not that I care one way or the other, it could be in Greek or Mandarin for all I care. I want only to have the IP address itself extracted, but like I said, my coding skills got rusty.
Here is the relevant area of what lynx -dump returns
[33]powered by
Ihre IP-Adresse lautet:
178.24.x.x
Ihre IPv6-Adresse lautet:
Ihre System-Informationen:
when running it as follows:
lynx -dump https://www.wieistmeineip.de/
Now, I need either awk or sed to find the 178.24.x.x part. (I know it can be done with python or Perl as well, but both are not part of a standard setting of my Linux, while awk and sed are.)
Since the script is there to extract the IP address, one needs to do the following either via sed or awk:
Search for "Ihre IP-Adresse lautet:"
Skip the next line.
Skip the whitespace at the beginning
Only return what is left of that line (without the lf at the end).
In the above example (that shows only the relevant part of the lynx dump, the whole dump is much larger but all above and below is irrelevant.) it would be "178.24.x.x" that should be returned.
Any help greatly appreciated to get my log-ip script back into working order.
Currently I have collected some other working URLs that report back the own internet IP. Any of these can also be used, but the area around the reported IP will differ from the above example. These are:
https://meineipinfo.de/
http://www.wie-ist-meine-ip.net/
https://www.dein-ip-check.de/
https://whatismyipaddress.com/
https://www.whatismyip.org/
https://www.whatismyip.net/
https://mxtoolbox.com/whatismyip/
https://www.whatismyip.org/my-ip-address
https://meineipadresse.de/
Even duckduckgo returns the IP address when e.g. asked this: https://duckduckgo.com/?q=ip+address&ia=answer
At least I know of no way of getting the own IP address when using the internet without retrieving an outside URL that reports that very IP address back to me.
You can do:
wget -O - v4.ident.me 2>/dev/null && echo
So, if you have a VM in some cloud provider you can solve this easily. I wrote some small Go app than echoes back an HTTP request. For instance :
$ curl 167.99.63.182:8888
Method ->
GET
Protocol ->
HTTP/1.1
Headers ->
User-Agent: [curl/7.54.0]
Accept: [*/*]
Content length (in Bytes) ->
0
Remote address ->
179.XXXXX
Payload
####################
####################
Where remote address is the address which the app received, hence, your IP.
And in case you are wondering, yes, 167.99.63.182 is the IP of the server and you can curl it right now and check it. I am disclosing the IP as anyway I get bombarded by brute force attacks for as long as I can remember and the machine does not have anything worth the break through.
Not exactly without relying on external services, but you could use dig to reach out to the resolver at opendns.com:
dig +short myip.opendns.com #resolver1.opendns.com
I think this is easier to integrate to a script.
I got a DNS internal query log file and I would like to do analysis on it, so onw record looks like this:
18-Jun-2017 04:00:10.139 client #XXXXXXXXXXXX ip-address#number (ip-address): view internal: query: ip-address IN AAAA + (ip-address in the format of ipv4)
I am really new to this so I tried to search on line but I couldn't find what does the "+" mean in the record, and as far as I understand, AAAA should represent the ipv6 ip format, why I got a following ipv4 address following it ?
Can some one explain it to me what does this record means or some links I could read myself ?
Many thanks !!
You're right AAAA (or the quad record) is for IPV6 addresses. The IPV6 addressing scheme allows for trailing or leading zeros to be omitted, so an IPV4 address of 192.169.1.1 may be represented as 00:00:00:00:C0:A8:01:01 which without the leading zeros is C0:A8:01:01 or 192.168.1.1 if your text editor converts Hex to Decimal.
I am setting my openwrt (Netgear R3800) to support policy routing with multiple outbound interfaces by the following shell scripts. I put ip lists for different outgoing interfaces in text file and use ip rule to set different ip tables.
e.g.
ip route add default via $VPN1 dev $VPNDEV1 table 100
ip route add default via $VPN2 dev $VPNDEV2 table 200
for IPTO in `cat /root/routes/policy_ips1` ; do
ip rule add to $IPTO table 100
done
for IPTO in `cat /root/routes/policy_ips2` ; do
ip rule add to $IPTO table 200
done
The script does work most time but sometimes not stable.
As the ip lists (file policy_ips1, policy_ips2) are considerable large. like 10-20 thousands lines, each line is a subnet like '1.0.1.0/24'. I am wondering whether this is an effective way as there will be tens of thousands rules and the router will query the routing policy databas RPDB for every packet.
A further question is that as the Priority of RPDB ranges from 0 to 32767 (ref:http://man7.org/linux/man-pages/man8/ip-rule.8.html), does it mean I cannot add more than 32765 rules? Is there any capability limit in using ip rule? And is there any limit with respect to the computational capability of the router(like router's cpu, ram)?
[I'm not exactly sure this question is appropriate for SO, but]
For this kind of setup I would use Netfilter (with ipsets) to mark packets and use fwmark-based rules to match packets in the RPDB. Like
ipset create nets100 hash:net
ipset create nets200 hash:net
for IPTO in `cat /root/routes/policy_ips1` ; do
ipset add nets100 "$IPTO"
done
for IPTO in `cat /root/routes/policy_ips2` ; do
ipset add nets200 "$IPTO"
done
iptables -A PREROUTING -t mangle -m set --match-set nets100 src -j MARK --set-mark 100
iptables -A PREROUTING -t mangle -m set --match-set nets200 src -j MARK --set-mark 200
ip route add default via $VPN1 dev $VPNDEV1 table 100
ip route add default via $VPN2 dev $VPNDEV2 table 200
ip rule add fwmark 100 table 100
ip rule add fwmark 200 table 200
IPsets can easily handle lots of addresses/networks and you have a simple RPDB as a result. Although it should be noted that this adds some Netfilter overhead to packet processing, but choosing between big RPDB and ipsets I would for ipsets.
As for priority, I would read it as "no more than 32767 rules".
I'm trying to find a way to reliably find the global IPv6 address of the local router. The command "ip -6 neigh show" doesn't display it. I have found the following works, so long as the router is using EUI-64 to generate a host address:
NET=$(ip -6 route|egrep "^[23]"|cut -d':' -f1-4)
EUI=$(ip -6 route|egrep "^default.*metric 1 "|cut -d':' -f3-6|cut -d' ' -f1)
ping6 $NET:$EUI -B -c 1 > /dev/null 2>&1
However this obviously doesn't work when EUI-64 isn't being used. Is there any other way I can find it?
There is no reliable way to determine this, as your local router doesn’t have to have a global IPv6 address at all. Best practice says it should, and the vast majority of routers out there will, but technically it’s possible that your router could only have link-local addresses on both interfaces and still route global prefixes. (Yes, I have done this before. And yes, it’s evil.)
Unless you manually added a default route pointing to a global address, you probably learned your default route via Router Advertisement which means the routing table and neighbour table (as you pointed out above) will contain a link-local address only.
However, if you perform a traceroute, the first hop is probably (key word: probably — it’s possible to spoof this stuff) your local router.
traceroute to 2600:: (2600::), 30 hops max, 80 byte packets
1 2001:db8:1::1 0.534 ms 0.510 ms 0.749 ms
2 2001:db8:2::1 32.494 ms 33.944 ms 35.406 ms
3 2600:: 36.935 ms 38.102 ms 39.860 ms
TL;DR it’s not possible to reliably determine this, so the short answer is “no”.
Is there no easy way to map a given IP adress to the corresponding ASN number? For example:
ping to find out the IP adress:
$ ping www.switch.ch
PING aslan.switch.ch (130.59.108.36) 56(84) bytes of data.
whois lookup for the ASN number:
$ whois -h whois.cymru.com -v 130.59.108.36
Warning: RIPE flags used with a traditional server.
AS | IP | BGP Prefix | CC | Registry | Allocated | AS Name
559 | 130.59.108.36 | 130.59.0.0/16 | CH | ripencc | 1993-09-22 | SWITCH SWITCH, Swiss Education and Research Network
So the mapping in this case would be 130.59.108.36 (IP)-> 559 (ASN). Easy. But what if I would like to create my own local mapping service with the public available information from the Regional Internet Registries? So, for the above example, it would be this list, right?
ftp://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-latest
And to find the matching entrie is also not a problem:
ripencc|CH|ipv4|130.59.0.0|65536|19930922|assigned
But how do I get the ASN number from the line above?? How are those two informations linked together?
ripencc|EU|asn|559|1|19930901|allocated
Thanks in advance for a reply!
I explain how to do this here: https://www.quaxio.com/bgp/ (formerly at https://alokmenghrajani.github.io/bgp/)
It basically involves downloading a dump from a router and then using an efficient data representation to map an IP address to a netmask.
I'd propose doing this based on MRT dumps collected from an actual BGP speaker.
There is this python library f.e. that can be used to easily parse MRT dumps: http://code.google.com/p/pyasn/
If you're not able to run your own BGP speaker, you can download dumps at http://archive.routeviews.org/
Make sure you checkout their other stuff too. They provide also DNS zonefiles that would enable you to do such lookups using a standard DNS server such as Bind or NSD: http://archive.routeviews.org/dnszones/
I hope that gets you started...
I have made a tool that appends ASNs to HTTP log lines. I explain how to build the database off RIPE raw data and use it with binary search. Also, C code is ready for use. 1.6M look-ups in a few seconds on a regular virtual instance:
https://github.com/psvz/tirexASN