I am trying to write a script that will take in a BIND zone file, grab all of the A records, in the format host ip. I've done that by grep -w 'A' "$A_ZONE"|awk '{print $1,$4}'|sort -V, to skip the IN A part. Now, I need to extract PTR records from all of the reverse zones that I have. Those are grouped by /24 subnets, so if I have a PTR record for 10.0.0.1, it would be in the 0.0.10.in-addr.arpa.zone file, as 10 IN PTR host.domain.tld. Seeing as that is a bit convoluted, I'm not sure how to extract the IP well, so that it would be in the format of the first file that I extracted, host ip.
Any suggestions?
You can use the following command:
egrep '^[0-9]+' 0.0.10.in-addr.arpa.zone | \
perl -p -e 's/^(\d+).*\s(\S+)\s*$/$2 10.0.0.$1/'
Output:
host.domain.tld. 10.0.0.10
It greps all the records that start with a number, and match the number and hostname and reverse them. The IP address is then constructed along with the hostname.
Note that in the command I showed, the subnet is hardcoded in the regexp, but you could apply similar strategy to extract it from your filename plug it into the regex.
You may also want to consider running your zone files through named-compilezone so as to make sure that they are in a canonical format suitable for scripting.
Related
Let's say I wrote in bash a simple script to whois multiple ip addresses I've in a huge list (list.txt) and the results of this search are written in a huge single files "results.txt".
My goal is to run the script but I need to write a single file for each ip address saved with the ip address i've searched.
I'm really stuck here, any help is appreciated.
Thanks
#!/bin/bash
for domain in `cat iplist.txt`
do
echo $domain
`whois $domain >> results.txt`
done
I know I got quite rusty when it comes to bash coding, especially the more elaborate needed trickery handling awk or sed parts.
I do have a script that logs the IP address currently in use for the interwebs.
It gets that by either using wget -q0 URL or lynx -dump URL.
The most easy one was a site that only returned the IP address in plain text and nothing else. Unfortunately that site no longer exists.
The code was simple as can be:
IP=$(wget -qO - http://cfaj.freeshell.org/ipaddr.cgi)
But alas! using the code returns nothing cause the site is gone, as lynx can tell us:
$ lynx -dump http://cfaj.freeshell.org/ipaddr.cgi
Looking up cfaj.freeshell.org
Unable to locate remote host cfaj.freeshell.org.
Alert!: Unable to connect to remote host.
lynx: Can't access startfile http://cfaj.freeshell.org/ipaddr.cgi
Some other sites I used to retrieve for the same purpose no longer work either.
And the one I want to use is a German speaking one, not that I care one way or the other, it could be in Greek or Mandarin for all I care. I want only to have the IP address itself extracted, but like I said, my coding skills got rusty.
Here is the relevant area of what lynx -dump returns
[33]powered by
Ihre IP-Adresse lautet:
178.24.x.x
Ihre IPv6-Adresse lautet:
Ihre System-Informationen:
when running it as follows:
lynx -dump https://www.wieistmeineip.de/
Now, I need either awk or sed to find the 178.24.x.x part. (I know it can be done with python or Perl as well, but both are not part of a standard setting of my Linux, while awk and sed are.)
Since the script is there to extract the IP address, one needs to do the following either via sed or awk:
Search for "Ihre IP-Adresse lautet:"
Skip the next line.
Skip the whitespace at the beginning
Only return what is left of that line (without the lf at the end).
In the above example (that shows only the relevant part of the lynx dump, the whole dump is much larger but all above and below is irrelevant.) it would be "178.24.x.x" that should be returned.
Any help greatly appreciated to get my log-ip script back into working order.
Currently I have collected some other working URLs that report back the own internet IP. Any of these can also be used, but the area around the reported IP will differ from the above example. These are:
https://meineipinfo.de/
http://www.wie-ist-meine-ip.net/
https://www.dein-ip-check.de/
https://whatismyipaddress.com/
https://www.whatismyip.org/
https://www.whatismyip.net/
https://mxtoolbox.com/whatismyip/
https://www.whatismyip.org/my-ip-address
https://meineipadresse.de/
Even duckduckgo returns the IP address when e.g. asked this: https://duckduckgo.com/?q=ip+address&ia=answer
At least I know of no way of getting the own IP address when using the internet without retrieving an outside URL that reports that very IP address back to me.
You can do:
wget -O - v4.ident.me 2>/dev/null && echo
So, if you have a VM in some cloud provider you can solve this easily. I wrote some small Go app than echoes back an HTTP request. For instance :
$ curl 167.99.63.182:8888
Method ->
GET
Protocol ->
HTTP/1.1
Headers ->
User-Agent: [curl/7.54.0]
Accept: [*/*]
Content length (in Bytes) ->
0
Remote address ->
179.XXXXX
Payload
####################
####################
Where remote address is the address which the app received, hence, your IP.
And in case you are wondering, yes, 167.99.63.182 is the IP of the server and you can curl it right now and check it. I am disclosing the IP as anyway I get bombarded by brute force attacks for as long as I can remember and the machine does not have anything worth the break through.
Not exactly without relying on external services, but you could use dig to reach out to the resolver at opendns.com:
dig +short myip.opendns.com #resolver1.opendns.com
I think this is easier to integrate to a script.
I've a huge list of IP's which is already sorted, but I still need to group them into subnet. For instance:
223.247.184.95
223.247.186.243
223.247.208.16
223.247.209.139
223.84.128.24
223.84.159.214 *
223.84.159.245 *
The market IP's with the "*" should all be grouped by '223.84.159.*'. There
is no database, just this text file with 10.000 entries !
I tested awk and uniq commands, but my results are all not what I want.
It is not clear, as you haven't shown the output. Could you please try following and let me know if this helps you, it will print only those lines which have * in them from Input_file.
awk '$2=="*"{print $1}' Input_file
Say your IP's are in a file called ips spread over different fields over different lines and the field separator is the default one, namely a space.
Example:
cat ips
223.247.184.95 223.247.186.243 223.247.208.16 223.247.209.139 223.84.128.24
223.84.159.214* 223.84.159.245*
then for
Unsorted IPs file: the following awk code
cat ips | awk '{for(i=1;i<=NF;i++){split($i,a,".");k=a[1]"."a[2]"."a[3];h[k]=h[k]" "$i}}END{for(k in h)printf(k": "h[k]"\n")}'
gives you a hash table of the different C domains:
223.247.184: 223.247.184.95
223.247.186: 223.247.186.243
223.84.159: 223.84.159.214* 223.84.159.245*
223.84.128: 223.84.128.24
223.247.208: 223.247.208.16
223.247.209: 223.247.209.139
Sorted IPs file: If the file containing your IP has been previously sorted, then there is a more efficient and faster one-liner
cat ips| awk '{for(i=1;i<=NF;i++){split($i,a,".");cn=a[1]"."a[2]"."a[3]; if(cn != c){c=cn;printf("\n"c": ")};printf($i" ")}}END{printf("\n")}'
giving you as well
223.247.184: 223.247.184.95
223.247.186: 223.247.186.243
223.247.208: 223.247.208.16
223.247.209: 223.247.209.139
223.84.128: 223.84.128.24
223.84.159: 223.84.159.214* 223.84.159.245*
Is there no easy way to map a given IP adress to the corresponding ASN number? For example:
ping to find out the IP adress:
$ ping www.switch.ch
PING aslan.switch.ch (130.59.108.36) 56(84) bytes of data.
whois lookup for the ASN number:
$ whois -h whois.cymru.com -v 130.59.108.36
Warning: RIPE flags used with a traditional server.
AS | IP | BGP Prefix | CC | Registry | Allocated | AS Name
559 | 130.59.108.36 | 130.59.0.0/16 | CH | ripencc | 1993-09-22 | SWITCH SWITCH, Swiss Education and Research Network
So the mapping in this case would be 130.59.108.36 (IP)-> 559 (ASN). Easy. But what if I would like to create my own local mapping service with the public available information from the Regional Internet Registries? So, for the above example, it would be this list, right?
ftp://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-latest
And to find the matching entrie is also not a problem:
ripencc|CH|ipv4|130.59.0.0|65536|19930922|assigned
But how do I get the ASN number from the line above?? How are those two informations linked together?
ripencc|EU|asn|559|1|19930901|allocated
Thanks in advance for a reply!
I explain how to do this here: https://www.quaxio.com/bgp/ (formerly at https://alokmenghrajani.github.io/bgp/)
It basically involves downloading a dump from a router and then using an efficient data representation to map an IP address to a netmask.
I'd propose doing this based on MRT dumps collected from an actual BGP speaker.
There is this python library f.e. that can be used to easily parse MRT dumps: http://code.google.com/p/pyasn/
If you're not able to run your own BGP speaker, you can download dumps at http://archive.routeviews.org/
Make sure you checkout their other stuff too. They provide also DNS zonefiles that would enable you to do such lookups using a standard DNS server such as Bind or NSD: http://archive.routeviews.org/dnszones/
I hope that gets you started...
I have made a tool that appends ASNs to HTTP log lines. I explain how to build the database off RIPE raw data and use it with binary search. Also, C code is ready for use. 1.6M look-ups in a few seconds on a regular virtual instance:
https://github.com/psvz/tirexASN
What is the easiest way using common linux tools to check if a bunch of ip addresses belongs to given network? I just need a number of how many of given addresses belongs to given subnet. Lets say network is 192.16.55.40/27 and addresses is 192.16.55.45, 192.16.55.115, 88.87.45.8, 192.16.55.37, 192.16.55.60 and 192.16.55.210..
I'm not sure whether you consider Ruby as a "common linux tool" but it has a nice module called IPAddr that has a method called include? for that.
require 'ipaddr'
net1 = IPAddr.new("192.168.2.0/24")
net2 = IPAddr.new("192.168.2.100")
net3 = IPAddr.new("192.168.3.0")
p net1.include?(net2) #=> true
p net1.include?(net3) #=> false
I needed this to, and decided to create a short script. I requires sed and bash. I'd call them both common linux tools.
Edit: Script too long to paste, apparently. You can find it here: http://folk.ntnu.no/olechrt/netaddr
$ cat ips
192.16.55.45
192.16.55.115
88.87.45.8
192.16.55.210.11
192.16.55.37
192.16.55.60
192.16.55.210
256.87.45.8
$ cat ips | netaddr 192.16.55.40/27
192.16.55.45
Warning: Input IP "192.16.55.210.11" is invalid.
192.16.55.37
192.16.55.60
Warning: Input IP "256.87.45.8" is invalid.
And finally, for the count you requested:
$ cat ips | netaddr 192.16.55.40/27 | wc -l
Warning: Input IP "192.16.55.210.11" is invalid.
Warning: Input IP "256.87.45.8" is invalid.
3