Using bash to parse a log for unique MAC addresses - linux

Question

I've got Debian box set up as a syslog server for a couple cisco ASAs. They are running DHCP and I'm attemping to track the unique instances of a MAC addresses being assigned a lease. I've set the ASAs to only log the message that the cisco DHCPd uses, and it sends that to the Debian server as %HOSTIPADDRESS%.log, which then rotates out daily. So I've got a directory that is filled with this:
-rw-r----- 1 syslog adm 536351 Aug 23 06:24 10.10.10.4.log.10
-rw-r----- 1 syslog adm 459634 Aug 22 06:24 10.10.10.4.log.11
-rw-r----- 1 syslog adm 176957 Aug 21 06:24 10.10.10.4.log.12
-rw-r----- 1 syslog adm 246654 Aug 20 06:24 10.10.10.4.log.13
-rw-r----- 1 syslog adm 459978 Aug 19 06:24 10.10.10.4.log.14
-rw-r----- 1 syslog adm 606987 Aug 18 06:21 10.10.10.4.log.15
-rw-r----- 1 syslog adm 599140 Aug 17 06:24 10.10.10.4.log.16
-rw-r----- 1 syslog adm 605837 Aug 16 06:24 10.10.10.4.log.17
-rw-r----- 1 syslog adm 607630 Aug 15 06:24 10.10.10.4.log.18
-rw-r----- 1 syslog adm 189493 Aug 14 06:24 10.10.10.4.log.19
In each of those logs I've got something that looks like this:
Aug 23 06:20:19 10.10.10.4 %ASA-6-604103: DHCP daemon interface inside: address granted 011c.9148.dbb4.15 (172.16.1.196)
Aug 23 06:20:41 10.10.10.4 %ASA-6-604103: DHCP daemon interface inside: address granted 0138.0f4a.986a.16 (172.16.1.126)
Aug 23 06:20:51 10.10.10.4 %ASA-6-604103: DHCP daemon interface inside: address granted 0190.b686.63c6.a9 (172.16.1.193)
Aug 23 06:20:55 10.10.10.4 %ASA-6-604103: DHCP daemon interface inside: address granted 0154.4e90.8a7a.00 (172.16.1.211)
Aug 23 06:21:11 10.10.10.4 %ASA-6-604103: DHCP daemon interface inside: address granted 012c.0e3d.fcf6.34 (172.16.1.189)
Aug 23 06:21:35 10.10.10.4 %ASA-6-604103: DHCP daemon interface inside: address granted 0154.4e90.8a7a.00 (172.16.1.211)
Aug 23 06:21:51 10.10.10.4 %ASA-6-604103: DHCP daemon interface inside: address granted 0154.4e90.8a7a.00 (172.16.1.211)
Aug 23 06:22:29 10.10.10.4 %ASA-6-604103: DHCP daemon interface inside: address granted 5caf.0664.cd18 (172.16.1.212)
Aug 23 06:24:00 10.10.10.4 %ASA-6-604103: DHCP daemon interface inside: address granted 01fc.dbb3.49af.eb (172.16.1.207)
Aug 23 06:24:21 10.10.10.4 %ASA-6-604103: DHCP daemon interface inside: address granted 01a0.3be3.03b4.74 (172.16.1.195)
Aug 23 06:24:39 10.10.10.4 %ASA-6-604103: DHCP daemon interface inside: address granted 01b4.79a7.1895.33 (172.16.1.157)
The problem, is that dhcp leases renew,as you can see by the multiple instances of the same device at 172.16.1.211, for instance. I thought I could get around this by setting longer leases, as my understanding of how DHCP works is that leases would not start the renewal process until they reached their half-life but that is not working.
I'm also running into issues of address pool depletion because my leases are so long, and the ASA model I'm using has a hard limit to the size of it's scope.
Long story short, I need to parse those logs and retrieve the number of unique MAC addresses that occur in one of those logs. Any ideas on how this can be accomplished with bash? If I knew how to pull that info from one of the files, I could get through setting up process to do it for all of them using cron or something. I am not a programmer, however, I'm a network engineer. Any help would be appreciated.
Thanks,

Answer 1

Long story short, I need to parse those logs and retrieve the number of unique MAC addresses that occur in one of those logs.
Yes, given the regular nature of the data in your log files, this is very easy to do with several different tools.
The most basic would be to use cut
cut -d" " -f13 | sort | uniq -c
A more advanced tool is awk, and it provides many logic enhancments that allow you to add as many conditional statements as you want to filter the data as needed. For your case, though it is still very simple,
awk '{print $12}' | sort | uniq -c
In both cases, cut and awk, I only had to count over the number of fields in your data to the value of interest, and then specify that as the column (field number in awk-speak).
(when testing these answers, I found that using cut required using -d" " and -f13 (for some reason). I thought cut defaulted to -d" " but I had to specify it explicitly for the code to work).
Of course in both examples, I'm using the sort and uniq utilities, (man uniq for the how-to). uniq, has several options, and the -c option indicates count, so the data needs to be sorted for the counts to accumulate correctly (I missed that in my original comment).
Just for example, you could extend your counter to filter by the date value at the front of each record with
awk '/^Aug 23/{print $12}' | sort | uniq -c
But there are many more filtering and logic tools that you can use with awk.
If you're going to be working with logfile data regularly (or other non-XML-like data), I'd recommend working thru the Grymoire's Awk Tutorial .
IHTH