Country and External IP Bash script - linux

I created a script based on what I could find on the INTERNET and some bash tutorials, that will show me my external IP and the country it's located in.
#
Script looks like this:
#!/bin/bash
wanip=$(dig +short myip.opendns.com #resolver1.opendns.com);
echo "$wanip" > /root/Documents/filewanip;
iplist="/root/Documents/filewanip"
while read IP;do
whois "$IP"
done < "$iplist" | grep "country" >geoloc
cat geoloc filewanip
rm filewanip geoloc
#
Output looks like this:
country: Holland
183.64.132.80
#
Problem is that I don't want to use files to do this as file structure obviously changes from system to system.
How can I make it in an elegant way so the check is made and stored into a variable(s) and then displayed directly into the shell?
John Connor

As I understand it, your goal is to eliminate the use of temporary files. In that case:
#!/bin/bash
wanip=$(dig +short myip.opendns.com #resolver1.opendns.com);
echo "$wanip" | while read ip; do
echo "$ip $(whois "$ip" | awk ' /[Cc]ountry/{print $2}')"
done
The above was written as if dig returns more than one address for your IP. If that is not the case, then the while loop is superfluous.
If you are only expecting one IP, then:
#!/bin/bash
ip=$(dig +short myip.opendns.com #resolver1.opendns.com);
echo "$ip $(whois "$ip" | awk ' /[Cc]ountry/{print $2}')"
Notes:
I converted IP to ip because it is best practices to use lower or mixed case names for your shell variables.
My whois returns a line with Country, not country. So, I made the search for country case insensitive.

Related

checking information of different computers in a network using bash script

I am trying to write a bash script to access a file (nodeNames) that contains ip addresses to different computers in a cluster network, ssh into each of these computers and output some basic information namely: Hostname, Host IPaddress, load average and the process using the most memory and append all these information into a file seperating each wit commas. Also, each of the computers have same user and password. This is my code so far but it isn't working, please I need help here
egrep -ve '^#|^$'nodeNames | while read a
do
ssh $a "$#" &
output1=`hostname`
#This will display the server's IP address
output2=`hostname -i`
#This will output the server's load average
output3=`uptime | grep -oP '(?<=average:).*'| tr -d ','`
#This outputs memory Information
output4=`ps aux --sort=-%mem | awk 'NR<=1{print $0}'`
#This concantenates all output to a single line of text written to
echo "$output1, $output2, $output3, $output4" | tee clusterNodeInfo
done
You need to understand what is executed on which computer. The shell script you start is executed on your host A and you want information from your host B. ssh $a "$#" & will not suddenly make all the commands execute on the remote host B. Therefore, the
output1=`hostname`
will be executed on host A and output1 will have the hostname of host A.
You may also want to put the tee outside the loop or use tee -a to prevent overwriting your output file.
For bash, use $() instead of `` .
So, that would make your script:
egrep -ve '^#|^$'nodeNames | while read a
do
output1=$(ssh $a hostname)
#This will display the server's IP address
output2=$(ssh $a hostname -i)
#This will output the server's load average
output3=$(ssh $a "uptime | grep -oP '(?<=average:).*'| tr -d ','")
#This outputs memory Information
output4=$(ssh $a "ps aux --sort=-%mem | awk 'NR<=1{print $0}'")
#This concantenates all output to a single line of text written to
echo "$output1, $output2, $output3, $output4" | tee -a clusterNodeInfo
done
(have not tested it, but it should be something like this)

formatting whois results to CSV using BASH and grep

I've been trying to get my head round this for a few hours now and thought I'd come here and ask for help.
I have a CSV file of IP addresses from a log file that I want to run through and get a WHOIS netrange and company name from and then append the result to the end of the CSV.
So far what I have managed to do is get the whois results to a separate csv
echo ip, company, > result.csv
for ip in $(grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" source.csv); do
whois $ip | grep -i -e 'netrange\|inetnum' -e 'org-name\|orgname' \
| awk 'BEGIN{FS="NetRange:|inetnum:|OrgName:|org-name:"} {print $2","$3}'
|xargs; done >> result.csv
my challenge is how to add my 2 new columns back into the source.csv? I have tried using
paste -d, source.csv result.csv
but all that happens is the values in result.csv overwrite the first few columns of source.csv
my source.csv looks something like the below
ip address requests number of visits
66.249.90.77 2149 200
66.249.66.1 216 233
My result.csv
ip range company
66.249.64.0 - 66.249.95.255 Google Inc.
66.249.64.0 - 66.249.95.255 Google Inc.
i would like my final csv to look like
ip requests number of visits ip range company
66.249.90.77 2149 200 66.249.64.0 - 66.249.95.255 Google Inc.
66.249.66.1 2161 233 66.249.64.0 - 66.249.95.255 Google Inc.
If possible I would prefer to accomplish this with BASH rather than installing any 3rd party tools etc. I have tried the python package ipwhois but my python knowledge is far less than my limited BASH knowledge so I abandoned it lest I continue wasting time!
Any help is much appreciated.
Try putting this in a script file and running it while inside the directory containing your data file.
#!/bin/bash -ue
# Pattern for spacing characters
sp="[[:space:]]*"
#Pattern for non-spacing characters
nsp="[^[:space:]]+"
# Iterate on each line in the data file
while IFS= read -r line
do
[[ "$line" =~ ^$sp($nsp)$sp($nsp)$sp($nsp)$sp$ ]] || continue
f1="${BASH_REMATCH[1]}"
f2="${BASH_REMATCH[2]}"
f3="${BASH_REMATCH[3]}"
# Extract information from whois
whois_data="$(whois "$f1")"
range="$(grep NetRange <<<"$whois_data" | cut -f2 -d":")"
company="$(grep OrgName <<<"$whois_data" | cut -f2 -d":")"
echo $f1,$f2,$f3,$range,$company
done <"source.csv"
The output is formatted as fields separated by commas, and there should be some trimming of the range and company variables before they are used (to remove spaces at the beginning and end), but this should give you the idea.
Basically, the code does not try to merge two files together, but rather extracts the fields from each line of the source file, performs the whois, extracts the two fields needed from it, and then outputs all five fields to the output without an intermediary step.

Mail output with Bash Script

SSH from Host A to a few hosts (only one listed below right now) using the SSH Key I generated and then go to a specific file, grep for a specific word with a date of yesterday .. then I want to email this output to myself.
It is sending an email but it is giving me the command as opposed to the output from the command.
#!/bin/bash
HOST="XXXXXXXXXXXXXXXXXX, XXXXXXXXXXXXX"
DATE=$(date -d "yesterday")
INVALID=' cat /xxx/xxx/xxxxx | grep 'WORD' | sed 's/$/.\n/g' | grep "$DATE"'
COUNT=$(echo "$INVALID" | wc -c)
for x in $HOSTS
do
ssh BLA#"$x" $COUNT
if [ "$COUNT" -gt 1 ];
then
EMAILTEXT=""
if [ "$COUNT" -gt 1 ];
then
EMAILTEXT="$INVALID"
fi
fi
done | echo -e "$EMAILTEXT" | mail XXXXXXXXXXX.com
This isn't properly an attempt to answer your question, but I think you should be aware of some fundamental problems with your code.
INVALID=' cat /xxx/xxx/xxxxx | grep 'WORD' | sed 's/$/.\n/g' | grep "$DATE"'
This assigns a simple string to the variable INVALID. Because of quoting issues, s/$/.\n/g is not quoted at all, and will probably be mangled by the shell. (You cannot nest single quotes -- the first single-quoted string extends from the first quote to the next one, and then WORD is outside of any quotes, followed by the next single-quoted string, etc.)
If your intent is to execute this as a command at this point, you are looking for a command substitution; with the multiple layers of uselessness peeled off, perhaps something like
INVALID=$(sed -n -e '/WORD/!d' -e "/$DATE/s/$/./p" /xxx/xxx/xxxx)
which looks for a line matching WORD and $DATE and prints the match with a dot appended at the end -- I believe that's what your code boils down to, but without further insights into what this code is supposed to do, it's impossible to tell if this is what you actually need.
COUNT=$(echo "$INVALID" | wc -c)
This assigns a number to $COUNT. With your static definition of INVALID, the number will always be 62; but I guess that's not actually what you want here.
for x in $HOSTS
do
ssh BLA#"$x" $COUNT
This attempts to execute that number as a command on a number of remote hosts (except the loop is over HOSTS and the variable containing the hosts is named just HOST). This cannot possibly be useful, unless you have a battery of commands named as natural numbers which do something useful on these remote hosts; but I think it's safe to assume that that is not what is supposed to be going on here (and if it was, it would absolutely be necessary to explain this in your question).
if [ "$COUNT" -gt 1 ];
then
EMAILTEXT=""
if [ "$COUNT" -gt 1 ];
then
EMAILTEXT="$INVALID"
fi
fi
So EMAILTEXT is either an empty string or the value of INVALID. You assigned it to be a static string above, which is probably the source of your immediate question. But even if it was somehow assigned to a command on the local host, why do you need to visit remote hosts and execute something there? Or is your intent actually to execute the command on each remote host and obtain the output?
done | echo -e "$EMAILTEXT" | mail XXXXXXXXXXX.com
Piping into echo makes no sense at all, because it does not read its standard input. You should probably just have a newline after done; though a possibly more useful arrangement would be to have your loop produce output which we then pipe to mail.
Purely speculatively, perhaps something like the following is what you actually want.
for host in $HOSTS; do
ssh BLA#"$host" sed -n -e '/WORD/!d' -e "/$DATE/s/$/./p" /xxx/xxx/xxxx |
grep . || echo INVALID
done | mail XXXXXXXXXXX.com
If you want to check that there is strictly more than one line of output (which is what the -gt 1 suggests) then this may need to be a little bit more complicated.
Your command substitution is not working. You should read up on how it works but here are the problem lines:
COUNT=$(echo "$INVALID" | wc -c)
[...]
ssh BLA#"$x" $COUNT
should be:
COUNT_CMD="'${INVALID} | wc -c'"
[...]
COUNT=$(ssh BLA#"$x" $COUNT_CMD)
This inserts the value of $INVALID into the string, and puts the whole thing in single quotes. The single quotes are necessary for the ssh call so the pipes aren't evaluated in the script but on the remote host. (COUNT is changed to COUNT_CMD for readability/clarity.)
EDIT:
I misread the question and have corrected my answer.

Extract parent domain name from a list of url through Bash ShellScripting

I have a list of urls like this:
http://noto.zrobimystrone.pl/pucenter/images/NGdocs/
http://visionwebmkt.com/unsubscribe.php?M=879552&C=b744d324e38f5f3b0bcf549f1d57a3ab&L=20&N=497
http://www.meguiatramandai.com.br/unsubscribe.php?M=722&C=8410431be55bf12faac13d18982d71cd&L=1&N=3
http://www.contatoruy.in/link.php?M=86457&N=4&L=1&F=H
http://www.maxxivrimoveis.com.br/
http://www.meguiatramandai.com.br/unsubscribe.php?M=722&C=8410431be55bf12faac13d18982d71cd&L=1&N=2
http://arm.smilecire.com/ch+urch38146263923bpa.stor/imp-roved258021029his+health212149011
http://hurl.zonalrems.com/ge.tyo-ur584372780599hea+lth247408058un/der+control21211901
http://harp.doomyjupe.com/see.this-better/life+58291551346csexdrive663295668+better/how.981692016
http://beefy.toneyvaws.com/no+tice/how/35306640b+see/app=5429204last/attempt=457943182
http://kirk.yournjuju.com/shop/sam.sclub-win=ter/58387369768esame+673844946.bett.er-loo.k981686408
http://idly.theirpoem.com/veri-fy/notice-7853508818b2glob/al=who.43639603inc.lusion-610549278
http://wva188.suleacatan.com/credit-score/review/-551694841511001sfdghsfdgsdfg63887839
http://cop.forterins.com/app.lyto=face962540097dtolo+oko.ung268570307yo.un-ger8752507
http://vni116.gaelsyaray.com/qertqetert//-dghjghjghd5531864856415612229498430
http://ticket.prategama.com/shop/sam.sclub-win=ter/752490935same+226373195.bett.er-loo.k212801
http://cbu125.quetxviii.com/cvbnvbn7551116db537203--swrtytry664896546
http://c5a.dicadodia.com.br/pass4sp09/NetAffProTeste-1.html
http://snub.woadsbevy.com/ama/zing-753773417oppe-tun/ity+217801.is-here/now=236922473
http://mkt.livrariacultura.com.br/pub/cc?_ri_=X0Gzc2X%3DWQpglLjHJlYQGgzfB7tPi0PuyyJ71ES
I wanna extract only the parents domain names, for example:
http://noto.zrobimystrone.pl/pucenter/images/NGdocs/
http://visionwebmkt.com/unsubscribe.php?M=879552&C=b744d324e38f5f3b0bcf549f1d57a3ab&L=20&N=497
http://www.meguiatramandai.com.br/unsubscribe.php?M=722&C=8410431be55bf12faac13d18
Into
zrobimystrone.pl
visionwebmkt.com
meguiatramandai.com.br
I have tried
awk '{gsub("http://|/.*","")}1' list.txt
and got the following results:
noto.zrobimystrone.pl
visionwebmkt.com
www.meguiatramandai.com.br
www.contatoruy.in
www.maxxivrimoveis.com.br
www.meguiatramandai.com.br
arm.smilecire.com
hurl.zonalrems.com
harp.doomyjupe.com
beefy.toneyvaws.com
but dont know how to get only the parent name from noto.zrobimystrone.pl for instance.
Using awk
awk -F \/ '{l=split($3,a,"."); print (a[l-1]=="com"?a[l-2] OFS:X) a[l-1] OFS a[l]}' OFS="." file|sort -u
contatoruy.in
dicadodia.com.br
doomyjupe.com
forterins.com
gaelsyaray.com
livrariacultura.com.br
maxxivrimoveis.com.br
meguiatramandai.com.br
prategama.com
quetxviii.com
smilecire.com
suleacatan.com
theirpoem.com
toneyvaws.com
visionwebmkt.com
woadsbevy.com
yournjuju.com
zonalrems.com
zrobimystrone.pl
You can use this awk:
awk -F'.' '{gsub("http://|/.*","")} NF>2{$1="";$0=substr($0, 2)}1' OFS='.' list.txt
zrobimystrone.pl
visionwebmkt.com
meguiatramandai.com.br
contatoruy.in
maxxivrimoveis.com.br
meguiatramandai.com.br
smilecire.com
zonalrems.com
doomyjupe.com
toneyvaws.com
yournjuju.com
theirpoem.com
suleacatan.com
forterins.com
gaelsyaray.com
prategama.com
quetxviii.com
dicadodia.com.br
woadsbevy.com
livrariacultura.com.br
A "simple" bash solution. Tested in bash shell on Solaris 11.2 x86.
#!/bin/bash
while IFS=/ read HTTP NULL FQDN PAGE
do
PARENT=${FQDN#*.}
if [[ $PARENT != *"."* ]]
then echo $FQDN
else echo $PARENT
fi
done < fileOfURLs.txt
Without the string contains pattern test, too much of the domain could be stripped away. The if paragraph can be reduced,so the whole script now looks like this:
#!/bin/bash
while IFS=/ read HTTP NULL FQDN PAGE
do
PARENT=${FQDN#*.}
[[ $PARENT != *"."* ]] && echo $FQDN || echo $PARENT
done < fileOfURLs.txt
The bash variable substitution is taking the contents of the variable FQDN and stripping from the left any character up to and including the first dot.
The test condition is asking if the contents of the PARENT variable does not contain a dot. If it does not hold a dot somewhere in the value, the test evaluates to true and will display the original FQDN contents. If the test evaluates to false, (there is still a dot in the value) the contents of PARENT are displayed.
I guess it depends on what you mean by parent. If by "parent", you mean the top of the zone apex in DNS (e.g., zrobimystrone.pl ), then the right way to do this is to look that up in DNS. There's a trick with DNS where you get back the parent zone SOA record if you ask for the SOA for any name.. So, try this:
for i in $(awk '{gsub("http://|/.*","")}1' list.txt); do dig soa $i | grep -v ^\; | grep SOA | awk '{print $1}'; done
This will give you a much more accurate list, but it runs way slower and is sub-optimal. The other answers don't take into account all the possible variations of TLD names used within TLDs, e.g., www.somecompany.org.uk, so it all depends on how accurate you need this to be.
An easy solution to get parent domain name
echo http://www.humkinar.pk | awk -F '/' '{print $3}'
www.humkinar.pk

Is there any better way to get mac address from arp table?

I want to get a mac address from arp table by using ip address. Currently I am using this command
arp -a $ipAddress | awk '{print $4}'
This command prints what I want. But I am not comfortable with it and I wonder if there is any built-in way or more stable way to do this.
You can parse the /proc/net/arp file using awk:
awk "/^${ipAddress//./\.}\>/"' { print $4 }' /proc/net/arp
but I'm not sure it's simpler (it saves one fork and a subshell, though).
If you want a 100% bash solution:
while read ip _ _ mac _; do
[[ "$ip" == "$ipAddress" ]] && break
done < /proc/net/arp
echo "$mac"
Well, you could write a program (such as in C) to actually use the ARP protocol (yes, I know that's redundant, like ATM machine or PIN number) itself to get you the information but that's likely to be a lot harder than a simple pipeline.
Perhaps you should examine your comfort level a little more critically, since it's likely to cause you some unnecessary effort :-)
The manpage for the Linux ARP kernel module lists several methods for manipulating or reading the ARP tabes, ioctl probably being the easiest.
The output of arp -a is locale dependent (i.e. it changes with your system language). So it might be a good idea to at least force it to the default locale:
LC_ALL=C arp -a $ipAddress | awk '{print $4}'
However, I share your fear that the output of arp -a is not meant to be parsed. If your program is restricted to linux system, another option would be to parse the file /proc/net/arp. This file is exported by the kernel and is what arp itself parses to get its information. The format of this file is described in the manpage proc(5), see man 5 proc.
This can be easily done with awk:
awk '$1==IPADDRESS {print $4}' /proc/net/arp
Here's an awk + sed solution which doesn't assume the column number is always 4.
#!/bin/bash
cat /proc/net/arp |\
# remove space from column headers
sed 's/\([^ ]\)[ ]\([^ ]\)/\1_\2/g' |\
# find HW_address column number and/or print that column
awk '{
if ( !column ) {
for (i = 1; i <= NF; i++ ) {
if ( $i ~ /HW_address/ ) { column=i }
};
print $column
}
else {
print $column
}
}'
There are still fragile assumptions here, such as the column name being "HW address".
Update, removed PIPE
sed -nr 's/^'${ipAddress//./\.}'.*(([0-9A-Za-z]{2}:){5}[0-9A-Za-z]{2}).*$/\1/p' /proc/net/arp
Solution for non-fixed column;
arp -a $ipAddress | sed -n 's/^.*\(\([0-9A-Z]\{2\}:\)\{5\}[0-9A-Z]\{2\}\).*$/\1/p'
Explanation
^.* - Match start of string ^ followed by any character .*.
[0-9A-Z]\{2\}: - Match any character of numeric alpha-numeric twice followed by colon.
\([0-9A-Z]\{2\}:\)\{5\} - Match the pattern between the ( ) five times.
[0-9A-Z]\{2\} - Match any character of numeric alpha-numeric twice.
.*$ - Match any characters zero or more times .* until end of string $.
\1/p - Return capture pattern 1 / p print the match.
You can use this one for scripting:
awk ' $1~/[[:digit:]]/ {print $4}' /proc/net/arp
what it do:
read /proc/net/arp (standard arp output)
searchig for strings with [0-9]
get the 4rd "column" with mac adresses
Enjoy!
I prefer to use the arping command to explicitly query the MAC of some IP address (this also updates the local ARP cache):
arping -c 1 192.168.2.24 | grep -Eo "([0-9a-fA-F]{2}:){5}[0-9a-fA-F]"
It's very useful to find if there exist two or more hosts using the same IP address (add -D option), or to check the current IP addresses used in the local VLAN with a simple script like:
for i in $(seq 1 254); do
IP="192.168.5.$i"
MAC=$(arping -c 1 $IP | grep -Eo "([0-9a-fA-F]{2}:){5}[0-9a-fA-F]")
if [ "$MAC" ] ; then
echo "$IP $MAC"
fi
done
Note that arping can't detect the IP address of the local host in this way (but we can add checks in the script to show it if exists in the range).
There exist several versions of arping with slightly different options and output. In Linux Ubuntu there are one in the package iputils-arping and other in the package arping.
Note: To answer the question and not the problem, when filtering /proc/net/arp you must use a regex that ensures the full match, like ending the expression with a space (otherwise, in this example, it will show also 2.240-2.249 addresses if present):
ipaddress="192.168.2.24"
grep "^${ipaddress} " /proc/net/arp | grep -Eo "([0-9a-fA-F]{2}:){5}[0-9a-fA-F]")

Resources