I'm currently working on a script that reads in several hundred IP addresses listed line-by-line in a file. The script is supposed to take the IP addresses and then output the IP address and its longitude and latitude to a new file. However, whenever I try to run the script I received multiple "curl: (3) Illegal characters found in URL". I've been troubleshooting it for a couple of days, and so far I've come up with nothing. Can anyone put me in the right direction to figuring out the problem?
Thanks in advance for any help.
This is the script I'm using.
#!/bin/bash
cat ipCheck.txt | while read line
do
curl "https://api.ipstack.com/"$line"access_key=9c04ea7631a32590cac23eb27ec6c104&foraat=1&fields=ip,latitude,longitude"
done >> locations.txt
I'm currently using a test text file with 10 IP addresses. It is as follows
101.249.211.209
102.165.32.39
102.165.35.37
102.165.49.193
103.27.125.18
103.3.61.193
103.78.132.4
104.143.83.13
104.143.83.8
104.149.216.71
Nothing jumps out as being wrong. Have you tried commenting out your curl line and running the loop with something like:
cat ipCheck.txt | while read line
do
#curl "https://api.ipstack.com/"$line"access_key=9c04ea7631a32590cac23eb27ec6c104&foraat=1&fields=ip,latitude,longitude"
echo "\"$line\""
done >> locations.txt
This should help to locate an extra space or something strange with the formatting.
If it isn't the formatting in your file, you could try:
cat ipCheck.txt | while read line
do
curl "https://api.ipstack.com/${line}access_key=9c04ea7631a32590cac23eb27ec6c104&foraat=1&fields=ip,latitude,longitude"
done >> locations.txt
This should give the same result, but you could have something strange going on with your environment.
According to a previous comment a solution is to remove any additional character like the carriage return \r and checking using printf:
while IFS= read -r line; do
# fix `\r` at end
line=$(echo ${line} | tr -d '\r')
printf 'Downloading %q\n' "$line"
./download.sh $FOLDER $line
done < "$DS"
Related
I've created a script that seperates the IP that it finds, based on NIC card name input.
#!/bin/bash
echo what is your NIC?
read NIC
IP=`ifconfig $NIC 2>/dev/null|awk '/inet addr:/ {print $2}'|sed 's/addr://'`
NEWSTRING=${IP:0:6}
ALPHARETTA="12.101"
EUFAULA="12.102"
if [ "${NEWSTRING}" = "${ALPHARETTA}" ] ; then
echo I'm in Alpharetta
else
echo I'm in Eufaula
fi
If eth0 were to be 12.101.1.1 it would only take (12.101)
I'm comparing 12.101 and 12.101 for my tests... and i'm getting this echo back....
what is your NIC?
eth0
Im in Alpharetta
else
echo Im in Eufaula
I'm obviously doing something silly, and not seeing it.. could somebody point me in the correct direction?
The bash parser is seeing the apostrophe characters you are trying to echo in the word "I'm" and it thinks you are trying to print one long string that spans from line 10 to line 12 of your script. You can even see how the syntax highlighting on this site is also indicating a problem. You should wrap the message you are echoing in quotes. For example:
echo "I'm in Alpharetta"
I have this part of script, which I cannot get to work. Been searching everywhere, but I must be missing something here.
export RULE=`cat iptables_sorted.txt`
ssh hostname << EOF
for line in $RULE; do
echo \$line >> test.txt (I have tried with and without slashes, etc...)
done
exit
EOF
After running this part of script, I get
stdin: is not a tty
-bash: line 2: syntax error near unexpected token `103.28.148.0/24'
-bash: line 2: `103.28.148.0/24'
...which is totally weird, because the iptables_sorted.txt is just full of ip ranges (when I run it locally, it works).
Newlines in $RULE cause the problem. Replace them by spaces:
RULE=$(< iptables_sorted.txt)
RULE=${RULE//$'\n'/ }
ssh hostname << EOF
for line in $RULE ; do
echo \$line >> test.txt
done
EOF
Note that this wouldn't work with lines containing whitespace.
Don't use for to iterate over a file; use while. This also demonstrates piping the output of the loop, not just every individual echo, to the remote host. cat is used to read the incoming data and redirect it to the final output file.
while IFS= read -r line; do
echo "$line"
done | ssh hostname 'cat > test.txt'
I have a very strange problem I've never encountered and could not find anything related on other posts here, I have a shell script that just greps a test host file(will use real one once this works) in a while loop but it will only output the last line:
#!/bin/bash
while read line
do
grep $line /etc/test/hosts
done < temp-file.txt
here is the /etc/test/hosts file:
device1 192.168.0.1
device2 192.168.0.2
device3 192.168.0.3
// cut
device50 192.168.0.50
the temp-file.txt is identical to the test host file for testing
And as mentioned, here is the output - only showing last line:
device3 192.168.0.50
If I change the grep command to just echo $line it outputs correctly. I have tried changing the number of lines in the test host file to be more and less but same result. I have also done the grep command from cli and it works fine on every device I grep for. I have also tried putting the $line in double quotes but that also has not changed anything.
I have also tried an alternate while loop using the while IFS="\n" method but same results. This seems like it should be extraordinarily simple but I'm having issues. Am I doing something wrong or is this maybe a bug in bash?
question answered by Eric Renouf via comment.
solution:
add sed -i 's/\r$//' temp-file.txt to top of script to remove the \r that were at the end of each line.
I have a list of IP's I want to snmpget into a file. I'm having issues writing output to the file.
OID=1.3.6.1.2.1.25.3.2.1.3.1
cat printers.csv | while read IP ; do
OUT=$(snmpget -v1 -c public $IP $OID)
printf '%s, %s\n' $IP $OUT >> printerNames.csv
done
I'm new to the printf command. I'm guessing that's where it's messing up bc output is being written just incorrectly. Also, when there is no response it echos to console and I'd like it written to the output file. Any help will be much appreciated.
Try this:
OID=1.3.6.1.2.1.25.3.2.1.3.1
while read IP ; do
OUT=$(snmpget -v1 -c public "$IP" "$OID") && printf '%s, %s\n' "$IP" "$OUT"
done < printers.csv 2>&1 > printerNames.csv
It's a good idea to always quote parameter expansions unless you have a good reason not to. The redirections are applied to the while loop. read will read a line at a time from the input file (no cat needed); anything written to standard error is instead copied to standard output, and the standard output (errors included) is redirected to the output file. The printf is only executed if the snmpget command succeeds (I'm assuming it has a non-zero exit status if the lookup fails; that may not be the case).
It sounds like printers.csv has DOS line endings (\r\n). The carriage return is included as the last character of each line. When you print $IP, it prints the address, then the carriage return, which moves the cursor back to the beginning of the line. This causes the remainder of the line (, $OUT) to overwrite the address. To remove the carriage return, run the input file through dos2unix, or use some other method to turn the DOS line endings into UNIX line endings (\n alone).
I have a weird bug, basically I have set up a function to run remote commands via SSH on a box and get the lan MAC address and some other info. I want to write this info into a csv file.
When I run BOXLANMAC=$(remote_command "ifconfig eth0 | grep HWaddr | cut -d' ' -f11") I can echo $BOXLANMAC and get the expected output.
However, when I run echo $BOXLANMAC,$BOXLANMAC I get ,XX:XX:XX:XX:XX:XX where I expect to see XX:XX:XX:XX:XX:XX,XX:XX:XX:XX:XX:XX. I have tried many permutations of the echo command, using quotes and escape characters for the comma, but not had any success. I'm sure this is really simple and I should have been able to figure it out, but google seems to just get me results about splitting strings on commas.
As das.cyklone and I mentioned above, you might want to see if the remote command is returning whitespaces which are affecting the output of the echo command. Perhaps using tr to remove any whitespaces from the $boxlanmac variable will solve the problem. See How to trim whitespace from a Bash variable? for ways to do this.
Your example seems to work fine during testing. I can offer two suggestions:
1) Use awk instead of cut for whitespace. It's more reliable and easier to test.
2) If echo isn't working, try printf. It gives you more control with statements.
~-> BOXLANMAC=$(ifconfig eth0 | grep HWaddr | awk '{ print $5 }')
~-> echo ${BOXLANMAC}
78:2B:CB:88:E5:AR
~-> echo ${BOXLANMAC},${BOXLANMAC}
78:2B:CB:88:E5:AR,78:2B:CB:88:E5:AR
~-> printf "%s,%s\n" ${BOXLANMAC} ${BOXLANMAC}
78:2B:CB:88:E5:AR,78:2B:CB:88:E5:AR
~->
Good luck.
Have you tried: echo ${BOXLANMAC},${BOXLANMAC} ?