Curl function cannot parse proxy coming from a variable in bash - linux

Question

I have a proxy txt file with the format:
102.129.249.120:3128
102.129.249.120:8080
101.4.136.34:8080
103.228.117.244:8080
etc
and I am trying to create a bash script that will do (for example): curl -x "$IP" google.com.
Unfortunately, curl is giving me an unsupported proxy syntax for all the proxies.
Any ideas?
BTW, I really doubt this question has been repeated as I have tried everything else to no avail.
My script:
Number=$(wc -l < ProxyList.txt)
for ((i=1;i<=$Number;++i)) do
ip=$(head -n ${i} ProxyList.txt | tail -n +${i})
curl -p -x "$ip" 'webpage' -H 'user-agent' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: wpml_referer_url=referer; _icl_current_language=es; PHPSESSID=tpikve1vl4ued06i082vprqdo1' -H 'If-Modified-Since: Mon, 16 May 2016 07:27:13 GMT' -H 'If-None-Match: "3d6-532f08d9d7640-gzip"' -H 'Cache-Control: max-age=0' -m 6
done
A small sample of my proxy list:
102.129.249.120:3128
102.129.249.120:8080
101.4.136.34:8080
103.228.117.244:8080
103.253.27.108:80
104.45.188.43:3128
104.250.34.179:80
105.27.238.161:80
104.154.143.77:3128
110.243.20.2:9999
111.68.26.237:8080
106.104.151.142:58198
113.252.95.19:8197
115.231.31.130:80
118.69.50.154:80
118.69.50.154:443
119.81.189.194:80
119.81.189.194:8123
119.81.199.81:8123
119.81.199.83:8123
119.81.199.80:8123
12.139.101.100:80
12.139.101.101:80
119.81.199.85:31288
119.81.199.86:8123
119.81.199.87:8123
12.139.101.102:80
124.156.98.172:443
13.228.91.252:3128
138.197.157.32:3128
138.197.157.32:8080
138.68.240.218:8080
138.68.240.218:3128
138.68.60.8:8080
138.68.60.8:3128

Answer 1

Your input file has carriage return characters at the end of each line.
Each line in your input file ends with \r\n instead of just \n.
You can check with od:
$ head -1 ProxyList.txt | od -c
0000000 1 0 2 . 1 2 9 . 2 4 9 . 1 2 0 :
0000020 3 1 2 8 \r \n
0000026
So in your script, $ip has in fact a value of 102.129.249.120:3128\r.
You can remove the \r characters with tr for example:
while read proxy; do
curl -p -x $proxy $webpage
done < <( tr -d '\r' < ProxyList.txt )

Answer 2

try this:
for ip in $(cat ProxyList.txt)
do
curl -p -x "$ip" 'webpage' -H 'user-agent' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: wpml_referer_url=referer; _icl_current_language=es; PHPSESSID=tpikve1vl4ued06i082vprqdo1' -H 'If-Modified-Since: Mon, 16 May 2016 07:27:13 GMT' -H 'If-None-Match: "3d6-532f08d9d7640-gzip"' -H 'Cache-Control: max-age=0' -m 6
done
but the problem with curl might be, that should set the environment variables http_proxy and https_proxy like this:
export http_proxy=http://1.2.3.4:3128/
export https_proxy=http://1.2.3.4:3128/

Answer 3

As per the curl man page, the -x (or --proxy) switch can be prefixed with the protocol in front of the argument (if omitted, I assume that it defaults to http://):
-x, --proxy [protocol://]host[:port]
A simple bash script with xargs would look like:
#!/bin/bash
webpage=${1:-http://google.com}
cat ProxyList.txt \
| xargs -n1 -I{} curl -p -x http://{} "$webpage" -H 'user-agent' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: wpml_referer_url=referer; _icl_current_language=es; PHPSESSID=tpikve1vl4ued06i082vprqdo1' -H 'If-Modified-Since: Mon, 16 May 2016 07:27:13 GMT' -H 'If-None-Match: "3d6-532f08d9d7640-gzip"' -H 'Cache-Control: max-age=0' -m 6