I have a x.example which serves traffic for both a.example and b.example.
x.example has certificates for both a.example and b.example. The DNS for a.example and b.example is not yet set up.
If I add an /etc/hosts entry for a.example pointing to x.example's ip and run curl -XGET https://a.example, I get a 200.
However if I run curl --header 'Host: a.example' https://x.example, I get:
curl: (51) SSL: no alternative certificate subject name matches target
host name x.example
I would think it would use a.example as the host. Maybe I'm not understanding how SNI/TLS works.
Because a.example is an HTTP header the TLS handshake doesn't have access to it yet? But the URL itself it does have access to?
Indeed SNI in TLS does not work like that. SNI, as everything related to TLS, happens before any kind of HTTP traffic, hence the Host header is not taken into account at that step (but will be useful later on for the webserver to know which host you are connecting too).
So to enable SNI you need a specific switch in your HTTP client to tell it to send the appropriate TLS extension during the handshake with the hostname value you need.
In case of curl, you need at least version 7.18.1 (based on https://curl.haxx.se/changes.html) and then it seems to automatically use the value provided in the Host header. It alo depends on which OpenSSL (or equivalent library on your platform) version it is linked to.
See point 1.10 of https://curl.haxx.se/docs/knownbugs.html that speaks about a bug but explains what happens:
When given a URL with a trailing dot for the host name part: "https://example.com./", libcurl will strip off the dot and use the name without a dot internally and send it dot-less in HTTP Host: headers and in the TLS SNI field.
The --connect-to option could also be useful in your case. Or --resolve as a substitute to /etc/hosts, see https://curl.haxx.se/mail/archive-2015-01/0042.html for am example, or https://makandracards.com/makandra/1613-make-an-http-request-to-a-machine-but-fake-the-hostname
You can add --verbose in all cases to see in more details what is happening. See this example: https://www.claudiokuenzler.com/blog/693/curious-case-of-curl-ssl-tls-sni-http-host-header ; you will also see there how to test directly with openssl.
If you have a.example in your /etc/hosts you should just run curl with https://a.example/ and it should take care of the Host header and hence SNI (or use --resolve instead)
So to answer your question directly, replace
curl --header 'Host: a.example' https://x.example
with
curl --connect-to a.example:443:x.example:443 https://a.example
and it should work perfectly.
The selected answer helped me find the answer, even though it does not contain the answer. The answer in the mail/archive link Patrick Mevzek provided has the wrong port number. So even following that answer will cause it to continue to fail.
I used this container to run a debugging server to inspect the requests. I highly suggest anyone debugging this kind of issue do the same.
Here is how to address the OP's question.
# Instead of this:
# curl --header 'Host: a.example' https://x.example
# Do:
host=a.example
target=x.example
ip=$(dig +short $target | head -n1)
curl -sv --resolve $host:443:$ip https://$host
If you want to ignore bad certificates matches, use -svk instead of -sv
curl -svk --resolve $host:443:$ip https://$host
Note: Since you are using https, you must use 443 in the --resolve argument instead of 80 as was stated on the mail/archive
I had a similar need. Didn't have sudo access to update hosts file.
I use resolve parameter and also added the DNS host name as a header parameter.
--resolve <dns name>:<port>:<ip addr>
curl --request POST --resolve dns_name:443:a.b.c.d 'https://dns_name/x/y' --header 'Host: dns_name' ....
Cheers..
Related
I am trying to download a serious of scripts ... unfortunately it doesn't work.
shell:
$ wget --secure-protocol=TLSv1 --user=username --password=password --no-check-certificate https://www.example.com/bla/foo/bar/secure/1.pdf
respond:
--2014-10-06 12:49:26-- https://www.example.com/bla/foo/bar/secure/1.pdf
Resolving www.example.com (www.example.com)... xxx.xxx.xx.xx
Connecting to www.example.com (www.example.com)| xxx.xxx.xx.xx|:443... connected.
OpenSSL: error:14094438:SSL routines:SSL3_READ_BYTES:tlsv1 alert internal error
Unable to establish SSL connection.
There can be lots of reasons why this fails with this error, among them:
server is unable to cope with newer TLS versions
server requires client authentication
server has a misbehaving SSL load balancer in front
there is a firewall between you and the server rejecting your traffic after initial inspection
That's all which can be said from the information you provide.
You might check the server against sslabs to get more information or provide more details in your question, like the real URL.
Edit: The requested server is www2.cs.fau.de. This server supports only SSLv3 and croaks on TLSv1 (instead of just responding with SSLv3), so you need to enforce SSLv3 with wget:
wget --secure-protocol=SSLv3 ...
The certificate of the server can be verified against the usual trusted CA on Linux, so you probably don't need the --no-check-certificate option.
Most browsers can access this site because they automatically downgrade to older SSL versions if connects with more modern versions does not succeed, but tools like curl or wget do not retry with downgraded versions.
I want to verify that my web application does not have a path traversal vulnerability.
I'm trying to use curl for that, like this:
$ curl -v http://www.example.com/directory/../
I would like the HTTP request to be explicitly made to the /directory/../ URL, to test that a specific nginx rule involving proxy is not vulnerable to path traversal. I.e., I would like this HTTP request to be sent:
> GET /directory/../ HTTP/1.1
But curl is rewriting the request as to the / URL, as can be seen in the output:
* Rebuilt URL to: http://www.example.com/
(...)
> GET / HTTP/1.1
Is it possible to use curl for this test, forcing it to pass the exact URL in the request? If not, what would be an appropriate way?
The curl flag you are looking for is curl --path-as-is .
I'm not aware of a way to do it via curl, but you could always use telnet. Try this command:
telnet www.example.com 80
You'll see:
Trying xxx.xxx.xxx.xxx...
Connected to www.example.com.
Escape character is '^]'.
You now have an open connection to www.example.com. Now just type in your command to fetch the page:
GET /directory/../ HTTP/1.1
And you should see your result. e.g.
HTTP/1.1 400 Bad Request
You can use an intercepting proxy to capture a request to your application and repeat the request with parameters changed, such as the raw URL that is requested from the application.
The free version of Burp Suite will allow this using the Repeater.
However, there are alternatives that should also allow this such as Zap, WebScarab and Fiddler2.
I have to make some request to a https site via openssl. In the browser there is no problem at all, it's a simple GET request without any cookies, I watch it via fiddler. My problem is that I have to make this request via openssl and it waits too long and then I get an empty response.
It looks like:
(cat <REQUEST>; sleep 5) | openssl s_client -quiet -connect <HOST>:<PORT> > <variable>
I have many https sites and only one of them causes this. What could be the problem?
Make sure you send all the headers your browser does. In particular, Referer and Accept may play a role in it (and Cookie, but you said there are no cookies).
Other than that, the server's SSL certificate may be unverifiable by your client. The browser is usually more forgiving, so your request would still be successful. If you can translate your request into a wget command (i.e. wget https://host:port/URL), if the certificate has a problem it will report something like "cannot verify <host>'s certificate". If you attempt the same request with wget --no-check-certificate https://host:port/URL and it succeeds, you'll know the certificate is the problem.
I have an app running on port 9100 on a remote server serving http pages. After I ssh into the server I can curl localhost 9100 and I receive the response.
However I am unable to access the same app from the browser using http://ip:9100
I am also unable to telnet from my local PC. How do I debug it? Is there a way to traceroute a particular IP and port combination, to see where it is being blocked?
Any linux tools / commands / utilities will be appreciated.
Thanks,
Murtaza
You can use the default traceroute command for this purpose, then there will be nothing to install.
traceroute -T -p 9100 <IP address/hostname>
The -T argument is required so that the TCP protocol is used instead of UDP.
In the rare case when traceroute isn't available, you can also use ncat.
nc -Czvw 5 <IP address/hostname> 9100
tcptraceroute xx.xx.xx.xx 9100
if you didn't find it you can install it
yum -y install tcptraceroute
or
aptitude -y install tcptraceroute
you can use tcpdump on the server to check if the client even reaches the server.
tcpdump -i any tcp port 9100
also make sure your firewall is not blocking incoming connections.
EDIT: you can also write the dump into a file and view it with wireshark on your client if you don't want to read it on the console.
2nd Edit: you can check if you can reach the port via
nc ip 9100 -z -v
from your local PC.
Firstly, check the IP address that your application has bound to. It could only be binding to a local address, for example, which would mean that you'd never see it from a different machine regardless of firewall states.
You could try using a portscanner like nmap to see if the port is open and visible externally... it can tell you if the port is closed (there's nothing listening there), open (you should be able to see it fine) or filtered (by a firewall, for example).
it can be done by using this command: tcptraceroute -p destination port destination IP. like: tcptraceroute -p 9100 10.0.0.50 but don't forget to install tcptraceroute package on your system. tcpdump and nc by default installed on the system. regards
If you use the 'openssl' tool, this is one way to get extract the CA cert for a particular server:
openssl s_client -showcerts -servername server -connect server:443
The certificate will have "BEGIN CERTIFICATE" and "END CERTIFICATE" markers.
If you want to see the data in the certificate, you can do: "openssl x509 -inform PEM -in certfile -text -out certdata" where certfile is the cert you extracted from logfile. Look in certdata.
If you want to trust the certificate, you can add it to your CA certificate store or use it stand-alone as described. Just remember that the security is no better than the way you obtained the certificate.
https://curl.se/docs/sslcerts.html
After getting the certificate use keytool to install it.
I have this proxy address: 125.119.175.48:8909
How can I perform a HTTP request using cURL like curl http://www.example.com, but specifying the proxy address of my network?
From man curl:
-x, --proxy <[protocol://][user:password#]proxyhost[:port]>
Use the specified HTTP proxy.
If the port number is not specified, it is assumed at port 1080.
General way:
export http_proxy=http://your.proxy.server:port/
Then you can connect through proxy from (many) application.
And, as per comment below, for https:
export https_proxy=https://your.proxy.server:port/
The above solutions might not work with some curl versions I tried them for myself(curl 7.22.0). But what worked for me was:
curl -x http://proxy_server:proxy_port --proxy-user username:password -L http://url
Hope it solves the issue better!
Beware that if you are using a SOCKS proxy, instead of a HTTP/HTTPS proxy, you will need to use the --socks5 switch instead:
curl --socks5 125.119.175.48:8909 http://example.com/
You can also use --socks5-hostname instead of --socks5 to resolve DNS on the proxy side.
as an adition to airween, another good idea is to add this into your .bashrc, so you'll be able to switch from non proxied to proxied environment:
alias proxyon="export http_proxy='http://YOURPROXY:YOURPORT';export https_proxy='http://YOURPROXY:YOURPORT'"
alias proxyoff="export http_proxy='';export https_proxy=''"
WHERE YOURPROXY:YOURPORT is exactly that, your ip and port proxy :-).
Then, simply doing
proxyon
your system will start to use the proxy, and just the opposite with:
proxyoff
use the following
curl -I -x 192.168.X.X:XX http://google.com
192.168.X.X:XX put your proxy server ip and port.
-v verbose mode it will give more details including headers and response.
I like using this in order to get the IP under which I am seen
curl -x http://proxy_server:proxy_port https://api.ipify.org?format=json && echo
Hope this helps someone.
For curl you can configure proxy in your ~/.curlrc (_curlrc on Windows) file by adding proxy value, the syntax is:
proxy = http://username:password#proxy-host:port
curl -I "https://www.google.com" -x 1.1.1.1:8080
Just summarizing all great mentioned answers:
curl -x http://<user>:<pass>#<proxyhost>:<port>/ -o <filename> -L <link>
With a proxy with authentication I use:
curl -x <protocol>://<user>:<password>#<host>:<port> --proxy-anyauth <url>
because, I don't know why curl doesn't use/catch http[s]_proxy environment variables.
You don't need to export the http[s]_proxy shell variable if you're just setting the proxy for a one off command. e.g.
http_proxy=http://your.proxy.server:port curl http://www.example.com
That said, I'd prefer curl -x if I knew I was always going to use a proxy.
sudo curl -x http://10.1.1.50:8080/ -fsSL https://download.docker.com/linux/ubuntu/gpg
This worked perfectly for me, the error comes because curl need to set
the proxy
Remmember replace the proxy with your proxy, mine, "example" was
http://10.1.1.50:8080/.
curl -vv -ksL "https://example.com" -x "http://<proxy>:<port>"
Depending on your workplace, you may also need to specify the -k or the --insecure option for curl in order to get past potential issues with CA certificates.
curl -x <myCompanyProxy>:<port> -k -O -L <link to file to download>
In case the proxy is using automatic proxy with PAC file. We can find the actual proxy from the javascript from the PAC URL.
And if the proxy needs authentication, we can first use a normal web-browser to access the website which will promote authentication dialog. After authentication, we can use wireshark to capture the http package sends to the proxy server, from the http package, we can get the auth token from http header: Proxy-Authorization
Then we can set the http_proxy environment variable and also include auth token in the http header: Proxy-Authorization
export http_proxy=http://proxyserver:port
curl -H "Proxy-Authorization: xxxx" http://targetURL
curl -x socks5://username:password#ip:port example.com
For http proxy tunnels (needed for the TLS protocol), you need to specify -p (aka --proxytunnel) instead of -x.
curl post about proxies
tl;dr the proxy tunnel uses a newer "CONNECT" keyword instead of a modified "GET"
This was needed for the node http-proxy-middleware library.
Only got a clue once I used wget which worked out of the box.