I have a x.example which serves traffic for both a.example and b.example.
x.example has certificates for both a.example and b.example. The DNS for a.example and b.example is not yet set up.
If I add an /etc/hosts entry for a.example pointing to x.example's ip and run curl -XGET https://a.example, I get a 200.
However if I run curl --header 'Host: a.example' https://x.example, I get:
curl: (51) SSL: no alternative certificate subject name matches target
host name x.example
I would think it would use a.example as the host. Maybe I'm not understanding how SNI/TLS works.
Because a.example is an HTTP header the TLS handshake doesn't have access to it yet? But the URL itself it does have access to?
Indeed SNI in TLS does not work like that. SNI, as everything related to TLS, happens before any kind of HTTP traffic, hence the Host header is not taken into account at that step (but will be useful later on for the webserver to know which host you are connecting too).
So to enable SNI you need a specific switch in your HTTP client to tell it to send the appropriate TLS extension during the handshake with the hostname value you need.
In case of curl, you need at least version 7.18.1 (based on https://curl.haxx.se/changes.html) and then it seems to automatically use the value provided in the Host header. It alo depends on which OpenSSL (or equivalent library on your platform) version it is linked to.
See point 1.10 of https://curl.haxx.se/docs/knownbugs.html that speaks about a bug but explains what happens:
When given a URL with a trailing dot for the host name part: "https://example.com./", libcurl will strip off the dot and use the name without a dot internally and send it dot-less in HTTP Host: headers and in the TLS SNI field.
The --connect-to option could also be useful in your case. Or --resolve as a substitute to /etc/hosts, see https://curl.haxx.se/mail/archive-2015-01/0042.html for am example, or https://makandracards.com/makandra/1613-make-an-http-request-to-a-machine-but-fake-the-hostname
You can add --verbose in all cases to see in more details what is happening. See this example: https://www.claudiokuenzler.com/blog/693/curious-case-of-curl-ssl-tls-sni-http-host-header ; you will also see there how to test directly with openssl.
If you have a.example in your /etc/hosts you should just run curl with https://a.example/ and it should take care of the Host header and hence SNI (or use --resolve instead)
So to answer your question directly, replace
curl --header 'Host: a.example' https://x.example
with
curl --connect-to a.example:443:x.example:443 https://a.example
and it should work perfectly.
The selected answer helped me find the answer, even though it does not contain the answer. The answer in the mail/archive link Patrick Mevzek provided has the wrong port number. So even following that answer will cause it to continue to fail.
I used this container to run a debugging server to inspect the requests. I highly suggest anyone debugging this kind of issue do the same.
Here is how to address the OP's question.
# Instead of this:
# curl --header 'Host: a.example' https://x.example
# Do:
host=a.example
target=x.example
ip=$(dig +short $target | head -n1)
curl -sv --resolve $host:443:$ip https://$host
If you want to ignore bad certificates matches, use -svk instead of -sv
curl -svk --resolve $host:443:$ip https://$host
Note: Since you are using https, you must use 443 in the --resolve argument instead of 80 as was stated on the mail/archive
I had a similar need. Didn't have sudo access to update hosts file.
I use resolve parameter and also added the DNS host name as a header parameter.
--resolve <dns name>:<port>:<ip addr>
curl --request POST --resolve dns_name:443:a.b.c.d 'https://dns_name/x/y' --header 'Host: dns_name' ....
Cheers..
I am managing a site hosted on aws ec2 using nginx. To avoid threats continuously monitoring nginx logs ( access.log & error.log). Though many threats are well managed by tweaking nginx.conf, but this specific one I am not even able to figure out how attacker manage to send such request.
access.log
xx.xxx.xx.xxx - - [18/Aug/2021:09:04:13 +0000] "GET http://xxxxxxxxx.com/ HTTP/1.1" 200 1400 "-" "Go-http-client/1.1"
In above case let's say name of my website is "h ttp://abc-xyz-1234.com", attacker is passing url in path (i.e. http://xxxxxxxxx.com/ ), and nginx responding with "200". I am still scratching my head how was request made and what was responded with 1400 of bytes ( response length still much lesser than website response site for path "/" ).
As I believe its not possible through browser, I tried to simulate using curl but it wouldn't work.
it is considered 2 separate request to curl
curl -A Mozilla h ttp://abc-xyz-1234.com/ http://xxxxxxxxx.com
invalid domain
curl -A Mozilla h ttp://abc-xyz-1234.comhttp://xxxxxxxxx.com
it will hit host with path /http://xxxxxxxxx.com and get rejected. Attacker is manage to send it without prefix "/" and thats what trying to simulate
curl -A Mozilla h ttp://abc-xyz-1234.com/http://xxxxxxxxx.com
You can use --request-target for this:
curl -A Mozilla http://abc-xyz-1234.com --request-target http://xxxxxxxxx.com
I have a Tornado server running on some port
And if I make a request via browser to non existing url, Tornado prints:
WARNING:tornado.access:404 POST /some_url/ (MY.REAL.IP) 0.64ms python
But I noticed another 404 error done via localhost:
WARNING:tornado.access:404 POST /some_url/ (127.0.0.1) 0.64ms python
Is it possible in theory, that this request was done by some "cool hacker" from remote server using curl --resolve or something?
The only way this address should be spoofable would be if you set xheaders=True in your HTTPServer constructor. If you use xheaders=True, you should also be using a frontend proxy that sanitizes headers appropriately so it will not allow X-Real-IP headers from outside sources.
This is a really basic question but I somehow can't figure it out:
I am using netcat to get HTTP response from a website. Example:
request="GET / HTTP/1.1"
echo -ne $request | nc 127.0.0.1 80
If the website requires authentication I can send a POST request and include user/pass in form-data.
But how about apache password protected directories? Let's say this is in my apache.conf:
<Directory "/var/www/html/">
...
AuthType Basic
AuthName "Authentication Required"
AuthUserFile "/etc/htpasswd/.htpasswd"
Require valid-user
</Directory>
When I visit 127.0.0.1, I get prompted for user/pass. Browser shows request as "stalled". After I provide my credentials, request is finished. If I inspect request headers, I can't find my credentials or any other form data there.
My question is: how are these credentials sent? How can I send them using netcat utility?
p.s.: I am not dead set on using netcat, if there is another command line utility that can achive this in a simple way, that's also great.
They are sent in a Authorization header, encoded in base64 (Wireshark auto-decodes them for you if you expand the header).
You can specify the username and password easily using curl:
curl --user name:password localhost
You can see how it works by using -v to see the headers, for example:
curl -sv --user name:password localhost
In the header output you will see a line like this:
> Authorization: Basic bmFtZTpwYXNzd29yZA==
Try to use CURL insted of netcat
I have this proxy address: 125.119.175.48:8909
How can I perform a HTTP request using cURL like curl http://www.example.com, but specifying the proxy address of my network?
From man curl:
-x, --proxy <[protocol://][user:password#]proxyhost[:port]>
Use the specified HTTP proxy.
If the port number is not specified, it is assumed at port 1080.
General way:
export http_proxy=http://your.proxy.server:port/
Then you can connect through proxy from (many) application.
And, as per comment below, for https:
export https_proxy=https://your.proxy.server:port/
The above solutions might not work with some curl versions I tried them for myself(curl 7.22.0). But what worked for me was:
curl -x http://proxy_server:proxy_port --proxy-user username:password -L http://url
Hope it solves the issue better!
Beware that if you are using a SOCKS proxy, instead of a HTTP/HTTPS proxy, you will need to use the --socks5 switch instead:
curl --socks5 125.119.175.48:8909 http://example.com/
You can also use --socks5-hostname instead of --socks5 to resolve DNS on the proxy side.
as an adition to airween, another good idea is to add this into your .bashrc, so you'll be able to switch from non proxied to proxied environment:
alias proxyon="export http_proxy='http://YOURPROXY:YOURPORT';export https_proxy='http://YOURPROXY:YOURPORT'"
alias proxyoff="export http_proxy='';export https_proxy=''"
WHERE YOURPROXY:YOURPORT is exactly that, your ip and port proxy :-).
Then, simply doing
proxyon
your system will start to use the proxy, and just the opposite with:
proxyoff
use the following
curl -I -x 192.168.X.X:XX http://google.com
192.168.X.X:XX put your proxy server ip and port.
-v verbose mode it will give more details including headers and response.
I like using this in order to get the IP under which I am seen
curl -x http://proxy_server:proxy_port https://api.ipify.org?format=json && echo
Hope this helps someone.
For curl you can configure proxy in your ~/.curlrc (_curlrc on Windows) file by adding proxy value, the syntax is:
proxy = http://username:password#proxy-host:port
curl -I "https://www.google.com" -x 1.1.1.1:8080
Just summarizing all great mentioned answers:
curl -x http://<user>:<pass>#<proxyhost>:<port>/ -o <filename> -L <link>
With a proxy with authentication I use:
curl -x <protocol>://<user>:<password>#<host>:<port> --proxy-anyauth <url>
because, I don't know why curl doesn't use/catch http[s]_proxy environment variables.
You don't need to export the http[s]_proxy shell variable if you're just setting the proxy for a one off command. e.g.
http_proxy=http://your.proxy.server:port curl http://www.example.com
That said, I'd prefer curl -x if I knew I was always going to use a proxy.
sudo curl -x http://10.1.1.50:8080/ -fsSL https://download.docker.com/linux/ubuntu/gpg
This worked perfectly for me, the error comes because curl need to set
the proxy
Remmember replace the proxy with your proxy, mine, "example" was
http://10.1.1.50:8080/.
curl -vv -ksL "https://example.com" -x "http://<proxy>:<port>"
Depending on your workplace, you may also need to specify the -k or the --insecure option for curl in order to get past potential issues with CA certificates.
curl -x <myCompanyProxy>:<port> -k -O -L <link to file to download>
In case the proxy is using automatic proxy with PAC file. We can find the actual proxy from the javascript from the PAC URL.
And if the proxy needs authentication, we can first use a normal web-browser to access the website which will promote authentication dialog. After authentication, we can use wireshark to capture the http package sends to the proxy server, from the http package, we can get the auth token from http header: Proxy-Authorization
Then we can set the http_proxy environment variable and also include auth token in the http header: Proxy-Authorization
export http_proxy=http://proxyserver:port
curl -H "Proxy-Authorization: xxxx" http://targetURL
curl -x socks5://username:password#ip:port example.com
For http proxy tunnels (needed for the TLS protocol), you need to specify -p (aka --proxytunnel) instead of -x.
curl post about proxies
tl;dr the proxy tunnel uses a newer "CONNECT" keyword instead of a modified "GET"
This was needed for the node http-proxy-middleware library.
Only got a clue once I used wget which worked out of the box.