curl command line API change on debian 9 regarding HTTPS - linux

Is there any command line API change in Debian 9 curl?
Recently I started to use Debian 9 (9.4, from Debian 8.x) and a script involving curl stopped working. I connect to internet through a squid proxy on localhost connected to a parent proxy.
My environment variables are configured like this
root#server:~# printenv | grep -i proxy
HTTP_PROXY=http://127.0.0.1:3128
FTP_PROXY=http://127.0.0.1:3128
https_proxy=https://127.0.0.1:3128
http_proxy=http://127.0.0.1:3128
HTTPS_PROXY=https://127.0.0.1:3128
ftp_proxy=http://127.0.0.1:3128
When I use wget, it works:
root#server:~# wget https://www.google.com.cu
--2018-03-14 09:08:53-- https://www.google.com.cu/
Connecting to 127.0.0.1:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’
index.html [ <=> ] 11.12K --.-KB/s in 0.001s
2018-03-14 09:08:54 (14.9 MB/s) - ‘index.html’ saved [11389]
when I use curl, this is what I get
root#server:~# curl -v https://www.google.com.cu
* Rebuilt URL to: https://www.google.com.cu/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to (nil) (127.0.0.1) port 3128 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
* Curl_http_done: called premature == 0
* Closing connection 0
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
I know these two commands are not equivalent, this is just to illustrate the HTTPS transfer problem.
I need to use curl because the script uses a web API, so it needs to use POST instead of GET request, and to set some headers and data to the POST request. (api.dropboxapi.com is the target site)
This all used to work on Debian 8 without a hitch, and besides wget WORKS, only curl is failing with the debian version change. All the other HTTPS clients seem unaffected (FF, Chrome, Edge, wget all seems to work as always)
Is there any workaround, fix, command line option change or whatever for making debian 9's version of curl work?
There must be a way, I can't conceive curl can't make a HTTPS connection to google. There must be a command line or something that allows the connection.
Output of "curl -V"
root#server:~# curl -V
curl 7.52.1 (x86_64-pc-linux-gnu) libcurl/7.52.1 OpenSSL/1.0.2l zlib/1.2.8 libidn2/0.16 libpsl/0.17.0 (+libidn2/0.16) libssh2/1.7.0 nghttp2/1.18.1 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL
Output of "curl --insecure" as suggested
root#server:~# curl --insecure -v https://www.google.com.cu
* Rebuilt URL to: https://www.google.com.cu/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to (nil) (127.0.0.1) port 3128 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
* Curl_http_done: called premature == 0
* Closing connection 0
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
"curl -v https://www.google.com.cu --sslv2" output
root#server:/etc/squid# curl -v https://www.google.com.cu --sslv2
* Rebuilt URL to: https://www.google.com.cu/
* Trying 192.168.4.65...
* TCP_NODELAY set
* Connected to (nil) (192.168.4.65) port 81 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
* Curl_http_done: called premature == 0
* Closing connection 0
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

Many, many thanks to Michael Hampton. It turns out the problem was in the proxy configuration. It should say
https_proxy=http://127.0.0.1:3128
HTTPS_PROXY=http://127.0.0.1:3128
So curl was trying to connect to squid using TLS and failing of course.
Original answer in https://serverfault.com/questions/901626/debian-version-change-affecting-scripts-using-curl-and-https

Related

URL loads in the browser, but not in the terminal (curl or Node.js)

I want to make a GET request to https://www.bnro.ro/nbrfxrates.xml (the National Bank of Romania) to get the exchange rates for today.
While the XML document loads fine in the browser (tested in Safari and Chrome), it somehow fails in the terminal (checked with Node.js and curl):
$ curl -vL http://www.bnro.ro/nbrfxrates.xml
* Expire in 0 ms for 6 (transfer 0x7f8a5c009c00)
* Expire in 1 ms for 1 (transfer 0x7f8a5c009c00)
...
* Expire in 5 ms for 1 (transfer 0x7f8a5c009c00)
* Trying 194.102.208.89...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x7f8a5c009c00)
* Connected to www.bnro.ro (194.102.208.89) port 80 (#0)
> GET /nbrfxrates.xml HTTP/1.1
> Host: www.bnro.ro
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 302 Found
< Date: Fri, 14 Jun 2019 11:13:13 GMT
< Location: https://www.bnro.ro/nbrfxrates.xml
< Server: BigIP
< Content-Length: 0
< X-Cache: MISS from HS-F0
< X-Cache-Lookup: MISS from HS-F0:0
< Via: 1.1 HS-F0 (squid/3.4.8)
< Connection: keep-alive
<
* Connection #0 to host www.bnro.ro left intact
* Issue another request to this URL: 'https://www.bnro.ro/nbrfxrates.xml'
* Expire in 1 ms for 1 (transfer 0x7f8a5c009c00)
* Expire in 0 ms for 1 (transfer 0x7f8a5c009c00)
...
* Expire in 1 ms for 1 (transfer 0x7f8a5c009c00)
* Trying 194.102.208.89...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x7f8a5c009c00)
* Connected to www.bnro.ro (194.102.208.89) port 443 (#1)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /opt/local/share/curl/curl-ca-bundle.crt
CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 1
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
Both, curl and my Node.js app comaplain about the certificate.
In my Node.js script I get this error:
{ Error: unable to verify the first certificate
at TLSSocket.<anonymous> (_tls_wrap.js:1104:38)
at emitNone (events.js:105:13)
at TLSSocket.emit (events.js:207:7)
at TLSSocket._finishInit (_tls_wrap.js:638:8)
at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:468:38) code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE' }
I tried to use the Copy as CURL command option from Chrome, and that does not work either.
Why is this happening? How can it be fixed?
a missing certificate bundle, read more about it here https://curl.haxx.se/docs/sslcerts.html but the TL;DR is download https://curl.haxx.se/ca/cacert.pem and run
curl -vL --cacert cacert.pem http://www.bnro.ro/nbrfxrates.xml
Probably it is a misconfigured server. They used a wrong cert file (i.e, cert.pem instead of fullchain.pem). For example, if you use the "Let's encrypt" CA and python's ssl.wrap_socket you should write something like
httpd.socket = ssl.wrap_socket (httpd.socket, certfile='path/to/fullchain.pem', keyfile=path/to/privkey.pem' ,server_side=True, ssl_version=ssl.PROTOCOL_TLSv1_2)
If firefox opens the server and you cannot change the server configs and have to use curl,
then download fullchain.pem via firefox (Security->View
Certificate-> Miscellaneous -> Download PEM Chain)
Now You can use curl with this chain.
curl --cacert downloaded_chain_for_the_site.pem https://site_name.com
concatenate the CA Intermediate Certificate into your domain certificate
cat intermediate.crt >> domain.crt
ssl_certificate ssl/domain.crt; //your nginx server block

HttpExceptionRequest (CertificateUnknown) running stack setup

$ stack setup -v
Version 1.4.0, Git revision e714f1dd3fade19496d91bd6a017e435a96a6bcd i386 hpack-0.17.0
2017-05-26 07:40:44.945473: [debug] Checking for project config at: C:\development\repos\eta\stack.yaml
#(Stack\Config.hs:935:9)
2017-05-26 07:40:44.948475: [debug] Loading project config file stack.yaml
#(Stack\Config.hs:960:13)
2017-05-26 07:40:44.954477: [debug] Trying to decode C:\sr\build-plan-cache\i386-windows\lts-6.27.cache
#(Data\Store\VersionTagged.hs:68:5)
2017-05-26 07:40:44.964480: [debug] Success decoding C:\sr\build-plan-cache\i386-windows\lts-6.27.cache
#(Data\Store\VersionTagged.hs:72:13)
2017-05-26 07:40:44.975485: [debug] Using standard GHC build
#(Stack\Setup.hs:600:9)
HttpExceptionRequest Request {
host = "raw.githubusercontent.com"
port = 443
secure = True
requestHeaders = []
path = "/fpco/stackage-content/master/stack/stack-setup-2.yaml"
queryString = ""
method = "GET"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
}
(InternalException (HandshakeFailed (Error_Protocol ("certificate rejected: [InvalidSignature SignatureInvalid]",True,CertificateUnknown))))
I'm running Windows 10 and get the same error in GitBash, Cygwin and DOS Command Line, both as User and as Administrator.
I have tried fetching the file into IE (https://raw.githubusercontent.com/fpco/stackage-content/master/stack/stack-setup-2.yaml) as suggested for similar issues regarding the retrieval of files from GitHub. I can get the file but stack still fails afterwards.
Following comments I also tried curl from GitBash and get
* About to connect() to raw.githubusercontent.com port 443 (#0)
* Trying 151.101.48.133...
* Connected to raw.githubusercontent.com (151.101.48.133) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: C:\Users\pauldenby\AppData\Local\Atlassian\SourceTree\git_local\bin\curl-ca-bundle.crt
CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
I am currently trying to import the missing certs into curl-ca-bundle.crt but I'm not sure that'll help with the fundamental problem. Is it documented anywhere where stack will look for trusted CAs?
I can also run curl --insecure. As a very temporary workaround, is there anything similar for stack?

How to request an applicationID using YARN REST API

Trying to execute the spark program using YARN RESTAPI in HORTONWORKS 2.5
i have followed this link: https://community.hortonworks.com/content/kbentry/28070/starting-spark-jobs-directly-via-yarn-rest-api.html
Issue I'm facing is :
In the 5step: Request an application ID from YARN
curl -ikvu "knoxuser:knoxpwd" -X POST "https://$KNOX_SERVER:8443/gateway/default/resourcemanager/v1/cluster/apps/new-application"
Error:
About to connect() to $KNOX_SERVER port 8443 (#0) * Trying 53.244.194.234... connected * Connected to $KNOX_SERVER (53.244.194.234) port 8443 (#0) * successfully set certificate verify
locations: * CAfile: none CApath: /etc/ssl/certs/ * SSLv3, TLS
handshake, Client hello (1): * error:140770FC:SSL
routines:SSL23_GET_SERVER_HELLO:unknown protocol * Closing connection
#0 curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Where do I find the url or I'm going wrong in the url part in the command?
Please let me know, it would be great help to me.

Is the outlook REST API throttling me?

I have created a basic script which checks for an email in my inbox, and yesterday everything was working fine. But today all i get is a HTTP404 when I run the same code.
I suspect that outlook.office365.com might be throttling my requests since I ran this 1 time every minute for about 36 hours. But how can i tell if that is the case?
Here are my curl command:
curl https://outlook.office365.com/api/1.0/Me/folders/Inbox/messages -u login#email.outlook:password --insecure --show-error -H Content-Type:application/json -v
and the output:
* About to connect() to outlook.office365.com port 443 (#0)
* Trying 132.245.48.18... connected
* Connected to outlook.office365.com (132.245.48.18) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
* subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; OU=Microsoft Corporation; CN=outlook.com
* start date: 2015-02-13 00:38:15 GMT
* expire date: 2016-02-13 00:38:15 GMT
* subjectAltName: outlook.office365.com matched
* issuer: C=US; ST=Washington; L=Redmond; O=Microsoft Corporation; OU=Microsoft IT; CN=Microsoft IT SSL SHA1
* SSL certificate verify ok.
* Server auth using Basic with user 'login#email.outlook'
> GET /api/1.0/Me/folders/Inbox/messages HTTP/1.1
> Authorization: Basic Ybase64stringU=
> User-Agent: curl/7.21.3 (x86_64-pc-linux-gnu) libcurl/7.21.3 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host: outlook.office365.com
> Accept: */*
> Content-Type:application/json
>
< HTTP/1.1 404 Not Found
< Content-Length: 0
< Server: Microsoft-IIS/8.0
< request-id: d7967869-c282-4a28-9da2-6b80edd7d965
< Set-Cookie: ClientId=XXQW0EIP0KK9NUIRICS8BQ; expires=Fri, 28-Oct-2016 13:04:08 GMT; path=/; secure; HttpOnly
< X-CalculatedBETarget: AM2PR09MB0452.eurprd09.prod.outlook.com
< X-BackEndHttpStatus: 404
< X-DiagInfo: AM2PR09MB0452
< X-BEServer: AM2PR09MB0452
< X-Powered-By: ASP.NET
< X-FEServer: HE1PR01CA0033
< Date: Thu, 29 Oct 2015 13:04:08 GMT
<
* Connection #0 to host outlook.office365.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
any ideas?
If you had exceeded a quota, I would expect you would receive a 429 status code. Microsoft don't usually publish an exact threshold for their services as they reserve the right to adjust them...
so users can consume the maximum number of resources without degrading the reliability and performance
(The above quote is specifically in respect of Sharepoint, but given the lack of published limits for pretty much all REST APIs I imagine the principle holds).
The 404 would usually indicate something isn't quite right... if your user account was reasonably new it could indicate it hadn't propagated to all servers yet (although in your case that doesn't seem to be the issue as you must have had the account for >24 hours).
From a URL point of view... the v1 URL is outlook.office.com/api/v1.0/me/messages and the new beat URL is outlook.office.com/api/v1.0/me/messages these addresses seem to work for me... but I get a 404 if I use the address you are using... so double-check the URL.

CURL returns "Unknown SSL protocol error"

I've looked at several similar questions here, but none of them have helped me. I want to connect to a web service that uses single-sign-on via an RSA SecurID keyfob. I start by trying to load any cookies provided by the initial GET request. Here's my command:
curl -A "Mozilla/5.0" -L -b cookies.txt -c cookies.txt -v -X GET \
https://sso.example.com/sso/login.htm
I get this in response:
* About to connect() to sso.example.com port 443 (#0)
* Trying 23.12.245.199... connected
* Connected to sso.example.com (23.12.245.199) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs/
* SSLv3, TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to sso.example.com:443
* Closing connection #0
curl: (35) Unknown SSL protocol error in connection to sso.example.com:443
Here's what I'm using:
# curl --version
curl 7.19.7 (x86_64-suse-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8j zlib/1.2.7 libidn/1.10
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
# more /etc/*-release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 3
# uname -r
3.0.101-0.21-default
Any ideas?
This is OpenSSL in your client that has a problem to understand what the server is saying and it errors out because of that.
You can sometimes work around these kinds of issues by forcing curl to speak SSLv3 (-3), TLS1.0 (-1) or even SSLv2 (-2)
... it is also conceivable that your severely outdated versions of curl and OpenSSL simply have a bug or two that cause this and that you can fix this problem by upgrading to modern versions (and then also fix the numerous security problems your versions contain).

Resources