Trying to execute the spark program using YARN RESTAPI in HORTONWORKS 2.5
i have followed this link: https://community.hortonworks.com/content/kbentry/28070/starting-spark-jobs-directly-via-yarn-rest-api.html
Issue I'm facing is :
In the 5step: Request an application ID from YARN
curl -ikvu "knoxuser:knoxpwd" -X POST "https://$KNOX_SERVER:8443/gateway/default/resourcemanager/v1/cluster/apps/new-application"
Error:
About to connect() to $KNOX_SERVER port 8443 (#0) * Trying 53.244.194.234... connected * Connected to $KNOX_SERVER (53.244.194.234) port 8443 (#0) * successfully set certificate verify
locations: * CAfile: none CApath: /etc/ssl/certs/ * SSLv3, TLS
handshake, Client hello (1): * error:140770FC:SSL
routines:SSL23_GET_SERVER_HELLO:unknown protocol * Closing connection
#0 curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Where do I find the url or I'm going wrong in the url part in the command?
Please let me know, it would be great help to me.
Related
I want to make a GET request to https://www.bnro.ro/nbrfxrates.xml (the National Bank of Romania) to get the exchange rates for today.
While the XML document loads fine in the browser (tested in Safari and Chrome), it somehow fails in the terminal (checked with Node.js and curl):
$ curl -vL http://www.bnro.ro/nbrfxrates.xml
* Expire in 0 ms for 6 (transfer 0x7f8a5c009c00)
* Expire in 1 ms for 1 (transfer 0x7f8a5c009c00)
...
* Expire in 5 ms for 1 (transfer 0x7f8a5c009c00)
* Trying 194.102.208.89...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x7f8a5c009c00)
* Connected to www.bnro.ro (194.102.208.89) port 80 (#0)
> GET /nbrfxrates.xml HTTP/1.1
> Host: www.bnro.ro
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 302 Found
< Date: Fri, 14 Jun 2019 11:13:13 GMT
< Location: https://www.bnro.ro/nbrfxrates.xml
< Server: BigIP
< Content-Length: 0
< X-Cache: MISS from HS-F0
< X-Cache-Lookup: MISS from HS-F0:0
< Via: 1.1 HS-F0 (squid/3.4.8)
< Connection: keep-alive
<
* Connection #0 to host www.bnro.ro left intact
* Issue another request to this URL: 'https://www.bnro.ro/nbrfxrates.xml'
* Expire in 1 ms for 1 (transfer 0x7f8a5c009c00)
* Expire in 0 ms for 1 (transfer 0x7f8a5c009c00)
...
* Expire in 1 ms for 1 (transfer 0x7f8a5c009c00)
* Trying 194.102.208.89...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x7f8a5c009c00)
* Connected to www.bnro.ro (194.102.208.89) port 443 (#1)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /opt/local/share/curl/curl-ca-bundle.crt
CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 1
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
Both, curl and my Node.js app comaplain about the certificate.
In my Node.js script I get this error:
{ Error: unable to verify the first certificate
at TLSSocket.<anonymous> (_tls_wrap.js:1104:38)
at emitNone (events.js:105:13)
at TLSSocket.emit (events.js:207:7)
at TLSSocket._finishInit (_tls_wrap.js:638:8)
at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:468:38) code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE' }
I tried to use the Copy as CURL command option from Chrome, and that does not work either.
Why is this happening? How can it be fixed?
a missing certificate bundle, read more about it here https://curl.haxx.se/docs/sslcerts.html but the TL;DR is download https://curl.haxx.se/ca/cacert.pem and run
curl -vL --cacert cacert.pem http://www.bnro.ro/nbrfxrates.xml
Probably it is a misconfigured server. They used a wrong cert file (i.e, cert.pem instead of fullchain.pem). For example, if you use the "Let's encrypt" CA and python's ssl.wrap_socket you should write something like
httpd.socket = ssl.wrap_socket (httpd.socket, certfile='path/to/fullchain.pem', keyfile=path/to/privkey.pem' ,server_side=True, ssl_version=ssl.PROTOCOL_TLSv1_2)
If firefox opens the server and you cannot change the server configs and have to use curl,
then download fullchain.pem via firefox (Security->View
Certificate-> Miscellaneous -> Download PEM Chain)
Now You can use curl with this chain.
curl --cacert downloaded_chain_for_the_site.pem https://site_name.com
concatenate the CA Intermediate Certificate into your domain certificate
cat intermediate.crt >> domain.crt
ssl_certificate ssl/domain.crt; //your nginx server block
Is there any command line API change in Debian 9 curl?
Recently I started to use Debian 9 (9.4, from Debian 8.x) and a script involving curl stopped working. I connect to internet through a squid proxy on localhost connected to a parent proxy.
My environment variables are configured like this
root#server:~# printenv | grep -i proxy
HTTP_PROXY=http://127.0.0.1:3128
FTP_PROXY=http://127.0.0.1:3128
https_proxy=https://127.0.0.1:3128
http_proxy=http://127.0.0.1:3128
HTTPS_PROXY=https://127.0.0.1:3128
ftp_proxy=http://127.0.0.1:3128
When I use wget, it works:
root#server:~# wget https://www.google.com.cu
--2018-03-14 09:08:53-- https://www.google.com.cu/
Connecting to 127.0.0.1:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’
index.html [ <=> ] 11.12K --.-KB/s in 0.001s
2018-03-14 09:08:54 (14.9 MB/s) - ‘index.html’ saved [11389]
when I use curl, this is what I get
root#server:~# curl -v https://www.google.com.cu
* Rebuilt URL to: https://www.google.com.cu/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to (nil) (127.0.0.1) port 3128 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
* Curl_http_done: called premature == 0
* Closing connection 0
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
I know these two commands are not equivalent, this is just to illustrate the HTTPS transfer problem.
I need to use curl because the script uses a web API, so it needs to use POST instead of GET request, and to set some headers and data to the POST request. (api.dropboxapi.com is the target site)
This all used to work on Debian 8 without a hitch, and besides wget WORKS, only curl is failing with the debian version change. All the other HTTPS clients seem unaffected (FF, Chrome, Edge, wget all seems to work as always)
Is there any workaround, fix, command line option change or whatever for making debian 9's version of curl work?
There must be a way, I can't conceive curl can't make a HTTPS connection to google. There must be a command line or something that allows the connection.
Output of "curl -V"
root#server:~# curl -V
curl 7.52.1 (x86_64-pc-linux-gnu) libcurl/7.52.1 OpenSSL/1.0.2l zlib/1.2.8 libidn2/0.16 libpsl/0.17.0 (+libidn2/0.16) libssh2/1.7.0 nghttp2/1.18.1 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL
Output of "curl --insecure" as suggested
root#server:~# curl --insecure -v https://www.google.com.cu
* Rebuilt URL to: https://www.google.com.cu/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to (nil) (127.0.0.1) port 3128 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
* Curl_http_done: called premature == 0
* Closing connection 0
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
"curl -v https://www.google.com.cu --sslv2" output
root#server:/etc/squid# curl -v https://www.google.com.cu --sslv2
* Rebuilt URL to: https://www.google.com.cu/
* Trying 192.168.4.65...
* TCP_NODELAY set
* Connected to (nil) (192.168.4.65) port 81 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
* Curl_http_done: called premature == 0
* Closing connection 0
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Many, many thanks to Michael Hampton. It turns out the problem was in the proxy configuration. It should say
https_proxy=http://127.0.0.1:3128
HTTPS_PROXY=http://127.0.0.1:3128
So curl was trying to connect to squid using TLS and failing of course.
Original answer in https://serverfault.com/questions/901626/debian-version-change-affecting-scripts-using-curl-and-https
OS: CentOS release 6.9 (Final)
CURL: curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Hello,
I have a host which I am trying to run a curl command from, for testing only
curl https://api.twitter.com/ -v
Returns the below
* About to connect() to api.twitter.com port 443 (#0)
* Trying 104.244.42.2... connected
* Connected to api.twitter.com (104.244.42.2) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* NSS error -5961
* Closing connection #0
* SSL connect error
curl: (35) SSL connect error
After some lengthy reading I have come to the conclusion this is possibly a SLL issue due to either my OS version or CURL version, or both!
First question would be is that right? Are there indeed known issues with this configuration?
Second, can I build from source a working combination? I am unable to update my OS due to legacy dependencies
Try with specific ssl/tls or cipher suits for example: curl --tlsv1.2 https://api.twitter.com/
Since you say this is for testing only, you could try -k to connect insecurely:
curl -k https://api.twitter.com/
Or, if you think it’s an outdated SSL cert bundle you could download a newer certificate and trust store from the internet and try that by using the --cert and --cacert options
Rebuild a fresh server running centos 7
Worked straight away no issues!
$ stack setup -v
Version 1.4.0, Git revision e714f1dd3fade19496d91bd6a017e435a96a6bcd i386 hpack-0.17.0
2017-05-26 07:40:44.945473: [debug] Checking for project config at: C:\development\repos\eta\stack.yaml
#(Stack\Config.hs:935:9)
2017-05-26 07:40:44.948475: [debug] Loading project config file stack.yaml
#(Stack\Config.hs:960:13)
2017-05-26 07:40:44.954477: [debug] Trying to decode C:\sr\build-plan-cache\i386-windows\lts-6.27.cache
#(Data\Store\VersionTagged.hs:68:5)
2017-05-26 07:40:44.964480: [debug] Success decoding C:\sr\build-plan-cache\i386-windows\lts-6.27.cache
#(Data\Store\VersionTagged.hs:72:13)
2017-05-26 07:40:44.975485: [debug] Using standard GHC build
#(Stack\Setup.hs:600:9)
HttpExceptionRequest Request {
host = "raw.githubusercontent.com"
port = 443
secure = True
requestHeaders = []
path = "/fpco/stackage-content/master/stack/stack-setup-2.yaml"
queryString = ""
method = "GET"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
}
(InternalException (HandshakeFailed (Error_Protocol ("certificate rejected: [InvalidSignature SignatureInvalid]",True,CertificateUnknown))))
I'm running Windows 10 and get the same error in GitBash, Cygwin and DOS Command Line, both as User and as Administrator.
I have tried fetching the file into IE (https://raw.githubusercontent.com/fpco/stackage-content/master/stack/stack-setup-2.yaml) as suggested for similar issues regarding the retrieval of files from GitHub. I can get the file but stack still fails afterwards.
Following comments I also tried curl from GitBash and get
* About to connect() to raw.githubusercontent.com port 443 (#0)
* Trying 151.101.48.133...
* Connected to raw.githubusercontent.com (151.101.48.133) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: C:\Users\pauldenby\AppData\Local\Atlassian\SourceTree\git_local\bin\curl-ca-bundle.crt
CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
I am currently trying to import the missing certs into curl-ca-bundle.crt but I'm not sure that'll help with the fundamental problem. Is it documented anywhere where stack will look for trusted CAs?
I can also run curl --insecure. As a very temporary workaround, is there anything similar for stack?
I've looked at several similar questions here, but none of them have helped me. I want to connect to a web service that uses single-sign-on via an RSA SecurID keyfob. I start by trying to load any cookies provided by the initial GET request. Here's my command:
curl -A "Mozilla/5.0" -L -b cookies.txt -c cookies.txt -v -X GET \
https://sso.example.com/sso/login.htm
I get this in response:
* About to connect() to sso.example.com port 443 (#0)
* Trying 23.12.245.199... connected
* Connected to sso.example.com (23.12.245.199) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs/
* SSLv3, TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to sso.example.com:443
* Closing connection #0
curl: (35) Unknown SSL protocol error in connection to sso.example.com:443
Here's what I'm using:
# curl --version
curl 7.19.7 (x86_64-suse-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8j zlib/1.2.7 libidn/1.10
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
# more /etc/*-release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 3
# uname -r
3.0.101-0.21-default
Any ideas?
This is OpenSSL in your client that has a problem to understand what the server is saying and it errors out because of that.
You can sometimes work around these kinds of issues by forcing curl to speak SSLv3 (-3), TLS1.0 (-1) or even SSLv2 (-2)
... it is also conceivable that your severely outdated versions of curl and OpenSSL simply have a bug or two that cause this and that you can fix this problem by upgrading to modern versions (and then also fix the numerous security problems your versions contain).