Is there a way to connect to Elasticsearch (https://IP:9200) by PySpark without certificate verification?
I didn't see any settings for this: https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html
Curl works fine:
curl -iv --insecure --user admin:password --noproxy '*' https://xx.xxx.xx.xxx:9200
* Trying xx.xxx.xx.xxx:9200...
* Connected to xx.xxx.xx.xxx (xx.xxx.xx.xxx) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /opt/conda/ssl/cacert.pem
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: OU=dev-1; CN=dev-1-es-http.es-dev.es.local
* start date: Jul 9 00:47:04 2020 GMT
* expire date: Jul 9 00:57:04 2021 GMT
* issuer: OU=dev-1; CN=dev-1-http
* SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Server auth using Basic with user 'admin'
> GET / HTTP/1.1
> Host: xx.xxx.xx.xxx:9200
> Authorization: Basic abc==
> User-Agent: curl/7.71.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
content-type: application/json; charset=UTF-8
< content-length: 544
content-length: 544
<
{
"name" : "dev-1-es-hot-2",
"cluster_name" : "dev-1",
"cluster_uuid" : "444",
"version" : {
"number" : "7.8.0",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "123",
"build_date" : "2020-06-14T19:35:50.234439Z",
"build_snapshot" : false,
"lucene_version" : "8.5.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
* Connection #0 to host xx.xxx.xx.xxx left intact
My last try:
reader = spark.read.format("org.elasticsearch.spark.sql") \
.option("es.port", "9200") \
.option("spark.es.net.http.auth.user", "admin") \
.option("spark.es.net.http.auth.pass", "password") \
.option("es.net.ssl", "true") \
.option("es.net.ssl.cert.allow.self.signed", "true") \
.option("es.net.proxy.https.use.system.props", "false") \
.option("es.nodes", "xx.xxx.xxx.xxx")
Ends with:
: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:348)
at org.elasticsearch.spark.sql.ElasticsearchRelation.cfg$lzycompute(DefaultSource.scala:225)
at org.elasticsearch.spark.sql.ElasticsearchRelation.cfg(DefaultSource.scala:223)
at org.elasticsearch.spark.sql.ElasticsearchRelation.lazySchema$lzycompute(DefaultSource.scala:229)
at org.elasticsearch.spark.sql.ElasticsearchRelation.lazySchema(DefaultSource.scala:229)
at org.elasticsearch.spark.sql.ElasticsearchRelation.$anonfun$schema$1(DefaultSource.scala:233)
at scala.Option.getOrElse(Option.scala:189)
at org.elasticsearch.spark.sql.ElasticsearchRelation.schema(DefaultSource.scala:233)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:448)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:308)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:308)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:240)
at jdk.internal.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.elasticsearch.hadoop.rest.EsHadoopTransportException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:128)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:432)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:428)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:388)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:392)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:168)
at org.elasticsearch.hadoop.rest.RestClient.mainInfo(RestClient.java:745)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:338)
... 23 more
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1356)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1231)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1174)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1408)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1314)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:819)
at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1189)
at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.doExecute(CommonsHttpTransport.java:685)
at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:664)
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:116)
... 30 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
at java.base/sun.security.validator.Validator.validate(Validator.java:264)
at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:233)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:110)
at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory$TrustManagerDelegate.checkServerTrusted(SSLSocketFactory.java:76)
at java.base/sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:1510)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1340)
... 54 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
... 62 more
I also trying to provide keystore/truestore without results.
I had the same issue as you, but in Java. As already stated in the comments below the question, the issue is that we need to use the server certificates in order to complete the TLS handshake.
I used the tool suggested by lubomand added the certificates to a .jks truststore, put a password then finally I added the truststore to all the worker servers of the spark cluster.
Then added the following properties
conf.put("spark.es.net.ssl.keystore.location", "<truststore path>");
conf.put("spark.es.net.ssl.keystore.pass","<truststore password>";
In this way I was able to communicate with the ElasticSearch Server.
P.S. If this answer is useful don't thank me, but #lubom which solved my issue in the comments thread below the question! I answered the question (after asking permission) just to clearly put how we both solved our common issue.
Related
I have a barebone simple example that works locally but not in Azure. My original question was a bit specific to an cors-error, that i have now worked around, and therefor edited away that part. But i still cant manage to make WS work on azure. And i need help.;
var server = http.Server(app);
const wss = new WebSocket.Server({server});
server.listen(8070, () => {
console.log("Listening on " + port)
});
wss.on('connection', (socket) => {
console.log('a user connected');
socket.on('message', (msg) => {
console.log(msg)
socket.send(JSON.stringify({hey:'you'}))
})
});
client connection-string:
SOCKET_ENDPOINT = 'wss://******.azurewebsites.net/';
I've turned on websockets in my app-configuration.
EDIT:
So i have now turned every stone. Even switched library from socket.io to ws. Same problem. I came a little further i think when trying different things, now i get following error "Firefox can’t establish a connection to the server at wss://*******.azurewebsites.net:8070/."
Edit2:
The http-upgrade is fine
Here's the error
Edit 3: Curl command:
* TCP_NODELAY set
* Connected to *******.azurewebsites.net (20.40.202.6) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=*.azurewebsites.net
* start date: Sep 28 19:00:01 2020 GMT
* expire date: Sep 28 19:00:01 2021 GMT
* subjectAltName: host "********.azurewebsites.net" matched cert's "*.azurewebsites.net"
* issuer: C=US; O=Microsoft Corporation; CN=Microsoft RSA TLS CA 01
* SSL certificate verify ok.
> GET /socket.io/?EIO=4 HTTP/1.1
> Host: *********.azurewebsites.net
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Server: Kestrel
< WWW-Authenticate: Bearer realm="********.azurewebsites.net" authorization_uri="https://login.windows.net/<tenant>/oauth2/authorize" resource_id="<resource>"
< Date: Tue, 22 Jun 2021 05:56:15 GMT
< Content-Length: 0
<
* Connection #0 to host *******.azurewebsites.net left intact
I am guessing that you host your app in Azure AppService?
Make sure you have websocket option turned on in platform:
AppService resource -> Settings -> General settings tab -> Platform settings section
Here is the tutorial that I am following: What is the best way to enable a website users send money to each other?
The problem is with my post request.
Let's break it down into pieces.
Headers
headers: {
"Authorization": `Bearer ${accessToken}`,
"Content-Type": "application/json"
}
I get the access token also via paypal api namely https://api.sandbox.paypal.com/v1/oauth2/token.
Body
body: {
intent: 'CAPTURE',
purchase_units: [{
amount: {
currency_code: 'USD',
value: '2.00'
},
payee: {
email_address: "myanothersandboxaccount#gmail.com"
}
}]
}
But, it doesn't work, as I get an error
{
name: 'INVALID_REQUEST',
message: 'Request is not well-formed, syntactically incorrect, or violates schema.',
debug_id: '2884e1b5eccee',
details: [
{
field: '/',
location: 'body',
issue: 'INVALID_SYNTAX',
description: 'MALFORMED_REQUEST_JSON'
}
],
links: [
{
href: 'https://developer.paypal.com/docs/api/orders/v2/#error-INVALID_SYNTAX',
rel: 'information_link',
encType: 'application/json'
}
]
}
curl request
curl -v -X POST https://api.sandbox.paypal.com/v2/checkout/orders \
-H "Content-Type: application/json" \
-H "Authorization: Bearer A23AALAej8Yg-4iKJBcWckiv5-ZlhYWlkmBsPuWaVngJcMigU7P-6f8P02vnOpIo8QlOJ-P3hd3K86vKo_lpSlu0-bZBj98eg" \
-d '{
"intent": "CAPTURE",
"purchase_units": [
{
"amount": {
"currency_code": "USD",
"value": "100.00"
},
"payee": {
"email": "myanothersandboxaccount#gmail.com"
}
}
]
}'
curl response
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 173.0.82.78...
* TCP_NODELAY set
* Connected to api.sandbox.paypal.com (173.0.82.78) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=California; L=San Jose; O=PayPal, Inc.; OU=PayPal Production; CN=api.sandbox.paypal.com
* start date: Jul 27 00:00:00 2020 GMT
* expire date: Aug 1 12:00:00 2022 GMT
* subjectAltName: host "api.sandbox.paypal.com" matched cert's "api.sandbox.paypal.com"
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
* SSL certificate verify ok.
> POST /v2/checkout/orders HTTP/1.1
> Host: api.sandbox.paypal.com
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Type: application/json
> Authorization: Bearer A23AALAej8Yg-4iKJBcWckiv5-ZlhYWlkmBsPuWaVngJcMigU7P-6f8P02vnOpIo8QlOJ-P3hd3K86vKo_lpSlu0-bZBj98eg
> Content-Length: 212
>
* upload completely sent off: 212 out of 212 bytes
< HTTP/1.1 201 Created
< Cache-Control: max-age=0, no-cache, no-store, must-revalidate
< Content-Length: 501
< Content-Type: application/json
< Date: Sat, 05 Dec 2020 10:30:13 GMT
< Paypal-Debug-Id: 6bd069526af1c
<
* Connection #0 to host api.sandbox.paypal.com left intact
{"id":"674004650C383744Y","status":"CREATED","links":[{"href":"https://api.sandbox.paypal.com/v2/checkout/orders/674004650C383744Y","rel":"self","method":"GET"},{"href":"https://www.sandbox.paypal.com/checkoutnow?token=674004650C383744Y","rel":"approve","method":"GET"},{"href":"https://api.sandbox.paypal.com/v2/checkout/orders/674004650C383744Y","rel":"update","method":"PATCH"},{"href":"https://api.sandbox.paypal.com/v2/checkout/orders/674004650C383744Y/capture","rel":"capture","method":"POST"}]}* Closing connection 0
And, with curl I get a successful response. I can see that all went well from my paypal sandbox dashboard.
What could it be?
I was using node-fetch to call the paypal api to create an order. Tried with axios, worked just fine.
The error basically means PayPal isn't able to parse the body object it received as JSON, so it appears something is being transmitted/received incorrectly.
You should first test with command-line curl to verify functionality and your JSON's correctness, which should work.
Then to get more details about what's actually going wrong in what you are sending, you will need to log the actual data being sent (not your code, the actual data the code sends when executed)
One way to do this is to instead of sending your request to https://api-m.paypal.com , instead send it to a request bin service like offered via https://requestbin.com/
I’m running into an issue with axios and Node 12. As I’m not sure this error is only related to axios, I followed the advice to ask on SO rather than opening a bug on axios’ GitHub.
Here is the code I’m trying to run :
const axios = require('axios')
axios({
method: 'get',
url: 'https://www.colisprive.com/moncolis/pages/detailColis.aspx?numColis=12345',
responseType: 'text'
}).then((response) => {
console.log(response)
})
This code fails on Node 12 with following error :
Error: write EPROTO 140121214769024:error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol:../deps/openssl/openssl/ssl/statem/statem_lib.c:1929:
at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:87:16)
Same code ran against Node 11 doesn’t throw any error.
When I curl -v I got this :
* Trying 91.208.224.32:443...
* TCP_NODELAY set
* Connected to www.colisprive.com (91.208.224.32) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: serialNumber=391029345; jurisdictionC=FR; businessCategory=Private Organization; C=FR; postalCode=13290; ST=Bouches-du-Rh�ne; L=AIX EN PROVENCE; street=1330 AV J R G GAUTIER DE LA LAUZIERE; street=ZI MILLES EUROPARC PICHAURY; O=COLIS PRIVE SAS; OU=0002 391029345; CN=www.colisprive.com
* start date: Sep 3 00:00:00 2018 GMT
* expire date: Sep 2 23:59:59 2020 GMT
* subjectAltName: host "www.colisprive.com" matched cert's "www.colisprive.com"
* issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO RSA Extended Validation Secure Server CA
* SSL certificate verify ok.
> GET /moncolis/pages/detailColis.aspx?numColis=12345 HTTP/1.1
> Host: www.colisprive.com
> User-Agent: curl/7.65.3
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Location: /moncolis/Default.aspx?numColis=12345&cp=
< Server: Microsoft-IIS/7.5
< Set-Cookie: ASP.NET_SessionId=eln3cq143d35lfj5tpqkkwcg; path=/; HttpOnly
< X-Powered-By: Colis Priv�
< Date: Fri, 24 Jan 2020 13:48:35 GMT
< Content-Length: 162
<
<html><head><title>Object moved</title></head><body>
<h2>Object moved to here.</h2>
</body></html>
* Connection #0 to host www.colisprive.com left intact
As you can see, it gives a 302 Found with a Location header pointing to another endpoint. I agree it should answer a 301 Moved to indicate document has moved, but this is not the case and it is handled as expected by axios on Node 11 (fetching endpoint under Location header).
I saw that Node 12 now includes TLS 1.3 as default, so this could be related to that…
Also, there is an unknown character in X-Powered-By header.
I tried to :
reproduce this issue with an express server always replying 302 Found with same headers : works as expected
fetch another .aspx web page with axios : works as expected
The problem is not just with axios but with got as well.
Node.js 12's default TLS settings are stricter now. The site doesn't handle TLS v1.2. Node 12 by default need 1.2.
You can change this via a command line flag (--tls-min-v1.0) when running your app.
something like this
node --tls-min-v1.0 app.js
Cloud Provider: Azure
Istio Version: 1.19
Kubernetes Version: 1.12.8
Below is my Virtual Service and Gateway yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: app-ingress-rules
namespace: istio-system
# Define the routes less restrictive first to more restrictive. this file gets read from bottom to top.
spec:
# https://istio.io/docs/reference/config/istio.networking.v1alpha3/#VirtualService
gateways: # The default `mesh` value for when left blank is doesn't seem to propigate the rule properly. For now, always use a list of FQDN gateways
- tls-gateway
hosts:
- "dev.myapp.westus.cld.something.com" # This host is part of installed HTTPS cert
- "nonprod.westus.cloudapp.azure.com" # This host is NOT part of installed HTTPS cert
http:
# Routes
- match:
- uri:
prefix: /myapp/status
rewrite:
uri: /status
route:
- destination:
host: myapp.myapp-ns.svc.cluster.local
--------------------------------------------
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: tls-gateway
namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- port:
name: https
number: 443
protocol: HTTPS
tls:
mode: SIMPLE
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
privateKey: /etc/istio/ingressgateway-certs/tls.key
hosts:
- "*"
If I do
curl -Iv https://dev.myapp.westus.cld.something.com/myapp/status
I get HTTP 404 as below
* Trying 123.12.123.12...
* TCP_NODELAY set
* Connected to dev.myapp.westus.cld.something.com (123.12.123.12.) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=***; L=*****; O=******, Inc.; CN=*********
* start date: Jul 26 20:32:59 2019 GMT
* expire date: Jul 26 20:32:59 2021 GM
* subjectAltName: host "dev.myapp.westus.cld.something.com" matched cert's "dev.myapp.westus.cld.something.com"
* issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign RSA OV SSL CA 2018
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f995e006600)
> HEAD /myapp/status HTTP/2
> Host: dev.myapp.westus.cld.something.com
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 404
HTTP/2 404
< location: https://dev.myapp.westus.cld.something.com/myapp/status
location: https://dev.myapp.westus.cld.something.com /myapp/status
< date: Wed, 31 Jul 2019 19:10:13 GMT
date: Wed, 31 Jul 2019 19:10:13 GMT
< server: istio-envoy
server: istio-envoy
<
* Connection #0 to host dev.myapp.westus.cld.something.com left intact
However when I try with
curl -Iv --insecure https://nonprod.westus.cloudapp.azure.com/myapp/status
I get HTTP 200. see below
* Trying 123.12.123.12...
* TCP_NODELAY set
* Connected to nonprod.westus.cloudapp.azure.com (123.12.123.12) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection:
ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=****; L=*****; O=******, Inc.; CN=*********
* start date: Jul 26 20:32:59 2019 GMT
* expire date: Jul 26 20:32:59 2021 GMT
* issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign RSA OV SSL CA 2018
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fb4b7003c00)
> HEAD /myapp/status HTTP/2
> Host: nonprod.westus.cloudapp.azure.com
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
HTTP/2 200
< request-context: appId=cid-v1:39296fe8-1dc6-4049-bb31-e7816e83b9ec
request-context: appId=cid-v1:39296fe8-1dc6-4049-bb31-e7816e83b9ec
< access-control-allow-origin: *
access-control-allow-origin: *
< access-control-expose-headers: Content-Disposition
access-control-expose-headers: Content-Disposition
< content-type: text/plain;charset=ISO-8859-1
content-type: text/plain;charset=ISO-8859-1
< content-length: 69
content-length: 69
< date: Wed, 31 Jul 2019 19:07:54 GMT
date: Wed, 31 Jul 2019 19:07:54 GMT
< x-envoy-upstream-service-time: 7
x-envoy-upstream-service-time: 7
< server: istio-envoy
server: istio-envoy
<
* Connection #0 to host nonprod.westus.cloudapp.azure.com left intact
Is my VS configuration wrong with regards to accepting HTTPS traffic? I am unable to look at envoy logs to see if there is any error.
How can I pinpoint where the issue is?
Update
dev.myapp.westus.cld.something.com is a CName record which ultimately points to nonprod.westus.cloudapp.azure.com
nonprod.westus.cloudapp.azure.com is a actual DNS which is mapped to a public IP.
Note sure if that makes any difference.
I am trying to access a remote server which responds properly in one network but not in another. I can access it on my local machine(In some networks) but not on aws:
Working Curl:
curl -vvv https://abc.example.com -H 'Content-Type:text/xml' --user ******:***** -d '<XML></XML>'
Output:
* Hostname was NOT found in DNS cache
* Trying xx.xx.xx.xx...
* Connected to abc.example.com (xx.xx.xx.xx) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
* subject: C=**; ST=***; L=***; x.x.x.x.x.x.x.x.x.x=**; x.x.x.x.x.x.x.x.x.x=ABCD; x.x.x.x.x.x.x.x.x.x.x=def; O=abc def ghi; businessCategory=abc def; serialNumber=123456; CN=abc.example.com
* start date: 2016-04-04 07:19:41 GMT
* expire date: 2018-04-04 07:49:39 GMT
* subjectAltName: abc.example.com matched
* issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2014 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1M
* SSL certificate verify ok.
* Server auth using Basic with user '******'
> POST /abc/def HTTP/1.1
> Authorization: Basic *********************
> User-Agent: curl/7.38.0
> Host: abc.example.com
> Accept: */*
> Content-Type:text/xml
> Content-Length: 16783
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Date: Sat, 23 Apr 2016 07:31:25 GMT
* Server Oracle-Application-Server-11g is not blacklisted
< Server: Oracle-Application-Server-11g
< X-ORACLE-DMS-ECID: *************************
< SOAPAction: "http://ghi.services//abc"
< X-Powered-By: Servlet/2.5 JSP/2.1
< Vary: Accept-Encoding,User-Agent
< Content-Type: text/xml; charset=utf-8
< Content-Language: en
< Transfer-Encoding: chunked
<
<XML>SUCCESSFUL OUTPUT</XML>
Not working CURL:
curl -vvv https://abc.example.com -H 'Content-Type:text/xml' --user ******:***** -d '<XML></XML>'
Output:
* Hostname was NOT found in DNS cache
* Trying xx.xx.xx.xx...
* Connected to abc.example.com (xx.xx.xx.xx) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
* subject: C=XX; ST=XXXXXXX; L=XXXXXXX; x.x.x.x.x.x.x=XX; x.x.x.x.x.x.x=XXXXXXX; x.x.x.x.x.x.x.x.x.x.x.x=XXXXX; O=Some company name; businessCategory=My unit; serialNumber=12345; CN=abc.example.com
* start date: 2016-04-04 07:19:41 GMT
* expire date: 2018-04-04 07:49:39 GMT
* subjectAltName: abc.example.com matched
* issuer: C=US; O=Entrust, Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2014 Entrust, Inc. - for authorized use only; CN=Entrust Certification Authority - L1M
* SSL certificate verify ok.
* Server auth using Basic with user '**********'
> POST /abc/def HTTP/1.1
> Authorization: Basic ******************
> User-Agent: curl/7.35.0
> Host: abc.example.com
> Accept: */*
> Content-Type:text/xml
> Content-Length: 16783
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 400 Bad Request
< Date: Sat, 23 Apr 2016 07:37:47 GMT
* Server Apache is not blacklisted
< Server: Apache
< Content-Length: 226
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>
* Closing connection 0
* SSLv3, TLS alert, Client hello (1):
I don't have any idea what's going on. Please help