How to get Python3 Requests module working for https request - python-3.x

I am trying to use requests module of Python3 to make an API call. I am using credentials that work when tried on same service from their web page. So credentials are verified.
Code that I am using:
#!/local/usr/bin/python3
import requests
url = "https://some.url.com/fol/something"
payload = "{'userName': \"user-name\", 'instanceName': \"instance-name\", 'password': \"user-password\", 'usersDomain': \"DOMAIN\"}"
headers = {'accept': 'application/json','content-type': 'application/json'}
response = requests.request("POST", url, data=payload, headers=headers, verify=False)
print(response)
Problem:
If I remove verify=False from line 7 I get ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749).
With verify=False (as in code above) I get following warning and response code is 401.
InsecureRequestWarning: Unverified HTTPS request is being made to host 'wifitracker.chartercom.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
<Response [401]>
What I have tried:
Tried using same code from different Linux servers with different Python and requests module version.
Tried ping {host} and nc -vz {host} 443 and both succeed. The API endpoint and Linux server from where I am trying to connect are both on the same network domain.
Tried adding verify="/etc/ssl/certs/ca-bundle.crt" and verify="/etc/ssl/certs/ca-bundle.trust.crt" as well as verify="/etc/ssl/certs" and verify="/etc/ssl/certs/". All these options don't cause any error/warning but the response code is still 401.
Used python module certifi and the found the installed certification authority budle and used it in the veryify= but I get ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)
Following suggestion by Steffen I tried openssl s_client -connect some.url.com:443 and result is (hope it's not too much):
USR--> openssl s_client -connect some.url.com:443
CONNECTED(00000003)
depth=2 C = <My Country Code>, O = <My Organization Name>, OU = <My Organization Name> Internal Tr<My Country Code>t Network, OU = (c) 2017 <My Organization Name> - For authorized <My Country Code>e only, CN = <My Organization Name> Root Certification Authority
verify error:num=19:self signed certificate in certificate chain
---
Certificate chain
0 s:/C=<My Country Code>/ST=<My State>/L=<My City>/O=<My Organization Name>/CN=some.url.com
i:/DC=com/DC=url/DC=corp/CN=<My Organization Name> Issuing CA1
1 s:/DC=com/DC=url/DC=corp/CN=<My Organization Name> Issuing CA1
i:/C=<My Country Code>/O=<My Organization Name>/OU=<My Organization Name> Internal Tr<My Country Code>t Network/OU=(c) 2017 <My Organization Name> - For authorized <My Country Code>e only/CN=<My Organization Name> Root Certification Authority
2 s:/C=<My Country Code>/O=<My Organization Name>/OU=<My Organization Name> Internal Tr<My Country Code>t Network/OU=(c) 2017 <My Organization Name> - For authorized <My Country Code>e only/CN=<My Organization Name> Root Certification Authority
i:/C=<My Country Code>/O=<My Organization Name>/OU=<My Organization Name> Internal Tr<My Country Code>t Network/OU=(c) 2017 <My Organization Name> - For authorized <My Country Code>e only/CN=<My Organization Name> Root Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
<CHUNK OF TEXT>
-----END CERTIFICATE-----
subject=/C=<My Country Code>/ST=<My State>/L=<My City>/O=<My Organization Name>/CN=some.url.com
issuer=/DC=com/DC=url/DC=corp/CN=<My Organization Name> Issuing CA1
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 6834 bytes and written 433 bytes
---
New, TLSv1/SSLv3, Cipher is <SOME-ALHPA-NUMERIC-CHARACTERS>
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : <SOME-ALHPA-NUMERIC-CHARACTERS>
Session-ID: <SOMELONGALHPANUMERIC64CHARACTERS>
Session-ID-ctx:
Master-Key: <SOMELONGALHPANUMERIC96CHARACTERS>
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: <epoch_time>
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
---
I tried finding options (from here on SO and other resources) and think I have tried those but obviously I am still missing something.
What else can I try to make a successful API call (status code 200)? Could it be that I need ca cert from the API provider? (It's a security related application)

Related

openssl certificate error for WinRM connection

I have on Windows server certificate which is valid and active and WinRM listener is active as well on port 5986 (telnet works) for WinRM connection that needs to be established from the Linux server.
I didn't copy that certificate anywhere on the linux server as I do not know where that should be or how it should be configured.
If I try to establish WinRM connection I am getting this error on the Linux server.
openssl s_client -connect 10.7.147.210:5986
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-384, 384 bits
---
SSL handshake has read 1367 bytes and written 447 bytes
Verification error: unable to verify the first certificate
I tried by referencing CAfile and CApath and cert option but without success:
openssl s_client -cert winrmcert.pem -key winrmcert.key -CApath . -connect 10.7.147.210:5986
openssl s_client -CAfile winrmcert.pem -connect 10.7.147.210:5986
Can you please help me what I need to do and configure on Linux server for certificate generated on Windows server for WinRM connection? I am not the expert for this topic so I would appreciate all useful instructions. Thank you

openLDAP Proxy fails with ldaps

This issue is driving me a bit insane.
I am trying to configure an openLDAP Proxy to an Active Directory, which works fine as long as I use unencrypted ldap to the AD. But I would like to secure the connection between the proxy and the AD via LDAPs.
What drives me crazy is that this worked before until I enabled LDAPs for the proxy itself. Now I can't get it running, even if I undo my changes.
This is how the slapd.conf looks like:
# Include schemas
include /etc/openldap/schema/corba.schema
include /etc/openldap/schema/core.schema
include /etc/openldap/schema/cosine.schema
include /etc/openldap/schema/duaconf.schema
include /etc/openldap/schema/dyngroup.schema
include /etc/openldap/schema/inetorgperson.schema
include /etc/openldap/schema/java.schema
include /etc/openldap/schema/misc.schema
include /etc/openldap/schema/nis.schema
include /etc/openldap/schema/openldap.schema
include /etc/openldap/schema/ppolicy.schema
include /etc/openldap/schema/collective.schema
include /etc/openldap/schema/local.schema
allow bind_v2
pidfile /var/run/openldap/slapd.pid
argsfile /var/run/openldap/slapd.args
modulepath /usr/lib/openldap
modulepath /usr/lib64/openldap
moduleload back_ldap
moduleload rwm
loglevel 256
#LDAPS Settings
TLSCipherSuite HIGH:MEDIUM:-SSLv3
TLSCertificateFile /etc/openldap/cacerts/server.crt
TLSCertificateKeyFile /etc/openldap/cacerts/server.key
TLSCACertificateFile /etc/openldap/cacerts/CAs.pem
### Database definition (Proxy to AD) #########################################
database ldap
readonly no
protocol-version 3
rebind-as-user
uri "ldaps://ad-server.internal.de:636"
suffix "dc=ad,dc=internal,dc=de"
Now when I search I get the below error (ldaps or ldap).
Both queries work just fine when I change the uri to "ldap://ad-server.internal.de:389"
ldapsearch -h localhost:389 -D "CN=admin,OU=User,DC=ad,DC=internal,DC=de" -W -b DC=ad,DC=internal,DC=de '(objectclass=person)'
ldapsearch -H ldaps://localhost:636 -D "CN=admin,OU=User,DC=ad,DC=internal,DC=de" -W -b DC=ad,DC=internal,DC=de '(objectclass=person)'
=>
ldap_bind: Server is unavailable (52)
additional info: Proxy operation retry failed
This is not a certificate error, as the direct ldapsearch from the same host works just fine:
ldapsearch -H ldaps://ad-server.internal.de:636 -D "CN=admin,OU=User,DC=ad,DC=internal,DC=de" -W -b DC=ad,DC=internal,DC=de '(objectclass=person)'
This is what the log looks like:
Jun 09 12:06:06 adproxy.internal.de slapd[94952]: conn=1000 fd=12 ACCEPT from IP=127.0.0.1:47458 (IP=0.0.0.0:389)
Jun 09 12:06:06 adproxy.internal.de slapd[94952]: conn=1000 op=0 BIND dn="cn=admin,ou=User,dc=ad,dc=internal,dc=de" method=128
Jun 09 12:06:06 adproxy.internal.de slapd[94952]: conn=1000 op=0 ldap_back_retry: retrying URI="ldaps://ad-server.internal.de:636" DN=""
Jun 09 12:06:06 adproxy.internal.de slapd[94952]: conn=1000 op=0 RESULT tag=97 err=52 text=Proxy operation retry failed
Jun 09 12:06:06 adproxy.internal.de slapd[94952]: conn=1000 op=1 UNBIND
Jun 09 12:06:06 adproxy.internal.de slapd[94952]: conn=1000 fd=12 closed
Any idea why this happens would be greatly appreciated.
Edit:
I did some more digging and it actually seems to be a certificate problem. The tcpdump for the connection returns this error:
TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Unknown CA)
The AD certificate is self signed and unfortunately I have no saying in that, so I cannot change that.
In the standard config I get the same error when running ldapsearch. I can get ldapsearch running with two methods:
Set TLS_REQCERT never in ldap.conf
Set environment variable LDAPTLS_REQCERT=never
I know this is not ideal but this is not supposed to be the final configuration, I just want to get it running first.
However slapd seems to ignore both, because I still get the same error in the tcpdump and slapd debugging still gives me the following:
TLS trace: SSL_connect:before/connect initialization
TLS trace: SSL_connect:SSLv2/v3 write client hello A
TLS trace: SSL_connect:SSLv3 read server hello A
TLS certificate verification: depth: 0, err: 20, subject: /CN=ad-server.internal.de, issuer: /CN=ad-server.internal.de
TLS certificate verification: Error, unable to get local issuer certificate
TLS trace: SSL3 alert write:fatal:unknown CA
TLS trace: SSL_connect:error in error
TLS trace: SSL_connect:error in error
TLS: can't connect: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (unable to get local issuer certificate).
I tried to add the self-signed certificate to the ldap.conf and to the trusted anchors for openssl, but both didn't work. I also noticed that I cannot get openssl to verify the self-signed certificate, while I can do the same on my PC...
My PC:
$ openssl version
OpenSSL 1.1.1h 22 Sep 2020
$ openssl verify server.crt
error server.crt: verification failed
CN = ad-server.internal.de
error 18 at 0 depth lookup: self signed certificate
$ openssl verify -CAfile server.crt server.crt
server.crt: OK
While on the server (REHL 7.9):
$ openssl version
OpenSSL 1.0.2k-fips 26 Jan 2017
$ openssl verify server.crt
server.crt: CN = ad-server.internal.de
error 20 at 0 depth lookup: unable to get local issuer certificate
$ openssl verify -CAfile server.crt server.crt
server.crt: CN = ad-server.internal.de
error 20 at 0 depth lookup: unable to get local issuer certificate
The result on my PC is the expected behaviour for self-signed certificates, as far as I know. I already checked, but it is the latest version available on RHEL 7.9
So does anyone have any idea how I can
get SLAPD to honor the TLS_REQCERT or LDAPTLS_REQCERT settings OR
openssl to verify the self-signed certificate
Thank you!
I finally found a possibility to ignore the certificate validation error. Which is not the optimal solution, but due to the self-signed certificate and the openssl issue in the RHEL7.9 version, I am for now content to have it running at all. The final solution should then be to make the AD department get a certificate from our CA, though I don't know if I can convince them to do so.
So while the LDAP proxy seems to ignore the TLS_REQCERT none setting in the ldap.conf, there is a possibility to do this directly in slapd.conf:
### Database definition (Proxy to AD) #########################################
database ldap
readonly no
protocol-version 3
rebind-as-user
uri "ldaps://ad-server.internal.de:636"
suffix "dc=ad,dc=internal,dc=de"
tls start tls_reqcert=never
The last line eliminates the verification error.

Hyperledger Fabric CA: http: TLS handshake error from 127.0.0.1:53356: tls: oversized record received with length 21536

we have a very simple setup where we have spin up an instance of fabric-ca-server by running
/bin/bash -c "fabric-ca-server start -b admin:adminpw"
in a docker container as explained e.g., here. but when we try to enroll a user by running
root#a7094a09a3b7:/etc/hyperledger/fabric-ca-client# fabric-ca-client enroll -u http://user:userpw#localhost:7054
we get below error:
2018/12/07 21:06:17 [INFO] generating key: &{A:ecdsa S:256}
2018/12/07 21:06:17 [INFO] encoded CSR
Error: POST failure of request: POST http://localhost:7054/enroll
{"hosts":["a7094a09a3b7"],"certificate_request":"-----BEGIN CERTIFICATE REQUEST-----\nMIIBQzCB6wIBADBfMQswCQYDVQQGEwJVUzEXMBUGA1UECBMOTm9ydGggQ2Fyb2xp\nbmExFDASBgNVBAoTC0h5cGVybGVkZ2VyMQ8wDQYDVQQLEwZGYWJyaWMxEDAOBgNV\nBAMTB3NqYWluNjgwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAASiWBUZ5vG5i9X9\nAV8fdfuxFAFsWHeg4JRpNFs/m7knB3QaDWDbbk/G53gEbgyv2scg61cYMNdLDaYK\nGi9ywX0noCowKAYJKoZIhvcNAQkOMRswGTAXBgNVHREEEDAOggxhNzA5NGEwOWEz\nYjcwCgYIKoZIzj0EAwIDRwAwRAIgTY0gn18UIsKfOLgS33F41ORAlKNz2tZS6ip5\nXS03XNICIAE3C9nhv2d4ZrCcO+7rycWLoR6wEU6cnAqZsSf1Txz4\n-----END CERTIFICATE REQUEST-----\n","profile":"","crl_override":"","label":"","NotBefore":"0001-01-01T00:00:00Z","NotAfter":"0001-01-01T00:00:00Z","CAName":""}: Post http://localhost:7054/enroll: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02\x16"
and the fabric-ca-server output shows:
http: TLS handshake error from 127.0.0.1:53356: tls: oversized record received with length 21536
wondering what are we doing wrong here and how can we fix this?
Looks like you started the fabric-ca-server with TLS enabled and you are not using TLS with the fabric-ca-client. You need to specify https in your URL:
fabric-ca-client enroll -u https://user:userpw#localhost:7054
Alternatively, disable TLS in the Fabric CA config.

TLS error enrolling with Fabric CA Client

I prepared multi-organization with Fabric CA environment by following tutorial. This tutorial itself was worked with no problem.
Deploying a Hyperledger Composer blockchain business network to Hyperledger Fabric (multiple
organizations)
https://hyperledger.github.io/composer/unstable/tutorials/deploy-to-fabric-multi-org.html
In this environment, I'm trying Fabric CA client commands on both ca_peerOrg1 and host environment. I copied tls crt, public key and private key from crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/ and tried following command, but I got error "x509: certificate is valid for ca.org1.example.com, not localhost".
linux1#fabric:~$ fabric-ca-client enroll --caname ca-org1 --tls.certfiles ca.crt --tls.client.certfile Admin#org1.example.com-cert.pem --tls.client.keyfile 0dbfd9d156b90331e6bc4a0529e07cb123f7de586935019499623062f182a8fa_sk -u https://admin:adminpw#localhost:7054
2017/12/19 16:34:35 [INFO] User provided config file: /home/linux1/fabric-ca/clients/admin/fabric-ca-client-config.yaml
2017/12/19 16:34:35 [INFO] generating key: &{A:ecdsa S:256}
2017/12/19 16:34:35 [INFO] encoded CSR
2017/12/19 16:34:35 [INFO] TLS Enabled
Error: POST failure [Post https://localhost:7054/enroll: x509: certificate is valid for ca.org1.example.com, not localhost]; not sending
POST https://localhost:7054/enroll
Authorization: Basic YWRtaW46YWRtaW5wdw==
{"hosts":["fabric"],"certificate_request":"-----BEGIN CERTIFICATE REQUEST-----\nMIIBOzCB4wIBADBdMQswCQYDVQQGEwJVUzEXMBUGA1UECBMOTm9ydGggQ2Fyb2xp\nbmExFDASBgNVBAoTC0h5cGVybGVkZ2VyMQ8wDQYDVQQLEwZGYWJyaWMxDjAMBgNV\nBAMTBWFkbWluMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEI8Y4aR/YxeyMXIQ3\nHGydrTLaoI+6tZ57ho1uv6gNonbFfftE7lrdsIbL8d8QLimztDjtqkg6rk612nso\nD66M9aAkMCIGCSqGSIb3DQEJDjEVMBMwEQYDVR0RBAowCIIGZmFicmljMAoGCCqG\nSM49BAMCA0cAMEQCIH9u2w9xO43Ba5rJ1SCKgtEFfokj1DKhg2cHjnbu8ucEAiBn\nwZKjjSbQage+ElofYbgagWpuCFz0t57Zwycz22M00w==\n-----END CERTIFICATE REQUEST-----\n","profile":"","crl_override":"","label":"","CAName":"ca-org1"}
linux1#fabric:~$
How can I avoid this error ? When I'm looking at the tutorial, I realized that composer connection profile has configuration like "hostnameOverride": "ca.org1.example.com". I guess I need to do same kind for fabric-ca-client.

NodeJS + Socket.io SSL fails after a few connections

I'm running NodeJS with socket.io, all under SSL. I've recently moved servers and what used to work fine now stops working after a couple of minutes - Chrome says the connection is reset, and curl says there are SSL problems.
I've tried with and without express - neither with any success after a couple of minutes.
So - my minimal code is:
var fs = require('fs');
var https = require('https');
const options = {
key: fs.readFileSync('XXX.key'),
cert: fs.readFileSync('XXX.crt'),
ca: fs.readFileSync('XXX.ca')
};
var app = https.createServer(options, function(req, res) {
res.writeHead(200);
res.end('(not imporant)\n');
}).listen(3000);
var io = require('socket.io')(app);
If I comment out the last line, the server always accepts new connections (albeit without socket.io) and everything works. If I leave that last line in, it works for a minute, various SSL checking tools are all ok, including:
openssl s_client -connect REDACTED.com:3000
CONNECTED(00000003)
depth=3 /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
0 s:/OU=Domain Control Validated/OU=Hosted by XILO Communications Ltd./OU=PositiveSSL Wildcard/CN=*.REDACTED.com
i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
Server certificate
-----BEGIN CERTIFICATE-----
REDACTED
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/OU=Hosted by XILO Communications Ltd./OU=PositiveSSL Wildcard/CN=*.REDACTED.com
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
---
No client certificate CA names sent
---
SSL handshake has read 5650 bytes and written 456 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : AES256-SHA
Session-ID: 704A23D4B00B3A445A8C9097427FF26B724458F24B105C847B76E576FCA9C803
Session-ID-ctx:
Master-Key: 04FB6B5B06954516A073C65A456010C819B157A20F5F308EDCC88C8B0FDCD9A990D3A23AC117714363A92EB56BC98272
Key-Arg : None
Start Time: 1474318665
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
But shortly after (and probably a few hundred client connections from people using the site), the same command gives (from a Mac):
CONNECTED(00000003)
65931:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-59.60.1/src/ssl/s23_lib.c:185:
And from linux/vagrant:
CONNECTED(00000003)
140236360926880:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 295 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---
curl https://REDACTED:3000 gives:
curl: (35) Unknown SSL protocol error in connection to REDACTED:3000
Does anyone have any ideas how to overcome this? The code works fine on our development machines, but as soon as it goes onto the live servers, it breaks pretty quickly.
Is it anything to do with the fact I'm using a wildcard certificate?
Thanks in advance.
Ed

Resources