Pass certificates into requests lib - python-3.x

I have https server written on Python which uses self singed certs:
server_cert = 'certs/server/server.crt'
server_key = 'certs/server/server.key'
client_certs = 'certs/client/client.crt'
context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
context.verify_mode = ssl.CERT_REQUIRED
context.load_cert_chain(certfile=server_cert, keyfile=server_key)
context.load_verify_locations(cafile=client_certs)
Steps to certs generate:
openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=localhost/UID=testnotifier" -keyout ${certs}/server/server.key -out ${certs}/server/server.crt
openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=localhost/UID=testnotifier" -keyout ${certs}/client/client.key -out ${certs}/client/client.crt
Also I have https client:
#!/usr/bin/python3
import socket
import ssl
host_addr = '127.0.0.1'
host_port = 8082
server_sni_hostname = 'localhost'
server_cert = 'certs/server/server.crt'
client_cert = 'certs/client/client.crt'
client_key = 'certs/client/client.key'
context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH, cafile=server_cert)
context.load_cert_chain(certfile=client_cert, keyfile=client_key)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
conn = context.wrap_socket(s, server_side=False, server_hostname=server_sni_hostname)
conn.connect((host_addr, host_port))
conn.send(b"Hello, world!")
conn.close()
This works, but I want to send different requests to some path, e.g. /rest/v1/fill.
So, I decided to use requests or urllib3 lib.
I'm trying to pass certificates to this libs
https = urllib3.PoolManager(cert_file=client_cert, cert_reqs='CERT_REQUIRED', ca_certs=server_cert, key_file=client_key)
https.request('GET', 'https://localhost:8082/rest/v1/fill')
but client hangs with both of libs:
/home/marat/.local/lib/python3.6/site-packages/urllib3/connection.py:362: SubjectAltNameWarning: Certificate for localhost has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
SubjectAltNameWarning
^CTraceback (most recent call last):
File "/home/marat/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 377, in _make_request
httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'
server's output:
Client connected: 127.0.0.1:40026
SSL established. Peer: {'subject': ((('countryName', 'US'),), (('stateOrProvinceName', 'Denial'),), (('localityName', 'Springfield'),), (('organizationName', 'Dis'),), (('commonName', 'localhost'),), (('userId', 'testnotifier'),)), 'issuer': ((('countryName', 'US'),), (('stateOrProvinceName', 'Denial'),), (('localityName', 'Springfield'),), (('organizationName', 'Dis'),), (('commonName', 'localhost'),), (('userId', 'testnotifier'),)), 'version': 3, 'serialNumber': '88F15B15B2D1ABE7', 'notBefore': 'Jun 5 11:35:38 2019 GMT', 'notAfter': 'Jun 4 11:35:38 2020 GMT'}
Received: b'GET / HTTP/1.1\r\nHost: localhost:8082\r\nAccept-Encoding: identity\r\n\r\n'
Closing connection
I would like to know how to pass certificates so that the client works as in the first version (with socket/ssl).

I worked incorrectly with certificates.
So, bash script for generate certificates:
generate_test_certs() {
echo "Generate test certs"
mkdir -p ${SCRIPT_PATH}/test/certs/{server,client,root}
local certs=${SCRIPT_PATH}/test/certs
local CN=localhost # common name
cat <<EOF > ${certs}/openssl.cnf
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = #alt_names
[alt_names]
DNS.1 = ${CN}
EOF
# Create root key
openssl genrsa -out ${certs}/root/rootCA.key 2048
openssl req -new -x509 -days 3650 -key ${certs}/root/rootCA.key -out ${certs}/root/rootCA.crt \
-subj "/C=US/ST=Denial/L=Springfield/O=Qwerty Inc./OU=Qwerty Operations/CN=Qwerty Root"
# Generate sign request
openssl req -new -newkey rsa:2048 -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=${CN}/UID=testnotifier" \
-keyout ${certs}/server/server.key -nodes -out ${certs}/server/server.csr
# Sign CSR by ROOT CA certificate
openssl x509 -req -days 3650 -in ${certs}/server/server.csr -CA ${certs}/root/rootCA.crt \
-CAkey ${certs}/root/rootCA.key -set_serial 01 -out ${certs}/server/server.crt \
-extfile ${certs}/openssl.cnf
# Generate client certificate
openssl req -new -newkey rsa:2048 -nodes -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=${CN}/UID=testnotifier" \
-keyout ${certs}/client/client.key -out ${certs}/client/client.csr
openssl x509 -req -days 3650 -in ${certs}/client/client.csr -CA ${certs}/root/rootCA.crt -CAkey ${certs}/root/rootCA.key \
-set_serial 01 -out ${certs}/client/client.crt
}
Nginx config for example:
server {
listen 451 ssl;
server_name tester;
ssl_certificate ../../../test/certs/server/server.crt;
ssl_certificate_key ../../../test/certs/server/server.key;
ssl_client_certificate ../../../test/certs/root/rootCA.crt;
ssl_verify_client optional;
ssl_verify_depth 3;
include ../../../conf/tester.rules.conf;
}
And finally python client
#!/usr/bin/python3
import requests
root_ca = 'certs/root/rootCA.crt'
client_cert = 'certs/client/client.crt'
client_key = 'certs/client/client.key'
resp = requests.get('https://localhost:451/qwe/rty',
cert=(client_cert, client_key), verify=root_ca)
print(resp.status_code)
print(resp.content)

Related

Self signed certificate only works with localhost, not 127.0.0.1

I'm trying to generate a self-signed certificate such that my local development environment uses HTTPS, but I'm having some trouble. The reason for this is that I want to test push notifications on my phone through my local network (through my local IP 192.168.1.155) and notifications only work via a secure context.
It only seems to work when I go to localhost:8080, and is still insecure when navigating to 127.0.0.1:8080. When I navigate to 127.0.0.1:8080 Chrome's Security Page says: This site is missing a valid, trusted certificate (net::ERR_CERT_COMMON_NAME_INVALID).
Here's my setup I use to generate the certificate:
req.cnf:
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = US
ST = VA
L = SomeCity
O = MyCompany
OU = MyDivision
CN = 127.0.0.1
[v3_req]
keyUsage = critical, digitalSignature, keyAgreement
extendedKeyUsage = serverAuth
subjectAltName = #alt_names
[alt_names]
DNS.1 = localhost
DNS.2 = 127.0.0.1
DNS.3 = 192.168.1.155
openssl req -newkey rsa:2048 -x509 -nodes -keyout key.pem -new -out cert.pem -config req.cnf -sha256 -days 3650
I'd imagine perhaps my CN or alt_names is incorrect, but I'm not sure what to change them to such that the site will always work securely (either via localhost, 127.0.0.1, or 192.168.1.155)
In an unforseen case of rubber duck debugging, I seem to have finally solved this issue momentarily after posting it. Here's what I did:
req.cnf:
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = US
ST = VA
L = SomeCity
O = MyCompany
OU = MyDivision
CN = localhost
[v3_req]
keyUsage = critical, digitalSignature, keyAgreement
extendedKeyUsage = serverAuth
subjectAltName = DNS:localhost,IP:192.168.1.155,IP:127.0.0.1
command prompt:
openssl req -newkey rsa:2048 -x509 -nodes -keyout key.pem -new -out cert.pem -config req.cnf -sha256 -days 3650
Then navigate to the page in Chrome, save the certificate (as it will still be invalid) as a DER file and then using mmc.exe, import it into the Trusted Root Certification Authorities on your machine (this is assuming you're using Windows)

NodeJs - Error: error:0906D06C:PEM routines:PEM_read_bio:no start line

I am trying to use godaddy ssl certificates on ec2. But getting this error.
Error: error:0906D06C:PEM routines:PEM_read_bio:no start line
Flow I used.
openssl req -new -newkey rsa:2048 -nodes -keyout foo.com.key -out
foo.com.csr
used foo.com.csr to generate the godaddy and got 3 files.
3b8bc5516b18ff0.crt , 3b8bc5516b18ff0.pem and gd_bundle-g2-g1.crt
var privateKey = fs.readFileSync('3b8bc5516b18ff0.pem');
var certificate = fs.readFileSync('3b8bc5516b18ff0.crt');
/**
* And converted gd_bundele to gd0, gd1, gd2
*/
var credentials = {
key: privateKey,
cert: certificate,
ca: [
fs.readFileSync('gd0.crt'),
fs.readFileSync('gd1.crt'),
fs.readFileSync('gd2.crt')
]
};
//http.createServer(app);
var httpsServer = https.createServer(credentials, app);
I got the solution .
openssl req -new -newkey rsa:2048 -nodes -keyout foo.com.pem -out foo.com.csr
foo.com.pem should be passed as privatekey.
NOTE : It is important save the key with which the csr was created and sent to the godaddy (I called foo.com.pem).

I have .pem privatekey and .der encrypted file. I want to decrypt using nodejs

I have used the following commands to generate the encrypted file "Example.txt.der".
openssl genrsa -out privatekey.pem 2048
openssl req -new -sha256 -key privatekey.pem -out csr.csr
openssl req -x509 -sha256 -days 365 -key privatekey.pem -in csr.csr -out certificate.pem
openssl smime -encrypt -aes-256-cbc -binary -in Example.txt -outform DER -out Example.txt.der certificate.pem
I want to decrypt "Example.txt.der" using : privatekey.pem.
Try the below command:
openssl smime -decrypt -in Example.txt.der -inform DER -inkey privatekey.pem -out DecryptedExample.txt
It works using exec command.
const exec = require('child_process').exec; exec('openssl smime -decrypt -in ./encryptedConfig.txt -inkey ./privatekey.pem', (error, stdout, stderr) => { if (error) { console.log("error:",error); reject(error); } else { console.log("Config decrypt completed. Data: ",stdout); resolve(stdout); }

Certificate SSL_ERROR_BAD_CERT_DOMAIN for localhost / 127.0.0.1

I am creating a local app that starts a webserver in the localhost:8080 address. I am trying to create a certificate for it so that I can access it using HTTPS, but I am having a hard time doing this.
First I created my own CA and then I created a certificate with the localhost:8080 common name. Then I added my CA to the trusted authorities on my computer (I am using Windows 10 by the way), however when I opened my site I got the BAD_CERT_DOMAIN error using Firefox and Chrome.
I also tried to created another certificate using 127.0.0.1:8080 as the common name, but it also didn't work.
What I am doing wrong? Do these browsers always reject certificates with localhost as the CN?
UPDATE
I created a configuration file like this:
[ req ]
default_bits = 2048
distinguished_name = req_distinguished_name
req_extensions = req_ext
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
stateOrProvinceName = State or Province Name (full name)
localityName = Locality Name (eg, city)
organizationName = Organization Name (eg, company)
commonName = Common Name (e.g. server FQDN or YOUR name)
[ req_ext ]
subjectAltName = #alt_names
[ alt_names ]
DNS.1 = localhost
DNS.2 = localhost:8080
DNS.3 = localhost:12125
DNS.4 = localhost:12126
DNS.5 = 127.0.0.1:8080
DNS.6 = 127.0.0.1:12125
DNS.7 = 127.0.0.1:12126
DNS.8 = 127.0.0.1
IP.1 = 127.0.0.1
And these are the commands that I am using to generate my certificate:
Sign request: openssl req -out myrequest.csr -newkey rsa:2048 -nodes -keyout mykey.key -config myconfig.conf
When I ran this command, the CN = localhost 127.0.0.1
Signining with my CA: openssl x509 -req -in myrequest.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out mycertificate.crt -days 36500 -sha256
However I am still getting the BAD_CERT_DOMAIN for both Firefox and Google Chrome, even after I tell them to trust my own CA.
I have followed #jww's advice and I re-wrote my config file as this:
[ req ]
default_bits = 2048
distinguished_name = req_distinguished_name
req_extensions = req_ext
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
stateOrProvinceName = State or Province Name (full name)
localityName = Locality Name (eg, city)
organizationName = Organization Name (eg, company)
commonName = Common Name (e.g. server FQDN or YOUR name)
[ req_ext ]
subjectAltName = #alt_names
[ alt_names ]
DNS.1 = localhost
IP.1 = 127.0.0.1
Then I used this commands to generate my certificate.
Sign request: openssl req -out myrequest.csr -newkey rsa:2048 -nodes -keyout mykey.key -config myconfig.conf
Signining with my CA: openssl x509 -req -in myrequest.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out mycertificate.crt -days 36500 -sha256
So the problem really was that I was adding the port numbers to the IP and DNS addresses of my certificate sign request.

Client certificate validation on server side, DEPTH_ZERO_SELF_SIGNED_CERT error

I'm using node 0.10.26 and trying to establish https connection with client validation.
Server's code:
var https = require('https');
var fs = require('fs');
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";
var options = {
key: fs.readFileSync('ssl/server1.key'),
cert: fs.readFileSync('ssl/server1.pem'),
requestCert: true,
rejectUnauthorized: false,
};
var server = https.createServer(options, function (req, res) {
if (req.client.authorized) {
res.writeHead(200, {"Content-Type":"application/json"});
res.end('{"status":"approved"}');
console.log("Approved Client ", req.client.socket.remoteAddress);
} else {
console.log("res.connection.authroizationError: " + res.connection.authorizationError);
res.writeHead(403, {"Content-Type":"application/json"});
res.end('{"status":"denied"}');
console.log("Denied Client " , req.client.socket.remoteAddress);
}
});
server.on('error', function(err) {
console.log("server.error: " + err);
});
server.on("listening", function () {
console.log("Server listeining");
});
server.listen(5678);
Client's code:
var https = require('https');
var fs = require('fs');
var options = {
host: 'localhost',
port: 5678,
method: 'GET',
path: '/',
headers: {},
agent: false,
key: fs.readFileSync('ssl/client2.key'),
cert: fs.readFileSync('ssl/client2.pem'),
ca: fs.readFileSync('ssl/ca.pem')
};
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";
var req = https.request(options, function(res) {
console.log(res.req.connection.authorizationError);
});
req.on("error", function (err) {
console.log('error: ' + err);
});
req.end();
I've created certificates with following commands, each time providing result of "uname -n" as "Common Name":
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key -days 999 -out ca.pem
openssl genrsa -out server1.key 1024
openssl req -new -key server1.key -out server1.csr
openssl x509 -req -days 999 -in server1.csr -CA ca.pem -CAkey ca.key -set_serial 01 -out server1.pem
openssl genrsa -out client1.key 1024
openssl req -new -key client1.key -out client1.csr
openssl x509 -req -days 999 -in client1.csr -CA ca.pem -CAkey ca.key -set_serial 01 -out client1.pem
openssl genrsa -out server2.key 1024
openssl req -new -key server2.key -out server2.csr
openssl x509 -req -days 999 -in server2.csr -CA server1.pem -CAkey server1.key - set_serial 02 -out server2.pem
openssl genrsa -out client2.key 1024
openssl req -new -key client2.key -out client2.csr
openssl x509 -req -days 999 -in client2.csr -CA client1.pem -CAkey client1.key -set_serial 02 -out client2.pem
I've run client and server with all compbinations of client's and server's certificates (that is: [(server1, client1), (server1, client2), (server2, client1), (server2, client2)] and for each combination of those server was tested both with default value of "agent" field and with "agent" set to "false".
Each time I ran client.js, res.req.connection.authorizationError was set to DEPTH_ZERO_SELF_SIGNED_CERT.
How can I establish secure connection in node with client's certificate authentication?
I believe you have two problems, one with your code and one with your certificates.
The code issue is in your server. You are not specifying the CA to check client certificates with an options property like you have in your client code:
ca: fs.readFileSync('ssl/ca.pem'),
The second problem is the one that really causes that DEPTH_ZERO_SELF_SIGNED_CERT error. You are giving all your certificates - CA, server, and client - the same Distinguished Name. When the server extracts the issuer information from the client certificate, it sees that the issuer DN is the same as the client certificate DN and concludes that the client certificate is self-signed.
Try regenerating your certificates, giving each one a unique Common Name (to make the DN also unique). For example, name your CA certificate "Foo CA", your server certificate the name of your host ("localhost" in this case), and your client something else (e.g. "Foo Client 1").
For those of that want to use a self-signed certificate, the answer is to add rejectUnauthorized: false to the https.request options.
This one worked for me:
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";
Note: Posting answer so it can help others in future.
Just add strictSSL: false to your options.
Despite of long lines of description in this page, I still got 'UNABLE_TO_VERIFY_LEAF_SIGNATURE' error in the client side with this recipe. Maybe I had something wrong in following rhashimoto's comment.
By referencing book of 'Professional Node.js', I found another way to test HTTPS with Client Certificate Verification successfully.
Here is my story.
By setting requestCert: true in server side, the server tries to validate client certificate. But the default CA doesn't validate the self-signed certificate of the client. I can succeed the test with simple trick -- copy the client certificate and say that is a Certificate Authority.
I reused original code and modified it slightly to make it work.
The big difference is in creating certificate files.
Creating certificate files:
# create client private key
openssl genrsa -out client2.key
openssl req -new -key client2.key -out client2.csr
# create client certificate
openssl x509 -req -in client2.csr -signkey client2.key -out client2.pem
# create server private key and certificate
openssl genrsa -out server1.key
openssl req -new -key server1.key -out server1.csr
openssl x509 -req -in server1.csr -signkey server1.key -out server1.pem
# * Important *: create fake CA with client certificate for test purpose
cp client2.pem fake_ca.pem
Server code:
var options = {
key: fs.readFileSync('ssl/server1.key'),
cert: fs.readFileSync('ssl/server1.pem'),
ca: [fs.readFileSync('ssl/fake_ca.pem')], // Line added
requestCert: true,
rejectUnauthorized: false,
};
Client code:
key: fs.readFileSync('ssl/client2.key'),
cert: fs.readFileSync('ssl/client2.pem'),
//ca: fs.readFileSync('ssl/ca.pem') // Line commented
};
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";
var req = https.request(options, function(res) {
//console.log(res.req.connection.authorizationError);
console.log("statusCode:", res.statusCode);
res.on('data', function(d) {
console.log('data:', d.toString());
});
});
req.on("error", function (err) {
console.log('error: ' + err);
});
req.end();
As mentioned above, there is a sledgehammer to hammer in your nail, using rejectUnauthorized: false.
A more sensible option, from a security point of view, would be to ask the user if (s)he would like to accept and store the self-signed server certificate, just like a browser (or SSH) does.
That would require:
(1) That NodeJS throws an exception that contains the server certificate, and
(2) that the application calls https.request with the stored certificate in the ca: property (see above for description of ca) after the certificate has been manually accepted.
It seems that NodeJS does not do (1), making (2) impossible?
Even better from a security point of view would be to use EFF's SSL observatory to make a crowd-sourced judgement on the validity of a self-signed certificate. Again, that requires NodeJS to do (1).
I think a developer needs to improve NodeJS with respect to (1)...
If you have only a .pem self-signed certificate (e.g. /tmp/ca-keyAndCert.pem) the following options will work:
var options = {
url: 'https://<MYHOST>:<MY_PORT>/',
key: fs.readFileSync('/tmp/ca-keyAndCert.pem'),
cert: fs.readFileSync('/tmp/ca-keyAndCert.pem'),
requestCert: false,
rejectUnauthorized: false
};

Resources