I am trying to connect to a mongodb replica set which is set up to authenticate clients using SSL. I can connect using the mongo shell, but for some reason cannot connect from node.js with the same keys.
I am using mongodb version 3.2.6 and node.js driver version 2.1.18, running on mac.
I followed this article, and was able to setup a cluster on my local machine by running the attached script:
# Prerequisites:
# a. Make sure you have MongoDB Enterprise installed.
# b. Make sure mongod/mongo are in the executable path
# c. Make sure no mongod running on 27017 port, or change the port below
# d. Run this script in a clean directory
##### Feel free to change following section values ####
# Changing this to include: country, province, city, company
dn_prefix="/C=CN/ST=GD/L=Shenzhen/O=MongoDB China"
ou_member="MyServers"
ou_client="MyClients"
mongodb_server_hosts=( "server1" "server2" "server3" )
mongodb_client_hosts=( "client1" "client2" )
mongodb_port=27017
# make a subdirectory for mongodb cluster
kill $(ps -ef | grep mongod | grep set509 | awk '{print $2}')
#rm -Rf db/*
mkdir -p db
echo "##### STEP 1: Generate root CA "
openssl genrsa -out root-ca.key 2048
# !!! In production you will want to use -aes256 to password protect the keys
# openssl genrsa -aes256 -out root-ca.key 2048
openssl req -new -x509 -days 3650 -key root-ca.key -out root-ca.crt -subj "$dn_prefix/CN=ROOTCA"
mkdir -p RootCA/ca.db.certs
echo "01" >> RootCA/ca.db.serial
touch RootCA/ca.db.index
echo $RANDOM >> RootCA/ca.db.rand
mv root-ca* RootCA/
echo "##### STEP 2: Create CA config"
# Generate CA config
cat >> root-ca.cfg <<EOF
[ RootCA ]
dir = ./RootCA
certs = \$dir/ca.db.certs
database = \$dir/ca.db.index
new_certs_dir = \$dir/ca.db.certs
certificate = \$dir/root-ca.crt
serial = \$dir/ca.db.serial
private_key = \$dir/root-ca.key
RANDFILE = \$dir/ca.db.rand
default_md = sha256
default_days = 365
default_crl_days= 30
email_in_dn = no
unique_subject = no
policy = policy_match
[ SigningCA ]
dir = ./SigningCA
certs = \$dir/ca.db.certs
database = \$dir/ca.db.index
new_certs_dir = \$dir/ca.db.certs
certificate = \$dir/signing-ca.crt
serial = \$dir/ca.db.serial
private_key = \$dir/signing-ca.key
RANDFILE = \$dir/ca.db.rand
default_md = sha256
default_days = 365
default_crl_days= 30
email_in_dn = no
unique_subject = no
policy = policy_match
[ policy_match ]
countryName = match
stateOrProvinceName = match
localityName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer:always
basicConstraints = CA:true
EOF
echo "##### STEP 3: Generate signing key"
# We do not use root key to sign certificate, instead we generate a signing key
openssl genrsa -out signing-ca.key 2048
# !!! In production you will want to use -aes256 to password protect the keys
# openssl genrsa -aes256 -out signing-ca.key 2048
openssl req -new -days 1460 -key signing-ca.key -out signing-ca.csr -subj "$dn_prefix/CN=CA-SIGNER"
openssl ca -batch -name RootCA -config root-ca.cfg -extensions v3_ca -out signing-ca.crt -infiles signing-ca.csr
mkdir -p SigningCA/ca.db.certs
echo "01" >> SigningCA/ca.db.serial
touch SigningCA/ca.db.index
# Should use a better source of random here..
echo $RANDOM >> SigningCA/ca.db.rand
mv signing-ca* SigningCA/
# Create root-ca.pem
cat RootCA/root-ca.crt SigningCA/signing-ca.crt > root-ca.pem
echo "##### STEP 4: Create server certificates"
# Now create & sign keys for each mongod server
# Pay attention to the OU part of the subject in "openssl req" command
# You may want to use FQDNs instead of short hostname
for host in "${mongodb_server_hosts[#]}"; do
echo "Generating key for $host"
openssl genrsa -out ${host}.key 2048
openssl req -new -days 365 -key ${host}.key -out ${host}.csr -subj "$dn_prefix/OU=$ou_member/CN=${host}"
openssl ca -batch -name SigningCA -config root-ca.cfg -out ${host}.crt -infiles ${host}.csr
cat ${host}.crt ${host}.key > ${host}.pem
done
echo "##### STEP 5: Create client certificates"
# Now create & sign keys for each client
# Pay attention to the OU part of the subject in "openssl req" command
for host in "${mongodb_client_hosts[#]}"; do
echo "Generating key for $host"
openssl genrsa -out ${host}.key 2048
openssl req -new -days 365 -key ${host}.key -out ${host}.csr -subj "$dn_prefix/OU=$ou_client/CN=${host}"
openssl ca -batch -name SigningCA -config root-ca.cfg -out ${host}.crt -infiles ${host}.csr
cat ${host}.crt ${host}.key > ${host}.pem
done
echo ""
echo "##### STEP 6: Start up replicaset in non-auth mode"
mport=$mongodb_port
for host in "${mongodb_server_hosts[#]}"; do
echo "Starting server $host in non-auth mode"
mkdir -p ./db/${host}
mongod --replSet set509 --port $mport --dbpath ./db/$host \
--fork --logpath ./db/${host}.log
let "mport++"
done
sleep 3
# obtain the subject from the client key:
client_subject=`openssl x509 -in ${mongodb_client_hosts[0]}.pem -inform PEM -subject -nameopt RFC2253 | grep subject | awk '{sub("subject= ",""); print}'`
echo "##### STEP 7: setup replicaset & initial user role\n"
myhostname=`hostname`
cat > setup_auth.js <<EOF
rs.initiate();
mport=$mongodb_port;
mport++;
rs.add("$myhostname:" + mport);
mport++;
rs.add("$myhostname:" + mport);
sleep(5000);
db.getSiblingDB("\$external").runCommand(
{
createUser: "$client_subject",
roles: [
{ role: "readWrite", db: 'test' },
{ role: "userAdminAnyDatabase", db: "admin" },
{ role: "clusterAdmin", db:"admin"}
],
writeConcern: { w: "majority" , wtimeout: 5000 }
}
);
EOF
cat setup_auth.js
mongo localhost:$mongodb_port setup_auth.js
kill $(ps -ef | grep mongod | grep set509 | awk '{print $2}')
sleep 3
echo "##### STEP 8: Restart replicaset in x.509 mode\n"
mport=$mongodb_port
for host in "${mongodb_server_hosts[#]}"; do
echo "Starting server $host"
mongod --replSet set509 --port $mport --dbpath ./db/$host \
--sslMode requireSSL --clusterAuthMode x509 --sslCAFile root-ca.pem \
--sslAllowInvalidHostnames --fork --logpath ./db/${host}.log \
--sslPEMKeyFile ${host}.pem --sslClusterFile ${host}.pem
let "mport++"
done
# echo "##### STEP 9: Connecting to replicaset using certificate\n"
cat > do_login.js <<EOF
db.getSiblingDB("\$external").auth(
{
mechanism: "MONGODB-X509",
user: "$client_subject"
}
)
EOF
# mongo --ssl --sslPEMKeyFile client1.pem --sslCAFile root-ca.pem --sslAllowInvalidHostnames --shell do_login.js
After running the cluster, I am able to connect to it using the mongo shell with this command (all keys\certs were generated in ./ssl dir):
mongo --ssl --sslPEMKeyFile ssl/client1.pem --sslCAFile ssl/root-ca.pem --sslAllowInvalidHostnames
and authenticate as follows:
db.getSiblingDB("$external").auth(
{
mechanism: "MONGODB-X509",
user: "CN=client1,OU=MyClients,O=MongoDB China,L=Shenzhen,ST=GD,C=CN"
}
)
When I try to connect from node.js I keep failing. I am running the following code to connect to mongo using the native mongo driver:
'use strict';
const mongodb = require('mongodb');
const P = require('bluebird');
const fs = require('fs');
function connect_mongodb() {
let user = 'CN=client1,OU=MyClients,O=MongoDB China,L=Shenzhen,ST=GD,C=CN';
let uri = `mongodb://${encodeURIComponent(user)}#localhost:27017,localhost:27018,localhost:27019/test?replicaSet=set509&authMechanism=MONGODB-X509&ssl=true`;
var ca = [fs.readFileSync("./ssl/root-ca.pem")];
var cert = fs.readFileSync("./ssl/client1.pem");
var key = fs.readFileSync("./ssl/client1.pem");
let options = {
promiseLibrary: P,
server: {
ssl: true,
sslValidate: false,
checkServerIdentity: false,
sslCA: ca,
sslKey: key,
sslCert: cert,
},
replset: {
sslValidate: false,
checkServerIdentity: false,
ssl: true,
sslCA: ca,
sslKey: key,
sslCert: cert,
}
};
return mongodb.MongoClient.connect(uri, options);
}
connect_mongodb();
When running the script I get the following error:
Unhandled rejection MongoError: no valid seed servers in list
When checking the mongodb log I see these errors:
2017-01-17T22:48:54.191+0200 I NETWORK [initandlisten] connection accepted from 127.0.0.1:63881 #99 (5 connections now open)
2017-01-17T22:48:54.207+0200 E NETWORK [conn99] no SSL certificate provided by peer; connection rejected
2017-01-17T22:48:54.207+0200 I NETWORK [conn99] end connection 127.0.0.1:63881 (4 connections now open)
I was trying different options described here, but with no success.
Thanks for the help
Upgrading to mongodb node js driver 2.2.22 solved the problem
Related
I have a IXWebSocket c++ wss client connecting to a Node.js wss server(websocket npm package). Everything is fine as long as the client connect to "wss://localhost:8080". Soon as I use the ip address of the Node.js wss server, I have the error "OpenSSL failed - error:0A000086:SSL routines::certificate verify failed"
Certificate chain creation
I created my own private root ca. I used those commands to generate root ca key/certificate and server key/certificate:
$ openssl genpkey -aes256 -out root-ca/private/ca.private.key -algorithm RSA -pkeyopt rsa_keygen_bits:2048
$ openssl req -config root-ca/root-ca.conf -key root-ca\private\ca.private.key -x509 -days 7500 -sha256 -extensions v3_ca -out root-ca\certs\ca.crt
$ openssl genpkey -out server/private/server.private.key -algorithm RSA -pkeyopt rsa_keygen_bits:2048
$ openssl req -key server\private\server.private.key -new -sha256 -out server\csr\server.csr
$ openssl ca -config root-ca\root-ca.conf -extensions server_cert -days 365 -notext -in server\csr\server.csr -out server\certs\server.crt
The configuration has a subjectAltName for both root and server and it looks like this :
[ca]
#\\root\\ca\\root-ca\\root-ca.conf
#see man ca
default_ca = CA_default
[CA_default]
dir = C:\\ca\\root-ca
certs = $dir\\certs
crl_dir = $dir\\crl
new_certs_dir = $dir\\newcerts
database = $dir\\index
serial = $dir\\serial
RANDFILE = $dir\\private\\.rand
private_key = $dir\\private\\ca.private.key
certificate = $dir\\certs\\ca.crt
crlnumber = $dir\\crlnumber
crl = $dir\\crl\\ca.crl
crl_extensions = crl_ext
default_crl_days = 30
default_md = sha256
name_opt = ca_default
cert_opt = ca_default
default_days = 365
preserve = no
policy = policy_loose
[ policy_strict ]
countryName = supplied
stateOrProvinceName = supplied
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional
[ policy_loose ]
countryName = optional
stateOrProvinceName = optional
localityName = optional
organizationName = optional
organizationalUnitName = optional
commonName = supplied
emailAddress = optional
[ req ]
# Options for the req tool, man req.
default_bits = 2048
distinguished_name = req_distinguished_name
string_mask = utf8only
default_md = sha256
# Extension to add when the -x509 option is used.
x509_extensions = v3_ca
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
stateOrProvinceName = State or Province Name
localityName = Locality Name
0.organizationName = Organization Name
organizationalUnitName = Organizational Unit Name
commonName = Common Name
emailAddress = Email Address
countryName_default = CA
stateOrProvinceName_default = Qc
0.organizationName_default = Adacel
[ v3_ca ]
# Extensions to apply when createing root ca
# Extensions for a typical CA, man x509v3_config
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign
subjectAltName = #alt_names
[ v3_intermediate_ca ]
# Extensions to apply when creating intermediate or sub-ca
# Extensions for a typical intermediate CA, same man as above
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
#pathlen:0 ensures no more sub-ca can be created below an intermediate
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign
[ server_cert ]
# Extensions for server certificates
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = #alt_names
[ alt_names ]
DNS.1 = localhost
IP.1 = 192.168.230.138
IP.2 = 127.0.0.1
The certificate chain looks valid between my root ca and my server:
$ openssl verify -CAfile root-ca\certs\ca.crt server\certs\server.crt
server\certs\server.crt: OK
Both ca.crt and server.crt have a reference to the wss server IP address, I used the subjectAltName parameter to defined it. I thought that my root ca would need it (I am not even sure that make sense to have domain on the root ca), but it doesn't make any difference.
What is not working
My IXWebSocket c++ client :
ix::initNetSystem();
ix::WebSocket webSocket;
ix::SocketTLSOptions tlsOptions;
tlsOptions.caFile = "ca.crt";
webSocket.setTLSOptions(tlsOptions);
std::string url("wss://localhost:8080");
//std::string url("wss://192.168.230.138:8080"); //Cannot connect to the ip
webSocket.setUrl(url);
What is working
wss javascript client:
I also coded a javascript client (using same npm package as my server, not ) and this little client can connect using the ip address!!
const options = {
ca: fs.readFileSync("./ca.crt"),
};
var client = new WebSocketClient();
client.on("connectFailed", function (error) {
console.log("Connect Error: " + error.toString());
});
client.on("connect", function (connection) {
console.log("WebSocket Client Connected");
connection.on("error", function (error) {
console.log("Connection Error: " + error.toString());
});
connection.on("close", function () {
console.log("echo-protocol Connection Closed");
});
connection.on("message", function (message) {
if (message.type === "utf8") {
console.log("Received: '" + message.utf8Data + "'");
}
});
});
client.connect("wss://192.168.230.138:8080/", null, null, null, options);
My Node.js server :
const httpsSignalServer = https.createServer(
{
key: fs.readFileSync("./server.private.key"),
cert: fs.readFileSync("./server.crt"),
},
(req, res) => {
console.log(`signal server : we have received the request ${req}`);
}
);
const signalWebsocket = new WebSockerServer.server({
httpServer: httpsSignalServer,
});
signalWebsocket.on("request", (request) => onRequest(request));
httpsSignalServer.listen(8080, () =>
console.log("---> My signal server is listening <---")
);
signalWebsocket.on("request", (request) => onRequest(request));
httpsSignalServer.listen(8080, () =>
console.log("---> My signal server is listening <---")
);
Questions :
Any idea why my c++ client cannot connect using an ip address to the server, while the javascript client can? (using the same certificate chain)
If no, any idea how I could debug this?
Would it be possible that the problem is a high level SSL stuff, where you actually need a real hostname and can't use an IP?
I am currently looking into Wolkenkit by following the tutorial to create a chat application.
After finishing writing the code and I ran sudo yarn wolkenkit start. This gave me the following error message:
Waiting for https://localhost:3000/ to reply...
(node:11226) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
Error: Failed to get lowest processed position.
at EventSequencer.getLowestProcessedPosition (/wolkenkit/eventSequencer/EventSequencer.js:71:13)
at /wolkenkit/app.js:63:41
at process._tickCallback (internal/process/next_tick.js:68:7)
Application code caused runtime error.
✗ Failed to start the application.
A bit above the error the command warns about:
▻ Application certificate is self-signed.
I would appreciate any help on how to solve this and get the demo application to run on my local machine.
My development machine is running Debian GNU/Linux 10 with
Node 13.8.0
Yarn 1.21.1
Docker 18.09.1
Wolkenkit 3.1.2
Because of the warnings, I suspect this could be related to the X.509 certificate used for TLS. I created it using openssl like follows:
$ openssl req -new -sha256 -nodes -out localhost.csr -newkey rsa:2048 -keyout localhost.key -config <(
cat <<-EOF
[req]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C=US
ST=New York
L=Rochester
O=Somthing
OU=Something Else
emailAddress=test#example.com
CN = localhost
[ req_ext ]
subjectAltName = #alt_names
[ alt_names ]
DNS.1 = localhost
EOF
)
$ openssl x509 -req -days 365 -in localhost.csr -signkey localhost.key -sha256 -out localhost.crt
Then I moved the localhost.crt and localhost.key into the following structure:
server/keys/localhost
├── certificate.pem
└── privateKey.pem
And set up a package.json like this:
{
"name": "chat",
"version": "0.0.0",
"wolkenkit": {
"application": "chat",
"runtime": {
"version": "3.1.0"
},
"environments": {
"default": {
"api": {
"address": {
"host": "localhost",
"port": 3000
},
"certificate": "/server/keys/localhost",
"allowAccessFrom": "*"
},
"fileStorage": {
"allowAccessFrom": "*"
},
"node": {
"environment": "development"
}
}
}
},
"dependencies": {
"wolkenkit": "^3.1.2"
}
}
Seems like this could be the same problem described here in this Github issue.
The problem is that due to a change in the start command, we now
assume that there must be a read model (which has not yet been
defined, if you follow the guide).
If you simply ignore this error, and follow on, the next thing is to
define the read model. Once you have done that, you can successfully
run wolkenkit start.
I am trying to setup a server and some clients using TLS in node. I am using self-signed certificates on the clients and the server. The server runs ok, but when I try to connect a client I end up with the following error on the client side:
Error: DEPTH_ZERO_SELF_SIGNED_CERT
Cert Generation
CA cert:
# Create key to sign our own certs. Act like as our own a CA.
echo "SecuresPassphrase" > ./tls/passphrase.txt
openssl ecparam -name secp521r1 -genkey -noout -out ./tls/certs/ca/ca.plain.key # Generate private key
openssl pkcs8 -topk8 -passout file:./tls/passphrase.txt -in ./tls/certs/ca/ca.plain.key -out ./tls/certs/ca/ca.key # Generate encrypted private key
rm ./tls/certs/ca/ca.plain.key; # Remove unencrypted private key
# Generate CA cert
openssl req -config ./openssl/oid_file -passin file:./tls/passphrase.txt -new -x509 -days 365 -key ./tls/certs/ca/ca.key -out ./tls/certs/ca/ca.crt
Server Cert:
openssl ecparam -name secp521r1 -genkey -noout -out ./tls/certs/servers/server.key # Generate server private key
openssl req -config ./openssl/oid_file -new -key ./tls/certs/servers/server.key -out ./tls/certs/servers/server.csr # Generate signing request
openssl x509 -passin file:./tls/passphrase.txt -req -days 365 -in ./tls/certs/servers/server.csr -CA ./tls/certs/ca/ca.crt -CAkey ./tls/certs/ca/ca.key -CAcreateserial -out ./tls/server.crt
mv ./tls/server.crt ./tls/certs/servers/
Client cert:
Client's certs are created inside a bash loop, the variable ${name} contains the name of the client and changes each iteration.
openssl ecparam -name secp521r1 -genkey -noout -out ./tls/certs/clients/${name}/client.key
openssl req -config ./openssl/oid_file -new -key ./tls/certs/clients/${name}/client.key -out ./tls/certs/clients/${name}/client.csr
openssl x509 -passin file:./tls/passphrase.txt -req -days 365 -in ./tls/certs/clients/${name}/client.csr -CA ./tls/certs/ca/ca.crt -CAkey ./tls/certs/ca/ca.key -CAcreateserial -out ./tls/client.crt
mv ./tls/client.crt ./tls/certs/clients/${name}/
I am also trying to use Perfect Forward Secrecy by using ephemereal Diffie-Hellman interchange. The parameters, for clients and server, are created as openssl dhparam -outform PEM -out ./tls/params/servers/server/dhparams.pem 2048
Server's code:
return new Promise(resolve => {
// Define parameters of TLS socket
const options = {
rejectUnauthorized: false,
requestCert: true,
// Secure Context Parameters
ca: [fs.readFileSync("tls/certs/ca/ca.crt")],
cert: fs.readFileSync("tls/certs/servers/server.crt"),
key: fs.readFileSync("tls/certs/servers/server.key"),
ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:!RC4:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!SRP:!CAMELLIA",
dhparam: fs.readFileSync("tls/params/servers/server/dhparams.pem"),
ecdhCurve: tls.DEFAULT_ECDH_CURVE,
minVersion: "TLSv1.2"
};
// Iniciar servidor TLS
this.SERVIDOR = tls.createServer(options, function (socket) {
console.log("Server created.");
});
this.SERVIDOR.listen(this.puerto, this.direccion, function () {
console.log("Listening");
});
this.SERVIDOR.on("secureConnection", (socket) => this.handleRequest(socket));
this.SERVIDOR.on("tlsClientError", (error) => console.log("Error client TLs. %s", error));
});
Client's code:
return new Promise(resolve => {
// Define parameters of TLS socket
const options = {
host: this.NODE,
port: this.NODE_PORT,
rejectUnauthorized: false,
secureContext: tls.createSecureContext({
ca: [fs.readFileSync("tls/certs/ca/ca.crt")],
cert: fs.readFileSync("tls/certs/clients/" + this.NAME + "/client.crt"),
key: fs.readFileSync("tls/certs/clients/" + this.NAME + "/client.key"),
ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:!RC4:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!SRP:!CAMELLIA",
dhparam: fs.readFileSync("tls/params/clients/" + this.NAME + "/dhparams.pem"),
ecdhCurve: tls.DEFAULT_ECDH_CURVE,
minVersion: "TLSv1.2"
})
};
// Initialize TLS socket
this.WEB_SOCKET = tls.connect(options, function () {
// Check if TLS auth worked
if (this.authorized) {
console.log("Connection authorized by a Cert. Authority ");
} else {
console.log("Authorization not granted. Error: " + this.authorizationError);
}
});
What I have tried:
Set rejectUnauthorized to false.
Tried to run the code with NODE_TLS_REJECT_UNAUTHORIZED = "0";. It didn't work and I think is not a good option for production.
Checked similar questions on SO, this one it looks pretty similar to my problem. But the most upvoted answer is from 2014 and I couldn't find too much information about Distinguished name on the docs.
I checked the certs using openssl x509 -in *.cert -text and they looked good.
¿Am I wrongly generating the certs? Any help is appreciated.
Thanks!
EDIT. 16/10/2019
There was a problem in the code, I didn't use resolve(); when the sockets were up. The same problem remains... BUT despite of getting an authorization error on the client, the server fires up the "secureConnection" event and messages are interchanged. ¿Does this makes sense? *Yes, it makes sense since the server accepts unauthorized connections. If I set the server to reject not certified connections the clients hung up *
The problem was I was using the same configuration file (./openssl/oid_file) for all the certificates. Using different configuration files and different Alternative names solved this issue.
I ended with an "UNABLE_TO_VERIFY_LEAF_SIGNATURE" error. The certificates were properly generated but it didn't work. I couldn't find a working example of self-signed certificates in nodejs. Most of them simply deprecated the use of certificates by disabling SSL or accepting unathorized transactions, which is the opposite of what TLS is supposed to do.
Finally, I used this tool to generate the certificates: https://github.com/FiloSottile/mkcert . The best and simplest way to generate self-signed certificates in a testing environment. You only need to set the node variable NODE_EXTRA_CA_CERTS to point the root certificate:
process.env.NODE_EXTRA_CA_CERTS = [os.homedir() + "/.local/share/mkcert/rootCA.pem"];
-
The question:
Can someone help me figure out why I can't get filebeats to talk to logstash over TLS/SSL?
The Error:
I can get the filebeat and logstash to talk to eachover with TLS/SSL disabled, but when i enable it and use the settings/config below, I get the following error (observed in logstash.log):
{:timestamp=>"2016-10-28T17:21:44.445000+0100", :message=>"Pipeline aborted due to error",
:exception=>java.lang.NullPointerException, :backtrace=>["org.logstash.netty.PrivateKeyCo
nverter.generatePkcs8(org/logstash/netty/PrivateKeyConverter.java:43)", "org.logstash.nett
y.PrivateKeyConverter.convert(org/logstash/netty/PrivateKeyConverter.java:39)", "java.lang
.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "RUBY.create_server(/usr/share
/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-3.1.0.beta4-java/lib/logstash/
inputs/beats.rb:139)", "RUBY.register(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/log
stash-input-beats-3.1.0.beta4-java/lib/logstash/inputs/beats.rb:132)", "RUBY.start_inputs(
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:311)", "org.jruby.RubyArray.eac
h(org/jruby/RubyArray.java:1613)", "RUBY.start_inputs(/usr/share/logstash/logstash-core/li
b/logstash/pipeline.rb:310)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/lo
gstash/pipeline.rb:187)", "RUBY.run(/usr/share/logstash/logstash-core/lib/logstash/pipelin
e.rb:145)", "RUBY.start_pipeline(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:2
40)", "java.lang.Thread.run(java/lang/Thread.java:745)"], :level=>:error}
{:timestamp=>"2016-10-28T17:21:47.452000+0100", :message=>"stopping pipeline", :id=>"main"
, :level=>:warn}
{:timestamp=>"2016-10-28T17:21:47.456000+0100", :message=>"An unexpected error occurred!",
:error=>#<NoMethodError: undefined method `stop' for nil:NilClass>, :backtrace=>["/us
r/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-3.1.0.beta4-java/lib/lo
gstash/inputs/beats.rb:173:in `stop'", "/usr/share/logstash/logstash-core/lib/logstash/inp
uts/base.rb:88:in `do_stop'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logst
ash/logstash-core/lib/logstash/pipeline.rb:366:in `shutdown'", "/usr/share/logstash/logsta
sh-core/lib/logstash/agent.rb:252:in `stop_pipeline'", "/usr/share/logstash/logstash-core/
lib/logstash/agent.rb:261:in `shutdown_pipelines'", "org/jruby/RubyHash.java:1342:in `each
'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:261:in `shutdown_pipelines'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:123:in `shutdown'", "/usr/share/
logstash/logstash-core/lib/logstash/runner.rb:237:in `execute'", "/usr/share/logstash/vend
or/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logsta
sh/logstash-core/lib/logstash/runner.rb:157:in `run'", "/usr/share/logstash/vendor/bundle/
jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bo
otstrap/environment.rb:66:in `(root)'"], :level=>:fatal}
The Setup:
Servers
2 servers.
$> uname -a
Linux elkserver 3.10.0-327.36.2.el7.x86_64 #1 SMP Mon Oct 10 23:08:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$> cat /etc/*-release
CentOS Linux release 7.2.1511 (Core)
SELinux is Permissive (soz).
Firewalls are of. (mazza soz).
One server runs elasticsearch and logstash; one runs filebeat.
Elasticsearch
$> /usr/share/elasticsearch/bin/elasticsearch -version
Version: 2.4.1, Build: c67dc32/2016-09-27T18:57:55Z, JVM: 1.8.0_111
Logstash
$> /usr/share/logstash/bin/logstash -V
logstash 5.0.0-alpha5
Filbeat
$> /usr/share/filebeat/bin/filebeat -version
filebeat version 5.0.0 (amd64), libbeat 5.0.0
Config:
Logstash
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/filebeat-forwarder.crt"
ssl_key => "/etc/pki/tls/private/filebeat-forwarder.key"
}
}
output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
}
Filebeat.yml
output:
logstash:
enabled: true
hosts:
- "<my ip address>:5044"
timeout: 15
tls:
certificate_authorities:
- /etc/pki/tls/certs/filebeat-forwarder.crt
filebeat:
prospectors:
-
paths:
- /var/log/syslog
- /var/log/auth.log
document_type: syslog
-
paths:
- /var/log/nginx/access.log
document_type: nginx-access
File: openssl_extras.cnf:
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = TG
ST = Togo
L = Lome
O = Private company
CN = *
[v3_req]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints = CA:TRUE
subjectAltName = #alt_names
[alt_names]
DNS.1 = *
DNS.2 = *.*
DNS.3 = *.*.*
DNS.4 = *.*.*.*
DNS.5 = *.*.*.*.*
DNS.6 = *.*.*.*.*.*
DNS.7 = *.*.*.*.*.*.*
IP.1 = <my ip address>
The command used to create the cert:
$> openssl req -subj '/CN=elkserver.system.local/' -config /etc/pki/tls/openssl_extras.cnf \
-x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout /etc/pki/tls/private/filebeat-forwarder.key \
-out /etc/pki/tls/certs/filebeat-forwarder.crt
In Filebeat 5.0 the tls configuration setting was changed to ssl to be consistent with the configuration setting used in Logstash and Elasticsearch. Try updating your Filebeat configuration.
References:
Securing Communication With Logstash by Using SSL
Breaking Changes in 5.0
I have started writing a wrapper for an API which requires all requests to be over HTTPS. Instead of making requests to the actual API while I am developing and testing it I would like to run my own server locally which mocks the responses.
I am confused about how to generate the certificates I need to create a HTTPS server and send requests to it.
My server looks something like this:
var options = {
key: fs.readFileSync('./key.pem'),
cert: fs.readFileSync('./cert.pem')
};
https.createServer(options, function(req, res) {
res.writeHead(200);
res.end('OK\n');
}).listen(8000);
The pem files were generated with:
openssl genrsa 1024 > key.pem
openssl req -x509 -new -key key.pem > cert.pem
And a request looks something like this:
var options = {
host: 'localhost',
port: 8000,
path: '/api/v1/test'
};
https.request(options, function(res) {
res.pipe(process.stdout);
}).end();
With this setup I get Error: DEPTH_ZERO_SELF_SIGNED_CERT, so I think I need to add a ca option for the request.
So my question is how should I generate the following:
The server key?
The server cert?
The ca for the request?
I have read a few things about generating self signed certificates with openssl, but can't seem to wrap my head around it and figure out which keys and certificates to use where in my node code.
Update
The API provides a CA certificate to use instead of the defaults. The following code works using their certificate and this is what I want to reproduce locally.
var ca = fs.readFileSync('./certificate.pem');
var options = {
host: 'example.com',
path: '/api/v1/test',
ca: ca
};
options.agent = new https.Agent(options);
https.request(options, function(res) {
res.pipe(process.stdout);
}).end();
Update (Nov 2018): Do you need self-signed certs?
Or would real certificates get the job done better? Have you considered any of these?
Let's Encrypt via Greenlock.js
Let's Encrypt via https://greenlock.domains
Localhost relay service such as https://telebit.cloud
(Note: Let's Encrypt can also issue certificates to private networks)
ScreenCast
https://coolaj86.com/articles/how-to-create-a-csr-for-https-tls-ssl-rsa-pems/
Full, Working example
creates certificates
runs node.js server
no warnings or errors in node.js client
no warnings or errors in cURL
https://github.com/coolaj86/nodejs-self-signed-certificate-example
Using localhost.greenlock.domains as an example (it points to 127.0.0.1):
server.js
'use strict';
var https = require('https')
, port = process.argv[2] || 8043
, fs = require('fs')
, path = require('path')
, server
, options
;
require('ssl-root-cas')
.inject()
.addFile(path.join(__dirname, 'server', 'my-private-root-ca.cert.pem'))
;
options = {
// this is ONLY the PRIVATE KEY
key: fs.readFileSync(path.join(__dirname, 'server', 'privkey.pem'))
// You DO NOT specify `ca`, that's only for peer authentication
//, ca: [ fs.readFileSync(path.join(__dirname, 'server', 'my-private-root-ca.cert.pem'))]
// This should contain both cert.pem AND chain.pem (in that order)
, cert: fs.readFileSync(path.join(__dirname, 'server', 'fullchain.pem'))
};
function app(req, res) {
res.setHeader('Content-Type', 'text/plain');
res.end('Hello, encrypted world!');
}
server = https.createServer(options, app).listen(port, function () {
port = server.address().port;
console.log('Listening on https://127.0.0.1:' + port);
console.log('Listening on https://' + server.address().address + ':' + port);
console.log('Listening on https://localhost.greenlock.domains:' + port);
});
client.js
'use strict';
var https = require('https')
, fs = require('fs')
, path = require('path')
, ca = fs.readFileSync(path.join(__dirname, 'client', 'my-private-root-ca.cert.pem'))
, port = process.argv[2] || 8043
, hostname = process.argv[3] || 'localhost.greenlock.domains'
;
var options = {
host: hostname
, port: port
, path: '/'
, ca: ca
};
options.agent = new https.Agent(options);
https.request(options, function(res) {
res.pipe(process.stdout);
}).end();
And the script that makes the certificate files:
make-certs.sh
#!/bin/bash
FQDN=$1
# make directories to work from
mkdir -p server/ client/ all/
# Create your very own Root Certificate Authority
openssl genrsa \
-out all/my-private-root-ca.privkey.pem \
2048
# Self-sign your Root Certificate Authority
# Since this is private, the details can be as bogus as you like
openssl req \
-x509 \
-new \
-nodes \
-key all/my-private-root-ca.privkey.pem \
-days 1024 \
-out all/my-private-root-ca.cert.pem \
-subj "/C=US/ST=Utah/L=Provo/O=ACME Signing Authority Inc/CN=example.com"
# Create a Device Certificate for each domain,
# such as example.com, *.example.com, awesome.example.com
# NOTE: You MUST match CN to the domain name or ip address you want to use
openssl genrsa \
-out all/privkey.pem \
2048
# Create a request from your Device, which your Root CA will sign
openssl req -new \
-key all/privkey.pem \
-out all/csr.pem \
-subj "/C=US/ST=Utah/L=Provo/O=ACME Tech Inc/CN=${FQDN}"
# Sign the request from Device with your Root CA
openssl x509 \
-req -in all/csr.pem \
-CA all/my-private-root-ca.cert.pem \
-CAkey all/my-private-root-ca.privkey.pem \
-CAcreateserial \
-out all/cert.pem \
-days 500
# Put things in their proper place
rsync -a all/{privkey,cert}.pem server/
cat all/cert.pem > server/fullchain.pem # we have no intermediates in this case
rsync -a all/my-private-root-ca.cert.pem server/
rsync -a all/my-private-root-ca.cert.pem client/
# create DER format crt for iOS Mobile Safari, etc
openssl x509 -outform der -in all/my-private-root-ca.cert.pem -out client/my-private-root-ca.crt
For example:
bash make-certs.sh 'localhost.greenlock.domains'
Hopefully this puts the nail in the coffin on this one.
And some more explanation: https://github.com/coolaj86/node-ssl-root-cas/wiki/Painless-Self-Signed-Certificates-in-node.js
Install private cert on iOS Mobile Safari
You need to create a copy of the root ca certificate a DER format with a .crt extension:
# create DER format crt for iOS Mobile Safari, etc
openssl x509 -outform der -in all/my-private-root-ca.cert.pem -out client/my-private-root-ca.crt
Then you can simply serve that file with your webserver. When you click the link you should be asked if you want to install the certificate.
For an example of how this works you can try installing MIT's Certificate Authority: https://ca.mit.edu/mitca.crt
Related Examples
https://github.com/coolaj86/nodejs-ssl-example
https://github.com/coolaj86/nodejs-ssl-trusted-peer-example
https://github.com/coolaj86/node-ssl-root-cas
https://github.com/coolaj86/nodejs-https-sni-vhost-example
(Multiple vhosts with SSL on the same server)
https://telebit.cloud
(get REAL SSL certs you can use TODAY for testing on localhost)
Try adding this to your request options
var options = {
host: 'localhost',
port: 8000,
path: '/api/v1/test',
// These next three lines
rejectUnauthorized: false,
requestCert: true,
agent: false
};
Your key generation looks okay. You shouldn't need a ca because you aren't rejecting unsigned requests.
Add .toString() to the end of your readFileSync methods so that you are actually passing a string, not a file object.
This procedure allows you to create both a certificate authority & a certificate :
grab this ca.cnf file to use as a configuration shortcut :
wget https://raw.githubusercontent.com/anders94/https-authorized-clients/master/keys/ca.cnf
create a new certificate authority using this configuration :
openssl req -new -x509 -days 9999 -config ca.cnf -keyout ca-key.pem -out ca-cert.pem
now that we have our certificate authority in ca-key.pem and ca-cert.pem, let's generate a private key for the server :
openssl genrsa -out key.pem 4096
grab this server.cnf file to use as a configuration shortcut :
wget https://raw.githubusercontent.com/anders94/https-authorized-clients/master/keys/server.cnf
generate the certificate signing request using this configuration :
openssl req -new -config server.cnf -key key.pem -out csr.pem
sign the request :
openssl x509 -req -extfile server.cnf -days 999 -passin "pass:password" -in csr.pem -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial -out cert.pem
I found this procedure here, along with more information on how to use these certificates.
Try adding
agent: false,
rejectUnauthorized: false
When you have the self-signed cert[s], you tell Node.js to use it with the Environment variable: NODE_EXTRA_CA_CERTS
Copy [cat] all the generated *.cert.pem files to a single file.
I put it the directory with all the keys & certs:
> (cd $keys; cat *.cert.pem > node_extra_ca_certs)
Tell node where to find them:
> export NODE_EXTRA_CA_CERTS=$keys/node_extra_ca_certs
Now, when you run node, it will accept your private certs as valid.