I have a mongodb database hosted on an Atlas MongoDB Cloud cluster. I'm currently accessing the database in my node.js application using mongoose:
mongoose.connect("mongodb://user:pw#cluster0-shard-00-00-***.mongodb.net:***,cluster0-shard-00-01-***.mongodb.net:***,cluster0-shard-00-02-***.mongodb.net:***/admin?ssl=true&replicaSet=Cluster0-shard-0&authSource=admin");
Because Atlas MongoDB Cloud have a whitelist, and Heroku doesn't provide the possibility to obtain a fixed IP address for my application, I'm using Fixie add-on. Basically Fixie acts as a proxy for outbound traffic.
This way, I can request resources via HTTP, which are tunneled through a fixed IP address provided by Fixie. But what I do need is to connect to the Atlas Cloud cluster using Fixie's proxy, in order to retrieve and modify data from the database.
Can it be done using mongoose?
The mongoose.connect function accepts an option parameter, but I couldn't find any option regarding the establishment of a connection through a proxy.
Just got a reply from Fixie's team:
Fixie is an http/https proxy, so it won't work for lower-level TCP connections like you'd need for your mongodb connection
When I asked about the possibility of using SOCKS for this case, they replied:
It looks like mongoose does not natively support socks proxies, and it does not accept a custom socket (which is how Node database drivers for MySQL and Postgres support it).
So apparently, in my case, there is no way to establish a connection to MongoDB Atlas cluster using mongoose through the proxy solution offered by Fixie (Heroku Add-on).
A lot has changed since this question was originally asked in 2017. Since then, Mongoose has added support for proxy options, and Fixie released Fixie Socks, a SOCKS5 proxy designed for proxying database connections.
To connect to a MongoDB server (including MondoDB Atlas) through Fixie Socks, you can do the following:
const mongoose = require('mongoose');
const fixieData = process.env.FIXIE_SOCKS_HOST.split(new RegExp('[/(:\\/#/]+'));
mongoose.connect(process.env.DB_CONNECTION,
{
proxyUsername: fixieData[0],
proxyPassword: fixieData[1],
proxyHost: fixieData[2],
proxyPort: fixieData[3]
},
(error) => {
if(error){
console.log(error);
} else {
console.log('Connected to database');
}
}
);
Related
To antecipate the question: do I need to get SSL support on Heroku in order to establish a connection between Heroku and Atlas MongoDB Cloud using SSL? (TSL/SSL connection is a requirement to access Atlas MongoDB Cloud service).
I am trying to connect my Heroku App, written in node.js, to a cluster hosted at Atlas MongoDB Cloud.
My current database is hosted at mLab (as a Heroku Add-on), and the MongoDB URI used to access the cluster through mongoose is (using xxx to omit confidential info):
MONGODB_URI="mongodb://xxx:xxx#xxx-a0.mlab.com:23266,xxx-a1.mlab.com:xxx/xxx?replicaSet=rs-xxx"
Now that I've migrated my data from mLab to Atlas MongoDB Cloud, I am currently accessing the cluster using the URI:
MONGODB_URI="mongodb://xxx:xxx#cluster0-shard-xxx.mongodb.net:xxx,cluster0-shard-xxx.mongodb.net:xxx,cluster0-shard-xxx.mongodb.net:xxx/xxx?replicaSet=xxx&ssl=true&authSource=admin"
When running my Heroku App locally in my machine I can access the database with no problem. I'm also able to connect to the cluster using mongo shell.
However, when running the App in Heroku, the connection cannot be established. In the Browser JS console, I get the 503 service unavailable message. In heroku, I get the error:
no primary found in replica set
I am aware that Atlas MongoDB Cloud requires SSL connection, differently from mLab. In my local machine, I suppose a self signed certificate is being used to connect successfully to the cluster.
My question is: do I need to get SSL support in Heroku in order to be able to access establish the secure connection between Heroku and MongoDB Atlas? Or the SSL suport in Heroku is only required to client/Heroku secure connection?
What I think might fix your problem
Disclaimer: I have used neither Heroku nor MongoDB Atlas but I am looking into them.
According to a Github issue I found, you will get that error message if you haven't whitelisted the server IP addresses in MongoDB Atlas.
Reading the MongoDB Atlas docs, the only way I see to do this in combination with Heroku dynos is to add 0.0.0.0/0 (i.e. all addresses) to your MongoDB Atlas whitelist.
Give that a try and please report back whether you can instantiate a connection.
On SSL
Trying to reply to the SSL question, I do not think that you need to enable it on Heroku based on what I read, although I am not totally sure.
If the MongoDB server performed certificate validation, the Node.js code for connecting to it would have to look like the following (taken from the Node.js driver documentation):
var MongoClient = require('mongodb').MongoClient,
f = require('util').format,
fs = require('fs');
// Read the certificates
var ca = [fs.readFileSync(__dirname + "/ssl/ca.pem")];
var cert = fs.readFileSync(__dirname + "/ssl/client.pem");
var key = fs.readFileSync(__dirname + "/ssl/client.pem");
// Connect validating the returned certificates from the server
MongoClient.connect("mongodb://localhost:27017/test?ssl=true", {
server: {
sslValidate:true
, sslCA:ca
, sslKey:key
, sslCert:cert
, sslPass:'10gen'
}
}, function(err, db) {
db.close();
});
If the MongoDB server does not check for any SSL certificates, you can simply use code like the following (also taken from the Node.js driver documentation):
var MongoClient = require('mongodb').MongoClient;
MongoClient.connect("mongodb://localhost:27017/test?ssl=true", function(err, db) {
db.close();
});
Given that the Atlas documentation contains the following example code for connecting to it from Node.js, I think that you do not have to enable SSL on Heroku:
var MongoClient = require('mongodb').MongoClient;
var uri = "mongodb://kay:myRealPassword#mycluster0-shard-00-00-wpeiv.mongodb.net:27017,mycluster0-shard-00-01-wpeiv.mongodb.net:27017,mycluster0-shard-00-02-wpeiv.mongodb.net:27017/admin?ssl=true&replicaSet=Mycluster0-shard-0&authSource=admin";
MongoClient.connect(uri, function(err, db) {
db.close();
});
You can find all IP ranges for Heroku with this command:
HEROKU_REGION=eu; sudo apt -qqy install curl jq 2>/dev/null 1>/dev/null; heroku regions --json 2>/dev/null | jq ".[] | select(.name==\"$HEROKU_REGION\") | .provider.region" | (REGION=$(cat); curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | jq ".prefixes[] | select(.region==$REGION) | .ip_prefix")
Also had to add 0.0.0.0/0 to the Mongo IP whitelist AND redeploy my app on Heroku for it to finally work (before changing IP, a CORS error was thrown).
very simple solution! just add to the white list IP in mongo atlas the adress "0.0.0.0/0"
it will open the mongo atlas to all the world..... so it os not for production but it helps for small tests
I solved this by installing an addon(i used Fixie Socks) for Static IP addresses for database requests and other TCP connections.
More options here: https://elements.heroku.com/addons#network
Since allowing access from anywhere is not secure and IP ranges could change, I ended up installing add-on QuotaGuard Static IP's (it provides 2 IP addresses for IP whitelist) so SOCKS5 Proxy can be used with QGTunnel.
QGTunnel should be downloaded and included in the codebase
curl https://s3.amazonaws.com/quotaguard/qgtunnel-latest.tar.gz | tar xz
Procfile should be updated
web: bin/qgtunnel npm start
Let’s say you want to access a replicated MongoDB cluster using QGTunnel with 3 replicas located on the hosts: rs01.mongodb.net:52115, rs02.mongodb.net:52115, and rs1.mongodb.net:52115. For this configuration, you will need to create 3 separate tunnels for each host on port 52115 in transparent mode. Once this is done, QGTunnel will alter the DNS resolution process to resolve these hostnames to the appropriate loopback address and auto-discovery for your replicated cluster should work as intended.
To antecipate the question: do I need to get SSL support on Heroku in order to establish a connection between Heroku and Atlas MongoDB Cloud using SSL? (TSL/SSL connection is a requirement to access Atlas MongoDB Cloud service).
I am trying to connect my Heroku App, written in node.js, to a cluster hosted at Atlas MongoDB Cloud.
My current database is hosted at mLab (as a Heroku Add-on), and the MongoDB URI used to access the cluster through mongoose is (using xxx to omit confidential info):
MONGODB_URI="mongodb://xxx:xxx#xxx-a0.mlab.com:23266,xxx-a1.mlab.com:xxx/xxx?replicaSet=rs-xxx"
Now that I've migrated my data from mLab to Atlas MongoDB Cloud, I am currently accessing the cluster using the URI:
MONGODB_URI="mongodb://xxx:xxx#cluster0-shard-xxx.mongodb.net:xxx,cluster0-shard-xxx.mongodb.net:xxx,cluster0-shard-xxx.mongodb.net:xxx/xxx?replicaSet=xxx&ssl=true&authSource=admin"
When running my Heroku App locally in my machine I can access the database with no problem. I'm also able to connect to the cluster using mongo shell.
However, when running the App in Heroku, the connection cannot be established. In the Browser JS console, I get the 503 service unavailable message. In heroku, I get the error:
no primary found in replica set
I am aware that Atlas MongoDB Cloud requires SSL connection, differently from mLab. In my local machine, I suppose a self signed certificate is being used to connect successfully to the cluster.
My question is: do I need to get SSL support in Heroku in order to be able to access establish the secure connection between Heroku and MongoDB Atlas? Or the SSL suport in Heroku is only required to client/Heroku secure connection?
What I think might fix your problem
Disclaimer: I have used neither Heroku nor MongoDB Atlas but I am looking into them.
According to a Github issue I found, you will get that error message if you haven't whitelisted the server IP addresses in MongoDB Atlas.
Reading the MongoDB Atlas docs, the only way I see to do this in combination with Heroku dynos is to add 0.0.0.0/0 (i.e. all addresses) to your MongoDB Atlas whitelist.
Give that a try and please report back whether you can instantiate a connection.
On SSL
Trying to reply to the SSL question, I do not think that you need to enable it on Heroku based on what I read, although I am not totally sure.
If the MongoDB server performed certificate validation, the Node.js code for connecting to it would have to look like the following (taken from the Node.js driver documentation):
var MongoClient = require('mongodb').MongoClient,
f = require('util').format,
fs = require('fs');
// Read the certificates
var ca = [fs.readFileSync(__dirname + "/ssl/ca.pem")];
var cert = fs.readFileSync(__dirname + "/ssl/client.pem");
var key = fs.readFileSync(__dirname + "/ssl/client.pem");
// Connect validating the returned certificates from the server
MongoClient.connect("mongodb://localhost:27017/test?ssl=true", {
server: {
sslValidate:true
, sslCA:ca
, sslKey:key
, sslCert:cert
, sslPass:'10gen'
}
}, function(err, db) {
db.close();
});
If the MongoDB server does not check for any SSL certificates, you can simply use code like the following (also taken from the Node.js driver documentation):
var MongoClient = require('mongodb').MongoClient;
MongoClient.connect("mongodb://localhost:27017/test?ssl=true", function(err, db) {
db.close();
});
Given that the Atlas documentation contains the following example code for connecting to it from Node.js, I think that you do not have to enable SSL on Heroku:
var MongoClient = require('mongodb').MongoClient;
var uri = "mongodb://kay:myRealPassword#mycluster0-shard-00-00-wpeiv.mongodb.net:27017,mycluster0-shard-00-01-wpeiv.mongodb.net:27017,mycluster0-shard-00-02-wpeiv.mongodb.net:27017/admin?ssl=true&replicaSet=Mycluster0-shard-0&authSource=admin";
MongoClient.connect(uri, function(err, db) {
db.close();
});
You can find all IP ranges for Heroku with this command:
HEROKU_REGION=eu; sudo apt -qqy install curl jq 2>/dev/null 1>/dev/null; heroku regions --json 2>/dev/null | jq ".[] | select(.name==\"$HEROKU_REGION\") | .provider.region" | (REGION=$(cat); curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | jq ".prefixes[] | select(.region==$REGION) | .ip_prefix")
Also had to add 0.0.0.0/0 to the Mongo IP whitelist AND redeploy my app on Heroku for it to finally work (before changing IP, a CORS error was thrown).
very simple solution! just add to the white list IP in mongo atlas the adress "0.0.0.0/0"
it will open the mongo atlas to all the world..... so it os not for production but it helps for small tests
I solved this by installing an addon(i used Fixie Socks) for Static IP addresses for database requests and other TCP connections.
More options here: https://elements.heroku.com/addons#network
Since allowing access from anywhere is not secure and IP ranges could change, I ended up installing add-on QuotaGuard Static IP's (it provides 2 IP addresses for IP whitelist) so SOCKS5 Proxy can be used with QGTunnel.
QGTunnel should be downloaded and included in the codebase
curl https://s3.amazonaws.com/quotaguard/qgtunnel-latest.tar.gz | tar xz
Procfile should be updated
web: bin/qgtunnel npm start
Let’s say you want to access a replicated MongoDB cluster using QGTunnel with 3 replicas located on the hosts: rs01.mongodb.net:52115, rs02.mongodb.net:52115, and rs1.mongodb.net:52115. For this configuration, you will need to create 3 separate tunnels for each host on port 52115 in transparent mode. Once this is done, QGTunnel will alter the DNS resolution process to resolve these hostnames to the appropriate loopback address and auto-discovery for your replicated cluster should work as intended.
I have mongodb in AWS EC2 instance.
I am trying to connec to mongodb instance from nodejs server using mongoClient. My node application is deployed on PCF platform.
When i try to connect to mongodb
it throws an error "502 Bad Gateway:
Registered endpoint failed to handle the request". My node application is working fine, one the connection with mongodb is causing the problem.
conn_str += req.params.dbname;
mongoClient.connect(conn_str, function(err, db) {
if(err) {
res.end(err);
} else {
res.end(db);
}
});
Help much appreciated.
Ok. I had faced a similar issue while I was trying to connect to my mongo cluster which was deployed on Google Compute Engine. The below checklist helped me resolve the issue
Check if your firewall and make sure that the port 27017 is open for external requests
Check if your MongoDB is up and running (just do a curl localhost:27017 from terminal) if it's running you should get the protocol error
Check if your MongoDB is configured to accept requests from applications that is outside it's network (if you're running on different VPC) -- By default, MongoDB doesn't allow external requests
To antecipate the question: do I need to get SSL support on Heroku in order to establish a connection between Heroku and Atlas MongoDB Cloud using SSL? (TSL/SSL connection is a requirement to access Atlas MongoDB Cloud service).
I am trying to connect my Heroku App, written in node.js, to a cluster hosted at Atlas MongoDB Cloud.
My current database is hosted at mLab (as a Heroku Add-on), and the MongoDB URI used to access the cluster through mongoose is (using xxx to omit confidential info):
MONGODB_URI="mongodb://xxx:xxx#xxx-a0.mlab.com:23266,xxx-a1.mlab.com:xxx/xxx?replicaSet=rs-xxx"
Now that I've migrated my data from mLab to Atlas MongoDB Cloud, I am currently accessing the cluster using the URI:
MONGODB_URI="mongodb://xxx:xxx#cluster0-shard-xxx.mongodb.net:xxx,cluster0-shard-xxx.mongodb.net:xxx,cluster0-shard-xxx.mongodb.net:xxx/xxx?replicaSet=xxx&ssl=true&authSource=admin"
When running my Heroku App locally in my machine I can access the database with no problem. I'm also able to connect to the cluster using mongo shell.
However, when running the App in Heroku, the connection cannot be established. In the Browser JS console, I get the 503 service unavailable message. In heroku, I get the error:
no primary found in replica set
I am aware that Atlas MongoDB Cloud requires SSL connection, differently from mLab. In my local machine, I suppose a self signed certificate is being used to connect successfully to the cluster.
My question is: do I need to get SSL support in Heroku in order to be able to access establish the secure connection between Heroku and MongoDB Atlas? Or the SSL suport in Heroku is only required to client/Heroku secure connection?
What I think might fix your problem
Disclaimer: I have used neither Heroku nor MongoDB Atlas but I am looking into them.
According to a Github issue I found, you will get that error message if you haven't whitelisted the server IP addresses in MongoDB Atlas.
Reading the MongoDB Atlas docs, the only way I see to do this in combination with Heroku dynos is to add 0.0.0.0/0 (i.e. all addresses) to your MongoDB Atlas whitelist.
Give that a try and please report back whether you can instantiate a connection.
On SSL
Trying to reply to the SSL question, I do not think that you need to enable it on Heroku based on what I read, although I am not totally sure.
If the MongoDB server performed certificate validation, the Node.js code for connecting to it would have to look like the following (taken from the Node.js driver documentation):
var MongoClient = require('mongodb').MongoClient,
f = require('util').format,
fs = require('fs');
// Read the certificates
var ca = [fs.readFileSync(__dirname + "/ssl/ca.pem")];
var cert = fs.readFileSync(__dirname + "/ssl/client.pem");
var key = fs.readFileSync(__dirname + "/ssl/client.pem");
// Connect validating the returned certificates from the server
MongoClient.connect("mongodb://localhost:27017/test?ssl=true", {
server: {
sslValidate:true
, sslCA:ca
, sslKey:key
, sslCert:cert
, sslPass:'10gen'
}
}, function(err, db) {
db.close();
});
If the MongoDB server does not check for any SSL certificates, you can simply use code like the following (also taken from the Node.js driver documentation):
var MongoClient = require('mongodb').MongoClient;
MongoClient.connect("mongodb://localhost:27017/test?ssl=true", function(err, db) {
db.close();
});
Given that the Atlas documentation contains the following example code for connecting to it from Node.js, I think that you do not have to enable SSL on Heroku:
var MongoClient = require('mongodb').MongoClient;
var uri = "mongodb://kay:myRealPassword#mycluster0-shard-00-00-wpeiv.mongodb.net:27017,mycluster0-shard-00-01-wpeiv.mongodb.net:27017,mycluster0-shard-00-02-wpeiv.mongodb.net:27017/admin?ssl=true&replicaSet=Mycluster0-shard-0&authSource=admin";
MongoClient.connect(uri, function(err, db) {
db.close();
});
You can find all IP ranges for Heroku with this command:
HEROKU_REGION=eu; sudo apt -qqy install curl jq 2>/dev/null 1>/dev/null; heroku regions --json 2>/dev/null | jq ".[] | select(.name==\"$HEROKU_REGION\") | .provider.region" | (REGION=$(cat); curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | jq ".prefixes[] | select(.region==$REGION) | .ip_prefix")
Also had to add 0.0.0.0/0 to the Mongo IP whitelist AND redeploy my app on Heroku for it to finally work (before changing IP, a CORS error was thrown).
very simple solution! just add to the white list IP in mongo atlas the adress "0.0.0.0/0"
it will open the mongo atlas to all the world..... so it os not for production but it helps for small tests
I solved this by installing an addon(i used Fixie Socks) for Static IP addresses for database requests and other TCP connections.
More options here: https://elements.heroku.com/addons#network
Since allowing access from anywhere is not secure and IP ranges could change, I ended up installing add-on QuotaGuard Static IP's (it provides 2 IP addresses for IP whitelist) so SOCKS5 Proxy can be used with QGTunnel.
QGTunnel should be downloaded and included in the codebase
curl https://s3.amazonaws.com/quotaguard/qgtunnel-latest.tar.gz | tar xz
Procfile should be updated
web: bin/qgtunnel npm start
Let’s say you want to access a replicated MongoDB cluster using QGTunnel with 3 replicas located on the hosts: rs01.mongodb.net:52115, rs02.mongodb.net:52115, and rs1.mongodb.net:52115. For this configuration, you will need to create 3 separate tunnels for each host on port 52115 in transparent mode. Once this is done, QGTunnel will alter the DNS resolution process to resolve these hostnames to the appropriate loopback address and auto-discovery for your replicated cluster should work as intended.
I'm using Mongodb with only one shard and 3 members in a replica set. I have each replica set member on a different machine. Two members each have a mongos router, one member (that has a router) has a config server, and all three have a shard server. I have one hostname that resolves to the two ip addresses of each one of the mongodb instances that has a mongos router. I'm using mongoose v.2.5.10 to connect to mongodb from my node.js application with the following code:
var mongoose = require('mongoose');
mongoose.connection.on('error', function(err){
console.error('Mongodb connection error: ', err);
});
var db = mongoose.connect('mongodb://username:password#' + mongoHost + '/database');
I simulate what would happen in a failover scenario by terminating one of the mongodb instances that has the mongos router, and when this happens, I detect that the instance is down and prune the instance's ip address from the DNS record. However, my application does not seem to correctly reconnect to mongodb, as the error event of the mongoose connection is not being emitted, and my application hangs until I restart my node server.
What might be going wrong here?
UPDATE:
I've just confirmed that the net client that the mongodb node.js client is using to connect (this.connection = net.createConnection(this.socketOptions.port, this.socketOptions.host);) is not receiving the close or error event when I manually terminate the Amazon instance. So this seems like this could be an internal node issue...