For a MEAN app, I need to pass the dbName depending on the logged-in user. The flow of events is as follows.
User logs-in and gets authenticated using an auth specific REST-API.
Auth-API returns to Angular the user account data which includes the userSpecificDbName.
Thereafter, Angular makes all calls to the app's main REST-API. With all calls to the API, account data is passed and the API shd use the dbName from the account data in the following statement of app.js
mongoose.connect(uri, { dbName: <userSpecificDbName> })
.then( () => {
console.log('Connection to the Atlas Cluster is successful!')
})
.catch( (err) => console.error(err));
Is this possible? And how do I pass data in app.js for this purpose?
It is certainly possible to do that, Although keep in mind you would need to open and close connections per request to do that, which is not a regular pattern in Node.js
Normally you would connect to the database once and then fire queries. But in your case, you would disconnect every time after you've sent a response and reconnect and the beginning to the DB.
Alternatively, you could most probably do a useDb call as suggested in the following answer:
switching database with mongoose
That would help you switch the database.
Related
I created a MEAN project in heroku.
In MongoDB I have a DB / Collection like this:
db == content / collection == android_main
I have verified that the collection is in the proper database (and not in admin) via the following MongoSH exchange:
Atlas atlas-xxxxxx-shard-0 [primary] content> use admin
switched to db admin
Atlas atlas-xxxxxx-shard-0 [primary] admin> show collections
Atlas atlas-xxxxxx-shard-0 [primary] admin> use content
switched to db content
Atlas atlas-xxxxxx-shard-0 [primary] content> show collections
android_main
In MongoDB I also have a user w/ specific privs to read this db/collection.
In the code (Node.js) I use the following connection string:
mongodb+srv://<USER>:<PASS>#<MONGO URL>/content?retryWrites=true&w=majority
The connection completes successfully. I pass in the user I mentioned above, e.g. the user with just the specific read privs on the database (content) and collection (android_main).
In the code, the mechanism I use to get the database variable via the connection and connection string specifically is:
mongodb.MongoClient.connect(process.env.MONGODB_URI, function (err, database) {
Now, in the code, in response to an `HTTP GET` I issue the following:
db.collection("android_main").find({}).toArray(function(err, docs) {
...
}
I get back this error:
ERROR: user is not allowed to do action [find] on [admin.android_main]
Question: how can I modify my code a/o setup to ensure the db.collection.find() call references the proper database? I would have thought this was taken care of in the connection, where the database is explicitly called out.
I'm going to post the following answer which solves the issue above.
I found it by just playing around with random ideas.
I'd like to thank Heroku and their pathetic documentation for turning a 30 minute task into a 3 day ordeal.
client.connect(
process.env.MONGODB_URI,
function (err, database) {
console.log("CONNECT...");
if (err) {
console.log(err);
process.exit(1);
}
db = database.db(DATABASE);
var server = app.listen(process.env.PORT || 8080,
function () {
var port = server.address().port;
console.log("CONNECT App now running on port", port);
}
);
console.log("CONNECT Done.");
}
);
Apparently, to properly set the variable db (the one which you will use in the queries later) you should note that the database you receive in the connection callback is the admin database (that which was used for auth), and from that you make the db() request passing in the database you intend to use. All of this is utterly redundant b/c we pass the database we intend to use in the connection string. What a complete CF; but it's certainly not the 1st one. Enjoy.
I am working on an application that needs to connect to different MongoDB databases based on the customer that is accessing my application. Each MongoDB database is located on separate servers.
For example, if my application has 3 different customers:
john (mongodb://john:john#server1/john)
tom (mongodb://tom:tom#server2/tom)
harry (mongodb://harry:harry#server3/harry)
when john uses my application, Express.js should connect to mongodb://john:john#server1/john or when harry uses my application, it should connect to mongodb://harry:harry#server3/harry.
I am wondering what an optimized way would be to achieve this. Normally, when we have just one MongoDB database instance, we put the mongoose.connect method inside the app.js and that gets called when the Express.js server starts, but in this case I don't want to connect all three databases at once since that will not be the optimized way.
According to the Mongoose docs, it does support connections to multiple hosts.
const conn1 = mongoose.createConnection('host1', options)
const conn2 = mongoose.createConnection('host2', options)
Let me give you an example (this is Express code) :
// CONNECTION SETTER MIDDLEWARE
app.use(async function (req, res, next) {
// Let's just assume you identified the user earlier with your favourite
// auth method and put their identity in req.user
const userCon = await mongoose.createConnection(`mongodb://${req.user.username}:${req.user.password}#${req.user.host}/${req.user.database}`) // Here you have it!
// Saving the connection in the request so the next
// middlewares/handlers can access it
req.user.connection = userCon
// Thank you, next (middleware/handler)
next()
});
While mongoose.connect() sets the default connection, mongoose.createConnection() only returns a connection.
This is what I would try, and this is Mongoose's recommended way to handle multiple connections. You should note that you can't share Models across different connections, but you can share Schemas.
If you want to just initialize the connection and connect later, the Mongoose docs also provide an example.
// initialize now, connect later
db = mongoose.createConnection();
db.openUri('localhost', 'database', port, [opts]);
I want to create a Database when a user launches an application if it does not already exist. The way I tried doing this is, because there's no way that I know of to create a Database without having an existing connection, is to connect to the standard postgres Database, which is a default pre-existing one (I think).
Once I'm connected, I execute this from the default connection:
`SELECT 'CREATE DATABASE dbName WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'dbName')\\gexec`
This supposingly creates the Database if it doesn't exist. Once this runs, and I guaranteed to have a dbName database that exists. I thus shut off the other connection for it is no longer needed with
postgresCon.end();
And I connect to the recently created Database, in hopes to then use it to do whatever queries I would like and have the user be able to add workouts/exercises/whatever else.
The Problem
Whenever I run the Select create database workout... statement, I catch the error, and what it gives me is "Error: Connection Terminated", safe to say that this error isn't really... descriptive. In fact, this happens when workout database already exists. If it doesn't, it simply complains (later - when I try to execute a query on this database) that it doesn't exist.
This is the code, my theory is that it's Asynchronous and so though the connection "started connecting", the rest of the Queries are already being run and returning with "doesn't exist", because it hasn't been fully created yet.
Here is the code on pastebin for some highlighting (though apparently it's not happy with '' so the highlighting may be a bit scuffed): https://pastebin.com/fiiK35j7
If you need more detail/code/more clarity, do ask!
EDIT I've been told that my Theory is correct and that I should use async/await, but I am unsure how I could stop queries from being ran before the connection has been successfully completed, I assume it'd have to be something along the lines of:
async connection() => {await db.connect()}
connection()
.then(can do queries here)
.catch(panic)
but I am unsure what will actually work
I don't believe you can call \gexec from application code. That's a command which can only be used from the psql cli tool.
You'll have to issue and query and then do the conditional in your application code to subsequently create (or not) the database.
As was commented above, connecting with the DB superuser from an application is a bad idea. But there are some scenarios where it might be warranted, i.e. from a CI/CD pipeline where you want to provision databases and users if they don't exist for new environments. I did the following (using ts-postgres instead of node-postgres):
const [host, username, password] = ['localhost', 'user', 'pass']
const client = new Client({host, user: username, password})
await client.connect()
/////
// There is no CREATE DATABASE IF NOT EXISTS in postgres, which is decidedly annoying
// instead, we have to query and see if the database and user exists, and if not, create them
const dbQuery = await client.query(`SELECT FROM pg_database WHERE datname = $1`, [applicationDatabaseName])
if (dbQuery.rows.length === 0) {
// database does not exist, make it:
await client.query(`CREATE DATABASE ${applicationDatabaseName}`)
}
const userQuery = await client.query(`SELECT FROM pg_roles where rolname = $1`, [applicationDatabaseUser])
if(userQuery.rows.length === 0) {
//user doesn't exist. make it
await client.query(`CREATE USER ${applicationDatabaseUser} with ENCRYPTED PASSWORD '${applicationDatabasePassword}'`)
await client.query(`GRANT ALL PRIVILEGES ON DATABASE ${applicationDatabaseName} TO ${applicationDatabaseUser}`)
}
await client.end()
I need to continuously update data on the client based on DB changes. I'm thinking about having a 5 second interval function that repeatedly gathers all the DB information and use Socket.IO to emit the data to the client.
Currently, I'm doing this on the client itself without socket.io, just repeatedly doing a REST call to the server which then handles the data.
My question is: Are either of these methods efficient or inefficient and is there a better solution to solve what I'm trying to achieve?
Ryan, you can try using MongoDB's collection.watch() which fires an event every time an update is made to a collection. You would need to do that within the socket connection event for it to work though. Something along these lines:
io.sockets.on('connection', function(socket) {
// when the socket is connected, start listening to MongoDB
const MongoClient = require("mongodb").MongoClient;
MongoClient.connect("mongodb://192.168.1.201")
.then(client => {
console.log("Connected correctly to server");
// specify db and collections
const db = client.db("your_db");
const collection = db.collection("your_collection");
const changeStream = collection.watch();
// start listening to changes
changeStream.on("change", function(change) {
console.log(change);
// this is where you can fire the socket.emit('the_change', change)
});
})
.catch(err => {
console.error(err);
});
});
Note that using this approach will require you to set up a replica set. You can follow those instructions or use a Dockerised replica set such as this one.
I need more details to make sure but it doesn't sound like a good solution.
If the data you need does not change rapidly, like let's say in seconds, each of your connection still polling every 5 seconds and that's kind of wasting.
In that case you might just trigger an event where the data got changed, then you can push the message through sockets that are active.
I am using mongodb-native-driver in express.js app. I have around 6 collections in the database, so I have created 6 js files with each having a collection as a javascript object (e.g function collection(){}) and the prototypes functions handling all the manipulation on those collections. I thought this would be a good architecture.
But the problem I am having is how to connect to the database? Should I create a connection in each of this files and use them? I think that would be an overkill as the connect in mongodb-native-driver creates a pool of connections and having several of them would not be justified.
So how do I create a single connection pool and use it in all the collections.js files? I want to have the connection like its implemented in mongoose. Let me know if any of my thought process in architecture of the app is wrong.
Using Mongoose would solve these problems, but I have read in several places thats it slower than native-driver and also I would prefer a schema-less models.
Edit: I created a module out of models. Each collection was in a file and it took the database as an argument. Now in the index.js file I called the database connection and kept a variable db after I got the database from the connection. (I used the auto-reconnect feature to make sure that the connection wasn't lost). In the same index.js file I exported each of the collections like this
exports.model1 = require('./model1').(db)
exprorts.model2 = require('./model2').(db)
This ensured that the database part was handled in just one module and the app would just call function that each model.js file exported like save(), fincdbyid() etc (whatever you do in the function is upto you to implement).
how to connect to the database?
In order to connect using the MongoDB native driver you need to do something like the following:
var util = require('util');
var mongodb = require('mongodb');
var client = mongodb.MongoClient;
var auth = {
user: 'username',
pass: 'password',
host: 'hostname',
port: 1337,
name: 'databaseName'
};
var uri = util.format('mongodb://%s:%s#%s:%d/%s',
auth.user, auth.pass, auth.host, auth.port, auth.name);
/** Connect to the Mongo database at the URI using the client */
client.connect(uri, { auto_reconnect: true }, function (err, database) {
if (err) throw err;
else if (!database) console.log('Unknown error connecting to database');
else {
console.log('Connected to MongoDB database server at:');
console.log('\n\t%s\n', uri);
// Create or access collections, etc here using the database object
}
});
A basic connection is setup like this. This is all I can give you going on just the basic description of what you want. Post up some code you've got so far to get more specific help.
Should I create a connection in each of this files and use them?
No.
So how do I create a single connection pool and use it in all the collections.js files?
You can create a single file with code like the above, lets call it dbmanager.js connecting to the database. Export functions like createUser, deleteUser, etc. which operate on your database, then export functionality like so:
module.exports = {
createUser: function () { ; },
deleteUser: function () { ; }
};
which you could then require from another file like so:
var dbman = require('./dbmanager');
dbman.createUser(userData); // using connection established in `dbmanager.js`
EDIT: Because we're dealing with JavaScript and a single thread, the native driver indeed automatically handles connection pooling for you. You can look for this in the StackOverflow links below for more confirmation of this. The OP does state this in the question as well. This means that client.connect should be called only once by an instance of your server. After the database object is successfully retrieved from a call to client.connect, that database object should be reused throughout the entire instance of your app. This is easily accomplished by using the module pattern that Node.JS provides.
My suggestion is to create a module or set of modules which serves as a single point of contact for interacting with the database. In my apps I usually have a single module which depends on the native driver, calling require('mongodb'). All other modules in my app will not directly access the database, but instead all manipulations must be coordinated by this database module.
This encapsulates all of the code dealing with the native driver into a single module or set of modules. The OP seems to think there is a problem with the simple code example I've posted, describing a problem with a "single large closure" in my example. This is all pretty basic stuff, so I'm adding clarification as to the basic architecture at work here, but I still do not feel the need to change any code.
The OP also seems to think that multiple connections could possibly be made here. This is not possible with this setup. If you created a module like I suggest above then the first time require('./dbmanager') is called it will execute the code in the file dbmanager.js and return the module.exports object. The exports object is cached and is also returned on each subsequent call to require('./dbmanager'), however, the code in dbmanager.js will only be executed the first require.
If you don't want to create a module like this then the other option would be to export only the database passed to the callback for client.connect and use it directly in different places throughout your app. I recommend against this however, regardless of the OPs concerns.
Similar, possibly duplicate Stackoverflow questions, among others:
How to manage mongodb connections in nodejs webapp
Node.JS and MongoDB, reusing the DB object
Node.JS - What is the right way to deal with MongoDB connections
As accepted answer says - you should create only one connection for all incoming requests and reuse it, but answer is missing solution, that will create and cache connection. I wrote express middleware to achieve this - express-mongo-db. At first sight this task is trivial, and most people use this kind of code:
var db;
function createConnection(req, res, next) {
if (db) { req.db = db; next(); }
client.connect(uri, { auto_reconnect: true }, function (err, database) {
req.db = db = databse;
next();
});
}
app.use(createConnection);
But this code lead you to connection-leak, when multiple request arrives at the same time, and db is undefined. express-mongo-db solving this by holding incoming clients and calling connect only once, when module is required (not when first request arrives).
Hope you find it useful.
I just thought I would add in my own method of MongoDB connection for others interested or having problems with different methods
This method assumes you don't need authentication(I use this on localhost)
Authentication is still easy to implement
var MongoClient = require('mongodb').MongoClient;
var Server = require('mongodb').Server;
var client = new MongoClient(new Server('localhost',27017,{
socketOptions: {connectTimeoutMS: 500},
poolSize:5,
auto_reconnect:true
}, {
numberOfRetries:3,
retryMilliseconds: 500
}));
client.open(function(err, client) {
if(err) {
console.log("Connection Failed Via Client Object.");
} else {
var db = client.db("theDbName");
if(db) {
console.log("Connected Via Client Object . . .");
db.logout(function(err,result) {
if(!err) {
console.log("Logged out successfully");
}
client.close();
console.log("Connection closed");
});
}
}
});
Credit goes to Brad Davley which goes over this method in his book (page 231-232)