Why mongoose opens two connections? - node.js

It's a simple file from mongoose quick guide
mongoose.js
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/Chat');
var userSchema = mongoose.Schema({
name: String
});
var User = mongoose.model('User', userSchema);
var user = new User({name: 'Andy'});
user.save(); // if i comment it mongoose will keep one connection
User.find({}, function(err, data) { console.log(data); }); // the same if i comment it
I tried to use db.once method, but effect the same.
Why mongoose opens the second connection in this case?

Mongoose uses native mongo driver underneath, and it in turn uses connection pooling - I believe the default is 5 connections (Check here).
So your mongoose connection will use up to 5 simultaneous connections when it has simultaneous requests.
And since both user.save and User.find are asynchronous, those will be done simultaneously. So what your "program" tells node:
1. Ok, you need to shoot a `save` request for this user.
2. Also, you need to fire this `find` request.
The node runtime then reads these, runs through the whole of your function (until a return). Then it looks at it's notes:
I was supposed to call this save
I also need to call this find
Hey, mongo native driver (which is written in C++) - here are two tasks for you!
and then the mongo driver fires the first request. And it sees it is allowed to open more connections then one, so it does, and fires the second request too, without waiting for the first to finish.
If you called the find within a callback to save, it would be sequential, and the driver would probably reuse the connection it already had.
Example:
// open the first connection
user.save(function(err) {
if (err) {
console.log('I always do this super boring error check:', err);
return;
}
// Now that the first request is done, we fire the second one, and
// we probably end up reusing the connection.
User.find(/*...*/);
});
Or similar with promises:
user.save().exec().then(function(){
return User.find(query);
})
.then(function(users) {
console.log(users);
})
.catch(function(err) {
// if either fails, the error ends up here.
console.log(err);
});
By the way, you can tell mongoose to use only one connection if you need to, for some reason:
let connection = mongoose.createConnection(dbUrl, {server: {poolSize: 1}});
That would be the gist of it.
Read more on MongoLab blog and Mongoose website.

Related

What is the proper way to handle connecting and closing the MongoDB Client from NodeJS (not using Mongoose!)?

export const client = new MongoClient(
process.env.ATLAS_URI,
// TODO: Figure out what this is and why it's needed to turn off deprecation warning
{
useUnifiedTopology: true,
}
);
Following this guide and all make sense...but she is just doing one 'call' and then close().
I need to keep doing repeated calls:
export const getAllProducts = async () => {
try {
await client.connect();
const cursor = await client.db("products").collection("data").find();
return await cursor.toArray();
} catch (err) {
throw new Error(err);
} finally {
await client.close();
}
};
The first call is fine. After that: Error: MongoError: Topology is closed, please connect
I honestly don't quite understand what Topology means, but evidently it's the close() that's contributing to the issue.
It doesn't make sense that I set up new MongoClient and the ATLAS_URI does have the 'database name' in there...so why I have to connect specify that again?
Anyway, the main part of my ❓ stands: Do I just keep a separate process going and not close it? Do I start back with a whole new MongoClient each time? 😕
I'll just put a brief answer here incase anyone runs into this.
The Mongodb documentation for the Node.js driver will give you simple examples that include the client.connect()and client.close() methods just to give you a runnable example of making a simple call to the database but in a real server application you are just opening the connection to the client once during start up and typically only closing when the server application is being closed.
So in short: You don't need to open and close and connection everytime you want to perform some action on your database.

Node.js: mongoose.once('open') doesn't execute callback function

I'm trying to save some json files inside my database using a custom function I've wrote. To achieve that I must connect to the database which I'm trying to do using this piece of code at the start of the function:
let url = "mongodb://localhost:27017/database";
(async () => {
const directory = await fs.promises.readdir(__dirname + '/files')
let database = await mongoose.createConnection(url, {useNewUrlParser:true, useUnifiedTopology:true});
database.on('error', error => {
throw console.log("Couldn't Connect To The Database");
});
database.once('open', function() {
//Saving the data using Schema and save();
Weirdly enough, when executing database.once('open', function()) the callback function isn't being called at all and the program just skips the whole saving part and gets right to the end of the function.
I've searched the web for a solution, and one solution suggested to use mongoose.createConnection instant of mongoose.connect.
As you can see it didn't really fixed the issue and the callback function is still not being called.
How can I fix it, and why it happens?
Thanks!
mongoose.createConnection creates a connection instance and allows you to manage multiple db connections as the documentation states. In your case using connect() should be sufficient (connect will create one default connection which is accessible under mongoose.connection).
By awaiting the connect-promise you don't actually need to listen for the open event, you can simply do:
...
await mongoose.connect(url, {useNewUrlParser:true, useUnifiedTopology:true});
mongoose.model('YourModel', YourModelSchema);
...

How to make two db query synchronous, so that if any of them fails then both fails in node.js

async.parallel([
function(callback){
con.Attandance.insert({'xxx':'a'}, function(err,data) {
console.log(data);
callback();
});
}, function(callback) {
console.log(data);
con.Profile.insert({'xxx':'a'},function(err){callback()});
}
], function(err) {
console.log('Both a and b are saved now');
});
Attendance.insert() works either Profile.insert() execute or fails. I want if any of them fails data should not be saved in any collection either in Attendance or in Profile
What you mean are transactions, which have nothing to do with synchronous / asynchronous.
Unfortunately, MongoDB simply does not support transactions. The only way to achieve something even remotely close, you have to perform either a two phase commit, or implement a custom rollback logic to undo all changes to Attandance if the changes to Profile failed.
The only possibility to at least achieve atomic (yet not transactions!) updates, is by changing your model. If the Profile is a container for all Attandance instances, you can update the entire object at one. It's impossible to update more than one object atomically with MongoDB, and neither is it possible to achieve a strict order of transactions.
If you need that, go for an SQL database instead. Pretty much all (except SQlite) support transactions.
I wrote a library that implements the two phase commit system (mentioned in a prior answer) described in the docs. It might help in this scenario. Fawn - Transactions for MongoDB.
var Fawn = require("Fawn");
// intitialize Fawn
Fawn.init("mongodb://127.0.0.1:27017/testDB");
/**
optionally, you could initialize Fawn with mongoose
var mongoose = require("mongoose");
mongoose.connect("mongodb://127.0.0.1:27017/testDB");
Fawn.init(mongoose);
**/
// after initialization, create a task
var task = Fawn.Task();
task.save("Attendance", {xxx: "a"})
.save("Profile", {xxx: "a"})
.run()
.then(function(results){
// task is complete
// result from first operation
var firstUpdateResult = results[0];
// result from second operation
var secondUpdateResult = results[1];
})
.catch(function(err){
// Everything has been rolled back.
// log the error which caused the failure
console.log(err);
});

Scaffolding a Node.js app properly without Express (the app doesn't receive requests)

[This question is quite vague, I apologize for it. I'm trying to address my various troubles by answering the question myself]
I am building a Node.js app which has to perform various tasks at given intervals. Here is the global scaffold (involves bluebird promises and mongoose for DB interactions) :
var Promise = require("bluebird");
var mongoose = require('mongoose');
mongoose.Promise = require('bluebird');
// Personal modules
var bootApp = require(...);
var doStuffA = require(...);
var doStuffB = require(...);
var doStuffC = require(...);
// running locally, but meant to be deployed at some point
mongoose.connect('mongodb://localhost:27017/myDatabase');
var db = mongoose.connection;
db.on('error', () => {
console.log("Error : lost connection !"));
process.exit(1);
});
db.once('open', () => {
bootApp() // always start by booting
.then( () => { // then start the infinite loop of events
setInterval(doStuffA, 1000*60*60); // 1x/1h
setInterval(doStuffB, 1000*60*10); // 1x/10min
setInterval(doStuffC, 1000*60*3); // 1x/3min
}).catch((e) => { // errors are handled by doStuffX(), so we should never catch anything here
console.log(e.message);
process.exit(1);
});
});
Each module doStuffX is a function returning a Promise, handling its own errors, and should finish at some point.
Expected behaviour for the entire app :
The app should be able to run forever
The app should try to doStuffX() at the given interval, regardless of whether it succeeded or failed last time.
[Optional :] The app should close smoothly without retrying any doStuff upon receiving a "shut down" signal.
My question : how to build a clean scaffold for such an app ? Can I get rid of setInterval and use promises instead ? One of my main concerns is to make sure the previous instance of doStuffX() is finished before starting the next one, even if it involves "killing" it in some way.
I am open to any link about scaffolding apps, but PLEASE DO NOT GIVE ME AN ANSWER/LINK INVOLVING EXPRESS : I don't need Express, since my app doesn't receive any request. (everything I found so far starts with Express :/)
If you don't want to start the next doStuffX() until the previous one is done, then you can replace your setInterval() with repeated setTimeout() calls.
function runA() {
setTimeout(function() {
doStuffA().then(runA).catch(function(err) {
// decide what to do differently if doStuffA has an error
});
}, 1000*60*60);
}
runA();
You could also add a timeout to this so that if doStuffA() doesn't respond within a certain amount of time, then you take some other action. This would involve using another timer and a timeout flag.
[I answer my own question, trying to put here everything I changed afterwards, in case someone falls into this page someday...]
For the Mongoose part of the scaffold, here is what I got so far for a reliable long-term DB connection :
The Mongoose documentation gives a fancy way to ensure the driver will never give up on trying to reconnect with reconnectTries
I don't really understand socketOptions and keepalive which seem related to replicas, so I leave them out of my code for now
Since Mongoose should autoreconnect whenever something goes wrong, I'll keep the db.once('open') as the access to the app code itself, even though I don't really understand yet the difference with db.on('connected')
I recommend reading this.
var Promise = require("bluebird");
var mongoose = require('mongoose');
mongoose.Promise = require('bluebird');
// Personal modules
var bootApp = require(...);
var doStuffA = require(...);
var doStuffB = require(...);
var doStuffC = require(...);
// running locally, but meant to be deployed at some point
var uri = 'mongodb://localhost:27017/myDatabase';
// the added option makes sure the app will always try to reconnect...
mongoose.connect(uri, { server: { reconnectTries: Number.MAX_VALUE } });
var db = mongoose.connection;
db.on('error', () => {
console.log("Error with Mongoose connection."));
});
db.once('open', () => {
bootApp() // always start by booting
.then( () => { // then start the infinite loop of events
//////////////////////////////////
/// Here goes the actual stuff ///
//////////////////////////////////
}).catch((e) => { // errors are handled by doStuffX(), so we should never catch anything here
console.log(e.message);
});
});
Now, for the actual repetitive stuff, my objective is to make sure everything runs smoothly, and that no process gets stuck. About the changes I made :
The methods used are not native ES6 but are specific to bluebird. You can read about .timeout() and .delay() which I find very useful for chaining timeouts and intervals in a clean code.
In my mind, .then(runA, runA) should always launch ONE UNIQUE instance of runA but I'm concerned about whether I could actually end up launching two parallel instances...
// Instead of using setInterval in a bluebird promised environment...
setInterval(doStuffA, 1000*60*60); // 1x/1h
// I would have liked a full promise chain, but as jfriend00 stated,
// It will end up crashing because the initial promise is never resolved...
function runA() {
return doStuffA()
.timeout(1000*60*30) // kill the running instance if it takes longer than 30min
.delay(1000*60*60) // wait 60min
.then(runA, runA); // whatever the outcome, restart the process
}
runA();
// Therefore, a solution like jfriend00's seems like the way to go :
function runA() {
setTimeout(function() {
doStuffA()
.timeout(1000*60*30)
.then(runA, runA)
}, 1000*60*60);
}
runA();

Does mongoose allow for multiple database requests concurrently?

I read that Mongoose will only open one connection at maximum per collection, and there's no option to change this.
Does this mean that a slow mongo query will make all subsequent queries wait?
I know everything in node.js is non-blocking, but I'm wondering whether a slow query will delay the execution of all subsequent queries. And whether there is a way to change this.
It does only use one connection, if you use the default method where you do mongoose.connect(). To get around this, you can create multiple connections, and then tie a model pointing to the same schema to that connection.
Like so:
var conn = mongoose.createConnection('mongodb://localhost/test');
var conn2 = mongoose.createConnection('mongodb://localhost/test');
var model1 = conn.model('Model', Schema);
var model2 = conn2.model('Model', Schema);
model1.find({long query}, function() {
console.log("this will print out last");
});
model2.find({short query}, function() {
console.log("this will print out first");
});
Hope that helps.
Update
Hey, that does work. Updating from the comments, you can create a connection pool using createConnection. It lets you do multiple queries from the same model concurrently:
var conn = mongoose.createConnection('mongodb://localhost/test', {server:{poolSize:2}});
var model = conn.model('Model', Schema);
model.find({long query}, function() {
console.log("this will print out last");
});
model.find({short query}, function() {
console.log("this will print out first");
});
Update 2 -- Dec 2012
This answer may be slightly outdated now--I noticed I've been continuing to get upvotes, so I thought I would update it. The mongodb-native driver that mongoose wraps now has a default connection pool size of 5, so you probably don't need to explicitly specify it in mongoose.

Resources