I'm writing some scripts for some command-line manipulation of Mongoose models with commander.js (eventually, I'd like to run these tools using Cron).
Now, I've written several scripts with commander and they all work fine, but if I connect to the MongoDB database using mongoose, they script just hangs after it's done. Now, I figured the database connection is keeping node alive, so I added a mongoose.disconnect() line and it still hangs.
The only thing I found that allows me to shutdown is to use process.exit(), but I'm reluctant to just terminate the process. Is there something in particular that I should do to trigger a graceful shutdown?
My reading of the API docs implies that .disconnect() must be given a callback function. It looks like it's called for each that's disconnected and may be passed an error.
There is a check in the code to make sure it's not called if it doesn't exist when things work out, but that check isn't being run on errors, so if Mongoose received an error message from the MongoDB client, it may be leaving a connection open and that's why it's not stopping execution.
If you're only opening a single connection to the database, you may just want to call [Connection object].close() since that function correctly inserts a no-op "callback" if no callback is given, and looks like it will correctly destruct things.
(The more I look into Mongoose, the more I want to just write a thin wrapper around the MongoDB client so I don't have to deal with Mongoose's "help.")
I use the async "Series" to perform operations and then call mongoose.connection.close() on completion. It prevents callback hell and allows you to neatly perform operations either one at a time or parallel followed by a function when all the other methods have completed. I use it all the time for scripts that require mongoose but are meant to terminate after all mongoose operations are finished.
Shutdown the node program directly is hiding the symptoms, not fixing the problem!
I finally isolated the problem and found it to be with Mongoose schema definitions. If you try to shutdown the connection too soon after Mongoose schemas are defined1, the application hangs and eventually produces some weird MongDB-related error.
Adding a small timeout before running the program.parse(argv) line to run the commander application fixes the problem. Just wrap the code like so:
var program = require('commander')
, mongoose = require('mongoose')
, models = null
;
// Define command line syntax.
program
.command(...)
;
mongoose.connect(
..., // connection parameters.
function() {
// connected to database, defined schemas.
models = require('./models');
// Wait 1 second before running the application code.
setTimeout(function(){
program.parser(process.argv);
}, 1000);
}
);
1: This is my initial interpretation, I have not (yet) extensively tested this theory. However, removing Mongoose schema definitions from the application successfully prevents the application from hanging.
Actually, just using process.nextTick() instead of the setTimeout() call fixes the situation nicely!
Related
I am a bit new to JavaScript web dev, and so am still getting my head around the flow of asynchronous functions, which can be a bit unexpected to the uninitiated. In my particular use case, I want execute a routine on the list of available databases before moving into the main code. Specifically, in order to ensure that a test environment is always properly initialized, I am dropping a database if it already exists, and then building it from configuration files.
The basic flow I have looks like this:
let dbAdmin = client.db("admin").admin();
dbAdmin.listDatabases(function(err, dbs){/*Loop through DBs and drop relevant one if present.*/});
return await buildRelevantDB();
By peppering some console.log() items throughout, I have determined that the listDatabases() call basically puts the callback into a queue of sorts. I actually enter buildRelevantDB() before entering the callback passed to listDatabases. In this particular example, it seems to work anyway, I think because the call that reads the configuration file is also asynchronous and so puts items into the same queue but later, but I find this to be brittle and sloppy. There must be some way to ensure that the listDatabases portion resolves before moving forward.
The closest solution I found is here, but I still don't know how to get the callback I pass to listDatabases to be like a then as in that solution.
Mixing callbacks and promises is a bit more advanced technique, so if you are new to javascript try to avoid it. In fact, try to avoid it even if you already learned everything and became a js ninja.
Dcumentation for listDatabases says it is async, so you can just await it without messing up with callbacks:
const dbs = await dbAdmin.listDatabases();
/*Loop through DBs and drop relevant one if present.*/
The next thing, there is no need to await before return. If you can await within a function, it is async and returns a promise anyway, so just return the promise from buildRelevantDB:
return buildRelevantDB();
Finally, you can drop database directly. No need to iterate over all databases to pick one you want to drop:
await client.db(<db name to drop>).dropDatabase();
I found the following on the ExpressJS guide:
var mysql = require('mysql');
var connection = mysql.createConnection({
host : 'localhost',
user : 'dbuser',
password : 's3kreee7'
});
connection.connect();
connection.query('SELECT 1 + 1 AS solution', function(err, rows, fields) {
if (err) throw err;
console.log('The solution is: ', rows[0].solution);
});
connection.end();
Isn't this supposed to be bad practice? The way I see it, it is possible for the connection to end before the query can be executed. Wouldn't that give an error?
As stated here :
Every method you invoke on a connection is queued and executed in sequence.
Closing the connection is done using end() which makes sure all remaining queries are executed before sending a quit packet to the mysql server.
So even though the call to the end() method can be made before the query has completed, it won't actually be executed until the query has finished executing.
This has to do more with the mysql package than NodeJS itself.
Your question How does async work in Express? and Isn't this supposed to be bad practice? can be answered in many ways, but for clarity I would like to explain that It depends !!!!
It generally is very bad practice, assuming you don't know the actual implementation.
If the the implementation is really simple, where it does exactly what you ask -- i.e. closes or ends the connection when end is executed then it could lead to rather ugly race conditions where it may or may not work depending on the load of the machines.
However, a clever implementation that does reference counting -- that is the end does not actually close the connection but just sets a flag to say -- "when last callback is done then close" -- then it may work.
If the mysql connector it implemented using reference counting then this may well work fine -- but that is not the same as saying that it is good practice for everything you find as a plugin.
I'm writing a script to batch process some text documents and insert them into a mysql database. I'm trying to use the async library because using a standard while loop blocks the event queue and prevents the insert queries from getting run until all are generated. Since that may take 10 minutes or more, I get a timeout. So, I am trying to use async to avoid blocking the main thread. However, it's not working as expected. When I run the simplest form of the code below, using node test.js, in the command line, it only executes once, instead of infinitely. It seems like the computer is terminating the node process early since it is non-blocking. This, of course, is not what I want. Why is this, and how can I get it to work correctly?
//this code should run forever, constantly printing "working". However it only runs once.
var async = require('async')
async.whilst(function(){return true},function(){console.log("working")})
The second parameter for whilst() is a function that takes in a callback that needs to be called when the current iteration is "done."
So if you modify the code this way, you'll get what you're expecting:
var async = require('async');
async.whilst(function() {
return true
}, function(cb) {
console.log("working");
cb();
});
In Node.js, when you do
var otherscript = require('otherscript');
it runs the script upon the require
I am wondering if there is a way to "require" a script without running it, so that you can run it later when you want to.
Is there any good reason why not?
If you can edit the 'otherscript' (no one else is using that script) then you can simply enclose the whole code inside a function and add it to exports.
Example:
otherscript:
module.exports = function(){
//original code goes here
};
Then use as:
var otherscript = require('otherscript');
var obj = otherscript();
When you require a file or module, the return of that file/module is cached. In other words, it is really only executed once, and subsequent calls to require() of that file/module will only return a reference to the exact same object.
A common example of this is the Mongoose package, where calling require('mongoose') will return an instance of mongoose, on which you can call connect() to connect to the database. Calling require('mongoose') again in a different part of your program will return the same instance with the same database connection made available.
I am using Mongoose 3 and the most obvious way to connect to database is
conn = mongoose.createConnection(...)
conn.on("open", ...)
Question is, do I need to define all my models in the open callback? If that is so, I will have to create a initMongoose.coffee that looks like
# initMongoose.coffee
mongoose = require "mongoose"
module.exports = mongoose.createConnection ...
# modelExample.coffee
conn = require "./initDatabase"
conn.on "open", ->
... define model?
modeule.exports = model # I think this does not work?
I think I read somewhere in Node docs that modules cannot be defined in a callback like that?
Since I am only using 1 connection, I think I can use
mongoose.connect ...
Which doesnt accepts any callbacks so I suppose is synchronous? Can I define all my models and thus queries right after connect()? It works at the moment, but it might be because its fast enough.
Mongoose buffers up commands until it is finished connecting, so you can treat it like it's synchronous and define your models and start using the library whenever you want; only once you want to start actually inserting or retrieving data do you need to make the connection.