I found the following on the ExpressJS guide:
var mysql = require('mysql');
var connection = mysql.createConnection({
host : 'localhost',
user : 'dbuser',
password : 's3kreee7'
});
connection.connect();
connection.query('SELECT 1 + 1 AS solution', function(err, rows, fields) {
if (err) throw err;
console.log('The solution is: ', rows[0].solution);
});
connection.end();
Isn't this supposed to be bad practice? The way I see it, it is possible for the connection to end before the query can be executed. Wouldn't that give an error?
As stated here :
Every method you invoke on a connection is queued and executed in sequence.
Closing the connection is done using end() which makes sure all remaining queries are executed before sending a quit packet to the mysql server.
So even though the call to the end() method can be made before the query has completed, it won't actually be executed until the query has finished executing.
This has to do more with the mysql package than NodeJS itself.
Your question How does async work in Express? and Isn't this supposed to be bad practice? can be answered in many ways, but for clarity I would like to explain that It depends !!!!
It generally is very bad practice, assuming you don't know the actual implementation.
If the the implementation is really simple, where it does exactly what you ask -- i.e. closes or ends the connection when end is executed then it could lead to rather ugly race conditions where it may or may not work depending on the load of the machines.
However, a clever implementation that does reference counting -- that is the end does not actually close the connection but just sets a flag to say -- "when last callback is done then close" -- then it may work.
If the mysql connector it implemented using reference counting then this may well work fine -- but that is not the same as saying that it is good practice for everything you find as a plugin.
Related
I have a MongoDB collection of 3257477 cities, and I'm using Mongoose on NodeJS to access it. I'm making requests to it repeatedly (once per 500ms). Requests are usually answered very quickly. However, when I make a bad typo the query takes a long time and requests start to pile up until the initial request is answered. Here are some logs I collected of requests and responses:
21:48:50 started query for "new"
21:48:50 finished query for "new"
21:48:52 started query for "newj ljl" // blockage
21:48:54 started query for "newj"
21:48:55 started query for "new"
21:48:57 started query for "new ye"
21:48:59 started query for "new york"
21:49:08 finished query for "newj ljl" // blockage removed, quick queries flood in
21:49:08 finished query for "new"
21:49:08 finished query for "new york"
21:49:08 finished query for "new ye"
21:49:23 finished query for "newj"
I'm able to cancel the requests made by the client so I'm not worried about queries coming back in the wrong order. And I'm not interested in how to make that query faster at this point, since queries for actual correct spellings are quick.
I'm wondering how a new request can cancel an old request that was made by the same client. In other words "newj ljl" gets canceled when "newj" arrives, "newj" gets canceled when "new" arrives, and so on. If it's just going to be thrown out, why tie up the database?
Is there a proper way to do this?
Update:
I'm aware of db.currentOp().inprog and I'm thinking I can use the client property of the documents within that array to know whether it's a repeat request, but I can't quite figure out how to access that from Mongoose. I'm also not sure when to do that, or how I know which request was spawned from this client (and therefore which to cancel). I'd like an actual code example using Mongoose, or the native NodeJS MongoDB driver if possible!
Here's some sample code to go off of:
models.City.find({ ... })
.exec(function (err, cities) {
});
Below is what I came up with to solve the issue.
I can easily do db.currentOp().inprog and db.killOp() from the Mongo shell, but I really need this to happen automatically, when it needs to, from Mongoose. Since you can reference the MongoDB driver using require('mongoose').connection.db, you can execute those commands by doing "queries" on the following collections:
db.collection('$cmd.sys.inprog');
db.collection('$cmd.sys.killop');
The full solution:
var db = require('mongoose').connection.db,
// get the client IP address
ip = request.headers['x-forwarded-for'] ||
request.connection.remoteAddress ||
request.socket.remoteAddress ||
request.connection.socket.remoteAddress;
// same thing as db.currentOp().inprog
db.collection('$cmd.sys.inprog').findOne(function (err, data) {
if (err) throw err;
data.inprog.filter(function (op) {
// get the operation's client IP address without the port
return ip == op.client.split(':')[0];
}).forEach(function(op){
// same thing as db.killOp()
db.collection('$cmd.sys.killop')
.findOne({ 'op': op.opid }, function (err, data) {
if (err) throw err;
});
});
// start the new cities query
models.City.find({ ... })
.exec(function (err, cities) {
});
});
Helpful links:
https://groups.google.com/forum/#!topic/mongodb-user/1wFp7AqWnM4
drop database with mongoose
How to determine a user's IP address in node
You can try using db.killOp()
http://docs.mongodb.org/manual/reference/method/db.killOp/#db.killOp
UPDATE: You can get the list of current operations from db.currentOp() and identify the operation to be cancelled by matching fields like op, query and client
http://docs.mongodb.org/manual/reference/method/db.currentOp/#db.currentOp
You can definitely do this with killop, and the above solution looks like it could work for the problem as stated. However, I think it may be worthwhile to dig a bit deeper.
The fact that you have a noticeably slow query when you've got a query that's going to return no results seems unusual. That reeks of a full collection scan. The questions to ask are, first, do you have indices set up, and second, are you querying with a general regex? MongoDB doesn't really handle regex searches like { "name" : /.*new york.*/ } particularly well.
Also, the whole "send an http request every time the user hits a key" approach is simple and elegant, but also causes some unnecessary server load. Perhaps a search button or a client-side timeout where you only send a request if a user hasn't hit a key for 1 second could help alleviate the need for the killop approach.
I can't figure out what I'm doing wrong, perhaps somone can point it out. I'm trying to figure out why my 'job complete' event isn't firing.
var kue = require('kue'),
jobs = kue.createQueue();
var util= require('util');
var job = jobs.create('test', util.puts('123')).on('complete', function(){
console.log("Job complete");
}).on('failed', function(){
console.log("Job failed");
}).on('progress', function(progress){
process.stdout.write('\r job #' + job.id + ' ' + progress + '% complete');
});
Now when I run this on node it prints 123 but it doesn't say job complete.
This question is old, but there is still no solution on the internet for those, who encounters this problem even with placing save()... I've made out three steps for myself to solve the problem:
1. Make sure that you call save() method on your jobs AFTER you set handlers on them.
var job = queue.create('some process', some_args);
job.on('complete', function(result) {
console.log('complete');
}).on('failed', function(result) {
console.log('failed');
}).removeOnComplete(true).save();
P.S. It's also a good practice to remove jobs on complete, otherwise you'll overfill redis memory.
2. Make sure your handlers are alright.
I myself experimented with event handlers, trying to pass them several arguments. My 'failed' event handler accepted both error code and other data I passed through done(err, data) method. This was not right. So check the documentation and official Kue examples to make sure your code isn't bugged.
3. If nothing helps, execute redis-cli flushall in your terminal.
And BEWARE!!! This will delete everything in your redis. I'm myself noob in it, so it is used only as a dependency for Kue on my system. I don't know definitely, but I suppose that this can destroy your data you may use in redis. Though it somehow fixes the problem, when nothing more helps.
Everyone, please, feel free to suggest any other secure ways to fix Kue with redis.
P.S. Haven't check, but I suppose, that changing process name for your jobs (it's 'some process' in my example) can also workaround the problem.
I think you need to run job.save() after the .create is executed.
As #James mentions you must call .save() after the event handlers has been set.
See the example.
I'm writing some scripts for some command-line manipulation of Mongoose models with commander.js (eventually, I'd like to run these tools using Cron).
Now, I've written several scripts with commander and they all work fine, but if I connect to the MongoDB database using mongoose, they script just hangs after it's done. Now, I figured the database connection is keeping node alive, so I added a mongoose.disconnect() line and it still hangs.
The only thing I found that allows me to shutdown is to use process.exit(), but I'm reluctant to just terminate the process. Is there something in particular that I should do to trigger a graceful shutdown?
My reading of the API docs implies that .disconnect() must be given a callback function. It looks like it's called for each that's disconnected and may be passed an error.
There is a check in the code to make sure it's not called if it doesn't exist when things work out, but that check isn't being run on errors, so if Mongoose received an error message from the MongoDB client, it may be leaving a connection open and that's why it's not stopping execution.
If you're only opening a single connection to the database, you may just want to call [Connection object].close() since that function correctly inserts a no-op "callback" if no callback is given, and looks like it will correctly destruct things.
(The more I look into Mongoose, the more I want to just write a thin wrapper around the MongoDB client so I don't have to deal with Mongoose's "help.")
I use the async "Series" to perform operations and then call mongoose.connection.close() on completion. It prevents callback hell and allows you to neatly perform operations either one at a time or parallel followed by a function when all the other methods have completed. I use it all the time for scripts that require mongoose but are meant to terminate after all mongoose operations are finished.
Shutdown the node program directly is hiding the symptoms, not fixing the problem!
I finally isolated the problem and found it to be with Mongoose schema definitions. If you try to shutdown the connection too soon after Mongoose schemas are defined1, the application hangs and eventually produces some weird MongDB-related error.
Adding a small timeout before running the program.parse(argv) line to run the commander application fixes the problem. Just wrap the code like so:
var program = require('commander')
, mongoose = require('mongoose')
, models = null
;
// Define command line syntax.
program
.command(...)
;
mongoose.connect(
..., // connection parameters.
function() {
// connected to database, defined schemas.
models = require('./models');
// Wait 1 second before running the application code.
setTimeout(function(){
program.parser(process.argv);
}, 1000);
}
);
1: This is my initial interpretation, I have not (yet) extensively tested this theory. However, removing Mongoose schema definitions from the application successfully prevents the application from hanging.
Actually, just using process.nextTick() instead of the setTimeout() call fixes the situation nicely!
I'm using node-redis. In code like this:
var store = require('redis').createClient();
store.watch('some:key');
store.get('some:key', function (err, results) {
var multi = store.multi();
// COMPUTE SOMETHING WITH results
multi.set('something:or:other', 25);
multi.exec(checkAllIsWell);
});
Should lines 1-2 read
store.watch('some:key', function (err, alwaysok) {
store.get('some:key', function (err, result) {
or will watch always have immediate effect?
EDIT: To reframe the question a little, is sequence guaranteed on seqential calls on the same Redis client? Or could the WATCH happen after the GET?
Having reframed my question, I realize that it must surely be sequence-preserving, and I'm actually duplicating this question: Are Redis updates synchronous?
So the answer is surely that I don't need to wait for WATCH to call back and my original code is OK.
Sorry to noise up the web, folks!
Watch always returns OK. http://redis.io/commands/watch
It is useful only if later you use MULTI/EXEC, to check the EXEC return value.
For more information about Redis transactions, visit http://redis.io/topics/transactions
In my app (node / express / redis), I use some code to update several items in DB at the same time:
app.put('myaction', function(req, res){
// delete stuff
db.del("key1");
db.srem("set1", "test");
// Add stuff
db.sadd("set2", "test2");
db.sadd("set3", "test3");
db.hmset("hash1", "k11", "v11", "k21", "v21");
db.hmset("hash2", "k12", "v12", "k22", "v22");
// ...
// Send response back
res.writeHead(200, {'content-type': 'application/json'});
res.write(JSON.stringify({ "status" : "ok" }));
res.end();
});
Can I be sure ALL those actions will be performed before the method returns ? My concern is the asynchronous processing. As I do not use callback function in the db actions, will this be alright ?
While all of the commands are sent and responses parsed asynchronously, it's useful to note that the callbacks are all invoked in order. So you can use the callback of the last Redis command to send the response to the client, and then you'll know that all of the Redis commands have been executed before responding.
Use the MULTI/EXEC command to create a queue of your commands and execute them in a row. Then use a callback to send a coherent response back (success/failure).
Note that you must use Redis' AOF to avoid that - in case of crash - the db state is not coherent with your logic because only a part of the commands in the queue were executed: i.e. MULTI/EXEC is not transactional upon execution. This is a useful reference.
I haven't worked with redis, but if this works(if you it doesn't call undefined function) and it should be asynchronous, then you can use it. But if there is an error in updating, then you can't handle it, this way.
No, you can't be sure if all those actions complete successfully, because your redis server might crash.. To speed things up, you can group all your update commands into one with pipelining (does your redis driver support that?), then get the success or failure of the whole operation via a callback and proceed..