I have found the following code that implements an asynchronous message queue (actually there is no queue, only files) with ZeroMQ and Node.js
setInterval(function() {
var value = { id: i++, date: new Date() };
WriteFile(value.id + ".dat", value);
client.send(value, function(result) {
console.log(value, result);
DeleteFile(value.id + ".dat");
});
}, 10000);
The code is from here.
The functions "WriteFile" and "DeleteFile" are defined later in the code, but there is nothing extraordinary there.
The function "client.send" is also defined in another file, where the callback is defined. Clearly there is a provision from ZeroMQ to have a callback when the message transmission is successful.
Now I want to do something like this but with Redis pubsub instead of ZeroMQ for simplicity. As I understand it, there is no callback in the "publish" function from the node_redis module.
My question is, is there a way to implement something like this? I really like the idea of writing files and then deleting them whet the transmission is complete, but I would like it done in Redis. I know I am grasping at straws, but if anyone has any ideas, I will gladly listen.
All of the redis module's commands have an optional callback as the last argument.
So doing something like
client.publish('channel', 'message', function(err) {
if (err) throw err;
});
should work as expected.
Related
I have a method that reads and write a log file, this method is called on every request by all users, then write the log the request path in a file. The questions are two:
Is safe read and then write a file in async mode considering concurrency questions?
If yes for the first question, the code bellow will work considering concurrency questions?
If yes for the first question how I have to do?
Please, disregard exceptions and performance questions, this is a didactic code.
var logFile = '/tmp/logs/log.js';
app.get("/", function(req){
var log = {path: req.path, date: new Date().getTime()};
log(log);
});
function log(data){
fs.exists(logFile, function(exists){
if(exists){
fs.readFile(logFile, function (err, data) {
if (err){
throw err
}
var logData = JSON.parse(data.toString());
logData.push(data);
writeLog(logData);
});
}else{
writeLog([data]);
}
});
}
function writeLog(base){
fs.writeFile(logFile, JSON.stringify(base, null, '\t'), function(err) {
if(err)
throw err;
});
}
I strongly suggest that you don't just "log asynchronously" because you want the log to be ordered based on the order things happened in your app, and there is no guarantee this will happen that way if you don't synchronize it somehow.
You can, for instance, use a promise chain to synchronize it:
var _queue = Promise.resolve();
function log(message){
_queue = _queue.then(function(){ // chain to the queue
return new Promise(function(resolve){
fs.appendFile("/tmp/logs/log.txt", new Date() + message + "\n", function(err, data){
if(err) console.log(err); // don't die on log exceptions
else resolve(); // signal to the queue it can progress
});
});
});
}
You can now call log and it will queue messages and write them some time asynchronously for you. It will never take more than a single file descriptor or exhaust the server either.
Consider using a logging solution instead of rolling your own logger btw.
In you're example you're already using the Asynchronous versions of those functions. If you're concerned about the order of your operations then you should use the synchronous versions of those functions.
readFileSync
writeFileSync
Also to note, JSON.parse() is a synchronous operation.You can make this "asynchronous" using the async module and doing a async.asyncify(JSON.parse(data.toString()));.
As noted by #BenjaminGruenbaum, async.asyncify(); doesn't actually make the operation of JSON.parse(); truly asynchronous but it does provide a more "async" style for the control flow of the operations.
I'm relatively new to node.js, wondering how to make my asynchronous code more reliable. Focusing on code that uses 'async' (no promises yet, due to legacy reasons).
One of my reliability concerns is failing to invoke callbacks. E.g.
async.series([
function(callback){
if(..)
console.log("invalid input, try again"); // BUG: NO CALLBACK!
else
callback(null, 'ok');
},
function(callback){
...
}
],
function(err,result){ handleErrorOrResult(err,result);} // MIGHT NOT BE REACHED
);
This code may never complete (never reach 'handleErrorOrResult') and I won't even know that anything went wrong. Since this pit is easy to fall into, I wonder whether there are ready-made library solutions for it? Any ideas would be welcome, but one direction might be timeouts: invoke some error handler if the whole thing isn't completed within, say, 5 minutes.
Thanks very much
Make your own wrapper for it that implements a timeout.
var myAsync = {
series: function (callbacks, timeout, done) {
var timeoutId = setTimeout(function () {
done("Maximum Time Exceeded for method!");
}, timeout);
async.series(callbacks, function () {
clearTimeout(timeoutId);
done.apply(this, arguments);
});
}
}
Now you can use it like:
myAsync.series(functions, functions.length * 500, callback);
I'm trying to determine when a node WriteStream is done writing:
var gfs = GFS(db);
var writeStream = gfs.createWriteStream(
{
filename: "thismyfile.txt",
root: "myfiles"
});
writeStream.on("finish", function()
{
console.log("finished");
response.send({ Success: true });
});
writeStream.write("this might work");
writeStream.end();
console.log("end");
In my console, I see "end", but never "finished", and there is never a response. The stream is writing properly, however, and it seems to be finishing (I see the completed file in the database). That even just isn't firing. I've tried moving the "this might work" into the call to end() and removing write(), I've also tried passing a string into end() as well. That string is written to the stream, but still no callback.
Why might this event not be getting fired?
Thank you.
The gridfs-stream module is designed and written primarily for node 0.8.x and below and does not use the stream2-style methods provided by require('stream').WritableStream in node >= 0.10.x. Because of this, it does not get the standardized finish event. It is up to the module implementation itself to emit finish, which it apparently does not.
I'm new to node.js.
I'm building a loop to query a database with chat messages every 3 seconds to then send required new messages to required users.
This is the loop I have - but currently it only loops once:
// New Database Chat Messages Send
var newDBMessagesInterval = 3000; // 3 Seconds
(function newDBMessagesSchedule() {
setTimeout(function() {
async(function() {
console.log('async is done!');
newDBMessagesSchedule();
});
}, newDBMessagesInterval)
})();
function async() {
console.log('in async function....');
}
Do I need to return something from the async function for the loop to continue?
Also is this a good/bad way to do a loop - my intention is to put a DB SELECT into the async function and don't what the DB calls to overlap.
Also is this non-blocking?
thx
There is nothing magical about aynchronous functions. You have to call the callback to an asynchronous function at some point. For testing purposes you should change async to:
function async(callback) {
callback();
}
Yes, this is non-blocking.
However, from your comment, I see that you're doing this to send messages to sockets. This isn't a great way to do that. You should look into getting a "pub/sub" system, and having each server subscribe and publish their own messages. Redis is a good choice for this.
I have the node.js code running on a server and would like to know if it is blocking or not. It is kind of similar to this:
function addUserIfNoneExists(name, callback) {
userAccounts.findOne({name:name}, function(err, obj) {
if (obj) {
callback('user exists');
} else {
// Add the user 'name' to DB and run the callback when done.
// This is non-blocking to here.
user = addUser(name, callback)
// Do something heavy, doesn't matter when this completes.
// Is this part blocking?
doSomeHeavyWork(user);
}
});
};
Once addUser completes the doSomeHeavyWork function is run and eventually places something back into the database. It does not matter how long this function takes, but it should not block other events on the server.
With that, is it possible to test if node.js code ends up blocking or not?
Generally, if it reaches out to another service, like a database or a webservice, then it is non-blocking and you'll need to have some sort of callback. However, any function will block until something (even if nothing) is returned...
If the doSomeHeavyWork function is non-blocking, then it's likely that whatever library you're using will allow for some sort of callback. So you could write the function to accept a callback like so:
var doSomHeavyWork = function(user, callback) {
callTheNonBlockingStuff(function(error, whatever) { // Whatever that is it likely takes a callback which returns an error (in case something bad happened) and possible a "whatever" which is what you're looking to get or something.
if (error) {
console.log('There was an error!!!!');
console.log(error);
callback(error, null); //Call callback with error
}
callback(null, whatever); //Call callback with object you're hoping to get back.
});
return; //This line will most likely run before the callback gets called which makes it a non-blocking (asynchronous) function. Which is why you need the callback.
};
You should avoid in any part of your Node.js code synchronous blocks which don't call system or I/O operations and which computation takes long time (in computer meaning), e.g iterating over big arrays. Instead move this type of code to the separate worker or divide it to smaller synchronous pieces using process.nextTick(). You can find explanation for process.nextTick() here but read all comments too.