i'am using nodejs with express for my webapp and i need to to run continuously
some code which checks if some data change and then update my mongodb.
How can i easily create a background process which runs the whole time together with the main task? So that the background task/process can inform the main task.
What i have tried already:
to solve this problem with a "setInterval" Function in the main process --> I works with no problem but think its not a good idea because it blocks the node event loop
Use child processes -> i could not found a good tutorial on them --> is there a easier method, perhaps a library which could help me?
some background worker libraries -->But do heavy-load tasks on the a child-process and finish but i need to do the work all the time
Update:
Final Solution:
UpdateEvent.js:
var events = require('events');
function Updater(time) {
this.time = time;
this.array = [
{number: 1},
{number: 2}
];
var that;
events.EventEmitter.call(this);
this.init = function()
{
that = this;
console.log("Contructor");
//Start interval
setInterval(that.run,that.time);
};
this.run = function()
{
that.array.forEach(function (item) {
if(item.number === 2)
{
that.emit('Event');
}
});
};
}
Updater.prototype.__proto__ = events.EventEmitter.prototype;
module.exports = Updater;
and then the code that uses it:
server.js:
var Updater = require('./UpdaterEvent');
var u = new Updater(10000);
u.init();
u.on('Event',function () {
console.log("Event catched!");
});
I followed the tutorial at:
http://www.sitepoint.com/nodejs-events-and-eventemitter/
The problem is the way you export your Updater constructor function:
exports.Updater = Updater;
When you require it, you do
var Updater = require('./UpdaterEvent');
and then try to run:
var u = new Updater(10000);
The problem is that you do not expose the function itself, but an object with a property called Updater which contains the function. Hence you either have to export it using
module.exports = Updater;
or you have to require it using:
var Updater = require('./UpdaterEvent').Updater;
Either way, then calling new Updater() will work. At the moment, you try to initialize a new object by calling an object instead of a constructor function, hence the error message:
TypeError: object is not a function
You should look into Events and EventEmitter
You could use child-process you don't really need to since JS is asynchronous. Just create a function for your background process and pass it your eventEmitter object. You can use setInterval or a while(true) loop to continuously check for the data change. When the data changes, call eventEmitter.emit('someEvent'); which will trigger a function in your main task to update your mongoDB.
Related
TL;DR: is there a way to wait for a module import with async functionality to complete before continuing with execution in the calling module in order to keep module functionality contained?
I'm working on a personal node project that I've been structuring in a modular/OOP way as the codebase has continued to grow. One requirement has been to enable logging across modules / objects, where different logfiles can be logged to at different times. I thought that I had solved the problem in a pretty clean way by creating a Logger.js file with an init function that I could use at any time by simply importing the Logger.js file in any module that I needed. Here is the stripped down code to illustrate this:
Logger.js
module.exports.init = function(location) {
var logFileBaseName = basePath + fullDatePathName;
var studentLogFile = fs.createWriteStream(logFileBaseName + '-student.log', {flags : 'a'});
var teacherLogFile = fs.createWriteStream(logFileBaseName + '-teacher.log', {flags : 'a'});
this.studentLog = function () {
arguments[0] = '[' + Utils.getFullDate() + '] ' + arguments[0].toString();
studentLogFile.write(util.format.apply(null, arguments) + '\n');
}
this.teacherBookLog = function () {
arguments[0] = '[' + Utils.getFullDate() + '] ' + arguments[0].toString();
teacherLogFile.write(util.format.apply(null, arguments) + '\n');
}
}
This seemed great, because in my main entrypoint I could simply do:
Main.js
const Logger = require('./utils/Logger');
Logger.init(path);
Logger.studentLog('test from Main');
// all my other code and more logging here
And in my other dozens of files I could do even less:
AnotherFile.js
const Logger = require('./utils/Logger');
Logger.studentLog('test from AnotherFile')
Then the requirement came to log not only to a file for the 'student logs', but to Discord (a chat client) as well. Seemed easy, I had this Logger file and I could just initialize Discord and log to Discord alongside the 'student logs', something like this:
Logger.js
module.exports.init = function(location) {
// code we've already seen above
var client = new Discord.Client();
client.login('my_login_string');
channels = client.channels;
this.studentLog = function () {
arguments[0] = '[' + Utils.getFullDate() + '] ' + arguments[0].toString();
var message = util.format.apply(null, arguments) + '\n';
studentLogFile.write(message);
channels.get('the_channel_to_log_to').send(message)
}
// more code we've already seen above
}
The problem is that if you were to re-run Main.js again, the studentLog would fail because the .login() function is asynchronous, it returns a Promise. The login has not completed and channels would be an empty Collection by the time we try to call Logger.studentLog('test from Main');
I've tried using a Promise in Logger.js, but of course execution of Main.js continues before the promise returns in Logger.js. I would love it if Main.js could simply wait until the Discord login was complete.
My question is, what is the best way to make this work while keeping with the pattern I've been using? I know that I could wrap my entire main function in a promise.then() that waits for Discord login to complete, but that seems a bit absurd to me. I'm trying to keep functionality contained into modules and would not like for this kind of Logger code / logic to spill out into my other modules. I want to keep it to a simple Logger import as I've been doing.
Any advice would be great!!
If the result of some async function is awaited and then used in the same caller function, the result is resolved first, then used. If the result is used in another function or module (e.g. the result is assigned to a global variable), it is not resolved. In your case, if client.login() assigns a value to client.channels asynchronously, that assignment is not awaited, and channels = client.channels assignment will assign undefined to channels.
To resolve this issue, you must use a callback or return a promise from client.login(), as stated in the comments.
You can refer to this article.
Let me offer my solution to the "asynchronously initialised logger" problem. Note that this only deals with logging and most likely cannot be generalised.
Basically, all messages are appended to a queue that is only sent to the remote location once a flag inidicating that the connection is ready is set.
Example:
//Logger.js
module.exports = {
_ready: false,
_queue: [],
init(): {
return connectToRemote().then(()=>{this._ready = true})
},
log(message): {
console.log(message);
_queue.push(message)
if (this._ready) {
let messagesToSend = this._queue;
this._queue = [];
this._ready = false;
sendToRemote(messagesToSend).then(()=>this._ready = true);
}
}
}
You can require the logger in any file and use the log funciton right away. The logs will be sent only after the init funciton that you can call anytime is resolved.
This is a very bare bones example, you may also want to limit the queue size and/or only send the logs in bulk in certain time intervals, but you get the idea.
So, I'm stilling learning the JS/Node way from a long time in other languages.
I have a tiny micro-service that reads from a redis channel, temp stores it in a working channel, does the work, removes it, and moves on. If there is more in the channel it re-runs immediately. If not, it sets a timeout and checks again in 1 second.
It works fine...but timeout polling doesn't seem to be the "correct" way to approach this. And I haven't found much about using BRPOPLPUSH to try to block (vs. RPOPLPUSH) and wait in Node....or other options like that. (Pub/Sub isn't an option here...this is the only listener, and it may not always be listening.)
Here's the short essence of what I'm doing:
var Redis = require('ioredis');
var redis = new Redis();
var redisLoop = function () {
redis.rpoplpush('channel', 'channel-working').then(function (result) {
if (result) {
processJob(result); //do stuff
//delete the item from the working channel, and check for another item
redis.lrem('channel-working', 1, result).then(function (result) { });
redisLoop();
} else {
//no items, wait 1 second and try again
setTimeout(redisLoop, 1000);
}
});
};
redisLoop();
I feel like I'm missing something really obvious. Thanks!
BRPOPLPUSH doesn't block in Node, it blocks in the client. In this instance I think it's exactly what you need to get rid of the polling.
var Redis = require('ioredis');
var redis = new Redis();
var redisLoop = function () {
redis.brpoplpush('channel', 'channel-working', 0).then(function (result) {
// because we are using BRPOPLPUSH, the client promise will not resolve
// until a 'result' becomes available
processJob(result);
// delete the item from the working channel, and check for another item
redis.lrem('channel-working', 1, result).then(redisLoop);
});
};
redisLoop();
Note that redis.lrem is asynchronous, so you should use lrem(...).then(redisLoop) to ensure that your next tick executes only after the item is successfully removed from channel-working.
var assert = require('assert');
var parseJSON = require('json-parse-async');
var contact = new Object();
contact.firstname = "Jesper";
contact.surname = "Aaberg";
contact.phone = ["555-0100", "555-0120"];
var contact2 = new Object();
contact2.firstname = "JESPER";
contact2.surname = "AABERG";
contact2.phone = ["555-0100", "555-0120"];
contact.toJSON = function(key) {
var replacement = new Object();
for (var val in this) {
if (typeof(this[val]) === 'string')
replacement[val] = this[val].toUpperCase();
else
replacement[val] = this[val]
}
return replacement;
};
var jsonText = JSON.stringify(contact);
contact = JSON.parse(jsonText);
console.log(contact);
console.log(contact2);
assert.deepEqual(contact, contact2, 'these two objects are the same');
What are the asynchronous equivalent functions of JSON.parse, JSON.stringify and assert.deepEqual? I am trying to create a race condition and non-deterministic behavior within the following code but I have not been able lto find non-blocking, asynchronous equivalents of the functions mentioned above.
node.js does not have an actual asynchronous JSON parser built-in. If you want something that will actually do the parsing outside the main node.js Javascript thread, then you would have to find a third party module that parses the JSON outside of the Javascript thread (e.g. in a native code thread or in some other process). There are some modules in NPM that advertise themselves as asynchronous such as async-json-parser or async-json-parse or json-parse-async. You would have to verify that whichever implementation you were interested in was truly an asynchronous implementation (your Javascript continues to run while the parsing happens in the background).
But, reading the detail in your question about the problem you're trying to solve, it doesn't sound like you actually need a parser that truly happens in the background. To give you your ability to test what you're trying to test, it seems to me like you just need an indeterminate finish that allows other code to run before the parsing finishes. That can be done by wrapping the synchronous JSON.parse() in a setTimeout() with a promise that has a random delay. That will give some random amount of time for other code to run (to try to test for your race conditions). That could be done like this:
JSON.parseAsyncRandom = function(str) {
return new Promise(function(resolve, reject) {
// use a random 0-10 second delay
setTimeout(function() {
try {
resolve(JSON.parse(str));
} catch(e) {
reject(e);
}
}, Math.floor(Math.random() * 10000));
});
}
JSON.parseAsyncRandom(str).then(function(obj) {
// process obj here
}, function(err) {
// handle err here
});
Note: This is not true asynchronous execution. It's an asynchronous result (in that it arrives some random time later and other code will run before the result arrives), but true asynchronous execution happens in the background in parallel with other JS running and this isn't quite that. But, given your comment that you just want variable and asynchronous results for testing purposes, this should do that.
I've recently faced this problem myself, so I decided to create a library to handle JSON parsing in a really asynchronous way.
The idea behind it is to divide the parsing process into chunks, and then run each separately in the event loop so that other events (user interactions, etc) can still be evaluated within a few milliseconds, keeping the UI interactive.
If you are interested, the library it's called RAJI and you can find it here: https://github.com/federico-terzi/raji
After installing RAJI, you can then convert your synchronous JSON.parse calls into async raji.parse calls, such as:
const object = await parse(payload);
These calls won't block the UI
You can use 'bluebird', like this example to convert calling function to promise.
I write code below using javascript es6.
const Promise = require('bluebird')
function stringifyPromise(jsonText) {
return Promise.try(() => JSON.stringify(jsonText))
}
function parsePromise(str) {
return Promise.try(() => JSON.parse(str))
}
stringifyPromise(contact)
.then(jsonText => parsePromise(jsonText))
.then(contact => {
assert.deepEqual(contact, contact2, 'these two objects are the same')
})
})
My use case is to read RSS feed items asynchronously and load them into a meteor collection.
I have the feedparser npm module that does the parsing. It emits three events .on('error'), .on('meta') and .on('readable) with three different outputs.
When I run it in fixtures.js, with just console.log statements to run the output, its working fine.
When I use the same code to insert into a collection, I get errors related to asynchronocity of the function (assuming something to do with fibers)
So, I want to make it into a meteor method using futures as below -
http://www.discovermeteor.com/patterns/5828399
I tried but could not wrap my head around handling multiple events in Futures.
If you just want to push something to db at one point, it's enough to synchronize this call. Other than that, you can do whatever you want asynchronously. For example:
var Fiber = Npm.require('fibers');
var item = {};
var onInit = function() {
// do whatever with item
};
var onData = function() {
// do whatever with item
};
var onFinish = function() {
new Fiber(function(){
Documents.insert(item);
}).run();
};
Although Meteor is a great tool, I think node and its async insight is brilliant, and the best tool for what you are doing. Keep as a plan b having this part of your project be a straight node app.
Otherwise,
async from meteor
and
Currently I have a Node.js module with the following form :
var events = require('events');
var emitter = new events.EventEmitter();
function emitSomething() {
emitter.emit("event");
}
exports.emitSomething = emitSomething;
exports.on = emitter.on;
However any callback registered through on do not get called when I call emitSomething.
I can get around this by changing the last line to
exports.on = function(event, callback) { emitter.on(event, callback); };
Is there a reason I can't delegate the on function to a function defined in another module?
This is a common JS mistake. Note that the on method depends on this (which is emitter in your example). However, all you export is the function itself, so it's context (emitter) is lost when you call it later.
Here is a small example for this issue:
var someObj = {
doSth: function() { console.log(this) }
};
someObj.doSth(); // "Object { ..."
var doSth = someObj.doSth;
doSth(); // "Window { ..."
When I call doSth as method of someObj, this references someObj as expected. After I copied the function to doSth, this references it's new context, in the browser the global context Window. This is what happens in your case.
As you already mentioned, you have to bind the emitter context to the function. This can also be done like this:
exports.on = emitter.on.bind(emitter);