I'm new to node.js.
I'm building a loop to query a database with chat messages every 3 seconds to then send required new messages to required users.
This is the loop I have - but currently it only loops once:
// New Database Chat Messages Send
var newDBMessagesInterval = 3000; // 3 Seconds
(function newDBMessagesSchedule() {
setTimeout(function() {
async(function() {
console.log('async is done!');
newDBMessagesSchedule();
});
}, newDBMessagesInterval)
})();
function async() {
console.log('in async function....');
}
Do I need to return something from the async function for the loop to continue?
Also is this a good/bad way to do a loop - my intention is to put a DB SELECT into the async function and don't what the DB calls to overlap.
Also is this non-blocking?
thx
There is nothing magical about aynchronous functions. You have to call the callback to an asynchronous function at some point. For testing purposes you should change async to:
function async(callback) {
callback();
}
Yes, this is non-blocking.
However, from your comment, I see that you're doing this to send messages to sockets. This isn't a great way to do that. You should look into getting a "pub/sub" system, and having each server subscribe and publish their own messages. Redis is a good choice for this.
Related
I'm building a socket.io Node JS application and my socket.io server will be listening for data from many socket.io clients, I need to save data to an API via my socket.io server as quickly as possible and figure that async/await is the best way forward.
Right now, I've got a function inside my .on('connection'), but is there a way I can make this an async function rather than have a nested function inside?
io.use((socket, next) => {
if (!socket.handshake.query || !socket.handshake.query.token) {
console.log('Authentication Error 1')
return
}
jwt.verify(socket.handshake.query.token, process.env.AGENT_SIGNING_SECRET, (err, decoded) => {
if (err) {
console.log('Authentication Error 2')
return
}
socket.decoded = decoded
next()
})
}).on('connection', socket => {
socket.on('agent-performance', data => {
async function savePerformance () {
const saved = await db.saveToDb('http://127.0.0.1:8000/api/profiler/save', data)
console.log(saved)
}
savePerformance()
})
})
Sort of, but you'll probably want to keep your current code if there can be multiple agent-performance events. You can modify the following, but it'd be messy and less readable. Event emitters still exist for a reason, they're not made obsolete by the introduction of promises. If it's performance you're after, your current code is probably faster and more resistant to backpressure and easier to error-handle.
events.on is a utility function that takes an event emitter (like socket) and returns an iterator that yields promises. You can await those with for await of.
events.once is a utility function that takes an event emitter (like socket) and returns a promise that resolves when the specified event is executed.
const { on, once } = require('events');
(async function() {
// This is an iterator that can emit infinite number of times.
const iterator = on(io, 'connection');
// Yield a promise, await it, run what is between `{ }` and repeat.
for await (const socket of iterator) {
const data = await once(socket, 'agent-performance');
const saved = await db.saveToDb(/* etc */);
}
})();
As the names imply, on is similar to socket.on and once is similar to socket.once. In the above example:
connected user 1, first agent-performance event: OK
connected user 1, second agent-performance event: not handled, there's no more event handler, since once is "used up".
connected user 2, first agent-performance event: OK
The documentation for on has a note about concurrency when using for await (x of on(...)), but I don't know if that would be a problem in your usecase.
// The execution of this inner block is synchronous and it
// processes one event at a time (even with await). Do not use
// if concurrent execution is required.
I'm using node express and postgress.
I'm not sure if what I'm trying to do is a good practice or a very big mistake.
Save data to database asynchronously after I already return a result to the client.
I tried to demonstrate it with console.log to check if my server will be blocked during the saving.
Here you can see status route and statusB route.
app.get("/statusB", async (req, res) => {
return res.status(200).send("testAAA");
});
app.get("/status", async (req, res) => {
const userStats = await UserController.getData("id")
const x = test();
return res.status(200).send(userStats);
});
async function test() {
return new Promise(() => {
for (let x = 0; x < 10000; x++) {
setTimeout( () => {
console.log(x)
}, 5000);
}
})
}
What should I want to happen is if I send /status and right after send statusB.
I expect the output to be:
/status will return userStats data
/StatusB return 'testAAA'
and the counter will run asynchronously.
But actual the output is:
- /status return userStats data
- The counter run
- /StatusB return 'testAAA' only after the counter finished
The console log is only test to know if I can fetching and saving data to the database asynchronously instead of the console log.
Depends on your business case.
If it's alright for your customer to get a 200 OK status code even if the saving might actually have failed, then sure, you can do it asynchronously after you've responded.
In other cases, you'll want to do the saving within the request and only respond after you're sure everything is safe and sound.
It's depending on your logic if you want for example to return the saved resource to the client you should wait (async/await or callback) until the data is saved to the database but for example, if you want just log an action without any returns to the frontend you can save it asynchronously
Yes, you should save data to db asynchronously, because of the way nodejs works. If you wait for an answer from db (synchronously), nodejs block event loop and doesn't handle new requests from clients. BUT if your business logic rely on the fact that you should return the answer from db to client, you should do it synchronously and maybe think about workarounds or choose another runtime, if that will become a problem.
Sorry, this one is a bit messy. My project is in nodejs. I have a test in mocha. In it I open a connection to geteventstore and subscribe to a stream. This essentially starts emitting events.
I wrap that event subscription in an rxjs observable and then write it to the console.
half of the time I get a stream full of events half of the time I don't.
I get the sense that the eventloop starts listening, doesn't hear anything and closes before the geteventstore can start blasting it with events.
I'm at a bit of a loss. I can tell the geteventstore is sending data cuz half the time I get it. My understanding is that as long as there is an someone is subscribed to an event, e.g. there is an eventlistener, the loop will stay open.
So perhaps the problem is with rxjs?
I don't know, any help would be greatly appreciated.
----EDIT
I don't know if this will help but the test looks like this.
context('when calling subscription', ()=> {
it('should stay open', function () {
mut = bootstrap.getInstanceOf('gesConnection');
var rx = bootstrap.getInstanceOf('rx');
var subscription = mut.subscribeToAllFrom();
rx.Observable.fromEvent(subscription, 'event').forEach(x=> console.log(x));
subscription.on('event', function (payload) {
console.log('event received by dispatcher');
console.log('event processed by dispatcher');
});
mut._handler._connectingPhase.must.equal('Connected');
})
});
so the mut is a connection to geteventstore, rx is rxjs, and the subscription object is an event emmiter that pumps data out of the geteventstore.
I understand that the problem is conflated by the fact that it deals wit at least two somewhat unusual products, the geteventstore, and the rxjs.
I mean I"m pretty confident that the gesConnection and subscription are, in fact, connecting and emitting. I just don't know how to test/investigate further.
Thanks
I don't see you making use of Mocha's async testing facilities.
MochaJs does not know that it should wait around for your test longer than it takes for your function to return.
Usually you'd return a promise:
it('must stay open', () => {
mut = bootstrap.getInstanceOf('gesConnection');
var rx = bootstrap.getInstanceOf('rx');
var subscription = mut.subscribeToAllFrom();
subscription.on('event', function (payload) {
console.log('event received by dispatcher');
console.log('event processed by dispatcher');
});
var promise = rx.Observable
.fromEvent(subscription, 'event')
.take(100) // stop test after 100 events
.do(x => console.log(x))
.finally(() => {
// do any cleanup here.
// such as close your connection
// or "subscription" variable
})
.toPromise();
mut._handler._connectingPhase.must.equal('Connected');
// tells Mocha to wait until the observable completes
return promise;
});
I am running a web application using express and nodejs. I have a request to a particular endpoint in which I use settimeout to call a particular function repeatedly after varying time intervals.
For example
router.get ("/playback", function(req, res) {
// Define callback here ...
....
var timeoutone = settimeout(callback, 1000);
var timeouttwo = settimeout(callback, 2000);
var timeoutthree = settimeout(callback, 3000);
});
The settimeout function returns an object with a circular reference. When trying to save this into mongodb i get a stack_overflow error. My aim is to be able to save these objects returned by settimeout into the database.
I have another endpoint called cancel playback which when called, will retrieve these timeout objects and call cleartimeout passing them in as an argument. How do I go about saving these timeout objects to the database ? Or is there a better way of clearing the timeouts than having to save them to the database. Thanks in advance for any help provided.
You cannot save live JavaScript objects in the database! Maybe you can store a string or JSON or similar reference to them, but not the actual object, and you cannot reload them later.
Edit: Also, I've just noticed you're using setTimeout for repeating stuff. If you need to repeat it on regular intervals, why not use setInterval instead?
Here is a simple solution, that would keep indexes in memory:
var timeouts = {};
var index = 0;
// route to set the timeout somewhere
router.get('/playback', function(req, res) {
timeouts['timeout-' + index] = setTimeout(ccb, 1000);
storeIndexValueSomewhere(index)
.then(function(){
res.json({timeoutIndex: index});
index++;
});
}
// another route that gets timeout indexes from that mongodb somehow
req.get('/playback/indexes', handler);
// finally a delete route
router.delete('/playback/:index', function(req, res) {
var index = 'timeout-' + req.params.index;
if (!timeouts[index]) {
return res.status(404).json({message: 'No job with that index'});
} else {
timeouts[index].cancelTimeout();
timeouts[index] = undefined;
return res.json({message: 'Removed job'});
}
});
But this probably would not scale to many millions of jobs.
A more complex solution, and perhaps more appropriate to your needs (depends on your playback job type) could involve job brokers or message queues, clusters and workers that subscribe to something they can listen to for their own job cancel signals etc.
I hope this helps you a little to clear up your requirements.
I have a node app that reads two files as streams. I use event.on('end') to then work with the results. The problem is I don't really know how I can wait for BOTH events to trigger 'end'.
What I have now is:
reader1.on('end', function(){
reader2.on('end',function(){
doSomething();
});
});
With small files this works, but if one of the files is very large the app aborts.
Your execution logic is somewhat flawed. You ought to do something like this instead
var checklist = [];
// checklist will contain sort of a counter
function reader_end(){
if(checklist.length == 2 )
// doSomething only if both have been added to the checklist
doSomething();
}
reader1.on('end', function() {
checklist.push('reader1');
// increment the counter
reader_end();
});
reader2.on('end', function() {
checklist.push('reader2');
reader_end();
});
Although there are libraries to better handle this sort of stuff, like Async and Promises.
With Async you'll need to use compose
var r12_done = async.compose(reader1.on, reader2.on);
r12_done('end', function(){
doSomething();
});
Edit: I just noticed that since probably reader1.on is a Stream 'end' event which doesn't have the standard callback argument signature of (err, results), this probably won't work. In that case you should just go with Promise.
With Promise you'll need to first Promisify and then join
var reader1Promise = Promise.promisify(reader1.on)('end');
var reader2Promise = Promise.promisify(reader2.on)('end');
var reader12Promise = Promise.join(reader1Promise, reader1Promise);
reader12Promise.then(function(){
doSomething();
});