integration test geteventstore using rxjs has race condition - node.js

Sorry, this one is a bit messy. My project is in nodejs. I have a test in mocha. In it I open a connection to geteventstore and subscribe to a stream. This essentially starts emitting events.
I wrap that event subscription in an rxjs observable and then write it to the console.
half of the time I get a stream full of events half of the time I don't.
I get the sense that the eventloop starts listening, doesn't hear anything and closes before the geteventstore can start blasting it with events.
I'm at a bit of a loss. I can tell the geteventstore is sending data cuz half the time I get it. My understanding is that as long as there is an someone is subscribed to an event, e.g. there is an eventlistener, the loop will stay open.
So perhaps the problem is with rxjs?
I don't know, any help would be greatly appreciated.
----EDIT
I don't know if this will help but the test looks like this.
context('when calling subscription', ()=> {
it('should stay open', function () {
mut = bootstrap.getInstanceOf('gesConnection');
var rx = bootstrap.getInstanceOf('rx');
var subscription = mut.subscribeToAllFrom();
rx.Observable.fromEvent(subscription, 'event').forEach(x=> console.log(x));
subscription.on('event', function (payload) {
console.log('event received by dispatcher');
console.log('event processed by dispatcher');
});
mut._handler._connectingPhase.must.equal('Connected');
})
});
so the mut is a connection to geteventstore, rx is rxjs, and the subscription object is an event emmiter that pumps data out of the geteventstore.
I understand that the problem is conflated by the fact that it deals wit at least two somewhat unusual products, the geteventstore, and the rxjs.
I mean I"m pretty confident that the gesConnection and subscription are, in fact, connecting and emitting. I just don't know how to test/investigate further.
Thanks

I don't see you making use of Mocha's async testing facilities.
MochaJs does not know that it should wait around for your test longer than it takes for your function to return.
Usually you'd return a promise:
it('must stay open', () => {
mut = bootstrap.getInstanceOf('gesConnection');
var rx = bootstrap.getInstanceOf('rx');
var subscription = mut.subscribeToAllFrom();
subscription.on('event', function (payload) {
console.log('event received by dispatcher');
console.log('event processed by dispatcher');
});
var promise = rx.Observable
.fromEvent(subscription, 'event')
.take(100) // stop test after 100 events
.do(x => console.log(x))
.finally(() => {
// do any cleanup here.
// such as close your connection
// or "subscription" variable
})
.toPromise();
mut._handler._connectingPhase.must.equal('Connected');
// tells Mocha to wait until the observable completes
return promise;
});

Related

How can I reassign the result of a listener function that is executed repeatedly?

I have a function which consumes a Queue from RabbitMQ and I want to save the result of every time it's executed to run some other code (e.g. save something in database).
The problem here is that it's only saved once, the listener keeps working fine and getting that information as the code within the function keeps being executed as I add more events to the queue but without reassigning the result of the execution to that variable:
Here's my code:
Controller (which calls the consumer)
async run() {
const eventData = await this.eventManager.consume(QueuesToConsume.USER_CREATED)
await this.createUserUseCase.run(eventData);
}
RabbitMQ consumer
async consume(queue: string): Promise<DomainEvent> {
let eventData: DomainEvent;
return new Promise<DomainEvent>(async (resolve, reject) => {
await this.channel.consume(queue, async (msg: Message) => {
console.log(`Message: \n ${Buffer.from(msg.content)} \n received successfully!`)
await this.channel.ack(msg)
eventData = JSON.parse(Buffer.from(msg.content).toString('utf8'))
console.log('Message acknowledged successfully')
resolve(eventData);
}).catch(err => {
console.log(`Error consuming the message: \n ${err}`)
reject(err)
});
})
}
So this is not working properly as eventData in the controller doesn't get every response and the useCase can only be executed the first time.
How can I fix this for eventData to get every result the consumer es returning?
PS: Note that I didn't copied the whole piece of code because it is not necessary, I can happily copy it if you need it to give me a proper answer!
I have found some other stack overflow forums that discussed this and in summary this can not be done, whether it is done with promises or with traditional async await, a variable can't just be reassigned to a value every time the listener listens to something.
I have found a workaround that allows me to achieve what I wanted at the very beginning using the Observer/Subject design pattern in which the RabbitMQ consumer that will be the subject will also notify the observers with the new event.
This way also helps a lot if you want that event to trigger two actions or usecases rather than one, you just have to add the new action as an observer and you're good to go!

Socket.io async/await for .on()

I'm building a socket.io Node JS application and my socket.io server will be listening for data from many socket.io clients, I need to save data to an API via my socket.io server as quickly as possible and figure that async/await is the best way forward.
Right now, I've got a function inside my .on('connection'), but is there a way I can make this an async function rather than have a nested function inside?
io.use((socket, next) => {
if (!socket.handshake.query || !socket.handshake.query.token) {
console.log('Authentication Error 1')
return
}
jwt.verify(socket.handshake.query.token, process.env.AGENT_SIGNING_SECRET, (err, decoded) => {
if (err) {
console.log('Authentication Error 2')
return
}
socket.decoded = decoded
next()
})
}).on('connection', socket => {
socket.on('agent-performance', data => {
async function savePerformance () {
const saved = await db.saveToDb('http://127.0.0.1:8000/api/profiler/save', data)
console.log(saved)
}
savePerformance()
})
})
Sort of, but you'll probably want to keep your current code if there can be multiple agent-performance events. You can modify the following, but it'd be messy and less readable. Event emitters still exist for a reason, they're not made obsolete by the introduction of promises. If it's performance you're after, your current code is probably faster and more resistant to backpressure and easier to error-handle.
events.on is a utility function that takes an event emitter (like socket) and returns an iterator that yields promises. You can await those with for await of.
events.once is a utility function that takes an event emitter (like socket) and returns a promise that resolves when the specified event is executed.
const { on, once } = require('events');
(async function() {
// This is an iterator that can emit infinite number of times.
const iterator = on(io, 'connection');
// Yield a promise, await it, run what is between `{ }` and repeat.
for await (const socket of iterator) {
const data = await once(socket, 'agent-performance');
const saved = await db.saveToDb(/* etc */);
}
})();
As the names imply, on is similar to socket.on and once is similar to socket.once. In the above example:
connected user 1, first agent-performance event: OK
connected user 1, second agent-performance event: not handled, there's no more event handler, since once is "used up".
connected user 2, first agent-performance event: OK
The documentation for on has a note about concurrency when using for await (x of on(...)), but I don't know if that would be a problem in your usecase.
// The execution of this inner block is synchronous and it
// processes one event at a time (even with await). Do not use
// if concurrent execution is required.

nodejs concurrency synchronous execution

I've two WebSockets getting data asynchronously, every time I get some message from the sockets I execute some code in CompareData.
The problem is that CompareData should be executed synchronously, or (better) only if it is not already running
This is my code:
function CompareData(data) {
console.log('data ', data);
AsyncFunction();
};
ws1 = new WebSocket(WS1_URL);
ws2 = new WebSocket(WS2_URL);
ws1.on('message', (data) => {
CompareData(data);
});
ws2.on('message', (data) => {
CompareData(data);
});
Can you help me, please? I'm very new to NodeJs
Node.js is single threaded. So you don't really get true concurrency issues occurring in Node programs as you might in other languages. In your example, there can only be at most one WebSocket callback for CompareData occurring at any given time.
You should not make synchronous call in node.js but you can make those call sequential. See below example might be helpful.
var messages = [];
var inProgress = false;
function CompareData(data) {
return new Promise((resolve, reject) => {
// do some stuff and resolve
setTimeout(() => {
resolve(data);
}, 1000);
});
};
const start = async () => {
if (!inProgress) {
if (messages.length !== 0) {
inProgress = true;
try {
const data = await CompareData(messages.shift());
console.log(data);
} catch (error) {
console.log(error);
}
inProgress = false;
await start();
}else{
console.log('Process Done');
}
}
}
const handler = (data) => {
messages.push(data);
start();
}
handler(1);
handler(2);
handler(3);
handler(4);
// ws1 = new WebSocket(WS1_URL);
// ws2 = new WebSocket(WS2_URL);
// ws1.on('message', handler);
// ws2.on('message', handler);
You should use some mutex in order to avoid that two async operations of compareData are executed at the same time, like node-mutex or mutexify.
My suggestions are:
First of all, you need to know when CompareData is finished. Reorganize your code to use promises or callbacks. If you're using third-party async functions, I'm almost sure they provide some feedback on completion - This is a must have in async world
Add inProgress = false flag somewhere to serve for you as simple lock. As someone posted, JS is single-threaded and you're guaranteed that your code won't get interrupted in the middle of operation. Thanks to that you can use really simple locks instead of complicated os-based mutexes known from multithreaded
langs.
In ws.on(...) check if inProgress is set. If not, lock it and run CompareData
In CompareData completion callback or on promise resolution set inProgress back to false, so you're no longer ignoring incoming data.
If you can simply discard the data, there is no need to complicate this scenario with extra queues, mutexes, etc.
If you need to serve it all, then queue incoming data and serve next piece after completion callback is fired.
This is basically what Rahul's suggests, but he uses features that are not established in current version of standard, so don't use it if you're not transpiling your code.

Node.js Async Loop to Query Database

I'm new to node.js.
I'm building a loop to query a database with chat messages every 3 seconds to then send required new messages to required users.
This is the loop I have - but currently it only loops once:
// New Database Chat Messages Send
var newDBMessagesInterval = 3000; // 3 Seconds
(function newDBMessagesSchedule() {
setTimeout(function() {
async(function() {
console.log('async is done!');
newDBMessagesSchedule();
});
}, newDBMessagesInterval)
})();
function async() {
console.log('in async function....');
}
Do I need to return something from the async function for the loop to continue?
Also is this a good/bad way to do a loop - my intention is to put a DB SELECT into the async function and don't what the DB calls to overlap.
Also is this non-blocking?
thx
There is nothing magical about aynchronous functions. You have to call the callback to an asynchronous function at some point. For testing purposes you should change async to:
function async(callback) {
callback();
}
Yes, this is non-blocking.
However, from your comment, I see that you're doing this to send messages to sockets. This isn't a great way to do that. You should look into getting a "pub/sub" system, and having each server subscribe and publish their own messages. Redis is a good choice for this.

How to close a readable stream (before end)?

How to close a readable stream in Node.js?
var input = fs.createReadStream('lines.txt');
input.on('data', function(data) {
// after closing the stream, this will not
// be called again
if (gotFirstLine) {
// close this stream and continue the
// instructions from this if
console.log("Closed.");
}
});
This would be better than:
input.on('data', function(data) {
if (isEnded) { return; }
if (gotFirstLine) {
isEnded = true;
console.log("Closed.");
}
});
But this would not stop the reading process...
Edit: Good news! Starting with Node.js 8.0.0 readable.destroy is officially available: https://nodejs.org/api/stream.html#stream_readable_destroy_error
ReadStream.destroy
You can call the ReadStream.destroy function at any time.
var fs = require("fs");
var readStream = fs.createReadStream("lines.txt");
readStream
.on("data", function (chunk) {
console.log(chunk);
readStream.destroy();
})
.on("end", function () {
// This may not been called since we are destroying the stream
// the first time "data" event is received
console.log("All the data in the file has been read");
})
.on("close", function (err) {
console.log("Stream has been destroyed and file has been closed");
});
The public function ReadStream.destroy is not documented (Node.js v0.12.2) but you can have a look at the source code on GitHub (Oct 5, 2012 commit).
The destroy function internally mark the ReadStream instance as destroyed and calls the close function to release the file.
You can listen to the close event to know exactly when the file is closed. The end event will not fire unless the data is completely consumed.
Note that the destroy (and the close) functions are specific to fs.ReadStream. There are not part of the generic stream.readable "interface".
Invoke input.close(). It's not in the docs, but
https://github.com/joyent/node/blob/cfcb1de130867197cbc9c6012b7e84e08e53d032/lib/fs.js#L1597-L1620
clearly does the job :) It actually does something similar to your isEnded.
EDIT 2015-Apr-19 Based on comments below, and to clarify and update:
This suggestion is a hack, and is not documented.
Though for looking at the current lib/fs.js it still works >1.5yrs later.
I agree with the comment below about calling destroy() being preferable.
As correctly stated below this works for fs ReadStreams's, not on a generic Readable
As for a generic solution: it doesn't appear as if there is one, at least from my understanding of the documentation and from a quick look at _stream_readable.js.
My proposal would be put your readable stream in paused mode, at least preventing further processing in your upstream data source. Don't forget to unpipe() and remove all data event listeners so that pause() actually pauses, as mentioned in the docs
Today, in Node 10
readableStream.destroy()
is the official way to close a readable stream
see https://nodejs.org/api/stream.html#stream_readable_destroy_error
You can't. There is no documented way to close/shutdown/abort/destroy a generic Readable stream as of Node 5.3.0. This is a limitation of the Node stream architecture.
As other answers here have explained, there are undocumented hacks for specific implementations of Readable provided by Node, such as fs.ReadStream. These are not generic solutions for any Readable though.
If someone can prove me wrong here, please do. I would like to be able to do what I'm saying is impossible, and would be delighted to be corrected.
EDIT: Here was my workaround: implement .destroy() for my pipeline though a complex series of unpipe() calls. And after all that complexity, it doesn't work properly in all cases.
EDIT: Node v8.0.0 added a destroy() api for Readable streams.
At version 4.*.* pushing a null value into the stream will trigger a EOF signal.
From the nodejs docs
If a value other than null is passed, The push() method adds a chunk of data into the queue for subsequent stream processors to consume. If null is passed, it signals the end of the stream (EOF), after which no more data can be written.
This worked for me after trying numerous other options on this page.
This destroy module is meant to ensure a stream gets destroyed, handling different APIs and Node.js bugs. Right now is one of the best choice.
NB. From Node 10 you can use the .destroy method without further dependencies.
You can clear and close the stream with yourstream.resume(), which will dump everything on the stream and eventually close it.
From the official docs:
readable.resume():
Return: this
This method will cause the readable stream to resume emitting 'data' events.
This method will switch the stream into flowing mode. If you do not want to consume the data from a stream, but you do want to get to its 'end' event, you can call stream.resume() to open the flow of data.
var readable = getReadableStreamSomehow();
readable.resume();
readable.on('end', () => {
console.log('got to the end, but did not read anything');
});
It's an old question but I too was looking for the answer and found the best one for my implementation. Both end and close events get emitted so I think this is the cleanest solution.
This will do the trick in node 4.4.* (stable version at the time of writing):
var input = fs.createReadStream('lines.txt');
input.on('data', function(data) {
if (gotFirstLine) {
this.end(); // Simple isn't it?
console.log("Closed.");
}
});
For a very detailed explanation see:
http://www.bennadel.com/blog/2692-you-have-to-explicitly-end-streams-after-pipes-break-in-node-js.htm
This code here will do the trick nicely:
function closeReadStream(stream) {
if (!stream) return;
if (stream.close) stream.close();
else if (stream.destroy) stream.destroy();
}
writeStream.end() is the go-to way to close a writeStream...
for stop callback execution after some call,
you have to use process.kill with particular processID
const csv = require('csv-parser');
const fs = require('fs');
const filepath = "./demo.csv"
let readStream = fs.createReadStream(filepath, {
autoClose: true,
});
let MAX_LINE = 0;
readStream.on('error', (e) => {
console.log(e);
console.log("error");
})
.pipe(csv())
.on('data', (row) => {
if (MAX_LINE == 2) {
process.kill(process.pid, 'SIGTERM')
}
// console.log("not 2");
MAX_LINE++
console.log(row);
})
.on('end', () => {
// handle end of CSV
console.log("read done");
}).on("close", function () {
console.log("closed");
})

Resources