How to implement rabbitMQ into node.js microservice app right way? - node.js

Greetings Stackoverflow.
I've been using stackoverflow for years to find answers, and this is my first attempts to make a question myself. So feel free to tell me if I'm doing it wrong way.
Currently I'm developing a data analytical system based on microservice architecture.
It is assumed that this system will consist of a dozen self-sufficient microservices communicating with each other by RabbitMQ. Each of them is encapsulated in a docker-container and the whole system is powered by docker-swarm in the production.
In particular each microservice is a node.js application and related database, connected with some ORM interface. Its task is to manage and serve data in a CRUD manner, and to provide results of some prepared queries based on the contained data. Nothing extraordinary.
To provide microservice-microservice communication I assume to use amqplib. But the way to use it is uncertain yet.
My current question is how to make use of amqplib in a OOP manner to link inter microservice communication network with application's object-related functionality? By OOP manner, I mean the possibility to replace amqplib (and RabbitMQ itself) in the long run without the need to make changes to the data-related logic.
What I really searching for is the example of currently working microservice application utilizing AMQP. I'd pretty much appreciate that if somebody could give a link to it.
And the second part of my question.
Does it make sense to build microservice application based on event-driven principals, and just pass messages from RabbitMQ to the application's main event queue? So that each procedure would be called the same way, despite the fact that it is an internal or external event.
As for the abstract example of single microservice:
Let's say I have an event service and a listener connected to this service:
class UserManager {
constructor(eventService) {
this.eventService = eventService;
this.eventServce.on("users.user.create-request", (payload) => {
User.create(payload); // User interface is omitted in this example
}
}
}
const eventService = new EventEmmiter();
const userManager = new UserManager(eventService);
On the other hand I've got RabbitMQ connection, that is waiting for messages:
const amqp = require('amqplib');
amqp.connect('amqp-service-in-docker').then(connection => {
connection.createChannel().then(channel => {
// Here we use topic type of exchange to be able to filter only related messages
channel.assertExchange('some-exchange', 'topic');
channel.assertQueue('').then(queue => {
// And here we are waiting only the related messages
channel.bind(queue.queue, 'some-exchange', 'users.*');
channel.consume(queue.queue, message => {
// And here is the crucial part
}
}
}
}
What I'm currently think off is to just parse and forward this message to eventService and use it's routing key as the name of the event, like this:
channel.consume(query.query, message => {
const eventName = message.fields.routingKey;
const eventPayload = JSON.parse(message.content.toString());
eventService.emit(eventName, eventPayload);
}
But how about RPC's? Should I make another exchange or even a channel for them with another approach, something like:
// In RPC channel
channel.consume(query.query, message => {
eventService.once('users.user.create-response', response => {
const recipient = message.properites.replyTo;
const correlationId = msg.properties.correlationId;
// Send response to specified recipient
channel.sendToQueue(
recipient,
Buffer.from(JSON.stringify(resonse)),
{
correlationId: correlationId
}
);
channel.ack(message);
});
// Same thing
const eventName = message.fields.routingKey;
const eventPayload = JSON.parse(message.content.toString());
eventService.emit(eventName, eventPayload);
}
And then my User class should fire 'users.user.create-response' event every time it creates a new user. Isn't this a crutch?

Related

Why am I receiving this error on Azure when using eventhubs?

I started using Azure recently and It has been an overwhelming experience. I started experimenting with eventhubs and I'm basically following the official tutorials on how to send and receive messages from eventhubs using nodejs.
Everything worked perfectly so I built a small web app (static frontend app) and I connected it with a node backend, where the communication with eventhubs occurs. So basically my app is built like this:
frontend <----> node server <-----> eventhubs
As you can see it is very simple. The node server is fetching data from eventhubs and sending it forward to the frontend, where the values are shown. It is a cool experience and I'm enjoying MS Azure until this error occured:
azure.eventhub.common.EventHubError: ErrorCodes.ResourceLimitExceeded: Exceeded the maximum number of allowed receivers per partition in a consumer group which is 5. List of connected receivers - nil, nil, nil, nil, nil.
This error is really confusing. Im using the default consumer group and only one app. I never tried to access this consumer group from another app. It said the limit is 5, I'm using only one app so it should be fine or am I missing something? I'm not checking what is happening here.
I wasted too much time googling and researching about this but I didn't get it. At the end, I thought that maybe every time I deploy the app (my frontend and my node server) on azure, this would be counted as one consumer and since I deployed the app more than 5 times then this error is showing up. Am I right or this is nonsense?
Edit
I'm using websockets as a communication protocol between my app (frontend) and my node server (backend). The node server is using the default consumer group ( I didn't change nothing), I just followed this official example from Microsoft. I'm basically using the code from MS docs that's why I didn't post any code snippet from my node server and since the error happens in backend and not frontend then it will not be helpful if I posted any frontend code.
So to wrap up, I'm using websocket to connect front & backend. It works perfectly for a day or two and then this error starts to happen. Sometimes I open more than one client (for example a client from the browser and client from my smartphone).
I think I don't understand the concept of this consumer group. Like is every client a consumer? so if I open my app (the same app) in 5 different tabs in my browser, do I have 5 consumers then?
I didn't quite understand the answer below and what is meant by "pooling client", therefore, I will try to post code examples here to show you what I'm trying to do.
Code snippets
Here is the function I'm using on the server side to communicate with eventhubs and receive/consume a message
async function receiveEventhubMessage(socket, eventHubName, connectionString) {
const consumerClient = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);
const subscription = consumerClient.subscribe({
processEvents: async (events, context) => {
for (const event of events) {
console.log("[ consumer ] Message received : " + event.body);
io.emit('msg-received', event.body);
}
},
processError: async (err, context) => {
console.log(`Error : ${err}`);
}
}
);
If you notice, I'm giving the eventhub and connection string as an argument in order to be able to change that. Now in the frontend, I have a list of multiple topics and each topic have its own eventhubname but they have the same eventhub namespace.
Here is an example of two eventhubnames that I have:
{
"EventHubName": "eh-test-command"
"EventHubName": "eh-test-telemetry"
}
If the user chooses to send a command (from the frontend, I just have a list of buttons that the user can click to fire an event over websockets) then the CommandEventHubName will be sent from the frontend to the node server. The server will receive that eventhubname and switch the consumerClient in the function I posted above.
Here is the code where I'm calling that:
// io is a socket.io object
io.on('connection', socket => {
socket.on('onUserChoice', choice => {
// choice is an object sent from the frontend based on what the user choosed. e.g if the user choosed command then choice = {"EventhubName": "eh-test-command", "payload": "whatever"}
receiveEventhubMessage(socket, choice.EventHubName, choice.EventHubNameSpace)
.catch(err => console.log(`[ consumerClient ] Error while receiving eventhub messages: ${err}`));
}
}
The app I'm building will be extending in the future to a real use case in the automotive field, that's why this is important for me. Therefore, I'm trying to figure out how can I switch between eventhubs without creating a new consumerClient each time the eventhubname changes?
I must say that I didn't understand the example with the "pooling client". I am seeking more elaboration or, ideally, a minimal example just to put me on the way.
Based on the conversation in the issue, it would seem that the root cause of this is that your backend is creating a new EventHubConsumerClient for each request coming from your frontend. Because each client will open a dedicated connection to the service, if you have more than 5 requests for the same Event Hub instance using the same consumer group, you'll exceed the quota.
To get around this, you'll want to consider pooling your EventHubConsumerClient instances so that you're starting with one per Event Hub instance. You can safely use the pooled client to handle a request for your frontend by calling subscribe. This will allow you to share the connection amongst multiple frontend requests.
The key idea being that your consumerClient is not created for every request, but shares an instance among requests. Using your snippet to illustrate the simplest approach, you'd end up hoisting your client creation to outside the function to receive. It may look something like:
const consumerClient = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);
async function receiveEventhubMessage(socket, eventHubName, connectionString) {
const subscription = consumerClient.subscribe({
processEvents: async (events, context) => {
for (const event of events) {
console.log("[ consumer ] Message received : " + event.body);
io.emit('msg-received', event.body);
}
},
processError: async (err, context) => {
console.log(`Error : ${err}`);
}
}
);
That said, the above may not be adequate for your environment depending on the architecture of the application. If whatever is hosting receiveEventHubMessage is created dynamically for each request, nothing changes. In that case, you'd want to consider something like a singleton or dependency injection to help extend the lifespan.
If you end up having issues scaling to meet your requests, you can consider increasing the number of clients for each Event Hub and/or spreading requests out to different consumer groups.

Is there a better solution than socket.io for slow-speed in-game chat?

I am creating a browser game with node.js (backend api) and angular (frontend). My goal is to implement an in-game chat to allow communication between players on the same map. The chat is not an essential part of the game, so messages don't need to be instant (few seconds of latency should be ok). It is just a cool feature to talk some times together.
A good solution should be to implement socket.io to have real-time communication. But as chat is not an essential component and is the only thing which will require websockets, i'm wondering if there is not an alternative to avoid server overload with sockets handling.
I thinked about polling every 2 or 3 seconds my REST API to ask for new messages, but it may overload server the same way... What are your recommandations?
Thank you for your advices
There's a pretty cool package called signalhub. It has a nodejs server component and stuff you can use in your users' browsers. It uses a not-so-well-known application of the http (https) protocol called EventSource. EventSource basically opens persistent http (https) connections to a web server.
It's a reliable and lightweight setup. (The README talks about WebRTC signalling, but it's useful for much more than that.)
On the server side, a simple but effective server setup might look like this:
module.exports = function makeHubServer (port) {
const signalhubServer = require('signalhub/server')
const hub = signalhubServer({ maxBroadcasts: 0 })
hub.on('subscribe', function (channel) {
/* you can, but don't have to, keep track of subscriptions here. */
})
hub.on('publish', function (channel, message) {
/* you can, but don't have to, keep track of messages here. */
})
hub.listen(port, null, function () {
const addr = hub.address()
})
return hub
}
In a browser you can do this sort of thing. It user GET to open a persistent EventSource to receive messages. And, when it's time to send a message, it POSTs it.
And, Chromium's devtools Network tab knows all about EventSource connections.
const hub = signalhub('appname', [hubUrl])
...
/* to receive */
hub.subscribe('a-channel-name')
.on('data', message => {
/* Here's a payload */
console.log (message)
})
...
/* to send */
hub.broadcast('a-channel-name', message)

Microservices Create Entity Implementation

This is a follow-up question to my issue outlined here.
The Gateway serves as an entry point to the application, to which every request from the client is made. The gateway then allocates the request to the responsible microservices and also handles authentication.
In this case the gateway listens for HTTP POST /bok and notifies the Microservice A to create a book. Thus Microservice A ist responsible for managing and storing everything about the book entity.
The following pseudo-code is a simplified implementation of this architecture:
Queue Communication
Gateway
router.post('/book', (req, res) => {
queue.publish('CreateBook', req.body);
queue.consume('BookCreated', (book) => {
const user = getUserFromOtherMicroService(book.userId);
res.json({ book, user });
});
});
Microservcie A
queue.consume('CreateBook', (payload) => {
const book = createBook(payload);
eventStore.insert('BookCreated', book);
const createdBook = updateProjectionDatabase(book);
queue.publish('BookCreated', createdBook);
})
But I am not quite sure about this because of the following reasons:
The listener for consuming BookCreated in the Gateway will be recreated every time a user requests to create a new book
What if 2 users simultaneously create a book and the wrong book will be returned?
I don't know how to fetch additional data (e.g. getUserFromOtherMicroService)
That's why I though about implementing this architecture:
Direct and Queue Communication
Gateway
router.post('/book', async (req, res) => {
const book = await makeHttpRequest('microservice-a/create-book', req.body);
const user = await makeHttpRequest('microservice-b/getUser', book.userId);
res.json({ book, user });
});
Microservice A
router.post('/create-book', (req, res) => {
const book = createBook(req.body);
eventStore.insert('BookCreated', book);
const createdBook = updateProjectionDatabase(book);
queue.publish('BookCreated', createdBook);
res.json(createdBook);
})
But I am also not really sure about this implementation because:
Don't I violate CQRS when I return the book after creation? (because I should only return OK or ERROR)
Isn't it inefficient to make another HTTP request in a microservices system?
Based on the comments above .
Approach 1
In this case your api gateway will be used to drop the message in the queue. This approach is more appropriate if your process is going to take long time and you have a queue workers sitting behind to pick up the messages and process. But your client side has to poll to get the results. Say you are looking for airline ticket . You drop the message. Your get an ID to poll. Your client will keep on polling until the results are available.
But in this case you will have a challenge , as you drop the message how you are going to generate the ID that client would poll ? Do you assign the ID to message at Gateway and drop in the queue and return the same ID for the client to poll to get result ? again this approach is good for web/worker kind of scenario.
Approach 2
Since Your API gateway is custom application that would handle the authentication and redirect the request to respective service. Your Microsvc A would create book and publish the event and your Microservice B and C would be using it . Your Gateway will wait for the Microservice A to return response with ID (or event metadata of newly created object) of book that is created so you don't poll for it later and client has it. If you want you can have additional information from other microservices you can fetch at this time and can send aggregated response.
For any data that is available in Microservice A,B,C you will be getting via Gateway. Make sure your gateway is highly available.
Hope that helps . Let me know if you have any questions !

Unique configuration per vhost for Micro

I have a few Zeit micro services. This setup is a RESTful API for multiple frontends/domains/clients
I need to, in my configs that are spread throughout the apps, differentiate between these clients. I can, in my handlers, setup a process.env.CLIENT_ID for example that I can use in my config handler to know which config to load. However this would mean launching a new http/micro process for each requesting domain (or whatever method I use - info such as client id will prob come in a header) in order to maintain the process.env.CLIENT_ID throughout the request and not have it overwritten by another simultaneous request from another client.
So I have to have each microservice check the client ID, determine if it has already launched a process for that client and use that else launch a new one.
This seems messy but not sure how else to handle things. Passing the client id around with code calls (i.e. getConfg(client, key) is not practical in my situation and I would like to avoid that.
Options:
Pass client id around everywhere
Launch new process per host
?
Is there a better way or have I made a mistake in my assumptions?
If the process per client approach is the better way I am wondering if there is an existing solution to manage this? Ive looked at http proxy, micro cluster etc but none seem to provide a solution to this issue.
Well I found this nice tool https://github.com/othiym23/node-continuation-local-storage
// Micro handler
const { createNamespace } = require('continuation-local-storage')
let namespace = createNamespace('foo')
const handler = async (req, res) => {
const clientId = // some header thing or host
namespace.run(function() {
namespace.set('clientId', clientId)
someCode()
})
})
// Some other file
const { getNamespace } = require('continuation-local-storage')
const someCode = () => {
const namespace = getNamespace('foo')
console.log(namespace.get('clientId'))
}

Realtime messaging with NodeJS across multiple processes

I'm trying to implement an API that interacts with a NodeJS server for realtime messaging. Now when that NodeJS app is deployed to a scalable environment like Heroku, multiple instances of this app may be running.
Is it possible to design the node app so that all clients subscribed to a "message channel" will receive this message, although multiple node instances are running - and therefore multiple copies of this channel?
Check out zeromq, it should provide some simple, high performance IPC abstractions to do what you want. In particular, the pub/sub example will be useful.
The main challenge as I imagine it, without knowing anything about how Heroku spawns multiple server instances, will be the logic to determine who is the publisher (the rest of the instances will be subscribers). So let's say, for argument's sake, that your hosting provider gives you an environment variable called INSTANCE_NUM which is an integer in [0,1024] indicating the instance number of the process; so we'll say that instance zero is the message publisher.
var zmq = require('zeromq')
if (process.env['INSTANCE_NUM'] === '0') { // I'm the publisher.
var emitter = getEventEmitter(); // e.g. an HttpServer.
var pub = zmq.createSocket('pub');
pub.bindSync('tcp://*:5555');
emitter.on('someEvent', function(data) {
pub.send(data);
});
} else { // I'm a subscriber.
var sub = zmq.createSocket('sub');
sub.subscribe('');
sub.on('message', function(data) {
// Handle the event data...
});
sub.connect('tcp://localhost:5555');
}
Note that I'm new to zeromq and the above code is totally untested, just for demonstration.

Resources