I am using Rascal.Js(it uses amqplib) for my messaging logic with rabbitMq on node.js app.
I am using something similar to their example on my project startup, which creates a permanent instance and "registers" all of my subscribers and redirects messages when they arrive to the queue (in the background).
My issue is with the publishers. There are http requests from outside which should trigger my publishers. A user clicks on create button of sorts which leads to certain flow of actions. At some point it reaches the point at which I need to use a publisher.
And here I am not sure about the right approach. Do I need to open a new connection every time I need to publish a message? and close it after it ends? Or maybe I should implement this in a way that it keeps the same connection open for all of the publishers? (I actually not so sure how to create it in a way that it can be accessed from other parts of my app).
At the moment I am using the following :
async publishMessage(publisherName, message) {
const dynamicSettings = setupDynamicVariablesFromConfigFiles(minimalPublishSettings);
const broker = await Rascal.BrokerAsPromised.create(Rascal.withDefaultConfig(dynamicSettings.rascal));
broker.on('error', async function(err) {
loggerUtil.writeToLog('error', 'publishMessage() broker_error_event: ' + publisherName + err + err.stack);
await broker.shutdown();
})
const publication = await broker.publish(publisherName, message);
try {
publication.on('error', async function(err) {
loggerUtil.writeToLog('error', 'publishMessage() publish_error_event: ' + err + err.stack);
await broker.shutdown();
}).on("success", async (messageId) => {
await broker.shutdown();
}).on("return", async (message) => {
loggerUtil.writeToLog('error', 'publishMessage() publish_return_event: ' + err + err.stack);
await broker.shutdown();
})
}
catch(err) {
loggerUtil.writeToLog('error', 'Something went wrong ' + err + err.stack);
await broker.shutdown();
}
}
I use this function from different parts of my app when I need to publish messages.
I thought to just add the broker.shutdown() for all of the endpoints but at some point after an error, I got an exception about closing a connection which was already closed, and this got me worried about the shutdown approach (which probably not a good one). I think it is related to this -
I tried doing that (the commented code) but I think it isnt working well in certain situations. If everything is ok it goes to "success" and then I can close it.
But one time I had an error instead of success and when I tried to use broker.shutdown() it gave me another exception which crashed the app. I think it is related to this -
https://github.com/squaremo/amqp.node/issues/111
I am not sure what might be the safest way to approach this?
Edit:
Actually now that I think about it, the exception might be related to me trying to shutdown the broker in the catch{} area as well. I will continue to investigate.
Rascal is designed to be initiated once at application startup, rather than created per HTTP request. Your application will be extremely slow if you use it in this way, and depending on how many concurrent requests you need to handle, could easily exceed max number of connections you can make to the broker. Furthermore you will get none of the benefits that Rascal provides, such as failed connection recovery.
If you can pre-determine the queue or exchange you need to publish to, then configure Rascal at application start-up (prior to your http server), and share the publisher between requests. If you are unable to determine the queue or exchange until your receive the http request, then Rascal is not an appropriate choice. Instead you're better off using amqplib directly, but should still establish a shared connection and channel. You will have to handle connection and channel errors manually though, otherwise they will crash your application.
Related
I started using Azure recently and It has been an overwhelming experience. I started experimenting with eventhubs and I'm basically following the official tutorials on how to send and receive messages from eventhubs using nodejs.
Everything worked perfectly so I built a small web app (static frontend app) and I connected it with a node backend, where the communication with eventhubs occurs. So basically my app is built like this:
frontend <----> node server <-----> eventhubs
As you can see it is very simple. The node server is fetching data from eventhubs and sending it forward to the frontend, where the values are shown. It is a cool experience and I'm enjoying MS Azure until this error occured:
azure.eventhub.common.EventHubError: ErrorCodes.ResourceLimitExceeded: Exceeded the maximum number of allowed receivers per partition in a consumer group which is 5. List of connected receivers - nil, nil, nil, nil, nil.
This error is really confusing. Im using the default consumer group and only one app. I never tried to access this consumer group from another app. It said the limit is 5, I'm using only one app so it should be fine or am I missing something? I'm not checking what is happening here.
I wasted too much time googling and researching about this but I didn't get it. At the end, I thought that maybe every time I deploy the app (my frontend and my node server) on azure, this would be counted as one consumer and since I deployed the app more than 5 times then this error is showing up. Am I right or this is nonsense?
Edit
I'm using websockets as a communication protocol between my app (frontend) and my node server (backend). The node server is using the default consumer group ( I didn't change nothing), I just followed this official example from Microsoft. I'm basically using the code from MS docs that's why I didn't post any code snippet from my node server and since the error happens in backend and not frontend then it will not be helpful if I posted any frontend code.
So to wrap up, I'm using websocket to connect front & backend. It works perfectly for a day or two and then this error starts to happen. Sometimes I open more than one client (for example a client from the browser and client from my smartphone).
I think I don't understand the concept of this consumer group. Like is every client a consumer? so if I open my app (the same app) in 5 different tabs in my browser, do I have 5 consumers then?
I didn't quite understand the answer below and what is meant by "pooling client", therefore, I will try to post code examples here to show you what I'm trying to do.
Code snippets
Here is the function I'm using on the server side to communicate with eventhubs and receive/consume a message
async function receiveEventhubMessage(socket, eventHubName, connectionString) {
const consumerClient = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);
const subscription = consumerClient.subscribe({
processEvents: async (events, context) => {
for (const event of events) {
console.log("[ consumer ] Message received : " + event.body);
io.emit('msg-received', event.body);
}
},
processError: async (err, context) => {
console.log(`Error : ${err}`);
}
}
);
If you notice, I'm giving the eventhub and connection string as an argument in order to be able to change that. Now in the frontend, I have a list of multiple topics and each topic have its own eventhubname but they have the same eventhub namespace.
Here is an example of two eventhubnames that I have:
{
"EventHubName": "eh-test-command"
"EventHubName": "eh-test-telemetry"
}
If the user chooses to send a command (from the frontend, I just have a list of buttons that the user can click to fire an event over websockets) then the CommandEventHubName will be sent from the frontend to the node server. The server will receive that eventhubname and switch the consumerClient in the function I posted above.
Here is the code where I'm calling that:
// io is a socket.io object
io.on('connection', socket => {
socket.on('onUserChoice', choice => {
// choice is an object sent from the frontend based on what the user choosed. e.g if the user choosed command then choice = {"EventhubName": "eh-test-command", "payload": "whatever"}
receiveEventhubMessage(socket, choice.EventHubName, choice.EventHubNameSpace)
.catch(err => console.log(`[ consumerClient ] Error while receiving eventhub messages: ${err}`));
}
}
The app I'm building will be extending in the future to a real use case in the automotive field, that's why this is important for me. Therefore, I'm trying to figure out how can I switch between eventhubs without creating a new consumerClient each time the eventhubname changes?
I must say that I didn't understand the example with the "pooling client". I am seeking more elaboration or, ideally, a minimal example just to put me on the way.
Based on the conversation in the issue, it would seem that the root cause of this is that your backend is creating a new EventHubConsumerClient for each request coming from your frontend. Because each client will open a dedicated connection to the service, if you have more than 5 requests for the same Event Hub instance using the same consumer group, you'll exceed the quota.
To get around this, you'll want to consider pooling your EventHubConsumerClient instances so that you're starting with one per Event Hub instance. You can safely use the pooled client to handle a request for your frontend by calling subscribe. This will allow you to share the connection amongst multiple frontend requests.
The key idea being that your consumerClient is not created for every request, but shares an instance among requests. Using your snippet to illustrate the simplest approach, you'd end up hoisting your client creation to outside the function to receive. It may look something like:
const consumerClient = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);
async function receiveEventhubMessage(socket, eventHubName, connectionString) {
const subscription = consumerClient.subscribe({
processEvents: async (events, context) => {
for (const event of events) {
console.log("[ consumer ] Message received : " + event.body);
io.emit('msg-received', event.body);
}
},
processError: async (err, context) => {
console.log(`Error : ${err}`);
}
}
);
That said, the above may not be adequate for your environment depending on the architecture of the application. If whatever is hosting receiveEventHubMessage is created dynamically for each request, nothing changes. In that case, you'd want to consider something like a singleton or dependency injection to help extend the lifespan.
If you end up having issues scaling to meet your requests, you can consider increasing the number of clients for each Event Hub and/or spreading requests out to different consumer groups.
I found similar questions but not found any good answer so I am posting it again and with more specific environment.
I am using Node.js + Express to build REST APi for my React Native app.
I am using pattern Router->Controller->Service->Database and I am not sure if I am supposed to throw specific errors from my services.
Example:
I am validating registration request.
Field validations are hapenning inside Controller (I am using ajv JSON schema validator).
Now I need to validate if user/email already exists
Am I supposed to do this from controller by calling for example service.emailExists(data.email) before calling service.createUser(data)??
Or I can let database fall on CREATE duplicate, return false from service.createUser(data) and inform user from controller that Email exists. If I do so, I am not able to inform user if there is Unspecified error inside service, because it will always return as Email exists error, even something else happens.
You can use try...catch, or if...else to handle the possibilities of errors.
This is how it worked for me. Service using express and sequelize
const { Keluhan } = require('../../models');
var getAllKeluhan = () => {
return Keluhan.findAll()
.then(data => {
if (data) {
return {
code: 200,
message: "Data Found!",
result: data
};
} else {
return {
code: 404,
message: "Data not Found!"
}
}
})
.catch(err => {
throw err;
})
}
module.exports = getAllKeluhan;
This kind of typical problem can be solved in different ways. i provide below few.
A general way of handling it to fail in Database layer and there will be a cascading of failure from database layer to service layer and then controller layer. In this case it is assumed that there is no graceful failure and in this case, people just broadcast a generic failure. All the errors and exceptions are reported in a log.
The another approach is to make a failure at the service layer itself if the email id already exists in the cache without connecting to database. It seems to be a good approach as long as there is a good synchronization between database and the cache layer.
Another approach would be a proactive failure where once the user enters the email id and makes a tab out in UI, you can connect to database to validate through a service.
Sometimes, it is also thought that let the user enters everything in the UI screen and let us validate at the end.
As far as design is concerned, I think approach should be given by the functional team who propose the idea. Technically, all the approaches are possible.
The most important thing here is to grab the error or exception in the controller and propagate as a bad request. It means user should be notified whether email id already exists or not.
Finally, you can think of the design like Fail-Safe or Fail-Fast as per your requirements.
I've written a simple service using redis to store data in memory or fetch from disc and then store in memory and want to set a timeout for slow requests. I'm hoping to find a way make a get request with a timeout to prevent this a request from hanging. Any help is appreciated.
So, there are a few things you can do here. But, first I wonder if you are attempting premature optimization. Redis in most normal situations is blazingly fast, and if you are finding performance issues on the client, then that indicates that you have some issues with your data or how you are processing it in redis. This should be fixed this in redis, there is nothing you should do in your client to handle slow requests.
So, if you are seeing occasional slowdowns, what are they coming from? This is not a normal redis issue, and should be addressed instead of looking for a javascript fix.
If you are still looking for a javascript fix, you could do something like this:
const client = require('redis').createClient(...);
export async function asyncSetEx(key) {
return new Promise((resolve, reject) => {
const timer = setTimeout(() => {
reject(new Error('Timed out'));
});
client.setEx(key, (res, err) => {
if (err) {
reject(err);
} else {
resolve(res);
}
clearTimeout(timer);
});
});
}
Though, I'd recommend generalizing this so that it works for any redis function with any number of parameters.
prevent this a request from hanging
If you set enable_offline_queue to false, all Redis commands will throw an exception immediately instead of waiting to reconnect. You can then catch that exception and do your fetching from disc or some DB.
Couldnt find anything in the documentation regarding this and found many such questions here on SO, hence posting in this old question.
Do keep in mind that, with enable_offline_queue set to false, the commands that you issue while there's some connection issue with the server will never be executed.
We need to update the client side UI to indicate that a message fails to deliver. How do I have Socket.IO JS client call a custom callback directly when the message fails to deliver? For example, something like:
socket.emit("event", data).onError(myCallback);
I know Socket.IO provides the Ack mechanism to confirm delivery success. Therefore, one can set up a timer with a handler which calls the failure callback, if the ack is not called after a certain amount of time. But this doesn't seem to be the best way to do.
Also there is the error event provided by Socket.IO, but it doesn't come with info regarding which emit caused the error.
Unfortunately there's no way to get errors from callbacks, the only way is to indeed create your own timeout:
var timeoutId = setTimeout(timeoutErrorFn, 500);
var acknCallbackFn = function(err, userData){
clearTimeout(timeoutId)
//manage UserData
}
socket.emit('getUserData', acknCallbackFn);
Source of the code
And there's another issue about this, open
So for the time being you have to stick with your manual setTimeout.
I earlier had all my code in socket.io+node.js server. I recently converted all the code to redis+socket.io+socket.io+node.js after noticing slow performance when too many users send messages across the server.
So, why socket.io alone was slow because it is not multi threaded, so it handles one request or emit at a time.
What redis does is distribute these requests or emits across channels. Clients subscribe to different channels, and when a message is published on a channel, all the client subscribed to it receive the message. It does it via this piece of code:
sub.on("message", function (channel, message) {
client.emit("message",message);
});
The client.on('emit',function(){}) takes it from here to publish messages to different channels.
Here is a brief code explaining what i am doing with redis:
io.sockets.on('connection', function (client) {
var pub = redis.createClient();
var sub = redis.createClient();
sub.on("message", function (channel, message) {
client.emit('message',message);
});
client.on("message", function (msg) {
if(msg.type == "chat"){
pub.publish("channel." + msg.tousername,msg.message);
pub.publish("channel." + msg.user,msg.message);
}
else if(msg.type == "setUsername"){
sub.subscribe("channel." +msg.user);
}
});
});
As redis stores the channel information, we can have different servers publish to the same channel.
So, what i dont understand is, if sub.on("message") is getting called every time a request or emit is sent, why is redis supposed to be giving better performance? I suppose even the sub.on("message") method is not multi threaded.
As you might know, Redis allows you to scale with multiple node instances. So the performance actually comes after the fact. Utilizing the Pub/Sub method is not faster. It's technically slower because you have to communicate between Redis for every Pub/Sign signal. The "giving better performance" is only really true when you start to horizontally scale out.
For example, you have one node instance (simple chat room) -- that can handle a maximum of 200 active users. You are not using Redis yet because there is no need. Now, what if you want to have 400 active users? Whilst using your example above, you can now achieve this 400 user mark, which is a "performance increase". In the sense you can now handle more users, but not really a speed increase. If that makes sense. Hope this helps!