I'm using the amqplib library for nodejs to work with RabbitMQ. I'm trying to check whether a queue exists by using the function checkQueue:
mychannel.checkQueue('xxx', function (err, ok) {
console.log(err);
console.log(ok)
});
But it not only throws an error, but also closes the channel. How can I safely check if the queue exists?
You can't without risking to destroy the channel. The workaround is to create a temporary channel which you can use to do the check.
A comment from amqp.node dev:
(https://github.com/squaremo/amqp.node/issues/280)
The behaviour of checkQueue is dictated by the protocol, but it can be
worked around. One tactic is to create a "sacrificial" extra channel with
which to test whether the queue exists. Once you have the answer, you can
throw the extra channel away, or keep it around for more tests.
Related
I have an event-driven system with my application implemented in nodejs using cosmosdb (azure-cosmosdb-sqlapi) to store the events. I have planning data coming via various events from kafka broker, to complete a planning documnet I need to combine data from 5 different events, I combine the events using planning id. In such a system for upsert operation we encounter 412 Precondition failure error very often, as we receive many events for a planning id.
The official Microsoft link says to retry. I am confused about which approach to take to retry
Handle the error code using a try catch and call the method handling the event from catch block for n number of times. If the retry fails after n times throw the exception back to broker, in that case the event is send again by the broker. The issue with this is I am not able to add test for the same. Secondly I need to manage all the retry logic in my code base. The advantage here is that I know that an event is failed and I can retry directly without sending the event back to broker. Below is the the snippet from planningservice.ts handlePlanningEvents method
try {
await repository.upsert(planningEntry, etag)
} catch (e: any) {
if (e.code === 412 and retries) {
this.handlePlanningEvents(event, retries-1)
} else {
throw e // throws exception back to broker
}
}
Not using try catch to handle the error in service code but propagating the error to controller where it sends a 500 error response to broker and the broker sends the event again. The issue with this case is that it's longer path as compared to using try catch where I can retry directly. But the advantage here is that I don't to worry about retry logic anymore its handled by broker, less and cleaner code.
Not sure which approach to take, also open to other suggestions.
I'm using an event handler, where each event code is in it's own files. I'm attaching the events to the client, so when that file has the event emitted, it will run that code:
// looping through all event files
client.on(file.split('.')[0], require(`events/${file}`).bind(null, client, Util);
If the event file was message.js, it would be similar to:
client.on('message', require('events/message.js').bind(null, client, Util);
So when the message event is emitted, it runs the message.js file, passing along the client and Util classes.
I also have a function that is attached to the client called report. It basically reports when there is an error. I would like it so whenever any event from the client has an error, it will run that function.
I've done this slightly with the commands: command.run(...).catch(error => client.report(error)).
Is there a similar way to do this, instead of having to put a try-catch around all code in all the event files?
Try this way
client.on('error', require('events/report.js').bind(null, client, Util);
Error handling should be context driven. This means your bot's response to the error should be dependent on what it was doing, in what channel, etc - both for debugging and for the end user's information on what happened. You'll miss out on all of the context by letting errors just travel all the way up into an uncaught exception, and without the ability to create an error message, the user will just see the bot not respond and think it's down or the command is broken.
My suggestion: Create helper methods for your most common error producing functions that wrap them with error handling. I did this mostly for sending messages, as there's a myriad of things that could cause a message send to fail out of your control and the handling consists of generating an error message and attempting to send it in the channel or DM it to the user if that fails.
How can I send message to deadletter queue?
serviceBusService.receiveQueueMessage(MESSAGE_QUEUE, {isPeekLock: true}, (error, message) => {
...... // want to put message to deadletter queue if there is exception
serviceBusService.deleteMessage(message, error => {
});
});
Mostly, you'd want to rely on the system to decide when to move a message to DLQ and make it the messaging engine's responsibility as much as possible (and not explicitly put a message on DLQ.) It also appears that the guidance in this scenario was provided via documentation here: How to handle application crashes and unreadable messages
Looks like you are using the older azure-sb package that relies on the HTTP REST apis. If you instead use the newer #azure/service-bus package which uses the faster AMQP implementation, there is a deadletter() method on the message you receive that you can use to send the message to the dead letter queue.
We have recently started working on Typescript language for one of the application where a queue'd communication is expected between a server and client/clients.
For achieving the queue'd communication, we are trying to use the ZeroMQ library version 4.6.0 as a npm package: npm install -g zeromq and npm install -g #types/zeromq.
The exact scenario :
The client is going to send thousands of messages to the server over ZeroMQ. The server in-turn will be responding with some acknowledgement message per incoming message from the client. Based on the acknowledgement message, the client will send next message.
ZeroMQ pattern used :
The ROUTER/DEALER pattern (we cannot use any other pattern).
Client side code :
import Zmq = require('zeromq');
let clientSocket : Zmq.Socket;
let messageQueue = [];
export class ZmqCommunicator
{
constructor(connString : string)
{
clientSocket = Zmq.socket('dealer');
clientSocket.connect(connString);
clientSocket.on('message', this.ReceiveMessage);
}
public ReceiveMessage = (msg) => {
var argl = arguments.length,
envelopes = Array.prototype.slice.call(arguments, 0, argl - 1),
payload = arguments[0];
var json = JSON.parse(msg.toString('utf8'));
if(json.type != "error" && json.type =='ack'){
if(messageQueue.length>0){
this.Dispatch(messageQueue.splice(0, 1)[0]);
}
}
public Dispatch(message) {
clientSocket.send(JSON.stringify(message));
}
public SendMessage(msg: Message, isHandshakeMessage : boolean){
// The if condition will be called only once for the first handshake message. For all other messages, the else condition will be called always.
if(isHandshakeMessage == true){
clientSocket.send(JSON.stringify(message));
}
else{
messageQueue.push(msg);
}
}
}
On the server side, we already have a ROUTER socket configured.
The above code is pretty straight forward. The SendMessage() function is essentially getting called for thousands of messages and the code works successfully but with load of memory consumption.
Problem :
Because the behavior of ZeroMQ is asynchronous, the client has to wait on the call back call ReceiveMessage() whenever it has to send a new message to ZeroMQ ROUTER (which is evident from the flow to the method Dispatch).
Based on our limited knowledge with TypeScript and usage of ZeroMQ with TypeScript, the problem is that because default thread running the typescript code (which creates the required 1000+ messages and sends to SendMessage()) continues its execution (creating and sending more messages) after sending the first message (handshake message essentially), unless all the 1000+ messages are created and sent to SendMessage() (which is not sending the data but queuing the data as we want to interpret the acknowledgement message sent by the router socket and only based on the acknowledgement we want to send the next message), the call does not come to the ReceiveMessage() call back method.
It is to say that the call comes to ReceiveMessage() only after the default thread creating and calling SendMessage() is done doing this for 1000+ message and now there is no other task for it to do any further.
Because ZeroMQ does not provide any synchronous mechanism of sending/receiving data using the ROUTER/DEALER, we had to utilize the queue as per the above code using a messageQueue object.
This mechanism will load a huge size messageQueue (with 1000+ messages) in memory and will dequeue only after the default thread gets to the ReceiveMessage() call at the end. The situation will only worsen if say we have 10000+ or even more messages to be sent.
Questions :
We have validated this behavior certainly. So we are sure of the understanding that we have explained above. Is there any gap in our understanding of either/or TypeScript or ZeroMQ usage?
Is there any concept like a blocking queue/limited size array in Typescript which would take limited entries on queue, and block any new additions to the queue until the existing ones are queues (which essentially applies that the default thread pauses its processing till the time the call back ReceiveMessage() is called which will de-queue entries from the queue)?
Is there any synchronous ZeroMQ methodology (We have used it in similar setup for C# where we pool on ZeroMQ and received the data synchronously)?.
Any leads on using multi-threading for such a scenario? Not sure if Typescript supports multi threading to a good extent.
Note : We have searched on many forums and have not got any leads any where. The above description may have multiple questions inside one question (against the rules of stackoverflow forum); but for us all of these questions are interlinked to using ZeroMQ effectively in Typescript.
Looking forward to getting some leads from the community.
Welcome to ZeroMQ
If this is your first read about ZeroMQ, feel free to first take a 5 seconds read - about the main conceptual differences in [ ZeroMQ hierarchy in less than a five seconds ] Section.
1 ) ... Is there any gap in our understanding of either/or TypeScript or ZeroMQ usage ?
Whereas I cannot serve for the TypeScript part, let me mention a few details, that may help you move forwards. While ZeroMQ is principally a broker-less, asynchronous signalling/messaging framework, it has many flavours of use and there are tools to enforce both a synchronous and asynchronous cooperation between the application code and the ZeroMQ Context()-instance, which is the cornerstone of all the services design.
The native API provides means to define, whether a respective call ought block, until a message processing across the Context()-instance's boundary was able to get completed, or, on the very contrary, if a call ought obey the ZMQ_DONTWAIT and asynchronously return the control back to the caller, irrespectively of the operation(s) (in-)completion.
As additional tricks, one may opt to configure ZMQ_SND_HWM + ZMQ_RCV_HWM and other related .setsockopt()-options, so as to meet a specific blocking / silent-dropping behaviours.
Because ZeroMQ does not provide any synchronous mechanism of sending/receiving data
Well, ZeroMQ API does provide means for a synchronous call to .send()/.recv() methods, where the caller is blocked until any feasible message could get delivered into / from a Context()-engine's domain of control.
Obviously, the TypeScript language binding/wrapper is responsible for exposing these native API services to your hands.
3 ) Is there any synchronous ZeroMQ methodology (We have used it in similar setup for C# where we pool on ZeroMQ and received the data synchronously) ?
Yes, there are several such :
- the native API, if not instructed by a ZMQ_DONTWAIT flag, blocks until a message can get served
- the native API provides a Poller()-object, that can .poll(), if given a -1 as a long duration specifier to wait for sought for events, blocking the caller until any such event comes and appears to the Poller()-instance.
Again, the TypeScript language binding/wrapper is responsible for exposing these native API services to your hands.
... Large memory consumption ...
Well, this may signal a poor resources management care. ZeroMQ messages, once got allocated, ought become also free-d, where appropriate. Check your TypeScript code and the TypeScript language binding/wrapper sources, if the resources systematically get disposed off and free-d from memory.
I'm using the node-amqp module to manage rabbitmq subscriptions. Specifically, I'm assigning an exclusive/private queue to each user/session, and providing binding methods through the REST interface. I.e. "bind my queue to this exchange/routing_key pair", and "unbind my queue to this exchange/routing_key pair".
The challenge here is to avoid keeping a reference to the queue object in memory (say, in an object with module-wide scope).
Simply retrieving the queue itself from the connection each time I need it, proved difficult, since the queue object keeps tabs on bindings internally, probably to avoid violating the following from the amqp 0.9.1 reference:
The client MUST NOT attempt to unbind a queue that does not exist. Error code: not-found
I tried to simply set the queue object as a property on a session object using connect-mongo, since it uses JSON.stringify/JSON.parse on its properties. Unfortunately, the queue object fails to "stringify" due to a circular structure.
What is the best practice for persisting a queue object from the node-amqp module? Is it possible to serialize/deserialize?
I would not try to store the queue object, instead of that use an unique name for the queue that you can store. After that whenever you want to make operations over the queue you have two options:
In the case you have a previously opened "channel" to the queue, you should be able to do:
queue = connection.queues[name].
I mean connection as a node-amqp connection against rabbitMQ.
In the case you dont have a channel opened in your connection with rabbitmq, just open the channel again:
connection.queue(name = queueName, options, function(queue) {
// for example do unbind
})
I am also using REST interface to manage rabbitMQ. My connection object maintains all the queues, channels, etc... So, only the first time I try to use a queue I call to connection.queue, and the following request just retrieve the queue through connection.queues.