Keeping Service Bus Connection alive for duration of app lifecycle - node.js

Hi I am using the new nodejs sdk to connect to servicebus. What is the proper way to keep receiving messages as long as my application is running? The example code shows 2 ways of listening for messages:
Method 1 - Receive Batch
const receiver = client.getReceiver();
try {
for (let i = 0; i < 10; i++) {
const messages = await receiver.receiveBatch(1, 5);
if (!messages.length) {
console.log("No more messages to receive");
break;
}
console.log(`Received message #${i}: ${messages[0].body}`);
await messages[0].complete();
}
await client.close();
} finally {
await ns.close();
}
}
Method 2 - Streaming Listener
try {
receiver.receive(onMessageHandler, onErrorHandler, { autoComplete: false });
// Waiting long enough before closing the receiver to receive messages
await delay(5000);
await receiver.close();
await client.close();
} finally {
await ns.close();
}
I went with method 2, on startup, and basically never close the client. But after a period of time the connection just stops working and the messages don't get received anymore (stuck in the queue).
What is the correct way to receive messages "forever"?:
Re-establish a new client (open and close eg every minute) with method 1, OR
Re-establish a new client (open and close eg every minute) with method 2, OR
Some kind of polling system (how)?

I know this is a late reply, but... Version 7 of #azure/service-bus SDK offers a solution in order to tackle this specific problem of disconnects and reconnection.
The subscribe method(which is equivalent to the receive method in your code snippet) can be leveraged which would run forever and is capable of recovering from fatal errors as well.
You can refer to the receiveMessagesStreaming.ts sample code that uses version 7 of #azure/service-bus SDK.
The latest version 7.0.0 of #azure/service-bus has been released recently.
#azure/service-bus - 7.0.0
Samples for 7.0.0
Guide to migrate from #azure/service-bus v1 to v7

Related

Node program doesn't stop after unregistering a listener from hyperledger fabric

I'm using SDK 1.4, and I'm creating a listener I want to receive a message and close my program but when I do the unregister the program doesn't stop and I can't see in the documentation any other way to stop listening.
Here is the documentation I'm using as guide
https://hyperledger.github.io/fabric-sdk-node/release-1.4/tutorial-listening-to-events.html
Here's my code
const contract = await fabricService.getContract(user, contractName);
const listener = await contract.addContractListener(
'tests',
'contract',
(_error: Error, event: any, _blockNumber?: string, _transactionId?: string, _status?: string) => {
console.log('message received');
const data = (event.payload as Buffer).toString('utf8');
expect(data).to.equal('xxxx');
listener.unregister();
}
);
listener.unregister();
I expected the program to end after de listener unregister but nothing happens and the program keeps running.
am I missing something? shouldn't the program stop?
Thanks

Correct way to process batches using receiveMessages

We are using the #azure/service-bus package to process message batches from multiple topics.
The code we use to take 20 messages from the topic every 2 seconds looks like this.
let isProcessing: boolean = false;
setInterval(async () => {
if (isProcessing === false) {
isProcessing = true;
try {
const messages: Array<ServiceBusMessage>
= await receiver.receiveMessages(Configuration.SB.batchSize as number);
if (messages.length > 0) {
this.logger.info(`[SB] ${topic} - ${messages.length} require processing`);
await Promise.all([
...messages.map(message => this.handleMsg(receiver, message, topic, moduleRef, handler))
]).catch(error => {
this.logger.error(error.message, error);
});
}
isProcessing = false;
} catch (error) {
this.logger.error(error.message, error);
isProcessing = false;
}
}
}, Configuration.SB.tickInterval as number);
My question is - Is this the best way to do this? Is there a better way? It works and is fairly performant BUT I think we are losing receiveAndDelete messages sometimes and I am trying to workout if its our implementation
Thanks for any help
It works and is fairly performant BUT I think we are losing receiveAndDelete messages sometimes and I am trying to workout if its our implementation
There are two modes to receive messages
Unsafe with ReceiveAndDelete
Safe with PeekLock
When ReceiveAndDelete mode is used, the moment messages are received by the client, they are automatically deleted from the server. So this is at-most-once delivery.
With PeekLock a message is "leased" to the client for a maximum of 5 minutes and the client has to either acknowledge successful processing by requesting message completion or by cancelling/dead-lettering if it can't handle it. If none of these operations take place within the defined lease time (which doesn't have to be strictly 5 minutes and could be less), the message is retried until a maximum number of delivery attempts (MaxDeliveryCount) is exceeded and the message is dead-lettered. Note that the message is never lost. Even if it failed to process and was dead-lettered. Therefore this is at-least-once-delivery which could be more suitable for your scenario. It will have a slight impact on how you code your client, but not a drastic change.

Nodejs Cluster Architecture reading from single REDIS instance

I'm using Nodejs cluster module to have multiple workers running.
I created a basic Architecture where there will be a single MASTER process which is basically an express server handling multiple requests and the main task of MASTER would be writing incoming data from requests into a REDIS instance. Other workers(numOfCPUs - 1) will be non-master i.e. they won't be handling any request as they are just the consumers. I have two features namely ABC and DEF. I distributed the non-master workers evenly across features via assigning them type.
For eg: on a 8-core machine:
1 will be MASTER instance handling request via express server
Remaining (8 - 1 = 7) will be distributed evenly. 4 to feature:ABD and 3 to fetaure:DEF.
non-master workers are basically consumers i.e. they read from REDIS in which only MASTER worker can write data.
Here's the code for the same:
if (cluster.isMaster) {
// Fork workers.
for (let i = 0; i < numCPUs - 1; i++) {
ClusteringUtil.forkNewClusterWithAutoTypeBalancing();
}
cluster.on('exit', function(worker) {
console.log(`Worker ${worker.process.pid}::type(${worker.type}) died`);
ClusteringUtil.removeWorkerFromList(worker.type);
ClusteringUtil.forkNewClusterWithAutoTypeBalancing();
});
// Start consuming on server-start
ABCConsumer.start();
DEFConsumer.start();
console.log(`Master running with process-id: ${process.pid}`);
} else {
console.log('CLUSTER type', cluster.worker.process.env.type, 'running on', process.pid);
if (
cluster.worker.process.env &&
cluster.worker.process.env.type &&
cluster.worker.process.env.type === ServerTypeEnum.EXPRESS
) {
// worker for handling requests
app.use(express.json());
...
}
{
Everything works fine except consumers reading from REDIS.
Since there are multiple consumers of a particular feature, each one reads the same message and start processing individually, which is what I don't want. If there are 4 consumers, 1 is marked as busy and can not consumer until free, 3 are available. Once the message for that particular feature is written in REDIS by MASTER, the problem is all 3 available consumers of that feature start consuming. This means that the for a single message, the job is done based on number of available consumers.
const stringifedData = JSON.stringify(req.body);
const key = uuidv1();
const asyncHsetRes = await asyncHset(type, key, stringifedData);
if (asyncHsetRes) {
await asyncRpush(FeatureKeyEnum.REDIS.ABC_MESSAGE_QUEUE, key);
res.send({ status: 'success', message: 'Added to processing queue' });
} else {
res.send({ error: 'failure', message: 'Something went wrong in adding to queue' });
}
Consumer simply accepts messages and stop when it is busy
module.exports.startHeartbeat = startHeartbeat = async function(config = {}) {
if (!config || !config.type || !config.listKey) {
return;
}
heartbeatIntervalObj[config.type] = setInterval(async () => {
await asyncLindex(config.listKey, -1).then(async res => {
if (res) {
await getFreeWorkerAndDoJob(res, config);
stopHeartbeat(config);
}
});
}, HEARTBEAT_INTERVAL);
};
Ideally, a message should be read by only one consumer of that particular feature. After consuming, it is marked as busy so it won't consume further until free(I have handled this). Next message could only be processed by only one consumer out of other available consumers.
Please help me in tacking this problem. Again, I want one message to be read by only one free consumer and rest free consumers should wait for new message.
Thanks
I'm not sure I fully get your Redis consumers architecture, but I feel like it contradicts with the use case of Redis itself. What you're trying to achieve is essentially a queue based messaging with an ability to commit a message once its done.
Redis has its own pub/sub feature, but it is built on fire and forget principle. It doesn't distinguish between consumers - it just sends the data to all of them, assuming that its their logic to handle the incoming data.
I recommend to you use Queue Servers like RabbitMQ. You can achieve your goal with some features that AMQP 0-9-1 supports: message acknowledgment, consumer's prefetch count and so on. You can set up your cluster with very agile configs like ok, I want to have X consumers, and each can handle 1 unique (!) message at a time and they will receive new ones only after they let the server (rabbitmq) know that they successfully finished message processing. This is highly configurable and robust.
However, if you want to go serverless with some fully managed service so that you don't provision like virtual machines or anything else to run a message queue server of your choice, you can use AWS SQS. It has pretty much similar API and features list.
Hope it helps!

ETIMEOUT error with servicebus connection while using topics

I am creating 5 connections to servicebus and putting them in an array. Then as the new messages keep on coming I get one connection from the array and use them to send the message. When I start the service and run a load test it works fine. I leave the service ideal for sometime and run the same load test again, it starts having this error. connect ETIMEDOUT xxx.xxx.xxx.xxx\\n at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1191:14)
I am not sure if it is a good way to cache the connection and reuse them, which would be causing this issue, or it is something else that causes this.
let serviceBusConnectionArray = [];
let executed = false;
let serviceBusService;
let count = 0;
let MAX_CONNECTIONS = 5;
class ServiceBus {
static createConnections(){
if(!executed){
for(let i=0; i< MAX_CONNECTIONS; i++){
serviceBusConnectionArray.push(azure.createServiceBusService(SERVICEBUS_CONNECTION_STRING).withFilter(new azure.ExponentialRetryPolicyFilter()));
}
executed = true;
}
}
static getConnectionString(){
ServiceBus.createConnections();
if(count < MAX_CONNECTIONS){
return serviceBusConnectionArray[count++];
}else{
count = 0;
return serviceBusConnectionArray[count];
}
}
static putMessageToServiceBus(topicName, message) {
return new Promise((resolve, reject) => {
serviceBusService = ServiceBus.getConnectionString();
serviceBusService.sendTopicMessage(topicName, message, function (error) {
if (error) {
log.error('Error in putting message to service bus, message: %s', error.stack);
reject(error);
}
resolve('Message added');
});
});
}
}
I am not sure what route should I choose now, to resolve this timeout errors.
Looking into the source code for azure-sdk-for-node, specifically these lines in order
servicebusservice.js#L455
servicebusservice.js#L496
serviceclient.js#L190
The SDK is just performing REST requests to Service Bus via its REST API. So, I don't really think pooling those objects really help.
The timeout seems to be a genuine timeout at that point of time raised by the request npm module used by the SDK.
You could probably try the newer SDK which uses amqp under the hood to connect to service bus. Note that this SDK is in preview.
As PramodValavala-MSFT has mentioned about #azure/service-bus SDK in the other answer, major version 7.0.0 of #azure/service-bus SDK(which was in the preview) depends on AMQP has been released recently.
Each instance of ServiceBusClient represents a connection, all the methods under ServiceBusClient use the same connection.
#azure/service-bus - 7.0.0
Samples for 7.0.0
Guide to migrate from #azure/service-bus v1 to v7

When calling a WCF channel from multiple threads some threads might get stuck for a long time

I have encountered a weird problem in one of my projects. I am creating one WCF channel and trying to consume it from multiple threads. The service I am targeting is shut down so I except to get an exception after the "Open timeout" (30 seconds in my case) at most. But what I have seen is that the first two calls to the channel are finished (with exception) really quickly. all the other calls are finished after 20 minutes (My receive timeout).
I am using the same channel because I don't want to wait for the channel to open for each request (Can take a few seconds in case of security and high latency). I have read that a channel is thread safe so I didn't think it should be a problem.
I am using dot net 4
Code sample:
EndpointAddress address = new EndpointAddress("net.tcp://localhost:9000/SomeService");
var netTcpBinding = new NetTcpBinding();
var channelFactory = new ChannelFactory<IService>(netTcpBinding, address);
IService channel = channelFactory.CreateChannel();
Parallel.For(0, 10, new ParallelOptions{MaxDegreeOfParallelism = 10}, i =>
{
try
{
channel.SomeOperation();
}
catch
{
}
});
I have tried to Close/Abort/Dispose the channel in the catch block but it didn't help.
Does anyone have any idea why this happens and how to fix it?
A Channel only has one connection, so even if it is thread-safe, you won't get the asynchronous benefits of using Parallel. Create a channel per loop and ensure that you close the channel after each request or you'll exhaust the connection pool on your machine from undisposed connections retained by the Channel.
Didn't find a standard solution but what I did find is that when I use async calls the problem doesn't happen (tested it several time with a 100 iterations loop.
Parallel.For(0, 10, new ParallelOptions{MaxDegreeOfParallelism = 10}, i =>
{
try
{
var result = channel.BeginSomeOperation();
channel.EndSomeOperation(result);
}
catch
{
}
});
Try this instead.
var tasks = from i in Enumerable.Range(0, 10)
select TaskEx.FromAsync(channel.BeginSomeOperation, channel.EndSomeOperation, null);
var results = from t in TaskEx.WhenAll(tasks)
select t.Result;
PS TaskEx is in the Async targeting pack.

Resources