I have two clients listening to a subscription of topic . I am seeing following issues.
Issue 1:
If one client receives a message and marks them completes other client won't receive that messages.
But i want all my clients to receive that message and acknowledge it so that once a client receives the message it shouldn't receive again
Issue 2:
If i don't acknowledge the message as complete . Some times i am receiving the message multiple times.
Issue 3:
During the Message Receive if see the Network disconnect. After some time it reconnects I am receiving the messages from starting again.
For Sending the Message Code
============
if (!namespaceManager.TopicExists("DataCollectionTopic"))
namespaceManager.CreateTopic("DataCollectionTopic");
if (!namespaceManager.SubscriptionExists("DataCollectionTopic", "one"))
namespaceManager.CreateSubscription("DataCollectionTopic", "one");
for(int i=0;i<100;i++)
{
BrokeredMessage bm = new BrokeredMessage("new Topic one");
bm.Label = "hELLLOOOO xcvxvxcvxvxvxc DummyMEssage"+i;
bm.Properties["StoreName"] = "asdasdasqwedas";
bm.Properties["MachineID"] = "Bajjiiiqweq567567wii";
if (namespaceManager == null)
{
Console.WriteLine("\nUnexpected Error");
return;
}
MessageSender sender = messageFactory.CreateMessageSender("DataCollectionTopic");
sender.Send(bm);
for receiving the message
===================
MessageReceiver receiver = await messageFactory.CreateMessageReceiverAsync("DataCollectionTopic/subscriptions/Vijay");
while (true) {
BrokeredMessage receivedMessage = receiver.Receive();
try
{
ProcessMessage(receivedMessage);
// receivedMessage.Complete();
}
catch (Exception e)
{
// receivedMessage.Abandon();
}
}
}
===============
TIA
Issue 1
If both clients receive using the same subscription name, they act as competing consumers. The first consumer to get the message and mark it as completed (processed) will be the winner. The rest of the clients won't process the same message. If intention is to receive a message (event) by all clients, then each client should have its own subscription and a copy of the message will be delivered to each subscription.
Issue 2
That is an expected behaviour. You receive the message up to MaxDeliverCount and after that message will be DLQ-ed. This is a default behaviour for PeekLock mode. With ReceiveAndDelete this doesn't happen, but it's a dangerous mode as you will lose the message if not processed successfully.
Issue 3
That is an expected behaviour. Each received message in a PeekLock mode has a LockDuration, time given to a processing code to complete the message or abandon it. If not completed, message becomes visible to other competing consumers and will be reprocessed up to DeliveryCount times.
To sum it up
Have multiple subscriptions per client for all of the clients to receive the same message (event)
Complete messages if done processing successfully. If you need time extension, renew the lock.
Related
I have an API that will call Azure Topic to schedule a message. Is there a way to receive that message before the schedule time? For example in my code below, I schedule a message to azure topic and it will be queue after 60mins/1hr. Is there a way to received that message before 1hr?
string queueName = "topic";
var client = new ServiceBusClient("", new ServiceBusClientOptions()
{
TransportType = ServiceBusTransportType.AmqpWebSockets
});
// create the sender
ServiceBusSender sender = client.CreateSender(queueName);
// create a message that we can send. UTF-8 encoding is used when providing a string.
ServiceBusMessage message = new ServiceBusMessage($"Hello world cancel 13 {DateTime.Now}");
// add 5 minutes delay
long seq = await sender.ScheduleMessageAsync(message,
DateTimeOffset.Now.AddMinutes(60)
);
The message sequence number you get back when scheduling is suitable for cancelling but doesn't allow receiving that message earlier. The service doesn't allow early receiving, as anything that gets the messages to get the active messages (not in the future). For this scenario, I would suggest keeping the data in a database and not leveraging the queue as the database.
I am working with an azure service bus queue configured to be FIFO (First input first output). I work on an order application with the following states "Pending", "Received" and "Sent". therefore I have grouped the messages by the "SessionId" service bus option, setting the orderId as sessionId so that it processes the messages in order in case of horizontal scaling.
So far it works perfectly, the problem I have found is when a message in "pending" or "Received" status fails due to a timeout and goes to the dead letter queue. The message in "sent" status is processed correctly and then when the support team re-sends the "Pending" or "Received" status message to the queue it is processed correctly marking the order in a previous status instead of "sent" ".
I can think of several ways to control this, for example that the support team looks at the status of the order before reprocessing the message from the dead letter queue :) but I would like to know if service bus offers the possibility that if there is a message in the dead letter queu all the messages in the session queue that have the same sessionId go to the dead letter queu. Finallly, my question is:
Is there a way to configure azure service bus so that if there are any messages in the dead letter queue it sends all messages with the same sessionId to the dead letter queue?
Thank you very much!!!
I would like to know if service bus offers the possibility that if there is a message in the dead letter queue all the messages in the session queue that have the same sessionId go to the dead letter queue.
No, there is no such offering by Service Bus by default.
Is there a way to configure azure service bus so that if there are any messages in the dead letter queue it sends all messages with the same sessionId to the dead letter queue?
Yes, you can do that. You can first peek the messages in your dead-letter queue to fetch all the session ids. Then you can receive the messages in your main queue whose session id is in the DLQ, and then move those messages to DLQ. Here's one such logic I've implemented in dot net using the latest version of Service Bus SDK.
var queueName = "<queue>";
var connectionString = "<connection-string>";
var client = new ServiceBusClient(connectionString);
var sessionIdInDLQList = new List<string>();
var receiver = client.CreateReceiver(queueName, new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter });
var message = await receiver.PeekMessageAsync();
while (message != null)
{
if (!sessionIdInDLQList.Contains(message.SessionId))
sessionIdInDLQList.Add(message.SessionId);
message = await receiver.PeekMessageAsync();
}
foreach (var sessionId in sessionIdInDLQList)
{
var session = await client.AcceptSessionAsync(queueName, sessionId);
message = await session.ReceiveMessageAsync(TimeSpan.FromSeconds(20));
while (message != null)
{
await session.DeadLetterMessageAsync(message, "Message with this session is to be dead-lettered!");
message = await session.ReceiveMessageAsync(TimeSpan.FromSeconds(20));
}
}
In your case, you need to do this before your consumers start reading the messages, probably you can write this in your consumer application or any trigger application like Azure Function or worker role. That’s upto your method of handling.
You can try this code to read Dead Letter from Queue.
public static async Task GetMessage()
{
string topic = "myqueue1";
string connectionString = "Endpoint = sb://xxx.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxx";
var servicebusclient = new ServiceBusClient(connectionString);
var reciveroptions = new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter };
var reciver = servicebusclient.CreateReceiver(topic, reciveroptions);
// 10 number of message read from Queue
await receiver.PeekMessageAsync(10);
}
after receiving message from Dead Letter you can send to queue.
As per Microsoft official documents
There's no automatic cleanup of the DLQ. Messages remain in the DLQ
until you explicitly retrieve them from the DLQ and call Complete() on
the dead-letter message.
These following document help you.
Thanks Casually Coding for posting post on Read Message from the Dead Letter Queue
Microsoft Documents Using Dead-Letter Queues to Handle Message Transfer Failures , Receive Message from Dead letter queue
Imagine that connection to a Kafka cluster has closed (e.g. eth cable was cut), and sooner or later the buffer queue is going to get full. How does one get those messages from the buffer and let's say send it to a local storage? Thanks!
If node-rdkafka cannot send messages that are in the buffer it will expire them after message.timeout.ms which is 300000 (5mins) by default. When expiring each message it will emit the delivery-report event and pass in the error and the message.
For example, you can set a listener on this event and have some backup logic in case of a failure:
// Register delivery report listener
producer.on('delivery-report', function(err, report) {
if (err) {
console.error('Delivery report: Failed sending message ' + JSON.stringify(report));
console.error(err);
// We could retry sending the message or store it locally
} else {
console.log('Message produced, offset: ' + report.offset);
}
});
The report object contains all the information (body, key, partition) needed to resent it if you want.
I'm using a Service Bus queue with Sessions enabled and I'm sending 5 messages with the same SessionId. My receiving code uses AcceptMessageSessionAsync to get a session lock so that it will receive all the messages for that session. It then uses session.ReceiveBatchAsync to try and get all the messages for the session. However, it only seems to get the first message, then when another attempt is made, it gets all the others. You should be able to see that there is a gap of almost a minute between the two batches even though all these messages were sent at once:
Session started:AE8DC914-8693-4110-8BAE-244E42A302D5
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_1_08:03:03.36523
Session started:AE8DC914-8693-4110-8BAE-244E42A302D5
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_2_08:03:04.22964
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_3_08:03:04.29515
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_4_08:03:04.33959
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_5_08:03:04.39587
My code to process these is a function in a WebJob:
[NoAutomaticTrigger]
public static async Task MessageHandlingLoop(TextWriter log, CancellationToken cancellationToken)
{
var connectionString = ConfigurationManager.ConnectionStrings["ServiceBusListen"].ConnectionString;
var client = QueueClient.CreateFromConnectionString(connectionString, "myqueue");
while (!cancellationToken.IsCancellationRequested)
{
MessageSession session = null;
try
{
session = await client.AcceptMessageSessionAsync(TimeSpan.FromMinutes(1));
log.WriteLine("Session started:" + session.SessionId);
foreach (var msg in await session.ReceiveBatchAsync(100, TimeSpan.FromSeconds(5)))
{
log.WriteLine("Message received:" + msg.MessageId);
msg.Complete();
}
}
catch (TimeoutException)
{
log.WriteLine("Timeout occurred");
await Task.Delay(5000, cancellationToken);
}
catch (Exception ex)
{
log.WriteLine("Error:" + ex);
}
}
}
This is called from my WebJob Main using:
JobHost host = new JobHost();
host.Start();
var task = host.CallAsync(typeof(Functions).GetMethod("MessageHandlingLoop"));
task.Wait();
host.Stop();
Why don't I get all my messages in the first call of ReceiveBatchAsync?
This was answered in the MSDN forum by Hillary Caituiro Monge: https://social.msdn.microsoft.com/Forums/azure/en-US/9a84f319-7bc6-4ff8-b142-4fc1d5f1e2fa/service-bus-session-receivebatchasync-only-receiving-1-message?forum=servbus
Service Bus does not guarantee you will receive the message count you
specify in receive batch even if your queue has them or more. Having
say that, you can change your code to try to get the 100 messages in
the first call, buy remember that your application should not assume
that as a guaranteed behavior.
Below this line of code varclient =
QueueClient.CreateFromConnectionString(connectionString, "myqueue");
add client.PrefetchCount = 100;
The reason that you are getting only 1 message at all times in the
first call is due to that when you accept a session it may be also
getting 1 prefetched message with it. Then when you do receive batch,
the SB client will give you that 1 message.
Unfortunately I found that setting the PrefetchCount didn't have an affect, but the reason given for only receiving one message seemed likely so I accepted it as the answer.
I've written some code to delete test messages off a service bus topic. I'm the only one using this topic. It's using ReceiveAndDelete mode so I am assuming it's going to delete them, but every time I run the code it goes through this cycle of receiving messages, so I know it's not deleting them. What am I doing wrong?
public void TruncateTopic()
{
// reset topic for testing..
SubscriptionClient client = SubscriptionClient.CreateFromConnectionString(
connStr, QUEUENAME, "AllMessages",ReceiveMode.ReceiveAndDelete);
BrokeredMessage message = client.Peek();
while (message != null)
{
client.Receive();
message = client.Peek();
}
client.Close();
}
In your code you only do Peek on the Topic/Queue. Peek action never deletes the messages.
As you can clearly read from documentation Peek method only peeks into the subscription without actually receiving the message.
The ReceiveAndDelete receive mode will well work when you not just Peek the messages but Receive them instead! That is why it is named ReceiveAndDelete but not PeekAndDelete.
Change your code to:
BrokeredMessage message = client.Receive();
while (message != null)
{
message = client.Receive();
}
And everything will be fine.