I am working with an azure service bus queue configured to be FIFO (First input first output). I work on an order application with the following states "Pending", "Received" and "Sent". therefore I have grouped the messages by the "SessionId" service bus option, setting the orderId as sessionId so that it processes the messages in order in case of horizontal scaling.
So far it works perfectly, the problem I have found is when a message in "pending" or "Received" status fails due to a timeout and goes to the dead letter queue. The message in "sent" status is processed correctly and then when the support team re-sends the "Pending" or "Received" status message to the queue it is processed correctly marking the order in a previous status instead of "sent" ".
I can think of several ways to control this, for example that the support team looks at the status of the order before reprocessing the message from the dead letter queue :) but I would like to know if service bus offers the possibility that if there is a message in the dead letter queu all the messages in the session queue that have the same sessionId go to the dead letter queu. Finallly, my question is:
Is there a way to configure azure service bus so that if there are any messages in the dead letter queue it sends all messages with the same sessionId to the dead letter queue?
Thank you very much!!!
I would like to know if service bus offers the possibility that if there is a message in the dead letter queue all the messages in the session queue that have the same sessionId go to the dead letter queue.
No, there is no such offering by Service Bus by default.
Is there a way to configure azure service bus so that if there are any messages in the dead letter queue it sends all messages with the same sessionId to the dead letter queue?
Yes, you can do that. You can first peek the messages in your dead-letter queue to fetch all the session ids. Then you can receive the messages in your main queue whose session id is in the DLQ, and then move those messages to DLQ. Here's one such logic I've implemented in dot net using the latest version of Service Bus SDK.
var queueName = "<queue>";
var connectionString = "<connection-string>";
var client = new ServiceBusClient(connectionString);
var sessionIdInDLQList = new List<string>();
var receiver = client.CreateReceiver(queueName, new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter });
var message = await receiver.PeekMessageAsync();
while (message != null)
{
if (!sessionIdInDLQList.Contains(message.SessionId))
sessionIdInDLQList.Add(message.SessionId);
message = await receiver.PeekMessageAsync();
}
foreach (var sessionId in sessionIdInDLQList)
{
var session = await client.AcceptSessionAsync(queueName, sessionId);
message = await session.ReceiveMessageAsync(TimeSpan.FromSeconds(20));
while (message != null)
{
await session.DeadLetterMessageAsync(message, "Message with this session is to be dead-lettered!");
message = await session.ReceiveMessageAsync(TimeSpan.FromSeconds(20));
}
}
In your case, you need to do this before your consumers start reading the messages, probably you can write this in your consumer application or any trigger application like Azure Function or worker role. That’s upto your method of handling.
You can try this code to read Dead Letter from Queue.
public static async Task GetMessage()
{
string topic = "myqueue1";
string connectionString = "Endpoint = sb://xxx.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxx";
var servicebusclient = new ServiceBusClient(connectionString);
var reciveroptions = new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter };
var reciver = servicebusclient.CreateReceiver(topic, reciveroptions);
// 10 number of message read from Queue
await receiver.PeekMessageAsync(10);
}
after receiving message from Dead Letter you can send to queue.
As per Microsoft official documents
There's no automatic cleanup of the DLQ. Messages remain in the DLQ
until you explicitly retrieve them from the DLQ and call Complete() on
the dead-letter message.
These following document help you.
Thanks Casually Coding for posting post on Read Message from the Dead Letter Queue
Microsoft Documents Using Dead-Letter Queues to Handle Message Transfer Failures , Receive Message from Dead letter queue
How to read the message from the service bus dead-letter queue? I'm able to read message-id and a sequence number of the message, but I need the actual message. Can someone help me with this? Is it possible to read the actual message?
Before reading message form deadletter queue, you should check what was the reason of fail? If some service was not available then Create WebJob and try below code and process messages.
public void GetDeadLetterMessagesAsync(string connectionString, string queueName)
{
var queueClient = QueueClient.CreateFromConnectionString(connectionString, QueueClient.FormatDeadLetterPath(queueName));
while (true)
{
BrokeredMessage bmessgage = queueClient.Receive();
if (bmessgage != null)
{
string msg = new StreamReader(bmessgage.GetBody<Stream>(), Encoding.UTF8).ReadToEnd();
//Custom business logic to prcess your message
bmessgage.Complete();
}
else
{
break;
}
}
}
If message having an issue then you should read and display message on UI so back office team can correct message otherwise it will fail again.
I have created a service bus queue in Azure and it works well. And if the message is not getting delivered within default try (10 times), it is correctly moving the message to the dead letter queue.
Now, I would like to resubmit this message from the dead letter queue back to the queue where it originated and see if it works again. I have tried the same using service bus explorer. But it gets moved to the dead letter queue immediately.
Is it possible to do the same, and if so how?
You'd need to send a new message with the same payload. ASB by design doesn't support message resubmission.
We had a batch of around 60k messages, which need to be reprocessed from the dead letter queue. Peeking and send the messages back via Service Bus Explorer took around 6 minutes per 1k messages from my machine. I solved the issue by setting a forward rule for DLQ messages to another queue and from there auto forward it to the original queue. This solution took around 30 seconds for all 60k messages.
Try to remove dead letter reason
resubmittableMessage.Properties.Remove("DeadLetterReason");
resubmittableMessage.Properties.Remove("DeadLetterErrorDescription");
full code
using Microsoft.ServiceBus.Messaging;
using System.Transactions;
namespace ResubmitDeadQueue
{
class Program
{
static void Main(string[] args)
{
var connectionString = "";
var queueName = "";
var queue = QueueClient.CreateFromConnectionString(connectionString, QueueClient.FormatDeadLetterPath(queueName), ReceiveMode.PeekLock);
BrokeredMessage originalMessage
;
var client = QueueClient.CreateFromConnectionString(connectionString, queueName);
do
{
originalMessage = queue.Receive();
if (originalMessage != null)
{
using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
{
// Create new message
var resubmittableMessage = originalMessage.Clone();
// Remove dead letter reason and description
resubmittableMessage.Properties.Remove("DeadLetterReason");
resubmittableMessage.Properties.Remove("DeadLetterErrorDescription");
// Resend cloned DLQ message and complete original DLQ message
client.Send(resubmittableMessage);
originalMessage.Complete();
// Complete transaction
scope.Complete();
}
}
} while (originalMessage != null);
}
}
}
Thanks to some other responses here!
We regularly need to resubmit messages. The answer from #Baglay-Vyacheslav helped a lot. I've pasted some updated C# code that works with the latest Azure.Messaging.ServiceBus Nuget Package.
Makes it much quicker/easier to process DLQ on both queues/topics/subscribers.
using Azure.Messaging.ServiceBus;
using System.Collections.Generic;
using System.Threading.Tasks;
using NLog;
namespace ServiceBus.Tools
{
class TransferDeadLetterMessages
{
// https://github.com/Azure/azure-sdk-for-net/blob/Azure.Messaging.ServiceBus_7.2.1/sdk/servicebus/Azure.Messaging.ServiceBus/README.md
private static Logger logger = LogManager.GetCurrentClassLogger();
private static ServiceBusClient client;
private static ServiceBusSender sender;
public static async Task ProcessTopicAsync(string connectionString, string topicName, string subscriberName, int fetchCount = 10)
{
try
{
client = new ServiceBusClient(connectionString);
sender = client.CreateSender(topicName);
ServiceBusReceiver dlqReceiver = client.CreateReceiver(topicName, subscriberName, new ServiceBusReceiverOptions
{
SubQueue = SubQueue.DeadLetter,
ReceiveMode = ServiceBusReceiveMode.PeekLock
});
await ProcessDeadLetterMessagesAsync($"topic: {topicName} -> subscriber: {subscriberName}", fetchCount, sender, dlqReceiver);
}
catch (Azure.Messaging.ServiceBus.ServiceBusException ex)
{
if (ex.Reason == Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityNotFound)
{
logger.Error(ex, $"Topic:Subscriber '{topicName}:{subscriberName}' not found. Check that the name provided is correct.");
}
else
{
throw;
}
}
finally
{
await sender.CloseAsync();
await client.DisposeAsync();
}
}
public static async Task ProcessQueueAsync(string connectionString, string queueName, int fetchCount = 10)
{
try
{
client = new ServiceBusClient(connectionString);
sender = client.CreateSender(queueName);
ServiceBusReceiver dlqReceiver = client.CreateReceiver(queueName, new ServiceBusReceiverOptions
{
SubQueue = SubQueue.DeadLetter,
ReceiveMode = ServiceBusReceiveMode.PeekLock
});
await ProcessDeadLetterMessagesAsync($"queue: {queueName}", fetchCount, sender, dlqReceiver);
}
catch (Azure.Messaging.ServiceBus.ServiceBusException ex)
{
if (ex.Reason == Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityNotFound)
{
logger.Error(ex, $"Queue '{queueName}' not found. Check that the name provided is correct.");
}
else
{
throw;
}
}
finally
{
await sender.CloseAsync();
await client.DisposeAsync();
}
}
private static async Task ProcessDeadLetterMessagesAsync(string source, int fetchCount, ServiceBusSender sender, ServiceBusReceiver dlqReceiver)
{
var wait = new System.TimeSpan(0, 0, 10);
logger.Info($"fetching messages ({wait.TotalSeconds} seconds retrieval timeout)");
logger.Info(source);
IReadOnlyList<ServiceBusReceivedMessage> dlqMessages = await dlqReceiver.ReceiveMessagesAsync(fetchCount, wait);
logger.Info($"dl-count: {dlqMessages.Count}");
int i = 1;
foreach (var dlqMessage in dlqMessages)
{
logger.Info($"start processing message {i}");
logger.Info($"dl-message-dead-letter-message-id: {dlqMessage.MessageId}");
logger.Info($"dl-message-dead-letter-reason: {dlqMessage.DeadLetterReason}");
logger.Info($"dl-message-dead-letter-error-description: {dlqMessage.DeadLetterErrorDescription}");
ServiceBusMessage resubmittableMessage = new ServiceBusMessage(dlqMessage);
await sender.SendMessageAsync(resubmittableMessage);
await dlqReceiver.CompleteMessageAsync(dlqMessage);
logger.Info($"finished processing message {i}");
logger.Info("--------------------------------------------------------------------------------------");
i++;
}
await dlqReceiver.CloseAsync();
logger.Info($"finished");
}
}
}
It may be "duplicate message detection" as Peter Berggreen indicated or more likely if you are directly moving the BrokeredMessage from the dead letter queue to the live queue then the DeliveryCount would still be at maximum and it would return to the dead letter queue.
Pull the BrokeredMessage off the dead letter queue, get the content using GetBody(), create in new BrokeredMessage with that data and send it to the queue. You can do this in a safe manor, by using peek to get the message content off the dead letter queue and then send the new message to the live queue before removing the message from the dead letter queue. That way you won't lose any crucial data if for some reason it fails to write to the live queue.
With a new BrokeredMessage you should not have an issue with "duplicate message detection" and the DeliveryCount will be reset to zero.
The Service Bus Explorer tool always creates a clone of the original message when you repair and resubmit a message from the deadletter queue. It could not be any different as by default Service Bus messaging does not provide any message repair and resubmit mechanism. I suggest you to investigate why your message gets ends up in the deadletter queue as well as its clone when you resubmit it. Hope this helps!
It sounds like it could be related to ASB's "duplicate message detection" functionality.
When you resubmit a message in ServiceBus Explorer it will clone the message and thereby the new message will have the same Id as the original message in the deadletter queue.
If you have enabled "Requires Duplicate Detection" on the queue/topic and you try to resubmit the message within the "Duplicate Detection History Time Window", then the message will immediately be moved to the deadletter queue again.
If you want to use Service Bus Explorer to resubmit deadletter messages, then I think that you will have to disable "Requires Duplicate Detection" on the queue/topic.
I have two clients listening to a subscription of topic . I am seeing following issues.
Issue 1:
If one client receives a message and marks them completes other client won't receive that messages.
But i want all my clients to receive that message and acknowledge it so that once a client receives the message it shouldn't receive again
Issue 2:
If i don't acknowledge the message as complete . Some times i am receiving the message multiple times.
Issue 3:
During the Message Receive if see the Network disconnect. After some time it reconnects I am receiving the messages from starting again.
For Sending the Message Code
============
if (!namespaceManager.TopicExists("DataCollectionTopic"))
namespaceManager.CreateTopic("DataCollectionTopic");
if (!namespaceManager.SubscriptionExists("DataCollectionTopic", "one"))
namespaceManager.CreateSubscription("DataCollectionTopic", "one");
for(int i=0;i<100;i++)
{
BrokeredMessage bm = new BrokeredMessage("new Topic one");
bm.Label = "hELLLOOOO xcvxvxcvxvxvxc DummyMEssage"+i;
bm.Properties["StoreName"] = "asdasdasqwedas";
bm.Properties["MachineID"] = "Bajjiiiqweq567567wii";
if (namespaceManager == null)
{
Console.WriteLine("\nUnexpected Error");
return;
}
MessageSender sender = messageFactory.CreateMessageSender("DataCollectionTopic");
sender.Send(bm);
for receiving the message
===================
MessageReceiver receiver = await messageFactory.CreateMessageReceiverAsync("DataCollectionTopic/subscriptions/Vijay");
while (true) {
BrokeredMessage receivedMessage = receiver.Receive();
try
{
ProcessMessage(receivedMessage);
// receivedMessage.Complete();
}
catch (Exception e)
{
// receivedMessage.Abandon();
}
}
}
===============
TIA
Issue 1
If both clients receive using the same subscription name, they act as competing consumers. The first consumer to get the message and mark it as completed (processed) will be the winner. The rest of the clients won't process the same message. If intention is to receive a message (event) by all clients, then each client should have its own subscription and a copy of the message will be delivered to each subscription.
Issue 2
That is an expected behaviour. You receive the message up to MaxDeliverCount and after that message will be DLQ-ed. This is a default behaviour for PeekLock mode. With ReceiveAndDelete this doesn't happen, but it's a dangerous mode as you will lose the message if not processed successfully.
Issue 3
That is an expected behaviour. Each received message in a PeekLock mode has a LockDuration, time given to a processing code to complete the message or abandon it. If not completed, message becomes visible to other competing consumers and will be reprocessed up to DeliveryCount times.
To sum it up
Have multiple subscriptions per client for all of the clients to receive the same message (event)
Complete messages if done processing successfully. If you need time extension, renew the lock.
I have an application subscribed on Azure Servicebus Topic who is constantly receiving messages from Stream Analytics. But this application isn't every time subscribed on this Topic. How do I receive only the last message from the topic when the application do the subscription?
Based on your question and your comments, this is what I can advice you:
When your application starts, connect to the Azure ServiceBus Subscription and get all messages in the queue.
Remove all the previous messages (just complete it) and process the last message.
Then you can start listening to new incoming messages.
Based on this answer (Clearing azure service bus queue in one go) :
// Get the message receiver
var messagingFactory = MessagingFactory.CreateFromConnectionString("ServiceBusConnectionString");
var messageReceiver = messagingFactory.CreateMessageReceiver(SubscriptionClient.FormatSubscriptionPath("TopicName", "SubscriptionName"));
BrokeredMessage lastMessage = null;
while (messageReceiver.Peek() != null)
{
if(lastMessage != null)
{
// This was not the last message so complete it.
lastMessage.Complete();
}
// Batch the receive operation
var brokeredMessages = messageReceiver.ReceiveBatch(300).ToList();
//Get the last message and remove it from the list
lastMessage = brokeredMessages.Last();
brokeredMessages.RemoveAt(brokeredMessages.Count -1);
// Complete all the other messages
var completeTasks = brokeredMessages.Select(m => Task.Run(() => m.Complete())).ToArray();
// Wait for the tasks to complete.
Task.WaitAll(completeTasks);
}
if (lastMessage != null)
{
// Process your message
}
// Start listening to new incoming message
messageReceiver.OnMessage(message =>
{
// Process new messages
}, new OnMessageOptions());