I am creating a sample Azure Service Bus application. I created a namespace, topic and Subscription. I have written test messages to the topic and if I go to the subscription in the portal, I see that I have a new message every time I write a new one using the writer application.
But when I go to pull the message, nothing is retrieved. In troubleshooting, I changed the subscription name to an incorrect value and received an error. I changed it back and I get no output and none of the messages are removed when I look in Azure portal. I'm stuck... this seems easy, but it isn't working.
string connectionString = "Endpoint=sb://redacted for obvious reasons";
SubscriptionClient Client = SubscriptionClient.CreateFromConnectionString(connectionString, "NewOrders", "AllOrders");
// Configure the callback options.
OnMessageOptions options = new OnMessageOptions();
options.AutoComplete = false;
options.AutoRenewTimeout = TimeSpan.FromMinutes(1);
Client.OnMessage((message) =>
{
try
{
Console.WriteLine("Body: " + message.GetBody<string>());
message.Complete();
Console.ReadLine();
}
catch (Exception)
{
// Indicates a problem, unlock message in subscription.
message.Abandon();
}
}, options);
It seems that it is not the code issue. I create a demo to retrieve message from topic, it works correctly.
Before I try to retrieve message from topic, I send the message to the topic or make sure that there are messages for subscription. We could check that from the portal
Send message code demo.
private static void SendMessages()
{
topicClient = TopicClient.Create(TopicName);
List<BrokeredMessage> messageList = new List<BrokeredMessage>
{
CreateSampleMessage("1", "First message information"),
CreateSampleMessage("2", "Second message information"),
CreateSampleMessage("3", "Third message information")
};
Console.WriteLine("Sending messages to topic...");
foreach (BrokeredMessage message in messageList)
{
while (true)
{
try
{
topicClient.Send(message);
}
catch (MessagingException e)
{
if (!e.IsTransient)
{
Console.WriteLine(e.Message);
throw;
}
}
Console.WriteLine($"Message sent: Id = {message.MessageId}, Body = {message.GetBody<string>()}");
break;
}
}
topicClient.Close();
}
I also try your mentioned code, it also works correctly for me.
We also could get the demo project from the cloud Project from the template. We also could get more info about How to use Service Bus topics and subscriptions from the document.
I have created a service bus queue in Azure and it works well. And if the message is not getting delivered within default try (10 times), it is correctly moving the message to the dead letter queue.
Now, I would like to resubmit this message from the dead letter queue back to the queue where it originated and see if it works again. I have tried the same using service bus explorer. But it gets moved to the dead letter queue immediately.
Is it possible to do the same, and if so how?
You'd need to send a new message with the same payload. ASB by design doesn't support message resubmission.
We had a batch of around 60k messages, which need to be reprocessed from the dead letter queue. Peeking and send the messages back via Service Bus Explorer took around 6 minutes per 1k messages from my machine. I solved the issue by setting a forward rule for DLQ messages to another queue and from there auto forward it to the original queue. This solution took around 30 seconds for all 60k messages.
Try to remove dead letter reason
resubmittableMessage.Properties.Remove("DeadLetterReason");
resubmittableMessage.Properties.Remove("DeadLetterErrorDescription");
full code
using Microsoft.ServiceBus.Messaging;
using System.Transactions;
namespace ResubmitDeadQueue
{
class Program
{
static void Main(string[] args)
{
var connectionString = "";
var queueName = "";
var queue = QueueClient.CreateFromConnectionString(connectionString, QueueClient.FormatDeadLetterPath(queueName), ReceiveMode.PeekLock);
BrokeredMessage originalMessage
;
var client = QueueClient.CreateFromConnectionString(connectionString, queueName);
do
{
originalMessage = queue.Receive();
if (originalMessage != null)
{
using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
{
// Create new message
var resubmittableMessage = originalMessage.Clone();
// Remove dead letter reason and description
resubmittableMessage.Properties.Remove("DeadLetterReason");
resubmittableMessage.Properties.Remove("DeadLetterErrorDescription");
// Resend cloned DLQ message and complete original DLQ message
client.Send(resubmittableMessage);
originalMessage.Complete();
// Complete transaction
scope.Complete();
}
}
} while (originalMessage != null);
}
}
}
Thanks to some other responses here!
We regularly need to resubmit messages. The answer from #Baglay-Vyacheslav helped a lot. I've pasted some updated C# code that works with the latest Azure.Messaging.ServiceBus Nuget Package.
Makes it much quicker/easier to process DLQ on both queues/topics/subscribers.
using Azure.Messaging.ServiceBus;
using System.Collections.Generic;
using System.Threading.Tasks;
using NLog;
namespace ServiceBus.Tools
{
class TransferDeadLetterMessages
{
// https://github.com/Azure/azure-sdk-for-net/blob/Azure.Messaging.ServiceBus_7.2.1/sdk/servicebus/Azure.Messaging.ServiceBus/README.md
private static Logger logger = LogManager.GetCurrentClassLogger();
private static ServiceBusClient client;
private static ServiceBusSender sender;
public static async Task ProcessTopicAsync(string connectionString, string topicName, string subscriberName, int fetchCount = 10)
{
try
{
client = new ServiceBusClient(connectionString);
sender = client.CreateSender(topicName);
ServiceBusReceiver dlqReceiver = client.CreateReceiver(topicName, subscriberName, new ServiceBusReceiverOptions
{
SubQueue = SubQueue.DeadLetter,
ReceiveMode = ServiceBusReceiveMode.PeekLock
});
await ProcessDeadLetterMessagesAsync($"topic: {topicName} -> subscriber: {subscriberName}", fetchCount, sender, dlqReceiver);
}
catch (Azure.Messaging.ServiceBus.ServiceBusException ex)
{
if (ex.Reason == Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityNotFound)
{
logger.Error(ex, $"Topic:Subscriber '{topicName}:{subscriberName}' not found. Check that the name provided is correct.");
}
else
{
throw;
}
}
finally
{
await sender.CloseAsync();
await client.DisposeAsync();
}
}
public static async Task ProcessQueueAsync(string connectionString, string queueName, int fetchCount = 10)
{
try
{
client = new ServiceBusClient(connectionString);
sender = client.CreateSender(queueName);
ServiceBusReceiver dlqReceiver = client.CreateReceiver(queueName, new ServiceBusReceiverOptions
{
SubQueue = SubQueue.DeadLetter,
ReceiveMode = ServiceBusReceiveMode.PeekLock
});
await ProcessDeadLetterMessagesAsync($"queue: {queueName}", fetchCount, sender, dlqReceiver);
}
catch (Azure.Messaging.ServiceBus.ServiceBusException ex)
{
if (ex.Reason == Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityNotFound)
{
logger.Error(ex, $"Queue '{queueName}' not found. Check that the name provided is correct.");
}
else
{
throw;
}
}
finally
{
await sender.CloseAsync();
await client.DisposeAsync();
}
}
private static async Task ProcessDeadLetterMessagesAsync(string source, int fetchCount, ServiceBusSender sender, ServiceBusReceiver dlqReceiver)
{
var wait = new System.TimeSpan(0, 0, 10);
logger.Info($"fetching messages ({wait.TotalSeconds} seconds retrieval timeout)");
logger.Info(source);
IReadOnlyList<ServiceBusReceivedMessage> dlqMessages = await dlqReceiver.ReceiveMessagesAsync(fetchCount, wait);
logger.Info($"dl-count: {dlqMessages.Count}");
int i = 1;
foreach (var dlqMessage in dlqMessages)
{
logger.Info($"start processing message {i}");
logger.Info($"dl-message-dead-letter-message-id: {dlqMessage.MessageId}");
logger.Info($"dl-message-dead-letter-reason: {dlqMessage.DeadLetterReason}");
logger.Info($"dl-message-dead-letter-error-description: {dlqMessage.DeadLetterErrorDescription}");
ServiceBusMessage resubmittableMessage = new ServiceBusMessage(dlqMessage);
await sender.SendMessageAsync(resubmittableMessage);
await dlqReceiver.CompleteMessageAsync(dlqMessage);
logger.Info($"finished processing message {i}");
logger.Info("--------------------------------------------------------------------------------------");
i++;
}
await dlqReceiver.CloseAsync();
logger.Info($"finished");
}
}
}
It may be "duplicate message detection" as Peter Berggreen indicated or more likely if you are directly moving the BrokeredMessage from the dead letter queue to the live queue then the DeliveryCount would still be at maximum and it would return to the dead letter queue.
Pull the BrokeredMessage off the dead letter queue, get the content using GetBody(), create in new BrokeredMessage with that data and send it to the queue. You can do this in a safe manor, by using peek to get the message content off the dead letter queue and then send the new message to the live queue before removing the message from the dead letter queue. That way you won't lose any crucial data if for some reason it fails to write to the live queue.
With a new BrokeredMessage you should not have an issue with "duplicate message detection" and the DeliveryCount will be reset to zero.
The Service Bus Explorer tool always creates a clone of the original message when you repair and resubmit a message from the deadletter queue. It could not be any different as by default Service Bus messaging does not provide any message repair and resubmit mechanism. I suggest you to investigate why your message gets ends up in the deadletter queue as well as its clone when you resubmit it. Hope this helps!
It sounds like it could be related to ASB's "duplicate message detection" functionality.
When you resubmit a message in ServiceBus Explorer it will clone the message and thereby the new message will have the same Id as the original message in the deadletter queue.
If you have enabled "Requires Duplicate Detection" on the queue/topic and you try to resubmit the message within the "Duplicate Detection History Time Window", then the message will immediately be moved to the deadletter queue again.
If you want to use Service Bus Explorer to resubmit deadletter messages, then I think that you will have to disable "Requires Duplicate Detection" on the queue/topic.
I have an application subscribed on Azure Servicebus Topic who is constantly receiving messages from Stream Analytics. But this application isn't every time subscribed on this Topic. How do I receive only the last message from the topic when the application do the subscription?
Based on your question and your comments, this is what I can advice you:
When your application starts, connect to the Azure ServiceBus Subscription and get all messages in the queue.
Remove all the previous messages (just complete it) and process the last message.
Then you can start listening to new incoming messages.
Based on this answer (Clearing azure service bus queue in one go) :
// Get the message receiver
var messagingFactory = MessagingFactory.CreateFromConnectionString("ServiceBusConnectionString");
var messageReceiver = messagingFactory.CreateMessageReceiver(SubscriptionClient.FormatSubscriptionPath("TopicName", "SubscriptionName"));
BrokeredMessage lastMessage = null;
while (messageReceiver.Peek() != null)
{
if(lastMessage != null)
{
// This was not the last message so complete it.
lastMessage.Complete();
}
// Batch the receive operation
var brokeredMessages = messageReceiver.ReceiveBatch(300).ToList();
//Get the last message and remove it from the list
lastMessage = brokeredMessages.Last();
brokeredMessages.RemoveAt(brokeredMessages.Count -1);
// Complete all the other messages
var completeTasks = brokeredMessages.Select(m => Task.Run(() => m.Complete())).ToArray();
// Wait for the tasks to complete.
Task.WaitAll(completeTasks);
}
if (lastMessage != null)
{
// Process your message
}
// Start listening to new incoming message
messageReceiver.OnMessage(message =>
{
// Process new messages
}, new OnMessageOptions());
I have a topic and a subscription that are handled in an azure web job, but some messages should be moved to a dead letter (queue or topic?) after a certain amount of retries. I have no idea what it takes to handle dead letter messages. Does somebody has a code example? Is this possible with azure web jobs?
I'm almost giving up and doing it manually using a retry counter.
For the time being, this what I'm doing, but I do not really like the idea to add the message back to the same queue:
public void SynchronizeConsumer(
[ServiceBusTrigger("topic")] Consumer consumer,
[ServiceBus("topic")] ICollector withError)
{
try
{
this.consumerSync.SyncConsumer(consumer);
}
catch (Exception ex)
{
consumer.NbOfRetries++; consumersWithError.Add(consumer);
}
}
Your messages are going to be moved to a deadletter subscription (= sub-queue). You can access messages from the deadletter subscription in the same way you access message from the normal subscription.
The path of your deadletter subscription is :
topicPath/Subscriptions/subscriptionName/$DeadLetterQueue
If you use ServiceBusTrigger, you function should looks like that:
public void ProcessMessage(
[ServiceBusTrigger("topicName", "subscriptionName")] BrokeredMessage message)
{
try
{
// Process you message
...
// Complete the message
message.Complete();
}
catch
{
message.Abandon();
}
}
So the function to access the deadletter subscription should be something like that:
public void ProcessDeadletterMessage(
[ServiceBusTrigger("topicName", "subscriptionName/$DeadLetterQueue")] BrokeredMessage message)
{
...
}
I've written some code to delete test messages off a service bus topic. I'm the only one using this topic. It's using ReceiveAndDelete mode so I am assuming it's going to delete them, but every time I run the code it goes through this cycle of receiving messages, so I know it's not deleting them. What am I doing wrong?
public void TruncateTopic()
{
// reset topic for testing..
SubscriptionClient client = SubscriptionClient.CreateFromConnectionString(
connStr, QUEUENAME, "AllMessages",ReceiveMode.ReceiveAndDelete);
BrokeredMessage message = client.Peek();
while (message != null)
{
client.Receive();
message = client.Peek();
}
client.Close();
}
In your code you only do Peek on the Topic/Queue. Peek action never deletes the messages.
As you can clearly read from documentation Peek method only peeks into the subscription without actually receiving the message.
The ReceiveAndDelete receive mode will well work when you not just Peek the messages but Receive them instead! That is why it is named ReceiveAndDelete but not PeekAndDelete.
Change your code to:
BrokeredMessage message = client.Receive();
while (message != null)
{
message = client.Receive();
}
And everything will be fine.