I have a current topic queue which send some messages to deadletter queue. I am creating another Webjob which listen to ServiceBusTrigger on the deadletter queue. The purpose is to resubmit the deadletter mesg to original queue to be re-processed
When a new message comes to the deadletter queue, I would clone the message and send it back to the original topic subscription to re-process.
I expect that the clone message would be sent to the original queue and stay there until it is processed, but turn out the message is completed as soon as the [originalTopicClient.Send(cloneMessage);] function called.
Am I missing anything?
public void ProcessQueueMessage([ServiceBusTrigger(topicName, dlqSubscription)] string message, TextWriter log)
{
MessagingFactory factory = MessagingFactory.Create(ServiceURI, tokenProvider);
string deadLetterQueuePath = SubscriptionClient.FormatDeadLetterPath(topicName, SubscriptionName);
SubscriptionClient dlqSubscriptionClient = factory.CreateSubscriptionClient(topicName, "mytopicsubscription/$DeadLetterQueue");
QueueClient deadletterQueueClient = factory.CreateQueueClient(deadLetterQueuePath);
BrokeredMessage cloneMessage;
Microsoft.ServiceBus.Messaging.TopicClient originalTopicClient = Microsoft.ServiceBus.Messaging.TopicClient.CreateFromConnectionString(serviceBusEndpoint, topicName);
BrokeredMessage dlqMessage;
if ((dlqMessage = dlqSubscriptionClient.Receive()) != null)
{
cloneMessage = dlqMessage.Clone();
cloneMessage.Properties.Remove("DeadLetterReason");
cloneMessage.Properties.Remove("DeadLetterErrorDescription");
originalTopicClient.Send(cloneMessage); //resend the clone message to original queue
dlqMessage.Complete(); //deadletter queue mesg completed
}
}
I faced a similar issue and then after quite a long research, I found that it worked perfectly fine in 3 ways.
The first one is to receive the message in ReceiveAndDelete mode. This deletes the message from the DLQ and so we can safely send it to the main queue without issues. The second way is to receive the message in PeekLock mode, Complete the message, and then Send it. This is nothing different from the first method except that we are completing rather than the Service Bus client doing it. The third way is to create a new BrokeredMessage and copy the necessary payload from the received message and send it as an entirely new message to the main queue.
While the reason for the Send to not work in your method is still unknown, I see that the Clone and Send work fine without Complete, and it seems like the clone is still referring to the original message.
Related
I've been recently having problems with my Service Bus queue. Random messages (one can pass and the other not) are placed on the deadletter queue with the error message saying:
"DeadLetterReason": "Moved because of Unable to get Message content There was an error deserializing the object of type System.String. The input source is not correctly formatted."
"DeadLetterErrorDescription": "Des"
This happens even before my consumer has the chance to receive the message from the queue.
The weird part is that when I requeue the message through Service Bus Explorer it passes and is successfully received and handled by my consumer.
I am using the same version of Service Bus either for sending and receiving the messages:
Azure.Messaging.ServiceBus, version: 7.2.1
My message is being sent like this:
await using var client = new ServiceBusClient(connString);
var sender = client.CreateSender(endpointName);
var message = new ServiceBusMessage(serializedMessage);
await sender.SendMessageAsync(message).ConfigureAwait(true);
So the solution I have for now for the described issue is that I implemented a retry policy for the messages that land on the dead-letter queue. The message is cloned from the DLQ and added again to the ServiceBus queue and for the second time there is no problems and the message completes successfully. I suppose that this happens because of some weird performance issues I might have in the Azure infrastructure. But this approach bought me some time to investigate further.
I have dot net code which read loan-numbers from Azure service queue and calls my API for each loan-number.
This is my code which calls the api
private async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
try
{
string loanNumber = Encoding.UTF8.GetString(message.Body);
_logger.LogInformation($"Received message: SequenceNumber:{message.SystemProperties.SequenceNumber} LoanNumber:{loanNumber}");
//API CALL HERE
await _apiClient.getResult(loanNumber);
await _queueClient.CompleteAsync(message.SystemProperties.LockToken);
}
catch (Exception ex)
{
//sending failed messages to Dead Letter queue
await _queueClient.AbandonAsync(message.SystemProperties.LockToken);
}
}
Failed loan-numbers are successfully sent to Dead Letter queue. When the server is down or bad request from API response.
I want to call the api after certain duration on the loan-numbers which are in Dead Letter Queue.
Is there any way to convert the messages in Dead Letter queue to active messages after some interval??
I am new to azure. Please help me to resolve the issue.
Thanks in advance.
Is there any way to convert the messages in Dead Letter queue to
active messages after some interval??
Automatically, no. However what you could do is read the messages from dead letter queue and then post them as new message in your main queue.
As far as automating the whole process, one possible solution would be to run a timer triggered Azure Function that reads messages from a dead letter queue periodically and post them in your main queue.
#akhil, it's worth noting that these messages will automatically be re-queued on the main queue until their DeliveryCount exceeds the MaxDeliveryCount of your main queue. The default value for MaxDeliveryCount is 10 so any failed requests to the API be retried ten times by this handler before being moved to the DLQ.
If you wanted to be a bit 'smarter' about this you could delay the retries using the ScheduledEnqueueTimeUtc property on the message in your catch block.
As #Gaurav Mantri has said, the framework offers nothing to process dead-letter messages 'for free'; you'll have to code a handler yourself as you would for a normal queue.
No, there is no way to convert the DLQ message to active message.
you need to process the DLQ and fix the reason as to why it ended in DLQ and then re transmit it back to your queue.
A sample from Github can be useful here.
https://github.com/Azure/azure-service-bus/tree/master/samples/DotNet/Microsoft.ServiceBus.Messaging/DeadletterQueue
(This sample shows how to move messages to the Dead-letter queue, how to retrieve messages from it, and resubmit corrected message back into the main queue.)
This could be an old answer but we can use ServiceBusExplorer from Microsoft. It can pull the messages from dead letter queue and then those can be requeued using
Let me know if you have any further questions.
C# .NetCore 2.2 -
Azure Service Bus 3.4.0
I have 3 queues in Azure Service Bus with same properties. While sending messages to these queues, the messages in one of the queues always get delivered to Dead letter queues, while other 2 queues receive active messages.
I have tried playing with the properties - increase TTL, maximum delivery count etc. The properties of all 3 queues are same, the only difference is the name of the queues.
I have used this tutorial - https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-get-started-with-queues
queue properties image
static async Task SendMessagesAsync(int numberOfMessagesToSend)
{
try
{
for (var i = 0; i < numberOfMessagesToSend; i++)
{
// Create a new message to send to the queue.
string messageBody = $"Message {i}";
var message = new Message(Encoding.UTF8.GetBytes(messageBody));
Console.WriteLine($"Sending message: {messageBody}");
// Send the message to the queue.
await queueClient.SendAsync(message);
}
}
catch (Exception exception)
{
Console.WriteLine($"{DateTime.Now} :: Exception: {exception.Message}");
}
}
How do I prevent messages from going to Dead Letter Queue? Why does it happen with only 1 queue, not the other 2?
When messages are dead-lettered, there is a reason user property gets added. Check that property to see the reason and troubleshoot accordingly. Specifically, check for DeadLetterReason and DeadLetterErrorDescription custom properties.
The common reasons for a message to be dead-lettered are
Maximum transfer hop count exceeded
Session Id Is Null
TTLExpiredException
HeaderSizeExceeded
The messages might also have got dead-lettered due to some errors while receiving the message from the Queue. As Sean Feldman mentioned, looking into the DeadLetterReason and DeadLetterDescription property will help you diagnose the error reason clearly.
Also try to increase or set the time to live of the message sent if the DeadLetterReason is TTLExpiredException. Because if you have set the time to live of the message to a lower value then it will override the time to live property of the Queue.
Check whether the Queue whether the queue where the messages are getting dead-lettered is a Session enabled queue and the message sent has the Session Id value set.
Without seeing your app / messages it's hard to help. But probably there's an error with the application that is trying to consume the message. As it could not be completed, the message goes to the dead letter queue.
Log the messages from this particular queue and see if there's any missing required properties. Sometimes you're trying to deserialize to an uncompatible type.
The purpose of the dead-letter queue is to hold messages that cannot
be delivered to any receiver, or messages that could not be processed.
In Azure WebJobs, in the OnMessageOptions class, I'm calling the QueueClient.Complete(Guid) method by setting the AutoComplete flag to true and messages seem to dequeue just fine when running the ProcessQueue function. Active messages count goes down by 1 after successful processing of each message. However, when I want to requeue a message (because it cannot be processed currently) back to the queue that triggers the service bus function, as a new brokered message after a minute, using BrokeredMessage.ScheduledEnqueueTimeUtc, it seems like it isn't working. Scheduled messages count seems to go up initially. I go back to the queue after a few hours and see active messages in the thousands. The copies are of the same message. What is happening? I'd expect the message to be taken off the queue because of QueueClient.Complete(Guid) and the new scheduled message to be its replacement.
Some detail:
To send the message I do the following:
var queueclient = QueueClient.CreateFromConnectionString(connectionString, queueName);
queueclient.Send(message);
queueclient.close();
Inside the WebJob I created a ServiceBusConfiguration object which requires a onMessageOptions object where I set the AutoComplete=true. I pass the ServiceBusConfiguration object to the JobHostConfiguration.UserServiceBus
method.
Inside the WebJob service bus queue triggered function I again do the following to requeue, by first creating a new instance of the brokered message again.
//if not available yet for processing please requeue...
var queueclient = QueueClient.CreateFromConnectionString(connectionString, queueName);
queueclient.Send(message);
queueclient.close();
I don't do the following/use callbacks which is may be why it isn't working?
var options = new OnMessageOptions();
options.AutoComplete = false; // to call complete ourselves
Callback to handle received messages
client.OnMessage(m =>
{
var clone = m.Clone();
clone.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddSeconds(60);
client.Send(clone);
m.Complete();
}, options);
when I want to requeue a message (because it cannot be processed currently) back to the queue that triggers the service bus function, as a new brokered message after a minute, using BrokeredMessage.ScheduledEnqueueTimeUtc, it seems like it isn't working
If you fail to process your message, do not re-queue it. Instead, abandon (with a reason) and it will be picked up again.
BrokeredMessage.ScheduledEnqueueTimeUtc is intended to be used for messages added to the queue. When you receive a message, you can complete, dead-letter, defer, or abandon. If you abandon a message, it will be retried, but you can't control when that will happen. If you have no other messages in the queue, it will be retried almost immediately.
Note: when you see a behaviour that you suspect is not right, having a simple repro to share would be very helpful.
I am encountering one major road block issue when trying to use ServiceBusTrigger in azureFunction. I am trying to abandon, or deadletter, a service bus message in V2 ServiceBusTrigger, How can I do so? I've tried the following solution, but I didn't get anywhere.
Here is the codeSample I used:
public async static Task Run(Message myQueueItem, TraceWriter log, ExecutionContext context)
{
log.Info($"C# ServiceBus queue trigger function processed message delivery count: {myQueueItem.SystemProperties.DeliveryCount}");
QueueClient queueClient = new QueueClient("[connectionstring]","[queueName]");
////await queueClient.DeadLetterAsync(myQueueItem.SystemProperties.LockToken);
await queueClient.AbandonAsync(myQueueItem.SystemProperties.LockToken);
}
Solution 1: I tried to substitute Message myQueueItem for BrokeredMessage like in V1, I then can call myQueueItem.Abandon, or deadletter, on the message lever. However It came back with exception:
Microsoft.Azure.WebJobs.Host: Exception binding parameter 'myQueueItem'. System.Private.DataContractSerialization: There was an error deserializing the object of type Microsoft.ServiceBus.Messaging.BrokeredMessage. The input source is not correctly formatted. System.Private.DataContractSerialization: The input source is not correctly formatted."
At least I can go one step further. to
solution 2. Solution 2: is to use:
QueueClient queueClient = new QueueClient("[connectionstring]","[queueName]");
////await queueClient.DeadLetterAsync(myQueueItem.SystemProperties.LockToken);
await queueClient.AbandonAsync(myQueueItem.SystemProperties.LockToken);
I can use the lock provided in the Message Object, however, when I try to send it with queueClient, It said the message gone from the queue. or no longer available.
Can anybody let me know if i am on the right track? If I am not, please kindly guide me in the right track.
Service Bus messages are automatically completed or abandoned by Azure Functions runtime based on the success/failure of the function call, docs:
The Functions runtime receives a message in PeekLock mode. It calls Complete on the message if the function finishes successfully, or calls Abandon if the function fails.
So, the suggested way to Abandon your message is to throw an exception from function invocation.