We are using apache-pulsar's retry topic and dead letter topic to handle failures in event processing. In case of a failure in processing, event is retried 3 times via retry topic and if it still fails then message is added in DLQ topic.
Is there a way to add failure message in event while adding that in DLQ after 3 failures?
Example:
Event:
{
"messageId": "message-1234",
"data": {...}
}
I want this event to have error field if it goes in DLQ topic after all retries are exhausted. This will help in understanding the reason of failure while analyzing the DLQ.
Event in DLQ:
{
"messageId": "message-1234",
"data": {...},
"errorReason": "Error reason from exception"
}
When you call the reconsumeLaterAsync(), you can pass the properties which is a customized key-value pair list. The properties will be copied to the message that sends to DLQ. Version 2.10.0 is required.
Here is an example.
Related
I have an Azure Function which listens to azure queue and, for example, something wrong. It re-adds a message to the queue again. But after 5 times message will be moved to poison queue.
I want to re-add a message to queue with a delay. For example, retry thru 1 hour. Because my Azure Function works with external resource, which can be unavailable for now. I don't want to do retry 5 times during 10 seconds at all, I want to retry after 1 hour. Of course, I write my own implementation of it, but probably this feature already exists.
#4c74356b41 has pointed out the right way. The host.json settings for queue is what you are looking for.
visibilityTimeout is The time interval between retries when processing of a message fails
maxDequeueCount is The number of times to try processing a message before moving it to the poison queue.
{
"version": "2.0",
"extensions": {
"queues": {
"visibilityTimeout" : "01:00:00",
"maxDequeueCount": 2
}
}
}
If your function is v1, similarly
{
"queues": {
"visibilityTimeout" : "01:00:00",
"maxDequeueCount": 2
}
}
Update
Since the problem is mainly about changing visibilityTimeout according to specific situation, setting the delay of CloudQueue.AddMessageAsync accordingly is the only way. Actually the visibilityTimeout does exactly the same thing but on the function app level(all queue), so we don't need to insist on it in this case.
I try to get 32 messages per request from Azure Queue.
queue.ApproximateMessageCount;
This code gives me the result of 1509. Telling me the connection is OK and it has records. Also I check in queue it really has 1509 records.
But when I try to retrieve records I don't get any record.
I do the following:
var messages = await queue.GetMessagesAsync(configuration.MessageBatchSize);
if (!messages.Any()) {
return;
}
It always goes in the if and returns.
What is going on here and what am I missing?
Do do that, receiving messages in batch mode, i use this kind of code :
var messages = await queueClient?.ReceiveBatchAsync(Max_Messages);
foreach (var message in messages)
{
await dispatcher.Dispatch(message); // do something with each message
}
But, for receiving messages with ReceiveBatchAsync, the queue have to be configured with the EnableBatchedOperations flag to true.
ApproximateMessageCount property represents the total number of messages available in queue at that particular moment. It does not represent that all messages (max #32 messages in a pull) are ready to be dequeued. You can use this property to infer that how many messages are in queue.
queue.ApproximateMessageCount;
If you could not retrieve the message by, GetMessagesAsync(numberOfMessages), then it says that all messages are not available or invisible for current QueueClient.
var cloudQueueMessages = await cloudQueue.GetMessagesAsync(numberOfMessages);
You could try polling the queue after sometime to see if messages came back to surface.
Note that, be advised of setting adequate visibility timeout for any message being dequeued to avoid indefinite starvation :)
I have an issue with my rabbit mq setup using node.
A brief gist of the code is
queue.subscribe({ack: true}, function(msg) {
// Do some processing
queue.shift(false);
}
I expect messages to be read and applied serially -> the second message should start processing only after first message is applied and acknowledged.
But this is not happening. I see that in some cases the second message starts processing before the first message is acknowledged. The default prefetch count is 1 so this should not happen.
What am I missing?
I am continuously receiving messages in peek mode and abandoning them if processing fails (Not the delivery). However, the message immediately becomes available again and is received for processing again. It fails quickly again and after max deliveries it is dead-lettered.
Is there a way to configure the topic/subscription to wait before releasing the message after it is abandoned? Preferably in exponential manner.
Of course I am open for suggestions via the code as well.
There is not a way to set an exponential back-off in the Service Bus configuration. I've experienced the same issue, and have done the following:
Dequeue the message, and marked the message as received.
Perform processing within a try/catch block. If there is an exception, enqueue a new message with a scheduled delivery at a certain point in the future.
We've wrapped our Service Bus Message Queue payload in a class that specifies the number of delivery attempts. We multiply the number of delivery attempts times a constant, then add that number to the current dateTime for scheduled delivery in the future. After the number of delivery attempts that we want to try are exceeded, we explicitly dead letter the message.
Edit 7-17-2020
Consider using Azure Durable Task Framework which has a customizable retry policy built right in.
Another option is using MassTransit which has support for Azure Service Bus.
Take a look at its extensive retry configuration.
Note that MassTransit is effectively doing the retries in memory after the message has been received so you'll need to adjust your topic subscription's MaxDeliveryCount and MaxAutoRenewDuration settings appropriately.
Your configuration might look something like:
var busControl = Bus.Factory.CreateUsingAzureServiceBus(cfg =>
{
var host = cfg.Host(serviceBusConnectionString, hst => { });
cfg.UseMessageRetry(retryConfigurator =>
RetryConfigurationExtensions.Exponential(retryConfigurator, ...);
cfg.SubscriptionEndpoint(
"subscriptionName",
"topicPath",
e =>
{
e.Consumer<SomeConsumer>();
// Let MassTransit do the retries
e.MaxDeliveryCount = 1;
e.MaxAutoRenewDuration = TimeSpan.FromMinutes(10);
});
});
You can use Scheduled Messages for this.
Essentially, when you need to retry a message you just schedule it in the future and Service Bus will add it to the queue again once enough time has passed:
ServiceBusSender queue = ...
int secs = // calculate the delay
var msg = new ServiceBusMessage("My scheduled message body")
{
ApplicationProperties =
{
["RetryCount"] = retryCount,
},
};
logger.LogInformation("Scheduling for " + secs + " secs");
await queue.ScheduleMessageAsync(msg, DateTimeOffset.Now.AddSeconds(secs));
You must add information about retry count in a header or the body. Otherwise you won't know how many times you have tried it and cannot really calculate the future date.
I am looking into this topic as well and I came across the class RetryExponential class from Microsoft.
RetryExponential Class
Namespace: Microsoft.ServiceBus
Assembly: Microsoft.ServiceBus.dll
Represents an implementation of a retry policy. For each time the messaging operation must be retried, the delay between retries grows in a staggered, exponential manner.
public sealed class RetryExponential : Microsoft.ServiceBus.RetryPolicy
Why would someone want to do that? I have to unit-test exception handling mechanism in our application.
I presumed that dead letter queue is literally azure service bus queue, where I could publish messages using QueueClient
string dlQ = #"sb://**.servicebus.windows.net/**/Subscriptions/DefaultSubscription/$DeadLetterQueue";
string connectionString = CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString");
NamespaceManager _namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
QueueDescription qd = _namespaceManager.GetQueue(dataPromotionDLQ);
var queueClient = QueueClient.CreateFromConnectionString(connectionString, "DefaultSubscription/$DeadLetterQueue");
BrokeredMessage brokeredMessage = new BrokeredMessage("Message to PublishToDLQ");
try
{
queueClient.Send(brokeredMessage);
}
catch (Exception)
{
}
But I get MessagingEntityNotFoundException. What could be wrong?
You would never want to publish directly to a dead letter queue. It's where poisoned messages that can't be processed are placed.
There are two ways of placing messages onto the dead letter queue. The service bus itself dead-letters messages that have exceeded the maximum number of delivery attempts. You can also explicitly dead-letter a message that you have received using the DeadLetter() method.
Create your messages with a very short TTL via the BrokeredMessage.TimeToLive property.
The Subscription must have EnableDeadLetteringOnMessageExpiration set to true.
Though late here, adding to the answers of #Mikee and #Ben Morris may help someone. You can make use of #Mike's suggestion of making use of message.DeadLetter() or message.DeadLetterAsync() to dead-letter a message. Another suggestion can be to set very less or 0 second TimeToLive to move the messages to Dead letter.
After you perform any of these and try to view the messages in the Active end queue, you may still find that message is available sometimes (Which you are currently facing). The reason is that the messages that are dead-lettered due to TTLExpiredException, HeaderSizeExceeded or any system defined Errors, or manually Dead-Lettered messages like DeadLetter() methods are cleaned up by an asynchronous "garbage collection" program periodically. This doesn't occur immediately which we expect it to.
When you perform Peek operation, you can still see that the message is in the Active queue. You have to wait for the garbage collector to run or you can perform a Receive operation which forces the garbage collector to run first, thereby moving the messages to dead-letter before retrieval is done.