I’m using Azure WebJobs SDK 2.0 and when I specify VisibilityTimeout then MaxDequeuCount, the message is not removed from the queue when it fails 3 times but only copied to poison queue. You can see the DequeueCount is greater than MaxDequeuCount and the message is still in the queue.
class Program
{
// Please set the following connection strings in app.config for this WebJob to run:
// AzureWebJobsDashboard and AzureWebJobsStorage
static void Main()
{
var config = new JobHostConfiguration();
if (config.IsDevelopment)
{
config.UseDevelopmentSettings();
}
config.Queues.BatchSize = 8;
config.Queues.MaxDequeueCount = 3;
config.Queues.VisibilityTimeout = TimeSpan.FromSeconds(5);
config.Queues.MaxPollingInterval = TimeSpan.FromSeconds(3);
var host = new JobHost(config);
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
}
The function is throwing an exception to mimic the error condition.
public class Functions
{
// This function will get triggered/executed when a new message is written
// on an Azure Queue called queue.
public static void ProcessQueueMessage([QueueTrigger("queue")] string message, TextWriter log)
{
log.WriteLine(message);
throw new Exception("There was an error processing the message");
}
}
After three tries the message is moved to the poison queue, which is expected but after 10 minutes or so the message that was moved to poison queue appears again in the queue.
console output
In the queue you can see that Dequeue count is greater than MaxDequeuCount and still the message is not deleted from the queue.queue
In the poison queue you can see M1 message processed twice.
message is not removed from the queue when it fails 3 times but only copied to poison queue
As far as I know, currently we cannot use Storage SDK 8.x with WebJobs SDK, others have reported the similar issue.
Poison messages stay undeleted-but-invisible with the latest
WindowsAzure.Storage
8.1.1
WebJobs V1.1.2 fails to remove poison messages from queue with
V8.0.1 of
Storage
If you are using Azure Storage SDK 8.x, please try to downgrad the version of Azure Storage SDK. Besides, in the second link, asiffermann shared us a workaround with sample: write a custom QueueProcessor to create a new CloudQueueMessage in CopyMessageToPoisonQueueAsync, please refer to it.
Related
How can we move a service bus queue message to the dead letter through service bus queue trigger function
https://github.com/Azure/azure-webjobs-sdk/issues/1986#issuecomment-433960534
In v3, you can bind to the MessageReceiver class, which exposes methods like DeadLetter, Abaondon, Complete, etc. Example:
public static async Task ProcessMessage(
[ServiceBusTrigger("myqueue")] string message, int deliveryCount,
MessageReceiver messageReceiver,
string lockToken)
{
. . .
await messageReceiver.DeadLetterAsync(lockToken);
. . .
}
In this example, the message is bound as a string and the various message properties including lockToken are bound as params. You can also bind the message as a Message Type and access the requisite message properties from there. In v2 the ServiceBus SDK exposed these message methods directly on the BrokeredMessage class itself, but in the latest version of their SDK those methods no longer exist, meaning you have to bind to MessageReceiver to access them.
Edit you also need to set AutoComplete to false when you do this. https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus-trigger?tabs=csharp#configuration
I have not tried it but what you can do is set the MaxDeliveryCount property on the queue to 1 and then throw an exception in the function as soon as it is triggered. That way the message's delivery count will increase by 1 and Service Bus will automatically dead letter the message.
In latest versions (5.5.1 for me), you must use the ServiceBusMessageActions class from the Microsoft.Azure.WebJobs.ServiceBus namespace. It looks like this:
[FunctionName("MyFunction")]
public static async Task Run(
[ServiceBusTrigger("myQueue", Connection = "myConnection")]
ServiceBusReceivedMessage message,
ServiceBusMessageActions messageActions)
{
...
await messageActions.DeadLetterMessageAsync(message);
...
}
The NuGet package you want to use is Microsoft.Azure.WebJobs.Extensions.ServiceBus.
Reading the Dead Letter Queue messages is by creating the Azure Function Trigger in the Azure portal. In the Function, provide the name of the DLQ as “QueueName/$DeadLetterQueue” as shown in the below image
Note: If you want to access the undelivered message from the Topic then, the syntax of reading the Dead Letter Queue will be “TopicName/Subscriptions/SubscriptionName/$DeadLetterQueue”.
Now, define what should be done with the DLQ messages. Here, as shown in the below screenshot, we are sending the DLQ messages of “myqueue” to the Topic named of “queue” using the Azure Service Bus
In this manner, we can handle the DLQ messages as required very easily using the Azure Functions.
In Azure WebJobs, in the OnMessageOptions class, I'm calling the QueueClient.Complete(Guid) method by setting the AutoComplete flag to true and messages seem to dequeue just fine when running the ProcessQueue function. Active messages count goes down by 1 after successful processing of each message. However, when I want to requeue a message (because it cannot be processed currently) back to the queue that triggers the service bus function, as a new brokered message after a minute, using BrokeredMessage.ScheduledEnqueueTimeUtc, it seems like it isn't working. Scheduled messages count seems to go up initially. I go back to the queue after a few hours and see active messages in the thousands. The copies are of the same message. What is happening? I'd expect the message to be taken off the queue because of QueueClient.Complete(Guid) and the new scheduled message to be its replacement.
Some detail:
To send the message I do the following:
var queueclient = QueueClient.CreateFromConnectionString(connectionString, queueName);
queueclient.Send(message);
queueclient.close();
Inside the WebJob I created a ServiceBusConfiguration object which requires a onMessageOptions object where I set the AutoComplete=true. I pass the ServiceBusConfiguration object to the JobHostConfiguration.UserServiceBus
method.
Inside the WebJob service bus queue triggered function I again do the following to requeue, by first creating a new instance of the brokered message again.
//if not available yet for processing please requeue...
var queueclient = QueueClient.CreateFromConnectionString(connectionString, queueName);
queueclient.Send(message);
queueclient.close();
I don't do the following/use callbacks which is may be why it isn't working?
var options = new OnMessageOptions();
options.AutoComplete = false; // to call complete ourselves
Callback to handle received messages
client.OnMessage(m =>
{
var clone = m.Clone();
clone.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddSeconds(60);
client.Send(clone);
m.Complete();
}, options);
when I want to requeue a message (because it cannot be processed currently) back to the queue that triggers the service bus function, as a new brokered message after a minute, using BrokeredMessage.ScheduledEnqueueTimeUtc, it seems like it isn't working
If you fail to process your message, do not re-queue it. Instead, abandon (with a reason) and it will be picked up again.
BrokeredMessage.ScheduledEnqueueTimeUtc is intended to be used for messages added to the queue. When you receive a message, you can complete, dead-letter, defer, or abandon. If you abandon a message, it will be retried, but you can't control when that will happen. If you have no other messages in the queue, it will be retried almost immediately.
Note: when you see a behaviour that you suspect is not right, having a simple repro to share would be very helpful.
Our code is just a copy paste from some online tutorial in getting the messages from an azure storage queue.
public int? GetQueueMessageCount(CloudQueue queue, TextWriter textWriter)
{
int? messageCount;
try
{
queue.FetchAttributes();
// Retrieve the cached approximate message count.
messageCount = queue.ApproximateMessageCount;
}
catch (Exception exception)
{
LogHelper.LogInfo(logger, textWriter, $"GetQueueMessageCount failed for {queue.Name}." + exception);
throw;
}
return messageCount;
}
However, we found that randomly some messages may get stuck in the queue and our queue trigger never got fired.
public static void ProcessUnitsForCacheItem(
[QueueTrigger(QueueClient.RefreshUnitsQueue)] string projectUnitsMessage, TextWriter textWriter)
When I open my queue with storage explorer, I can see the explorer will not show any message, instead just display a status text "displaying 0 of 199 messages". So storage explorer must somehow know that these messages are not right (expired or something).
Is there some status I can retrieve to see the status of the message or anyone know how storage explorer decide to show a message or not?
Storage explorer shows info exactly what it retrieves from Storage account/emulator.
displaying 0 of 199 messages means the messages are invisible for now because they have been dequeued and being processed, it's a feature of queue message and handled by Storage service automatically once your queue trigger gets messages from a queue. See Storage queue doc.
Typically, when a consumer retrieves a message via Get Messages, that message is usually reserved for deletion until the visibilitytimeout interval expires, but this behavior is not guaranteed. After the visibilitytimeout interval expires, the message again becomes visible to other consumers.
As for the problem
get stuck in the queue and our queue trigger never got fired
If I understand correctly, your code from some tutorial is a custom queuetrigger, which may have no guarantee on the behavior. Have a look at Azure Function Queuetrigger example.
I am encountering one major road block issue when trying to use ServiceBusTrigger in azureFunction. I am trying to abandon, or deadletter, a service bus message in V2 ServiceBusTrigger, How can I do so? I've tried the following solution, but I didn't get anywhere.
Here is the codeSample I used:
public async static Task Run(Message myQueueItem, TraceWriter log, ExecutionContext context)
{
log.Info($"C# ServiceBus queue trigger function processed message delivery count: {myQueueItem.SystemProperties.DeliveryCount}");
QueueClient queueClient = new QueueClient("[connectionstring]","[queueName]");
////await queueClient.DeadLetterAsync(myQueueItem.SystemProperties.LockToken);
await queueClient.AbandonAsync(myQueueItem.SystemProperties.LockToken);
}
Solution 1: I tried to substitute Message myQueueItem for BrokeredMessage like in V1, I then can call myQueueItem.Abandon, or deadletter, on the message lever. However It came back with exception:
Microsoft.Azure.WebJobs.Host: Exception binding parameter 'myQueueItem'. System.Private.DataContractSerialization: There was an error deserializing the object of type Microsoft.ServiceBus.Messaging.BrokeredMessage. The input source is not correctly formatted. System.Private.DataContractSerialization: The input source is not correctly formatted."
At least I can go one step further. to
solution 2. Solution 2: is to use:
QueueClient queueClient = new QueueClient("[connectionstring]","[queueName]");
////await queueClient.DeadLetterAsync(myQueueItem.SystemProperties.LockToken);
await queueClient.AbandonAsync(myQueueItem.SystemProperties.LockToken);
I can use the lock provided in the Message Object, however, when I try to send it with queueClient, It said the message gone from the queue. or no longer available.
Can anybody let me know if i am on the right track? If I am not, please kindly guide me in the right track.
Service Bus messages are automatically completed or abandoned by Azure Functions runtime based on the success/failure of the function call, docs:
The Functions runtime receives a message in PeekLock mode. It calls Complete on the message if the function finishes successfully, or calls Abandon if the function fails.
So, the suggested way to Abandon your message is to throw an exception from function invocation.
I would like to have my queue retry failed webjobs every 90 minutes and only for 3 attempts.
When creating the queue i use the following code
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
IRetryPolicy linearRetryPolicy = new LinearRetry(TimeSpan.FromSeconds(5400), 3);
queueClient.DefaultRequestOptions.RetryPolicy = linearRetryPolicy;
triggerformqueue = queueClient.GetQueueReference("triggerformqueue");
triggerformqueue.CreateIfNotExists();
However when simulating a failed webjob attempt the queue uses the default retry policy.
I'm i missing something.
I think you might be thinking about this backwards. Queues don't actually perform behavior. Instead what I am guessing you want to do is have a web job that is configured to pull messages from a queue and then if it fails to process the message from a queue for some reason have the web job retry 90 minutes later. In this case you just need to set the invisibility timeout to be 90 minutes (default is 30 seconds) which will ensure that if the message isn't fully processed (ie - GetMessage and DeleteMessage are both called) then the message will reappear on the queue 90 minutes later.
Take a look at this Getting Started with Queue Storage document for more information.
There is something like Azure WebJobs SDK Extensions and ErrorTriggerAttribute (it isn't yet available in nuget 1.0.0-beta1 package, but you have access to public repository)
public static void ErrorMonitor(
[ErrorTrigger("0:30:00", 10, Throttle = "1:00:00")] TraceFilter filter,
TextWriter log)
https://github.com/Azure/azure-webjobs-sdk-extensions#errortrigger
You need to use your RetryPolicy when you add an item to the queue, not on the queue itself, eg.
var queue = queueClient.GetQueueReference("myQueue");
queue.CreateIfNotExists();
options = new QueueRequestOptions { RetryPolicy = linearRetryPolicy };
await queue.AddMessageAsync(yourMessage, null, new TimeSpan(0, delayMinutes, 0), options, null);