I need to keep track of how many failed attempts have been made to process a message in an azure storage queue and delete the message after N unsuccesful attempts.
I have searched, but have not found any particular property that does this automaticaly and was wondering if there was a way other than using a counter in a storage table.
Each cloud queue message has a DequeueCount property. Does this help?
REST API reference here.
As for how to delete messages automatically after n attempts: There's nothing that automatically does this. You'll need to implement your own poison-message handling in Windows Azure queues, based on DequeueCount.
Alternatively, Azure Service Bus queues have a dead-letter queue for undeliverable messages (or ones that can't be processed). More info here.
Related
We have a Queue created in storage account. an external system is creating messages in the Queue.
So I need to process the messages for 2 environments (say PROD and Stage). So I have a Queue trigger available in both environments but the messages available in only one Queue.
If we copy the messages to another Queue also, the Queue trigger will be executed in both environments.
Is it possible to copy messages from one Queue to another Queue or how it can be done?
Thanks
Is it possible to copy messages from one Queue to another Queue or how
it can be done?
Natively Azure Storage does not support copying messages from one queue to another. What you will have to do is create a new message based on the contents/properties of an existing message.
You can create the message in another queue when you are processing the message in your queue trigger.
However a better way to handle this is to make use of Azure Service Bus Topics and Subscriptions. In this case you will send a single message to a Service Bus Topic and create 2 Subscriptions in that Topic. The message will be sent to both Subscriptions. Then you can have two Functions each of which is listening to one of the Subscriptions.
That way you do not have to worry about manually copying the message to another Queue.
I am using peek method to view the messages in the queue; the messages returned contain both the ones sitting in the queue and the ones being processed;
I also looked at the management api which provides active message count again it includes both sitting and being processed messages.
is there a way to get both the counts separately?
You can use Azure Monitor API: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-metrics-azure-monitor
Management API: https://learn.microsoft.com/en-us/rest/api/monitor/metrics/list
Example:
/subscriptions/{subscriptionID}/resourceGroups/{resourcegroup}/providers/Microsoft.ServiceBus/namespaces/{namespacename}/providers/microsoft.Insights/metrics?timespan=2020-08-06T05:18:00.000Z/2020-08-06T06:18:00.000Z&interval=FULL&metricnames=ActiveMessages&aggregation=average&metricNamespace=microsoft.servicebus%2Fnamespaces&top=10&$filter=EntityName eq '{your entityname}'&rollupby=EntityName&validatedimensions=false&api-version=2019-07-01
My scenario: I have an Azure Storage Queue where messages can come in at any time. If I have 10 items in that queue, it's imperative that they be processed in order. I'm using c# and the windows azure storage SDK.
If the first item fails after, say, 2 seconds it remains invisible on the queue for another 28 seconds (30 second invisibility by default).
Now, my worker will just continue to check a queue for messages and process them as and when. If a queue message fails, it remains invisible and so the next queue item will be processed before the first message is retried.
This seems like really basic functionality for anyone needing a queue where the items are processed in order.
No, I can't set the timeout to a smaller amount because tasks can take varying lengths of time.
George, if you are looking for a messaging queue solution that processes items in order, you should consider using Azure Service Bus Queues:
As a solution architect/developer, you should consider using Service Bus queues when:
Your solution must be able to receive messages without having to poll the queue. With Service Bus, this can be achieved through the use of the long-polling receive operation using the TCP-based protocols that Service Bus supports.
Your solution requires the queue to provide a guaranteed first-in-first-out (FIFO) ordered delivery.
You want a symmetric experience in Azure and on Windows Server (private cloud).
For more information, see Service Bus for Windows Server.
Your solution must be able to support automatic duplicate detection.
There is a good article comparing both Storage Queues and Service Bus: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted , you may find the latter better suitable for your case.
First service adds messages to queue if user does not exist in DB, second service gets message from queue and create user. Possible situation, when first service adds 2 messages for create users before second gets it. How to resolve it? As I understand, no way to review queue...
I use Azure Storage queues
Azure Queue message doesn't support peek-lock to be processed. Once it is read, it becomes invisible. You need to look into Azure Service Bus as it allows you to control message one by one and in order if required.
When using an azure webjobs queue, is it possible to queue a single message with a particular visibility time (i.e. when the message becomes available on the queue for processing)?
For the sake of retrying messages that we fail to process, it would be helpful to be able to re-queue with some sort of back-off so that transient problems have a chance to resolve themselves.
It is indeed possible when working with Service Bus Queues. Azure Storage Queues do NOT have this behavior. The QueueClient ScheduleMessageAsync method will allow you to do so.