When using an azure webjobs queue, is it possible to queue a single message with a particular visibility time (i.e. when the message becomes available on the queue for processing)?
For the sake of retrying messages that we fail to process, it would be helpful to be able to re-queue with some sort of back-off so that transient problems have a chance to resolve themselves.
It is indeed possible when working with Service Bus Queues. Azure Storage Queues do NOT have this behavior. The QueueClient ScheduleMessageAsync method will allow you to do so.
Related
We have a usecase where we need to schedule jobs which will be sent as a message from Azure web api service to Azure Service Bus Queue. As we need to schedule it at later point in time one solution is to use Scheduled Delivery ScheduledEnqueueTimeUtc.
What i understand is message gets engqueued only after the time specified expires . My concern is what happens if Web API crashes or undergoes upgrade meanwhile.
1.Will the messages be lost as its not enqueued yet?
2.Where does this messages are stored in the intermediate time ?
Second solution is to use visibilityTimeOut of storage queue where messages are enqueued and will not be impacted by Web API.
From stability and scalability perspective which would be a better option ?
The message is sent to Service Bus, which is enqueued (available to receive) according to the schedule. So, to answer your queries
Nope
In the queue, just not available to receive
visibilityTimeOut is for storage queues. Refer the comparison doc for making the decision.
Note that while you cannot receive scheduled messages, you can peek them.
I have a slightly philosophical problem. We are using Storage Queues for processing the "tickets". The way we have implemented that is we have a background service (worker role) that is polling the storage queue and finding out if there is any ticket to be processed. The nature of the work we do is seasonal. Which means that there won't be tickets all the time to be processed. The problem we are facing with this is - since multiple worker role instances are continuously polling the storage queue, we have cost impact as it's just too many GetMessage() calls.
I came across the Service Bus queue which has event-based capability. There we have the concept of OnMesage() which gets called every time a new message becomes available on a service bus queue.
But my question is - does OnMessage() goes ahead and calls Receive() internally? Which means is it just syntax sugar and internally it is still polling going on and would there be a cost impact in Service Bus Queue case as well?
Any insights into this will be helpful.
Azure Service Bus client is using long polling to retrieve messages from the broker.
By default, it's set to 1 minute or when a message arrives. So if you have a message showing up earlier than 1 minute elapses, it will be retrieved and another poll for 1 minute will be issues. OnMessage/MessageHandler are no exception. It's a higher level abstraction on top of low level receive operation.
My scenario: I have an Azure Storage Queue where messages can come in at any time. If I have 10 items in that queue, it's imperative that they be processed in order. I'm using c# and the windows azure storage SDK.
If the first item fails after, say, 2 seconds it remains invisible on the queue for another 28 seconds (30 second invisibility by default).
Now, my worker will just continue to check a queue for messages and process them as and when. If a queue message fails, it remains invisible and so the next queue item will be processed before the first message is retried.
This seems like really basic functionality for anyone needing a queue where the items are processed in order.
No, I can't set the timeout to a smaller amount because tasks can take varying lengths of time.
George, if you are looking for a messaging queue solution that processes items in order, you should consider using Azure Service Bus Queues:
As a solution architect/developer, you should consider using Service Bus queues when:
Your solution must be able to receive messages without having to poll the queue. With Service Bus, this can be achieved through the use of the long-polling receive operation using the TCP-based protocols that Service Bus supports.
Your solution requires the queue to provide a guaranteed first-in-first-out (FIFO) ordered delivery.
You want a symmetric experience in Azure and on Windows Server (private cloud).
For more information, see Service Bus for Windows Server.
Your solution must be able to support automatic duplicate detection.
There is a good article comparing both Storage Queues and Service Bus: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted , you may find the latter better suitable for your case.
Our application consist of 7 microservices that have some intercommunication. Currently we're using simple storage queues that a microservice publish events to (the number of events is relative low). Then we have a azurefunction for each queue that might call another microservices. This is working fine for us right now the services uses about 20 queues with a corresponding function.
Now we need to handle an blobstorage event, and I did some googling and a started to get really confused. Suddenly there was a lot of questions:
Should we switch to Azure Event Grid
It handles blobstorage without any limitations (functions blobstorage trigger has some)
It allows for multiple subscribers (storage queues does not)
It has a lot of fuz - maybe this is the new recommended way
I like the idea of one central thing, but it reminds me a bit about biztalk...
Should I switch to Azure Service Bus
It has a nice tool (ServiceBusExplorer) for monitoring the queues and listners, and I could to a repost of any failed events
It visulizes my azure functions subscribers nicely
Should I continue with only storage queues
A bit difficult to monitor, but it works nice
I'll be really thankful for any advice or insights to this question.
Thanks
EventGrid is great when you have notifications floating to multiple subscribers. Is that the case for you?
An example would be deferring messages. With queues you can defer a message, not with EventGrid. Whenever to choose Storage Queues or Service Bus depends on the specific requirement that you have. Do you need de-duplication? Or ordered delivery? If you do, Service Bus is the way. Otherwise Storage Queues is enough.
First of All, I would like to recommend these two articles, it will clarify most of your doubts about these services:
Choose between Azure services that deliver messages
Storage queues and Service Bus queues compared
Regarding Event Grid, it acts like a bridge between the publisher and the subscriber, where publisher will send messages and forget whether it has been processed or not, and the Event Grid will handle the retry if the receiver\subscriber does not acknowledge that it was processed successfully.
As you mentioned, storage queues has limitations, as such blob triggered functions, and maybe Service Bus, but it will depend on your design requirements. I would like to point out some things you might consider before moving to Event Grid.
Storage queues & Service Bus does not care about your message schema, in Event Grid you have to create a custom event based on their schema to wrap your event, so the publisher and subscriber has to understand Event Grid for that, not that is a big deal, but now you have both sides coupled to Event Grid.
If you want to send the event straight to your micro-service, you have to implement the subscription validation in your service, otherwise the service won't be able to receive the events
Event Grid only retry the delivery of your messages for 24 hours, if your service is down or not process the message correctly for longer than 24h, it will make the event dead. Currently, there is no way to query dead messages. Storage Queues and Service Bus are configurable how long you keep the message and it can be kept for many days.
Your service web-hook must acknowledge the receipt(http 200 or 202) of an event within 60 seconds, otherwise it will consider failed. If your operation is longer that that, you should send it to a queue and handle the locking from your service.
Probably there are more limitations, but these are the ones I remember right now that might change anytime soon, I think Event Grid is a great technology still on early days, and there is much to improve, I would recommencement only as a hub for Azure management events, I don't think it is ready for use as an application integrator.
Regarding your comment for queue manager, for Service Bus your have the Service Bus Explorer, and for Azure Storage you have the Azure Storage Explorer, where you can check the messages in the queue, is not the same as Service Bus, but helps.
It very much depends on how are you consuming the queue messages, you can take a look at this comparison: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted
If you don't need ordering and if you don't have a strong limit on message volume, size or TTL, you can stick to storage queues.
I'm trying to work with both one WebJob and one Worker Role.
The WebJob will have a BlobTrigger, every time a blob is added to the container a new message will be added to an Azure Storage Queue (call it pending blobs).
Also, there will be a Worker Role which will be pooling messages from the pending blobs queue and will add the blob names to an internal blocking collection that will be processed concurrently by several tasks triggered by the Worker Role.
I've thought in this solution setting my mind in scalability and because there will be a lot of blobs arriving to the container so I don't want to have peaks of CPU consumption.
Some questions came to my mind while developing the solution:
Is there a way to check if the Azure Storage Queue has messages inside?
If I call the GetMessage method and the queue does not have any message the excecution will be blocked until a new message arrive?
Is there a way to manually delete blob receipts?
Is there a way to check if the Azure Storage Queue has messages
inside?
Queues have an ApproximateMessageCount property you can check, for queue depth (note: this isn't 100% accurate, because messages may be added/deleted while checking).
If I call the GetMessage method and the queue does not have any
message the excecution will be blocked until a new message arrive?
GetMessage() is non-blocking. If there are no messages, the call returns. Note: Since you're planning on creating your own reader in a worker role, just be careful when dealing with an empty queue: If you put yourself in a tight loop and continue to blast the queue, you run the risk of exhausting the queue's 2000 transaction/second limit (and you'll probably see excessive network traffic and cpu utilization). How you implement a backoff strategy is up to you, but you'll want to incorporate a backoff of some type.
Maybe it would be better if you use Azure Functions to handle your queue messages. It will only be triggered if a new message appear on the queue.
https://azure.microsoft.com/en-us/documentation/articles/functions-bindings-storage/
FetchAttributes method in CloudQueue class can be used to get different attributes of Queue. The count attribute is one of those attributes.
According to documentation
The ApproximateMessageCount property returns the last value retrieved by the FetchAttributes method, without calling the Queue service.