We have a usecase where we need to schedule jobs which will be sent as a message from Azure web api service to Azure Service Bus Queue. As we need to schedule it at later point in time one solution is to use Scheduled Delivery ScheduledEnqueueTimeUtc.
What i understand is message gets engqueued only after the time specified expires . My concern is what happens if Web API crashes or undergoes upgrade meanwhile.
1.Will the messages be lost as its not enqueued yet?
2.Where does this messages are stored in the intermediate time ?
Second solution is to use visibilityTimeOut of storage queue where messages are enqueued and will not be impacted by Web API.
From stability and scalability perspective which would be a better option ?
The message is sent to Service Bus, which is enqueued (available to receive) according to the schedule. So, to answer your queries
Nope
In the queue, just not available to receive
visibilityTimeOut is for storage queues. Refer the comparison doc for making the decision.
Note that while you cannot receive scheduled messages, you can peek them.
Related
I have a slightly philosophical problem. We are using Storage Queues for processing the "tickets". The way we have implemented that is we have a background service (worker role) that is polling the storage queue and finding out if there is any ticket to be processed. The nature of the work we do is seasonal. Which means that there won't be tickets all the time to be processed. The problem we are facing with this is - since multiple worker role instances are continuously polling the storage queue, we have cost impact as it's just too many GetMessage() calls.
I came across the Service Bus queue which has event-based capability. There we have the concept of OnMesage() which gets called every time a new message becomes available on a service bus queue.
But my question is - does OnMessage() goes ahead and calls Receive() internally? Which means is it just syntax sugar and internally it is still polling going on and would there be a cost impact in Service Bus Queue case as well?
Any insights into this will be helpful.
Azure Service Bus client is using long polling to retrieve messages from the broker.
By default, it's set to 1 minute or when a message arrives. So if you have a message showing up earlier than 1 minute elapses, it will be retrieved and another poll for 1 minute will be issues. OnMessage/MessageHandler are no exception. It's a higher level abstraction on top of low level receive operation.
Currently, I have a problem handling data which I have sent from application to azure queue. The data I sent required to be sent FIFO but the Azure Queue cannot guarantee to be in order. Whereas Azure Service Bus Queue was guaranteed to be FIFO.
I am not sure is Azure Service Bus Queue has any differences with the Azure Queue.
As a solution architect/developer, you should consider using Storage queues when:
Your application must store over 80 GB of messages in a queue, where the messages have a lifetime shorter than 7 days.
Your application wants to track progress for processing a message inside of the queue. This is useful if the worker processing a message crashes. A subsequent worker can then use that information to continue from where the prior worker left off.
You require server side logs of all of the transactions executed against your queues.
As a solution architect/developer, you should consider using Service Bus queues when:
Your solution must be able to receive messages without having to poll the queue. With Service Bus, this can be achieved through the use of the long-polling receive operation using the TCP-based protocols that Service Bus supports.
Your solution requires the queue to provide a guaranteed first-in-first-out (FIFO) ordered delivery.
You want a symmetric experience in Azure and on Windows Server (private cloud). For more information, see Service Bus for Windows Server.
Your solution must be able to support automatic duplicate detection.
You want your application to process messages as parallel long-running streams (messages are associated with a stream using the SessionId property on the message). In this model, each node in the consuming application competes for streams, as opposed to messages. When a stream is given to a consuming node, the node can examine the state of the application stream state using transactions.
Your solution requires transactional behavior and atomicity when sending or receiving multiple messages from a queue.
The time-to-live (TTL) characteristic of the application-specific workload can exceed the 7-day period.
Your application handles messages that can exceed 64 KB but will not likely approach the 256 KB limit.
You deal with a requirement to provide a role-based access model to the queues, and different rights/permissions for senders and receivers.
Your queue size will not grow larger than 80 GB.
You want to use the AMQP 1.0 standards-based messaging protocol. For more information about AMQP, see Service Bus AMQP Overview.
You can envision an eventual migration from queue-based point-to-point communication to a message exchange pattern that enables seamless integration of additional receivers (subscribers), each of which receives independent copies of either some or all messages sent to the queue. The latter refers to the publish/subscribe capability natively provided by Service Bus.
Your messaging solution must be able to support the "At-Most-Once" delivery guarantee without the need for you to build the additional infrastructure components.
You would like to be able to publish and consume batches of messages.
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted
Messages in Storage queues are typically first-in-first-out, but sometimes they can be out of order; for example, when a message's visibility timeout duration expires (for example, as a result of a client application crashing during processing). When the visibility timeout expires, the message becomes visible again on the queue for another worker to dequeue it. At that point, the newly visible message might be placed in the queue (to be dequeued again) after a message that was originally enqueued after it.
You will find this article helpful in making the decision for your case: Storage Queues and Service Bus Queue comparison. It compares some of the fundamental queuing capabilities provided by Storage queues and Service Bus queues.
Also read Get started with Service Bus Queues.
My scenario: I have an Azure Storage Queue where messages can come in at any time. If I have 10 items in that queue, it's imperative that they be processed in order. I'm using c# and the windows azure storage SDK.
If the first item fails after, say, 2 seconds it remains invisible on the queue for another 28 seconds (30 second invisibility by default).
Now, my worker will just continue to check a queue for messages and process them as and when. If a queue message fails, it remains invisible and so the next queue item will be processed before the first message is retried.
This seems like really basic functionality for anyone needing a queue where the items are processed in order.
No, I can't set the timeout to a smaller amount because tasks can take varying lengths of time.
George, if you are looking for a messaging queue solution that processes items in order, you should consider using Azure Service Bus Queues:
As a solution architect/developer, you should consider using Service Bus queues when:
Your solution must be able to receive messages without having to poll the queue. With Service Bus, this can be achieved through the use of the long-polling receive operation using the TCP-based protocols that Service Bus supports.
Your solution requires the queue to provide a guaranteed first-in-first-out (FIFO) ordered delivery.
You want a symmetric experience in Azure and on Windows Server (private cloud).
For more information, see Service Bus for Windows Server.
Your solution must be able to support automatic duplicate detection.
There is a good article comparing both Storage Queues and Service Bus: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted , you may find the latter better suitable for your case.
When using an azure webjobs queue, is it possible to queue a single message with a particular visibility time (i.e. when the message becomes available on the queue for processing)?
For the sake of retrying messages that we fail to process, it would be helpful to be able to re-queue with some sort of back-off so that transient problems have a chance to resolve themselves.
It is indeed possible when working with Service Bus Queues. Azure Storage Queues do NOT have this behavior. The QueueClient ScheduleMessageAsync method will allow you to do so.
You can subscribe to asynchronous updates from Azure topics and queues by using SubscriptionClient/QueueClient's .OnMessage call which will presumably create a separate thread polling the topic/queue with default settings and calling a defined callback if it receives anything.
Azure website says that receiving a message is a billable action, which is understandable. However, it isn't clear enough if each those poll requests are considered billable even when they do not return anything, i.e. the queue in question has no pending messages.
Based on the Azure Service Bus Pricing FAQ - the answer to your question is yes
In general, management operations and “control messages,” such as
completes and deferrals, are not counted as billable messages. There
are two exceptions:
Null messages delivered by the Service Bus in
response to requests against an empty queue, subscription, or message
buffer, are also billable. Thus, applications that poll against
Service Bus entities will effectively be charged one message per poll.
Setting and getting state on a MessageSession will also result in
billable messages, using the same message size-based calculation
described above.
Given the price is $0.01 per 10,000 messages, I don't think you should worry too much about that.