I have a webjob (App service) that acts on messages from a queue in Azure. I like to scale the webjob when I have a long queue. How do I do that?
From the criteria, I can see 2 options:
- Messages processed/Instance (AVG)
- Active Messages Processed/Instance (AVG)
However,I don't think they are relevant what so ever. Am I wrong?
Azure Service Bus queue/subscription can contain messages with several statuses: Active, Deferred, and Scheduled. Active messages processed would be for the messages that were with an Active state. I.e. messages that were not deferred or scheduled. Messages Processed metric would include messages with any state. The documented metric names are not the same as the ones you see, though I get a feeling that's a UI discrepancy.
Related
We have a .NET Core 3.1 web-API application that listens to an Azure Service Bus queue. We would like to get more information about the consumer of the messages on the queue - within our workflow, there should only be one (designated) consumer of a given queue, however we suspect there are multiple consumers. We suspect this because there are no application log entries on the designated consumer application logs when (and after) a message on the queue is being processed.
To attempt to determine the consumer of a queued entry I have looked at the Service Bus Logs and queries related to AzureActivity and AzureDiagnostics however I couldn't find specific information about a message(s)/item(s) on a given queue
Can anyone shed any light on how I can find details about the consumer of a queued item ?
I am using Azure Service Bus Queues in one of my .net core application to receive the messages from the Queue in the FIFO order. Once received the message from the Queue, then I have processed the business logic with each message. Once processed the business logic with each message then I am trying to delete the message from Queue with following line of code:
await _messageReceiver.CompleteAsync(message.SystemProperties.LockToken);
But for the few messages, I am unable to delete the messages from Queue. Because lock on the message expired even though I configured the “Lock Duration” 3 minutes on Service Bus Queue.
I am getting the following error while deleting/completing the message from Service Bus Queue:
Exception Type: System.InvalidOperationException
Exception Message: Operation is not valid due to the current state of the object.
These are properties I configured on the Azure Service Bus Queue:
Max Delivery Count =1
Lock Duration=3 minutes
There are a few reasons why you are not able to delete the messages in Azure Service Bus Queue.
Challenge with Lock Duration
The first challenge is Time, specifically lock duration for every message once they have been retrieved. The problem is that the messages must be displayed in the UI and then provide time for the customer to choose the messages to be resubmitted or deleted. The solution would be to ask the customer to increase the lock duration, but this is not possible due to several reasons. Firstly, increasing lock duration might not be possible due to customer integration, so we cannot depend on that.
Read more on this blog post on how to overcome the challenge.
What happens if you have let's say 4 instances of a webapp, and a lot of messages queued up in the service bus queue (meant to reach all the 4 instances) and the webapp scales down to only 2 instances. Will the messages, in the service bus that were meant for the other two instances that were removed, be stuck in the queue until the time to live is exceeded and then removed or does the service bus understand that there are not 4 instances anymore and therefore it doesn't need to send out messages to 4 instances?
I'm not sure if this is the correct but from my understanding there usually is a topic and then multiple subscriptions? will the service bus understand when one of the instances (who has a subscription) is gone and then removes the message meant for that subscription while the message is queued up (a lot of other messages before it)?
Sorry if the question is a little dumb but I couldn't find any answers on the internet.
Will the messages, in the service bus that were meant for the other two instances that were removed, be stuck in the queue until the time to live is exceeded and then removed
Service Bus queues and topic subscriptions provide a secondary sub-queue, called a dead-letter queue (DLQ).
The purpose of the dead-letter queue is to hold messages that cannot be delivered to any receiver, or simply messages that could not be processed. More details please refer to the document.
does the service bus understand that there are not 4 instances anymore and therefore it doesn't need to send out messages to 4 instances?
Service Bus Queues offer First In, First Out (FIFO) message and Service Bus queue message is received and processed by only one message consumer. If we have multiple web instance, when a quest arrive then WebApp LoadBlance (ARR) will assign the request to the corresponding instance to process the message. If the instance has been removed, the WebApp loadBlance (ARR) will assign the request to the existing instance.
will the service bus understand when one of the instances (who has a subscription) is gone and then removes the message meant for that subscription while the message is queued up (a lot of other messages before it)?
As mentioned above, it is assigned by WebApp LoadBlance not Service bus. How to consume message is depended on your WebApp.
We are trying to use Azure Topic/Subscriptions to distribute changes in one domain to others services that need to update their local cache. So we have one guy publishing a message and a bunch, not knowing of eachother, listening to this topic queue.
I might have missunderstod the idea of the azure TopicDescription.DefaultMessageTimeToLive but I thought it indicated that as long as the message is still within this timeout, it will be delivered, regardless if the subscriber is "online" at the time of publishing.
But this does not seem do be the case?
What I want to accomplish is that if I have a DefaultMessageTimeToLive set to 10 minutes, all subscribers are guaranteed to get all published messages if they have a downtime lower than 10 minutes.
When I try it, I do not receive messages unless I am listening at the time of publishing. (Added remark: Each receiving queue has its own unique name)
Have I got it wrong or is there a configuration I missed?
If you want a topic available to N subscribers (listeners), then you need N subscriptions. Subscriptions are "virtual" FIFO queues. If a subscription has more than one listener, they "compete" for the next message. This pattern is often termed the “competing consumer” pattern. Read more at Service Bus Queues, Topics, and Subscriptions and How to Use Service Bus Topics/Subscriptions.
The TopicDescription.DefaultMessageTimeToLive just defines how long a message is available before it is moved to the dead-letter queue.
You can subscribe to asynchronous updates from Azure topics and queues by using SubscriptionClient/QueueClient's .OnMessage call which will presumably create a separate thread polling the topic/queue with default settings and calling a defined callback if it receives anything.
Azure website says that receiving a message is a billable action, which is understandable. However, it isn't clear enough if each those poll requests are considered billable even when they do not return anything, i.e. the queue in question has no pending messages.
Based on the Azure Service Bus Pricing FAQ - the answer to your question is yes
In general, management operations and “control messages,” such as
completes and deferrals, are not counted as billable messages. There
are two exceptions:
Null messages delivered by the Service Bus in
response to requests against an empty queue, subscription, or message
buffer, are also billable. Thus, applications that poll against
Service Bus entities will effectively be charged one message per poll.
Setting and getting state on a MessageSession will also result in
billable messages, using the same message size-based calculation
described above.
Given the price is $0.01 per 10,000 messages, I don't think you should worry too much about that.