I am using peek method to view the messages in the queue; the messages returned contain both the ones sitting in the queue and the ones being processed;
I also looked at the management api which provides active message count again it includes both sitting and being processed messages.
is there a way to get both the counts separately?
You can use Azure Monitor API: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-metrics-azure-monitor
Management API: https://learn.microsoft.com/en-us/rest/api/monitor/metrics/list
Example:
/subscriptions/{subscriptionID}/resourceGroups/{resourcegroup}/providers/Microsoft.ServiceBus/namespaces/{namespacename}/providers/microsoft.Insights/metrics?timespan=2020-08-06T05:18:00.000Z/2020-08-06T06:18:00.000Z&interval=FULL&metricnames=ActiveMessages&aggregation=average&metricNamespace=microsoft.servicebus%2Fnamespaces&top=10&$filter=EntityName eq '{your entityname}'&rollupby=EntityName&validatedimensions=false&api-version=2019-07-01
Related
I'm trying to find a solution for receiving large messages on Azure Service Bus. The essential pattern I was thinking is to publish a large messages in parts -- along with a correlation id, a page, and an "of".
So if I have a four-part message, they would all have the same correlation id, each would have an "of" of 4, and the page would be 0 - 3. The set would be published as a batch.
The listener could listen for only messages with a page of 0, and then pull the remaining messages according to the transaction id.
Publishing these messages is easy enough. ServiceBusMessage has a CorrelationId field, and a dictionary field called ApplicationProperties that I can add my custom "page" and "of" fields to. I can assemble them into a ServiceBusMessageBatch before publishing.
What I'm not sure about is how to receive the messages. I'm using Function Apps, so it's easy to setup a listener.
[FunctionName("GeneralLogger")]
public static void Run([ServiceBusTrigger("queueName", Connection = "AzureWebJobsServiceBus")] string myQueueItem, ApplicationProperties ap, ILogger log)
{ /// process message }
But I don't see how to filter here. Also, I can pull messages by adding a handler to the message processor, described here: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-get-started-with-queues But likewise I don't see how to filter.
The only Azure Service Bus filtering I see how to do is between a topic and subscription. There is a lot of capability there, but nothing dynamically I can set during runtime.
I feel like I'm either trying to miss-use something or re-inventing the wheel. Is anyone else doing something like this with Azure Service Bus?
I'm trying to find a solution for receiving large messages on Azure Service Bus.
A solution is already there. It's Azure Service Bus premium tier. Capable of sending messages up to 100MB in size. It comes with a price. Assuming you're looking to spit up the file either because the premium is much to pay for or because messages could be larger than 100MB, the claim-check pattern is the way to go. There's just one issue when the claim-check pattern is used over the premium tier - you cannot have a deterministic clean-up when a message is an event, and there are multiple receivers. You'd need to come up with some policy to clean up those blobs, given that those are large blobs and will quickly add to the storage consumption over time, depending on the number of messages flowing through the system. With the premium tier, the problem of clean-up doesn't exist. Nor do you have to provide a storage account. Therefore, if your large messages will not exceed 100MB, it could be a more suitable solution for your production environment.
It isn't possible to apply filters on a queue; they only operate on topics/subscriptions.
Generally, the Claim Check pattern is recommended when you're looking to send a payload too large for a single message. In a nutshell, you would write your payload to some form of durable storage and then your Service Bus message would provide the location for consumers.
An example implementation using the Azure.Messaging.ServiceBus package can be found in this sample.
How to read multiple messages from a service bus topic subscription under peek lock with http request not via service bus connector and using managed identity authentication?
Using REST API, it is not possible to get multiple messages in a single request. The REST API operation is Peek-Lock Message which only returns a single message.
If you have to fetch multiple messages, either call this operation multiple times on a topic subscription or make use of AMQP which supports fetching multiple messages.
First service adds messages to queue if user does not exist in DB, second service gets message from queue and create user. Possible situation, when first service adds 2 messages for create users before second gets it. How to resolve it? As I understand, no way to review queue...
I use Azure Storage queues
Azure Queue message doesn't support peek-lock to be processed. Once it is read, it becomes invisible. You need to look into Azure Service Bus as it allows you to control message one by one and in order if required.
I have following requirement
Message published to the Topic/Queue
Multiple consumers subscribed to the Topic/Queue. So our requirement is to only one consumer should listen to the message. That means no other consumer can get the same message.
I feel queue would be the best fit. But I have advise from our architect to check whether we can achieve it from Topics?.
So any body please let me know whether we can achieve it through Topics and also pros and constrains?
Thanks.
Azure Service Bus Queue is a single message queue. You send it a message and the message receiver will get the message and be able to process it accordingly. Each message will only be handled once.
Azure Service Bus Topic is a more robust message queue than Azure Service Bus Queue. With Topics there can be multiple Subscriptions configured to catch messages based on a Filter. If multiple Subscriptions have a Filter that matches an incoming message, then each of those Subscriptions will get a copy of the messages. With Topics it's up to you to configure the Subscription Filters according to your projects needs.
If you know a message only needs to be handled once in your system and the message queue is being used by a single message receiver application (single or multiple hosted instances) then Azure Service Bus Queue is likely the tool for the job.
I need to keep track of how many failed attempts have been made to process a message in an azure storage queue and delete the message after N unsuccesful attempts.
I have searched, but have not found any particular property that does this automaticaly and was wondering if there was a way other than using a counter in a storage table.
Each cloud queue message has a DequeueCount property. Does this help?
REST API reference here.
As for how to delete messages automatically after n attempts: There's nothing that automatically does this. You'll need to implement your own poison-message handling in Windows Azure queues, based on DequeueCount.
Alternatively, Azure Service Bus queues have a dead-letter queue for undeliverable messages (or ones that can't be processed). More info here.