Our Application is using Azure service bus for messaging purpose. We created few topics and subscribers. We will send around 500 messages per day, but in the graph it shows hundreds of thousands of requests for 500 messages. Our billing price is also around 800$ per month. This is too much for 500 * 30 messages. I am not able to understand why the price is this much and also I want to know what does the requests in the graph mean?.
If the reason for the price is because of number of requests then how can I reduce the number of requests?. Messages I am seeing correctly. The problem is only with requests.
This is Just a sample Graph for your reference(Not original). In original graph I see around 100k of requests for 500 messages.
Here, under FAQ:
https://azure.microsoft.com/en-us/pricing/details/service-bus/
How is the Operations meter calculated for queues and topics?
For brokered entities (queues and topics/subscriptions), an operation is any API interaction with Service Bus service on any protocol.
A send, receive delete for a message that is less than or equal to 64KB in size is considered as one billable operation. If the message is greater than 64KB in size, the number of billable operations is calculated according to the message size in multiples of 64KB. For example, an 8 KB message sent to the Service Bus will be billed as one operation, but a 96 KB message sent to the Service Bus will be billed as two operations. Reading the 8KB message with a lock and then completing or explicitly abandoning the message will be billed as two operations. Renewing the lock on a message also incurs an operation.
Multiple deliveries of the same message (for example, message fan out to multiple subscribers or message retrieval after abandon, deferral, or dead lettering) will be counted as independent operations. For example, in the case of a topic with three subscriptions, a single 64KB message sent and subsequently received will generate four billable operations, one “in” plus three “out”, assuming all messages are delivered to all subscriptions and deleted during the read.
Additionally creating, reading (listing), updating and deleting a queue, topic or subscription will each incur an operation charge.
Operations are API calls made against queue or topic/subscription service endpoints. This includes Management, Send/Receive and Session State Operations.
This is what I found in the docs, but not fully sure if that's what you were looking for:
When creating a queue or subscription client, you can specify a receive mode: Peek-lock or Receive and Delete. The default receive mode is PeekLock. When operating in the default mode, the client sends a request to receive a message from Service Bus. After the client has received the message, it sends a request to complete the message.
When setting the receive mode to ReceiveAndDelete, both steps are combined in a single request. These steps reduce the overall number of operations, and can improve the overall message throughput. This performance gain comes at the risk of losing messages.
Related
If a single Azure Service Bus Topic with 10 subscriptions exists. I put a Message on the Topic and it goes to all 10 subscriptions. From the docs I assume this incurs the cost of 10 operations.
https://azure.microsoft.com/en-gb/pricing/details/service-bus
However if we added a filter to all 10 to only allow certain Messages, would it still incur the cost of one operation regardless, i.e to process the filter even if the Message does not go on the Subscription?
if we added a filter to all 10 to only allow certain Messages, would it still incur the cost of one operation?
Yes even if we are using filters on a subscription and if we receive message from a topic, i.e., message retrieval after abandon, deferral or dead lettering will be counted as independent operations and will considered as billable operations.
So, it will incur an operational charge assuming all messages are delivered to all subscriptions.
Currently, I have a problem handling data which I have sent from application to azure queue. The data I sent required to be sent FIFO but the Azure Queue cannot guarantee to be in order. Whereas Azure Service Bus Queue was guaranteed to be FIFO.
I am not sure is Azure Service Bus Queue has any differences with the Azure Queue.
As a solution architect/developer, you should consider using Storage queues when:
Your application must store over 80 GB of messages in a queue, where the messages have a lifetime shorter than 7 days.
Your application wants to track progress for processing a message inside of the queue. This is useful if the worker processing a message crashes. A subsequent worker can then use that information to continue from where the prior worker left off.
You require server side logs of all of the transactions executed against your queues.
As a solution architect/developer, you should consider using Service Bus queues when:
Your solution must be able to receive messages without having to poll the queue. With Service Bus, this can be achieved through the use of the long-polling receive operation using the TCP-based protocols that Service Bus supports.
Your solution requires the queue to provide a guaranteed first-in-first-out (FIFO) ordered delivery.
You want a symmetric experience in Azure and on Windows Server (private cloud). For more information, see Service Bus for Windows Server.
Your solution must be able to support automatic duplicate detection.
You want your application to process messages as parallel long-running streams (messages are associated with a stream using the SessionId property on the message). In this model, each node in the consuming application competes for streams, as opposed to messages. When a stream is given to a consuming node, the node can examine the state of the application stream state using transactions.
Your solution requires transactional behavior and atomicity when sending or receiving multiple messages from a queue.
The time-to-live (TTL) characteristic of the application-specific workload can exceed the 7-day period.
Your application handles messages that can exceed 64 KB but will not likely approach the 256 KB limit.
You deal with a requirement to provide a role-based access model to the queues, and different rights/permissions for senders and receivers.
Your queue size will not grow larger than 80 GB.
You want to use the AMQP 1.0 standards-based messaging protocol. For more information about AMQP, see Service Bus AMQP Overview.
You can envision an eventual migration from queue-based point-to-point communication to a message exchange pattern that enables seamless integration of additional receivers (subscribers), each of which receives independent copies of either some or all messages sent to the queue. The latter refers to the publish/subscribe capability natively provided by Service Bus.
Your messaging solution must be able to support the "At-Most-Once" delivery guarantee without the need for you to build the additional infrastructure components.
You would like to be able to publish and consume batches of messages.
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted
Messages in Storage queues are typically first-in-first-out, but sometimes they can be out of order; for example, when a message's visibility timeout duration expires (for example, as a result of a client application crashing during processing). When the visibility timeout expires, the message becomes visible again on the queue for another worker to dequeue it. At that point, the newly visible message might be placed in the queue (to be dequeued again) after a message that was originally enqueued after it.
You will find this article helpful in making the decision for your case: Storage Queues and Service Bus Queue comparison. It compares some of the fundamental queuing capabilities provided by Storage queues and Service Bus queues.
Also read Get started with Service Bus Queues.
My scenario: I have an Azure Storage Queue where messages can come in at any time. If I have 10 items in that queue, it's imperative that they be processed in order. I'm using c# and the windows azure storage SDK.
If the first item fails after, say, 2 seconds it remains invisible on the queue for another 28 seconds (30 second invisibility by default).
Now, my worker will just continue to check a queue for messages and process them as and when. If a queue message fails, it remains invisible and so the next queue item will be processed before the first message is retried.
This seems like really basic functionality for anyone needing a queue where the items are processed in order.
No, I can't set the timeout to a smaller amount because tasks can take varying lengths of time.
George, if you are looking for a messaging queue solution that processes items in order, you should consider using Azure Service Bus Queues:
As a solution architect/developer, you should consider using Service Bus queues when:
Your solution must be able to receive messages without having to poll the queue. With Service Bus, this can be achieved through the use of the long-polling receive operation using the TCP-based protocols that Service Bus supports.
Your solution requires the queue to provide a guaranteed first-in-first-out (FIFO) ordered delivery.
You want a symmetric experience in Azure and on Windows Server (private cloud).
For more information, see Service Bus for Windows Server.
Your solution must be able to support automatic duplicate detection.
There is a good article comparing both Storage Queues and Service Bus: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted , you may find the latter better suitable for your case.
Does the Azure Service Bus and its on-premise version, Service Bus for Windows Server, replicate a message for every subscriber?
For example, let's say that there is a single topic with five subscribers, then is that message stored in the service bus' database five times - once for each subscriber - or is that message only stored once with business logic to determine which subscribers have read the message?
It would be nice if there is an official site and/or documentation to provide as a reference.
The behavior the Azure Service Bus seems to be that it is keeping a copy per subscriber. I tested this by creating a topic with two subscriptions. I sent in a single message and I see that the size of the Topic in Bytes is 464 (using topic.SizeInBytes). When I receive one message of a subscription the size the drops in half to 232. I tested it with three subscriptions and same behavior occurred: 696 bytes.
Even if they aren't keeping a copy of the message per subscription they are counting the size of the message times the number of subscriptions against the maximum size of the topic, which may be what you were trying to determine.
I agree it would be nice if they documented the behavior, especially for Service Bus for Windows Server since that could affect planning for the amount of storage you need to set aside. As for the Azure Service Bus side, I'm not sure the implementation behind the scenes matters as much as knowing how it factors towards the max size of the topic.
A subscription to a topic resembles a virtual queue that receives
copies of the messages that were sent to the topic. You can optionally
register filter rules for a topic on a per-subscription basis, which
allows you to filter/restrict which messages to a topic are received
by which topic subscriptions.
I think it copies messages. If it does not copy, it should check always, did all subscribers get the messages ? Additionally, if there is filter, it should check just these subscribers to delete message. I think, copying and applying simple consume implemation cost is less than without copying cost.
Article
You can subscribe to asynchronous updates from Azure topics and queues by using SubscriptionClient/QueueClient's .OnMessage call which will presumably create a separate thread polling the topic/queue with default settings and calling a defined callback if it receives anything.
Azure website says that receiving a message is a billable action, which is understandable. However, it isn't clear enough if each those poll requests are considered billable even when they do not return anything, i.e. the queue in question has no pending messages.
Based on the Azure Service Bus Pricing FAQ - the answer to your question is yes
In general, management operations and “control messages,” such as
completes and deferrals, are not counted as billable messages. There
are two exceptions:
Null messages delivered by the Service Bus in
response to requests against an empty queue, subscription, or message
buffer, are also billable. Thus, applications that poll against
Service Bus entities will effectively be charged one message per poll.
Setting and getting state on a MessageSession will also result in
billable messages, using the same message size-based calculation
described above.
Given the price is $0.01 per 10,000 messages, I don't think you should worry too much about that.