So, I'm analyzing whether using Functions to consume Service bus messages from a topic is viable.
The Service Bus pricing plan we chose has a limit on the number of 'Brokered connections' (max 1000) per month.
My understanding is that in the typical usage scenario the consumer/listener/subscriber connects to a topic and stays connected persistently, receiving multiple messages during a long period of time (a day, even a week) without breaking the connection, and that this is counted as 1 'brokered connection'. In the end you can receive thousands of messages on a single brokered connection.
How does this work with an Azure function binding? From what I read in the documentation, a function can be idle (ie not running), so it wouldnt be able to maintain this persistent connection.
Is there a separate component for Functions that keeps this connection alive listening for incoming messages? Or are we going to get billed for a new brokered connection each time the function goes idle and then restarts?
I'm including a screenshot of the current plan features:
https://azure.microsoft.com/en-us/pricing/details/service-bus/
Later in the same link:
edit
From Docs:
Service Bus charges for the peak number of concurrent brokered connections that exceed the included quantity (1,000 in the Standard tier). Peaks are measured on an hourly basis, prorated by dividing by 744 hours in a month, and added up over the monthly billing period. The included quantity (1,000 brokered connections per month) is applied at the end of the billing period against the sum of the prorated hourly peaks.
They specifically mention "1,000 brokered connections per month" in the last sentence.
This is an example:
Each of 10,000 devices connects via a single AMQP connection, and receives commands from a Service Bus topic. The devices send telemetry events to an Event Hub. If all devices connect for 12 hours each day, the following connection charges apply (in addition to any other Service Bus topic charges): 10,000 connections * 12 hours * 31 days / 744 = 5,000 brokered connections. After the monthly allowance of 1,000 brokered connections, you would be charged for 4,000 brokered connections, at the rate of $0.03 per brokered connection, for a total of $120.
So I guess all of this is for 10.000 subscribers connecting to the topic at the same time during 12hs, if they connected for 24hs each day then 9,000 brokered connections would be billed (10,000 minus the included 1,000)?
In any case, I'm also trying to validate whether persistent connections are possible with functions (I'm told they are with webjobs).
Azure Functions has a separate component known as the ScaleController that monitors your Service Bus for events 24/7.
Since the underlying SB message receiver in Functions is implemented in WebJobs, there will be a single connection that can retrieve multiple messages throughout the lifetime of the Function instance, although due of its current limitation, it is passing along those messages one at a time to your Function code for processing.
You are only charged when your Function code actually runs. Here's a link that gives you an overview of the ScaleController:https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale#runtime-scaling
If you are anticipating relatively high load, have you considered using Event Hub instead of Service Bus? ServiceBus-Triggers in Azure Functions currently can only process one message at a time, which is not optimal for high load scenarios. Here's a GitHub issue that is opened to track this feature request: https://github.com/Azure/azure-webjobs-sdk/issues/1024
EventHub-Triggers can process messages in batches and that should give your more bang for the buck per Function execution. See https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs
Related
I have an application that sends messages in batches to the Azure Service Premium queue. I am noticing that if I am sending 1000 messages in total (20 batches with 50 messages each, per-message size is 10 KB), I am noticing an extra count of messages in the queue. The extra count varies for every run. There are strictly no other senders. Is there any reason why this could happen? Or, if my logic needs to be checked thoroughly? I just want to eliminate any reasons inherent to the operations of the Azure Service bus.
I am using Azure.Messaging.ServiceBus latest version in .net 4.8
If a single Azure Service Bus Topic with 10 subscriptions exists. I put a Message on the Topic and it goes to all 10 subscriptions. From the docs I assume this incurs the cost of 10 operations.
https://azure.microsoft.com/en-gb/pricing/details/service-bus
However if we added a filter to all 10 to only allow certain Messages, would it still incur the cost of one operation regardless, i.e to process the filter even if the Message does not go on the Subscription?
if we added a filter to all 10 to only allow certain Messages, would it still incur the cost of one operation?
Yes even if we are using filters on a subscription and if we receive message from a topic, i.e., message retrieval after abandon, deferral or dead lettering will be counted as independent operations and will considered as billable operations.
So, it will incur an operational charge assuming all messages are delivered to all subscriptions.
Currently we are trying to determine which message service we should use - service bus or event hubs. And then we found that there are ambiguities on the pricing so hope we could get answer here.
Service bus charge
https://azure.microsoft.com/en-us/pricing/details/service-bus/
in this link above we see that standard tier service bus charges on broker connection. and it measures per hour with the peak of the concurrent connections. So problem is this ,say if I have 5K client, but they don't keep the connection alive, say they they send the message once per minute and only open/close the connection at that time, then the peak of concurrent connections might even lower than 100. In this case, how would connections be calculated. Is it going to be calculated in accumulative way or as the peak of certain time in this hour.
in FAQ "What is a brokered connection and how do I get charged for them?" mentioned that "An HTTP call to receive a message from a Service Bus topic or queue that has a receive timeout value greater than zero.", so what about using HTTP call to send message, how it would be count?
And for HTTP call limitation. We could find that there is 5K AMQP connection limit, but what about HTTP? Is it the net messaging refer to? https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-quotas .
Event Hubs charge
https://azure.microsoft.com/en-us/pricing/details/event-hubs/
In this link above we did not see how it charges on connection, but in FAQ it is saying the charge on connection, free on HTTP while AMQP charge is the same is service bus. So here comes two questions,
Is this charge tier matching? Charge on standard tier of event hub on connection is same as standard tier service bus and premium event hubs same as premium service bus? Asking because premium service bus does not charge on connection.
2 For HTTP messaging, same question as in service bus, how many concurrent http connection event hub could support concurrently?
Service Bus Charge
As the FAQ What is a brokered connection and how do I get charged for them? depicts, you will get charged by the hour and prorated for 730 hours in the month.
If all your clients connect to send messages per minute continuously, the charge would be
5000 * 24 * 30.5 / 730 = 5013
Since 1K connections are included, you would be charged for the remaining 4013 connections.
In this case, this call would just count towards a messaging operation.
As mentioned in the docs, REST operations don't count. You could however hit the concurrent receive requests limit.
Event Hubs Charge
Yes
Same as above. No concurrent limit.
I was trying to estimate no. of brokered connections while arrive at pricing for Service Bus (STANDARD TIER) for the following scenario:
There are 32 clients sending messages to/ receiving messages from a Service Bus queue through AMQP protocol.
Each client connects to the Service Bus several times in an hour, with duration of a single connection (including timeout) ~= 1 minute.
I understand from documentation that Microsoft charges based on the peak number of concurrent brokered connections:
Microsoft charges for the peak number of concurrent brokered
connections that exceed the included quantity (1,000 in the standard
and premium tier).
Now for one month, what will be the no. of brokered connections?
is it 32? - as, at any given point of time, there are only 32 peak live connections and based on the formula given in Pricing document. Therefore total connections being 32 connections * 24 hours per day * 30.5 days / 730 = 32.
or is it 1920? - as, in an hour, each client connects to Service Bus 60 times (no. of minutes in an hour [=60] / duration of a single connection [=1]). Therefore, total connections being (32 * 60) connections * 24 hours per day * 30,5 days / 730 = 1920.
Please help me understand how to calculate the no. of brokered connections in this scenario!
In this case, it will be 32 since there is hourly metering on Service Bus. (Even if it was per minute metering, it would still be 32).
Hope this helps.
Our Application is using Azure service bus for messaging purpose. We created few topics and subscribers. We will send around 500 messages per day, but in the graph it shows hundreds of thousands of requests for 500 messages. Our billing price is also around 800$ per month. This is too much for 500 * 30 messages. I am not able to understand why the price is this much and also I want to know what does the requests in the graph mean?.
If the reason for the price is because of number of requests then how can I reduce the number of requests?. Messages I am seeing correctly. The problem is only with requests.
This is Just a sample Graph for your reference(Not original). In original graph I see around 100k of requests for 500 messages.
Here, under FAQ:
https://azure.microsoft.com/en-us/pricing/details/service-bus/
How is the Operations meter calculated for queues and topics?
For brokered entities (queues and topics/subscriptions), an operation is any API interaction with Service Bus service on any protocol.
A send, receive delete for a message that is less than or equal to 64KB in size is considered as one billable operation. If the message is greater than 64KB in size, the number of billable operations is calculated according to the message size in multiples of 64KB. For example, an 8 KB message sent to the Service Bus will be billed as one operation, but a 96 KB message sent to the Service Bus will be billed as two operations. Reading the 8KB message with a lock and then completing or explicitly abandoning the message will be billed as two operations. Renewing the lock on a message also incurs an operation.
Multiple deliveries of the same message (for example, message fan out to multiple subscribers or message retrieval after abandon, deferral, or dead lettering) will be counted as independent operations. For example, in the case of a topic with three subscriptions, a single 64KB message sent and subsequently received will generate four billable operations, one “in” plus three “out”, assuming all messages are delivered to all subscriptions and deleted during the read.
Additionally creating, reading (listing), updating and deleting a queue, topic or subscription will each incur an operation charge.
Operations are API calls made against queue or topic/subscription service endpoints. This includes Management, Send/Receive and Session State Operations.
This is what I found in the docs, but not fully sure if that's what you were looking for:
When creating a queue or subscription client, you can specify a receive mode: Peek-lock or Receive and Delete. The default receive mode is PeekLock. When operating in the default mode, the client sends a request to receive a message from Service Bus. After the client has received the message, it sends a request to complete the message.
When setting the receive mode to ReceiveAndDelete, both steps are combined in a single request. These steps reduce the overall number of operations, and can improve the overall message throughput. This performance gain comes at the risk of losing messages.