I am trying to read the Azure Service Bus queue tech specs and pricing but it is not clear to me that
1) Is there any size limit for a message?
2) Is it possible that pricing is not depend on the message sizes? It seems, but I can not believe.
(I would like to send approx 10K - 100K data per message. Message queue will be consumed by concurrent identical worker roles running in multiple instances.
Thx in advance
1-)
Service Bus queues support a maximum message size of 256 KB (the
header, which includes the standard and custom application properties,
can have a maximum size of 64 KB). There is no limit on the number of
messages held in a queue but there is a cap on the total size of the
messages held by a queue. This queue size is defined at creation time,
with an upper limit of 5 GB.
http://www.windowsazure.com/en-us/documentation/articles/service-bus-dotnet-how-to-use-queues/
2-)
Queues and Topics
$0.01 for every 10,000 messages
Messages exceeding 64KB in size will result in an additional message being charged for every 64KB in message. Billable messages include any “no message available” replies from Service Bus in response to receive requests made against empty queues/subscriptions.
Relay
$0.10 for every 100 relay hours
$0.01 for every 10,000 messages
Relay hours start when the first listener connects to a given relay address and end when the last listener disconnects from that address, and are rounded up to the next clock hour. As with queues/topics, messages greater than 64KB in size will generate additional billable messages per 64KB of additional content.
http://www.windowsazure.com/en-us/pricing/details/service-bus/
Related
Is there any message receiving limit per device on Azure IoTHub?
If any, can I remove or raise the upper limit without registering additional devices?
I tested 2 things to make sure if I can place enough load (ideally, 18000 message/s)on Azure IoT Hub in the future load tests.
① Send a certain amount of mqtt messages from a VM.
② Send a certain amount of mqtt messages from two VMs.
I expected that the traffic of ② would be twice as large as that of ①. But it wasn't. Maximum messages per minute on IoTHub of ② is not so different from that of ①. Both of them are around 3.6k [message/min]. At that time, I registered only one device on IoT Hub. So I added another device and tested ② again to see if the second device could increase the traffic. As a result, it increased the traffic and IoT Hub had bigger messages per minute.
Judging from this result, I thought IoTHub has some kind of limit on receiving message per device. But I am not sure. So if anyone know about the limit, could you tell me what kind of limit it is and how to raise the upper limit without registering additional devices because in production we use only one device.
For your information, I know there is a "unit" to increase the throughput in IoTHub. To increase the load I changed the number of unit from 2 to 20 in both ① and ②. However, it did not make messages/min in IotHub bigger. I'd also like to know why the "unit" did not work as expected.
Thank you for reading, in advance. Any comment would be my help.
Every basic (B1,B2, B3) or standard unit of IoT Hub SKU (S1, S2, S3) has specific daily message quota as per https://azure.microsoft.com/en-us/pricing/details/iot-hub/. A single IoTHub can support 1 million devices and there is no per device cost associated, only the msg/day quota as above.
e.g. S1 SKU has 400,000 msg/day quota and you can add multiple units of S1 to increase capacity. S2 has 6000,000 msg/day and S3 has 300,000,000 msg/day quota per unit and more units can be added.
Before this limit is reached IoTHub will raise alert which can be used to automatically add more units or jump to higher SKU.
Regarding your test, there are specific throttling limits to avoid misuse of the service here -
https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-quotas-throttling
As an example, for 18000 msg/sec you will need 3 units of S3 SKU (each with 6000 msg/sec rate limit). In addition there are other limits like how quickly connections can be attempted, if using Azure IoT SDK's the built-in retry logic helps overcome this otherwise you need to have retry policy. Basically you dont want million device trying to connect at the same time, IoTHub will only accept connections at a certain rate. This is not concurrent connection limit but a rate at which new connnections are accepted.
Azure Service Bus allows to use sessioned entities(queues and topics), we use them to keep batching processing by the same consumer(if we send a batch from producer, it must be processed by the same consumer service). I cannot find any recommendations about max allowed sessions number per entity, we have 4 sessions for 1000 000 messages for now and I am wondered if we can increase the number. If we can, what is max possible value for my case?
Short info about my process:
* 1 000 000 messages to be sent to ASB entity
* Batch size is 100 messages
* Number of sessions is 4
* ASB entities have enabled sessions and partitioning
Thank you
Azure Service Bus quotas documentation doesn't spell out a limit on number of concurrent session you can have.
You are limitted by two factors:
The number of concurrent connecitons on a namespace (5,000 for AMQP and 1,000 for SBMP)
Number of concurrent receive request per entity (5,000) you can have.
As long as you don't exceed those quotas, you should be able to scale out your receiver to handle more sessions.
Looking at Azure Service Bus to replace some unreliable RabbitMQ servers and the one thing that isn't clear about the pricing is what exactly do they count as a "Brokered connection" for billing purposes?
We have about a dozen machines that are processing messages off over 1,000 queues (and one machine populating the queue), so does 1 application running on 1 machine count as one brokered connection (regardless of how many queues it's listening to)? Or would each machine count as 1,000+ brokered connections (which could add up very quickly)?
So let's say I did something like this:
var queues = queueNames.Select(q =>
{
if (!manager.QueueExists(q))
{
manager.CreateQueue(q);
}
return new QueueClient(ServiceBusConnectionString, q);
}).ToArray();
Add queueNames is an array of, let's say, 10 strings. Is that 10 brokered connections? 1 ? Or something else?
The polling of the queue encounters the cost for a brokered connection when you hold the queue connection open and sit waiting for a message (eg if you kept the queue open for 30 seconds waiting for a message). If you use the default zero timeout (which will just return null when there is no message to receive) it doesn't count as a brokered connection. An example of a non zero timeout would be using something like Receive(TimeSpan) where you wait for the timeout specified.
You could use OnMessageAsync to listen for messages rather than polling, which doesn't look like a brokered connection.
As for how its calculated, looks like this is concurrent connections averaged hourly over the month. There are some good example calculations in the pricing guide.
From this pricing guide
A brokered connection is defined as one of the following:
An AMQP connection from a client to a Service Bus queue or topic/subscription.
An HTTP call to receive a message from a Service Bus topic or queue that has a
receive timeout value greater than zero.
Service Bus charges for the peak number of concurrent brokered connections that exceed the included quantity (1,000 in the Standard tier). Peaks are measured on an hourly basis, prorated by dividing by 744 hours in a month, and added up over the monthly billing period. The included quantity (1,000 brokered connections per month) is applied at the end of the billing period against the sum of the prorated hourly peaks.
You should probably confirm this with the Azure team via support in the billing Portal, they are generally pretty good with requests like these.
I am working on the POC for Azure Event hubs to implement the same into our application.
Quick Brief on flow.
Created tool to read the CSV data from local folder and send it to event hub.
We are sending Event Data in Batch to event hub.
With 12 instance of tool (Parallel), I can send a total of 600 000 lines of messages to Event hub within 1 min.
But, On receiver side, to receive the 600 000 lines of data, it takes more than 10 mins.
Need to achieve
I would like to Match/double my egress speed on the receiver to
process the data. Existing Configuration
The configuration I have made user are
TU - 10 One Event hub with 32 Partition.
Coding logic goes same as mentioned in MSDN
Only difference is, I am sending line of data in a batch.
EventProcessorhost with options {MaxBatchSize= 1000000,
PrefetchCount=1000000
To achieve higher egress rate (aka faster processing pipeline) in eventhubs:
Create a Scaled-out pipeline - each partition in EventHub is the unit-of-scale for processing events out of EventHub. With the Scale you described (6Lakh events per min --> 10K events per sec - with 32 partitions - you already got this right). Make sure you create as many partitions as you envision your pipeline need in near future. Imagine analyzing traffic on a Highway and no. of lanes is the only limitation for the amount of traffic.
Equal load distribution across partitions: if you are using SendToASpecificPartition or SendUsingPartitionKey - you will need to take care of equal load distribution. If you use EventHubClient.Send(EventDataWithOutPartitionKey) - EventHubs service will make sure all of your partitions are equally loaded. If a single EventHub Partition is heavily loaded - the amount of time you can process all events on EventHub will be bound by no. of events on this Partition.
Scale-out physical resources on the Receiver/EventProcessorHost: most importantly Network (Sockets & bandwidth) & after-a-point, CPU & Memory. Use PartitionManagerOptions.MaxReceiveClients to increase the maximum number of EventHubClients (which has a dedicated MessagingFactory, which maps to 1 socket) created per EventProcessorHost instance. By default it is 16.
Let me know how it went... :)
More on Event Hubs.
I have a process(Process A) that keeps sending events to an ASB topic. There are multiple consumers of the topic and therefore multiple subscriptions. So lets say that one of the consumer's process is down. And due to this the topic gets full as the messages are not consumed. Does this mean then Process A also fails as it is not able to send messages to ASB topic as its full?
Two more things to check:
Make sure that your dead letter queue is not full that counts towards the size of the entity.
Make sure that you have at least one subscription that works for each message. For example, if you send a message with ID=1, but you only have a subscription with ID=2, the messages will get backed up.
I think you are correct, once the limit is reached the queue stops.
However, with partitioning (using all 16 partitions * 5 GB), you can store up to 80 GB:
https://azure.microsoft.com/en-us/blog/partitioned-service-bus-queues-and-topics/
Another solution is to use auto forwarding, so the topic forwards all messages to another queue/topic
https://azure.microsoft.com/en-us/documentation/articles/service-bus-auto-forwarding/
This way each subscriber can have it's own queue of 5GB (or 80GB if you use partition)
Some more info:
https://azure.microsoft.com/nl-nl/documentation/articles/service-bus-azure-and-service-bus-queues-compared-contrasted/
https://azure.microsoft.com/en-us/documentation/articles/service-bus-quotas/