I want to consume multiple Azure-EventHub with different subscriptions in one program.
can I create multiple eph new connections with different subscriptions ? and will I be able to consume events from multiple event hubs belongs to different subscriptions parallel ?
Yes, we can pull multiple event hub belongs to different subscription in our program parallel , the only thing we need to careful about storage container(for lease checker), as we have to use storage container name for each event hub and container should be in same resource group in which event hub is available.
Related
I am working in a microservice architecture and deploying them in k8s. For communication among different application, we are using Azure eventhub to publish and subscribe events. My question is if multiple instances(POD) of an application are running, then the subscribed event callback will be triggered in single POD or in each POD?
Do i need to segregate the PODs in different consumer group?
At any given time only one processor must read events from a consumer group. Do not share a consumer group between different receivers! Important: They need to actively read, there is no "callback".
So if you have multiple consumers that you want to receive each event, then yes, you need to have multiple consumer groups, one for each pod in your case.
But since you are talking about publish-subscribe, maybe something like Azure Service Bus Topics might actually better suited for your scenario?!
I have several instances of an application (Consumer) connected to the Event Hubs instance. I would like to know how many applications are connected. I didn't find this option on the Azure Portal.
Is there a consumer viewer?
Service doesn't provide an API such that you can fetch currently consumed partitions or the number of receivers. I recommend creating consumer groups instead of creating multiple receivers on the same partitions. With consumer groups, standard namespace allows 20 parallel receivers.
Maybe I'm missing something obvious, but I need to distribute messages to all nodes in a cluster. All those nodes will subscribe to an event based on a name, but would not be able to disambiguate in any other way. That is, I can't have individual subscriptions set up for each node because it's dynamic - nodes come and go.
Azure's Service Bus (topics) does not seem appropriate for this because for multiple subscribers using the same subscription name, it will deliver in a round-robin fashion.
Is there anything in Azure that will allow me to publish an event and have all current subscribers to the same "subscription" receive it?
I need to send messages from few thousands of devices to central hub and be able to get live stream of messages for specific device from that hub. So far, Azure Event Hubs seems to the cheapest option in terms of messages count. Event Hub namespace allows to create distinct event hubs in it.
Can I create few thousands of such hubs, one per device?
Is it a good idea? What could be potential drawbacks?
How price is calculated - per namespace or per event hub? (I think per namespace, but I cannot find this info)
If per namespace, does it mean that purchased throughput units are shared among all event hubs? If yes, will single event hub namespace with 1000 event hubs will consume same amount of resources as single event hub namespace with single event hub but which receives messages from 1000 devices?
No, you are limited to 10 Event Hubs per namespace.
Event Hub per device is not the recommended usage. Usual scenario is to put all messages from all devices to the same Event Hub, and then you can separate them again in the processing side. This will scale much better.
Event Hubs quotas
Azure Event hub is an event ingestion service to which you can send events from the event publishers.The events will be available in the event hub partitions to which different consumer groups subscribe.The partitions can be designed to accept only specific kind of events.
You can also create multiple event hubs within an event hub namespace. You can create a maximum of 10 event hubs per Event hub namespace, 32 event hub partitions within an event hub and 20 consumer groups per event hub. So, You can use event hub partitions to separate the events from the event publishers and consume the events in the processing side very easily.
The pricing is at event hub level and not at namespace level. Based on the tier you choose you will be provided with variable features like:
Basic tier:
You can have only 1 consumer group
Standard and Dedicated tier:
You can create up to 20 consumer groups.
For example,
If you choose Basic or Standard tier and region as East US, You will be charged $0.028 per million events for ingress and $0.015 per unit/hour for throughput.
If you choose Dedicated tier, you will be charged 6.849$ per hour which includes the unlimited ingress and throughput charges, but the minimum hours charged are 4hrs.
The main advantage of using dedicated tier is the message retention period is 7 days whereas in basic and standard tier it is just 1 day, and the message size is up to 1 MB whereas in basic and standard tier it is just 256 KB.
Refer https://azure.microsoft.com/en-in/pricing/details/event-hubs/.
In my application, we have different type of messages, like failure events, state change events etc. These events can further classified as process failed, machine failed etc. And we have different logic to handle different type of events (Process failed is handled differently and machine failed is handled differently).
As of now we have create multiple Service Bus Topic on main type- one for failure events and another for state change events. And since we can have multiple consumer for same type, we use subscriptions for them. But on consumer side, now we have some consumer that need to listen to both kind of events (like Notification Consumer). But Azure does not allow to have a subscription from multiple topic.
My question, is "Should this notification consumer listen to multiple subscriptions or we should create one more topic (and subscription) and publisher should publish message on multiple topic?"
Logically, Notification consumer should listen to multiple Subscription, But that add complexity when we need to scale this consumer. It will depend on multiple Queue depth.