I have several instances of an application (Consumer) connected to the Event Hubs instance. I would like to know how many applications are connected. I didn't find this option on the Azure Portal.
Is there a consumer viewer?
Service doesn't provide an API such that you can fetch currently consumed partitions or the number of receivers. I recommend creating consumer groups instead of creating multiple receivers on the same partitions. With consumer groups, standard namespace allows 20 parallel receivers.
Related
I'm evaluating the use of Azure Event Hub vs Kafka as a Service Broker. I was hoping I would be able to create two local apps side by side, one that consumes messages using Kafka with the other one using Azure Event Hub. I've got a docker container set up which is a Kafka instance and I'm in the process of setting up Azure Event hub using my Azure account (as far as I know there's no other way to create a local/development instance for Azure Event Hub).
Does anyone have any information regarding the two that might be useful when comparing their features?
Can't add a comment directly, but the currently top rate answer has the line
Kafka can have multiple topics each Azure Event Hub is a single topic.
This is misleading as it makes it sound like you can't have multiple topics, which you can.
As per https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview#kafka-and-event-hub-conceptual-mapping an "Event Hub" is a topic while an "Event Hub Namespace" is the Kafka cluster.
This decision usually is driven by a broader architectural choice if you are choosing azure as your iaas and paas solution then event hub provides a great integration within the azure ecosystem but if you don't want a vendor lock in kafka is better option.
Operationally also if you want fully managed service then with event hub it's out of the box but with kafka you also get this with confluent platform.
Maturity wise kafka is older and with large community you have a larger support.
Feature wise what kafka ecosystem provides azure ecosystem has those things but if you talk about only event hub then it lacks few features compared to kafka
I think this link can help you extend your understanding https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview
While Apache Kafka is software you typically need to install and operate, Event Hubs is a fully managed, cloud-native service. There are no servers, disks, or networks to manage and monitor and no brokers to consider or configure, ever. You create a namespace, which is an endpoint with a fully qualified domain name, and then you create Event Hubs (topics) within that namespace. For more information about Event Hubs and namespaces, see Event Hubs features. As a cloud service, Event Hubs uses a single stable virtual IP address as the endpoint, so clients don't need to know about the brokers or machines within a cluster. Even though Event Hubs implements the same protocol, this difference means that all Kafka traffic for all partitions is predictably routed through this one endpoint rather than requiring firewall access for all brokers of a cluster. Scale in Event Hubs is controlled by how many throughput units you purchase, with each throughput unit entitling you to 1 Megabyte per second, or 1000 events per second of ingress and twice that volume in egress. Event Hubs can automatically scale up throughput units when you reach the throughput limit if you use the Auto-Inflate feature; this feature work also works with the Apache Kafka protocol support.
You can find more on feature comparison here - https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview
Kafka can have multiple topics each Azure Event Hub is a single topic. Kafka running inside a container means you have to manage it. Azure Event Hub is a PaaS which means they managed the platform side. If you don't know how to make Kafka redundant, reliable, and scalable you may want to go with Azure Event Hubs or any PaaS that offers a similar pub/sub model. Event Hub platform is already scalable, reliable, and redundant.
You should compare
the administration capabilites / effort (as previously said)
the functional capabilities such as competing customer and pub/sub patterns
the performance : you should consider kafka if you plan to exceed the event hub quotas
I am working in a microservice architecture and deploying them in k8s. For communication among different application, we are using Azure eventhub to publish and subscribe events. My question is if multiple instances(POD) of an application are running, then the subscribed event callback will be triggered in single POD or in each POD?
Do i need to segregate the PODs in different consumer group?
At any given time only one processor must read events from a consumer group. Do not share a consumer group between different receivers! Important: They need to actively read, there is no "callback".
So if you have multiple consumers that you want to receive each event, then yes, you need to have multiple consumer groups, one for each pod in your case.
But since you are talking about publish-subscribe, maybe something like Azure Service Bus Topics might actually better suited for your scenario?!
I want to consume multiple Azure-EventHub with different subscriptions in one program.
can I create multiple eph new connections with different subscriptions ? and will I be able to consume events from multiple event hubs belongs to different subscriptions parallel ?
Yes, we can pull multiple event hub belongs to different subscription in our program parallel , the only thing we need to careful about storage container(for lease checker), as we have to use storage container name for each event hub and container should be in same resource group in which event hub is available.
Maybe I'm missing something obvious, but I need to distribute messages to all nodes in a cluster. All those nodes will subscribe to an event based on a name, but would not be able to disambiguate in any other way. That is, I can't have individual subscriptions set up for each node because it's dynamic - nodes come and go.
Azure's Service Bus (topics) does not seem appropriate for this because for multiple subscribers using the same subscription name, it will deliver in a round-robin fashion.
Is there anything in Azure that will allow me to publish an event and have all current subscribers to the same "subscription" receive it?
I have a single client application that will publish the message to a single location/target, and I need that message to then end up in 3 separate other queues as well ( and subsequently processed)
Basically here is the use case:
A website collects customers information in a lead form. That lead information is pushed to an restful web api. The restfull web api then publishes a message to a single location and then returns a success result to client. In the background, the message ends up on 4 queues, and ultimately sent to 4 different other web services (some external, some internal)
The system needs to be performant with respect to how quickly all 4 of the queues are processed from the 4 queues. But the volume of leads is not necessarily extremely high. (perhaps a few hundred leads per day)
Here is an image of what I am thinking
You could use a Topic in combination with the autoforwarding feature.
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-auto-forwarding
A single publish to a topic could then be setup to auto-forward to 4 separate queues.
Instead of uses Queues, you should use Topics:
a queue is often used for point-to-point communication, topics are useful in publish/subscribe scenarios.
Topics and subscriptions provide a "one-to-many" form of communication, using a publish/subscribe pattern.
If you really need to use a Queue there is no other way than to send copies of the message to the different queues.
One best solution to solve your business scenario, is using a Service Bus Topic with four Topic Subscriptions. You can send the message to the topic. You can create filters (or) Topic subscription rules to filter the messages received by the Service Bus Topic.
You can then set the auto-forward property of each topic subscription to the desired Service Bus Queue.