I am working in a microservice architecture and deploying them in k8s. For communication among different application, we are using Azure eventhub to publish and subscribe events. My question is if multiple instances(POD) of an application are running, then the subscribed event callback will be triggered in single POD or in each POD?
Do i need to segregate the PODs in different consumer group?
At any given time only one processor must read events from a consumer group. Do not share a consumer group between different receivers! Important: They need to actively read, there is no "callback".
So if you have multiple consumers that you want to receive each event, then yes, you need to have multiple consumer groups, one for each pod in your case.
But since you are talking about publish-subscribe, maybe something like Azure Service Bus Topics might actually better suited for your scenario?!
Related
I have several instances of an application (Consumer) connected to the Event Hubs instance. I would like to know how many applications are connected. I didn't find this option on the Azure Portal.
Is there a consumer viewer?
Service doesn't provide an API such that you can fetch currently consumed partitions or the number of receivers. I recommend creating consumer groups instead of creating multiple receivers on the same partitions. With consumer groups, standard namespace allows 20 parallel receivers.
Need a bit of architectural guidance. I have a set of stateless services that do various functions. My architecture allows for multiple copies of each service to run at the same time (as they are stateless), allowing me to:
scale up as needed for handling larger workloads
have fault-tolerance (if one instance of a service fails, no problem as there will be others to take on that work).
However, I don't want duplication of work.
If Service A, Instance 1 has already taken Job ABC, I don't want Service A, Instance 2, to take on that same job. So, I could avoid this problem by using Azure Service Bus Queues. Only a single worker would get a particular item from the queue and would only be reassigned to another worker, if the worker didn't mark it as complete in a set time.
So what's an appropriate use-case for Topics (Pub/Sub)? It seems like if I ever have multiple copies of the same service, I must rely on Queues. Is that right?
Asked another way, is there a way to use Topics in Azure Service Bus or similar products/services but avoid duplication of work? Also, if there is a way to lock a message (for a short period of time) when using Topics, is it possible to lock that message to just one instance of Service A (so no other instances of Service A will have access to it) but the message will be broadcast to Service B, Service, C, etc.?
is there a way to use Topics in Azure Service Bus or similar
products/services but avoid duplication of work?
Yes, there is. Basically with that you would need to use each subscription as a queue. What you will need to do is define proper filters so that one kind of message is sent to a single subscription (that way it acts as a queue) and have multiple listeners (service instances in your case) listen to a specific subscription only.
Also, if there is a way to lock a message (for a short period of time)
when using Topics, is it possible to lock that message to just one
instance of Service A (so no other instances of Service A will have
access to it) but the message will be broadcast to Service B, Service,
C, etc.?
It is certainly possible to lock a message. For that you will need to fetch messages in Peek-Lock mode. However if multiple subscribers (services) are involved, then only one subscriber will be able to lock the message and access it. For other subscribers, the message will be invisible. You can't have a scenario where one service acquires the lock and other services still receive the message.
Azure function triggers would provide all what you are looking for out of the box.
If you are not leveraging any advanced queuing features of service bus then I would recommend you look at storage queues to save some money.
If you need service bus then you can use service bus triggers.
Hope that helps.
Maybe I'm missing something obvious, but I need to distribute messages to all nodes in a cluster. All those nodes will subscribe to an event based on a name, but would not be able to disambiguate in any other way. That is, I can't have individual subscriptions set up for each node because it's dynamic - nodes come and go.
Azure's Service Bus (topics) does not seem appropriate for this because for multiple subscribers using the same subscription name, it will deliver in a round-robin fashion.
Is there anything in Azure that will allow me to publish an event and have all current subscribers to the same "subscription" receive it?
I want to use the Azure service-bus event-hub to send a single message to many instances of an application but not with the goal of load-balancing.
What I want to do is broadcast a message to all application instances regardless of which partition it is.
I have heard that I can do it using consumer groups but I couldn't find any tutorials on it.
This scenario is relatively straight forward assuming you can consistently inject the same unique name into each application instance such as on the command line. If you want to use an EventProcessorHost just specify a different Consumer Group string for each application instance in the constructor. Then it will "load balance" to only one machine per group causing each instance to receive all the messages. There are fancier things you can do if you end up with more machines
I suggest looking at this for some additional discussion of Consumer Groups, or this which discusses using EventHubs as a backplane (basically what you're doing).
I'm wondering how I can get certain behavior with Azure Service Bus topics and subscriptions. For our app we have 1 topic and many subscriptions. Subscriptions use filters. When a message is published it can be filtered by one or more subscriptions.
We already use peeklock and maxconcurrentcalls of 1, but that does not prevent multiple consumers from running in parallel.
What we want to do is let multiple consumer instances subscribe to the same subscription. But
* Only one instance may process a message at a given time. (in that way the order of processing is garanteed).
* We hope that those instances are load balanced.
So, at the end we want to load balance our consumers while keeping message order and no parallel processing.
How can we achieve this?
Edit: I want to make sure that I have failover of the consumers. And hopefully the consumers are load balanced: I don't want that all active instances are on 1 machine.
Multiple consumers will work in parallel, so it is very difficult to let just one work and have the others standby.
The solution is a single started consumer.
Failover of a consumer should be build on the consumer side, Azure has no idea of the state of your consumer.
So a second machine should check if the first one is still working, if not, take over, and make sure to lock the first one out in case this machine starts consuming again.