I want to use the Azure service-bus event-hub to send a single message to many instances of an application but not with the goal of load-balancing.
What I want to do is broadcast a message to all application instances regardless of which partition it is.
I have heard that I can do it using consumer groups but I couldn't find any tutorials on it.
This scenario is relatively straight forward assuming you can consistently inject the same unique name into each application instance such as on the command line. If you want to use an EventProcessorHost just specify a different Consumer Group string for each application instance in the constructor. Then it will "load balance" to only one machine per group causing each instance to receive all the messages. There are fancier things you can do if you end up with more machines
I suggest looking at this for some additional discussion of Consumer Groups, or this which discusses using EventHubs as a backplane (basically what you're doing).
Related
I am working in a microservice architecture and deploying them in k8s. For communication among different application, we are using Azure eventhub to publish and subscribe events. My question is if multiple instances(POD) of an application are running, then the subscribed event callback will be triggered in single POD or in each POD?
Do i need to segregate the PODs in different consumer group?
At any given time only one processor must read events from a consumer group. Do not share a consumer group between different receivers! Important: They need to actively read, there is no "callback".
So if you have multiple consumers that you want to receive each event, then yes, you need to have multiple consumer groups, one for each pod in your case.
But since you are talking about publish-subscribe, maybe something like Azure Service Bus Topics might actually better suited for your scenario?!
Need a bit of architectural guidance. I have a set of stateless services that do various functions. My architecture allows for multiple copies of each service to run at the same time (as they are stateless), allowing me to:
scale up as needed for handling larger workloads
have fault-tolerance (if one instance of a service fails, no problem as there will be others to take on that work).
However, I don't want duplication of work.
If Service A, Instance 1 has already taken Job ABC, I don't want Service A, Instance 2, to take on that same job. So, I could avoid this problem by using Azure Service Bus Queues. Only a single worker would get a particular item from the queue and would only be reassigned to another worker, if the worker didn't mark it as complete in a set time.
So what's an appropriate use-case for Topics (Pub/Sub)? It seems like if I ever have multiple copies of the same service, I must rely on Queues. Is that right?
Asked another way, is there a way to use Topics in Azure Service Bus or similar products/services but avoid duplication of work? Also, if there is a way to lock a message (for a short period of time) when using Topics, is it possible to lock that message to just one instance of Service A (so no other instances of Service A will have access to it) but the message will be broadcast to Service B, Service, C, etc.?
is there a way to use Topics in Azure Service Bus or similar
products/services but avoid duplication of work?
Yes, there is. Basically with that you would need to use each subscription as a queue. What you will need to do is define proper filters so that one kind of message is sent to a single subscription (that way it acts as a queue) and have multiple listeners (service instances in your case) listen to a specific subscription only.
Also, if there is a way to lock a message (for a short period of time)
when using Topics, is it possible to lock that message to just one
instance of Service A (so no other instances of Service A will have
access to it) but the message will be broadcast to Service B, Service,
C, etc.?
It is certainly possible to lock a message. For that you will need to fetch messages in Peek-Lock mode. However if multiple subscribers (services) are involved, then only one subscriber will be able to lock the message and access it. For other subscribers, the message will be invisible. You can't have a scenario where one service acquires the lock and other services still receive the message.
Azure function triggers would provide all what you are looking for out of the box.
If you are not leveraging any advanced queuing features of service bus then I would recommend you look at storage queues to save some money.
If you need service bus then you can use service bus triggers.
Hope that helps.
I have an online service hosted on Azure, that asynchronously sends data to on-premise clients.
Each client is identified by an unique code.
Actually there is a single topic, with a subscription for each client which has a filter on the unique code, that is sent as a parameter in the message. No message will ever be broadcasted to all the clients.
I feel that using topic this way is wrong.
The alternative that comes to my mind is to use a dedicated queue for each client, that is created on first contact
Could this be a better approach?
Thanks
In my opinion using Topics and Subscriptions is the right way to go. Here's the reason why:
Currently the routing logic (which message needs to go to which subscription) is handled by Azure Service Bus based on the rules you have configured. If you go with queues, the routing logic will need to come to your hosted service. You'll need to ensure that the queue exists before sending each message. I think it will increase the complexity at your service level somehow.
Furthermore, topics and subscriptions would enable you to do build an audit trail kind of functionality (not sure if you're looking for this kind of functionality). You can create a separate subscription that has a rule to deliver all messages (True SQL Rule) to that subscription along with client specific subscription.
Creating a separate Queue for each client is not advisable. This is the problem solved by Topics.
If you have separate Queue for each client, then you need to send messages to multiple Queues from Server. This will become tedious when the number of clients increases.
Having a single Topic and multiple Subscriptions is easy to manage as the message will be sent only to a single Topic from Server.
I have a web api application which performs different type of actions an a Domain entity, called Application. Actions like "Copy", "Update", "Recycle", "Restore" etc.
This actions needs to be executed, per Application, in First In First Out order, not randomly or simultaneous. However, it can process simultaneously two Actions as long as they are for two separate Applications.
Is some kind of a queue, but not a big queue for all the requests, but a queue of actions for each Application in database.
Knowing this, i think that azure service bus queue is a good solution for this scenario.
However, the solution i can think of right now is to programmatically create a queue for each Application i have in database, and start listening to that queue.
Is possible to get messages from the queue based on a filter? (using FIFO principle) So i have to subscribe only to one queue? (instead of having a queue for each Application - which is very hard to maintain)
What you want is Azure Service Bus Topics/Subscriptions.
Subscriptions allow you to filter messages that are published to a topic using a SqlFilter on the message headers.
The article linked above should provide enough examples to meet your needs.
I think u can solve this by using Sessions.
I just came across this very clear article: https://dev.to/azure/ordered-queue-processing-in-azure-functions-4h6c which explains in to detail how Azure Service Bus Queue sessions work.
In short: by defining a SessionId on the messages you can force the ordering of the processing within a session, the downside is that there will be no parallelization for all messages in a session between multiple consumers of the queue.
I'm wondering how I can get certain behavior with Azure Service Bus topics and subscriptions. For our app we have 1 topic and many subscriptions. Subscriptions use filters. When a message is published it can be filtered by one or more subscriptions.
We already use peeklock and maxconcurrentcalls of 1, but that does not prevent multiple consumers from running in parallel.
What we want to do is let multiple consumer instances subscribe to the same subscription. But
* Only one instance may process a message at a given time. (in that way the order of processing is garanteed).
* We hope that those instances are load balanced.
So, at the end we want to load balance our consumers while keeping message order and no parallel processing.
How can we achieve this?
Edit: I want to make sure that I have failover of the consumers. And hopefully the consumers are load balanced: I don't want that all active instances are on 1 machine.
Multiple consumers will work in parallel, so it is very difficult to let just one work and have the others standby.
The solution is a single started consumer.
Failover of a consumer should be build on the consumer side, Azure has no idea of the state of your consumer.
So a second machine should check if the first one is still working, if not, take over, and make sure to lock the first one out in case this machine starts consuming again.