I'm wondering how I can get certain behavior with Azure Service Bus topics and subscriptions. For our app we have 1 topic and many subscriptions. Subscriptions use filters. When a message is published it can be filtered by one or more subscriptions.
We already use peeklock and maxconcurrentcalls of 1, but that does not prevent multiple consumers from running in parallel.
What we want to do is let multiple consumer instances subscribe to the same subscription. But
* Only one instance may process a message at a given time. (in that way the order of processing is garanteed).
* We hope that those instances are load balanced.
So, at the end we want to load balance our consumers while keeping message order and no parallel processing.
How can we achieve this?
Edit: I want to make sure that I have failover of the consumers. And hopefully the consumers are load balanced: I don't want that all active instances are on 1 machine.
Multiple consumers will work in parallel, so it is very difficult to let just one work and have the others standby.
The solution is a single started consumer.
Failover of a consumer should be build on the consumer side, Azure has no idea of the state of your consumer.
So a second machine should check if the first one is still working, if not, take over, and make sure to lock the first one out in case this machine starts consuming again.
Related
I am working in a microservice architecture and deploying them in k8s. For communication among different application, we are using Azure eventhub to publish and subscribe events. My question is if multiple instances(POD) of an application are running, then the subscribed event callback will be triggered in single POD or in each POD?
Do i need to segregate the PODs in different consumer group?
At any given time only one processor must read events from a consumer group. Do not share a consumer group between different receivers! Important: They need to actively read, there is no "callback".
So if you have multiple consumers that you want to receive each event, then yes, you need to have multiple consumer groups, one for each pod in your case.
But since you are talking about publish-subscribe, maybe something like Azure Service Bus Topics might actually better suited for your scenario?!
Need a bit of architectural guidance. I have a set of stateless services that do various functions. My architecture allows for multiple copies of each service to run at the same time (as they are stateless), allowing me to:
scale up as needed for handling larger workloads
have fault-tolerance (if one instance of a service fails, no problem as there will be others to take on that work).
However, I don't want duplication of work.
If Service A, Instance 1 has already taken Job ABC, I don't want Service A, Instance 2, to take on that same job. So, I could avoid this problem by using Azure Service Bus Queues. Only a single worker would get a particular item from the queue and would only be reassigned to another worker, if the worker didn't mark it as complete in a set time.
So what's an appropriate use-case for Topics (Pub/Sub)? It seems like if I ever have multiple copies of the same service, I must rely on Queues. Is that right?
Asked another way, is there a way to use Topics in Azure Service Bus or similar products/services but avoid duplication of work? Also, if there is a way to lock a message (for a short period of time) when using Topics, is it possible to lock that message to just one instance of Service A (so no other instances of Service A will have access to it) but the message will be broadcast to Service B, Service, C, etc.?
is there a way to use Topics in Azure Service Bus or similar
products/services but avoid duplication of work?
Yes, there is. Basically with that you would need to use each subscription as a queue. What you will need to do is define proper filters so that one kind of message is sent to a single subscription (that way it acts as a queue) and have multiple listeners (service instances in your case) listen to a specific subscription only.
Also, if there is a way to lock a message (for a short period of time)
when using Topics, is it possible to lock that message to just one
instance of Service A (so no other instances of Service A will have
access to it) but the message will be broadcast to Service B, Service,
C, etc.?
It is certainly possible to lock a message. For that you will need to fetch messages in Peek-Lock mode. However if multiple subscribers (services) are involved, then only one subscriber will be able to lock the message and access it. For other subscribers, the message will be invisible. You can't have a scenario where one service acquires the lock and other services still receive the message.
Azure function triggers would provide all what you are looking for out of the box.
If you are not leveraging any advanced queuing features of service bus then I would recommend you look at storage queues to save some money.
If you need service bus then you can use service bus triggers.
Hope that helps.
I want to create a Service Bus Topic with a couple of subscriptions using filters for different message types. However I need to guarantee that all messages sent to the Topic will be received and successfully processed by at least one subscription, even if all of the subscribing processes go offline.
Is there a better way than auto-forwarding to queues for each filter, and a way to capture messages ignored by all filtering subscribers without capturing all messages?
Edit: my motivation is to provide a queue-like mechanism with prioritisation without creating a queue for each message type/priority level, or at least manage the complexity of multiple queues on the listening side. A queue generally guarantees a consumer. Rather than have the publisher have to push to different queues I would like to use a topic and use filters to manage priority.
Based on my current knowledge of the SB I suspect that I just need to make sure the subscriptions are in place for a topic including an inverse catch-all filter subscription before exposing the topic for use. I don't know whether subscriptions are completely reliable.
However I need to guarantee that all messages sent to the Topic will be received and successfully processed by at least one subscription, even if all of the subscribing processes go offline.
There's a problem in that statement. Topics and subscriptions are there to implement pub/sub and decouple publishers from subscribers. The broker itself does not guarantee there will be subscribers.
While topics support EnableFilteringMessagesBeforePublishing (TopicDescription.EnableFilteringMessagesBeforePublishing) it is not recommended for production use.
Update
Based on the updated question, the general answer remains the same. Topics/subscriptions are for pub/sub and decoupling. If you want to ensure that no message is lost once subscriber is coming online, you will need to ensure that subscription is created first.
I don't know whether subscriptions are completely reliable.
Yes, subscriptions are reliable. Behind the scenes subscription is a queue.
In case you want to route your messages to different processors based on message type, publishing that message to a topic and having forwarding subscriptions is a good approach. You do need to be mindful of the quotas (how many subscriptions per topic you can create), but those are fairly high. And if you get to that point, it's possible to reduce number of subscriptions when a given processor handles multiple message types by having more complex SQL filtering rules.
I want to use the Azure service-bus event-hub to send a single message to many instances of an application but not with the goal of load-balancing.
What I want to do is broadcast a message to all application instances regardless of which partition it is.
I have heard that I can do it using consumer groups but I couldn't find any tutorials on it.
This scenario is relatively straight forward assuming you can consistently inject the same unique name into each application instance such as on the command line. If you want to use an EventProcessorHost just specify a different Consumer Group string for each application instance in the constructor. Then it will "load balance" to only one machine per group causing each instance to receive all the messages. There are fancier things you can do if you end up with more machines
I suggest looking at this for some additional discussion of Consumer Groups, or this which discusses using EventHubs as a backplane (basically what you're doing).
I have a web api application which performs different type of actions an a Domain entity, called Application. Actions like "Copy", "Update", "Recycle", "Restore" etc.
This actions needs to be executed, per Application, in First In First Out order, not randomly or simultaneous. However, it can process simultaneously two Actions as long as they are for two separate Applications.
Is some kind of a queue, but not a big queue for all the requests, but a queue of actions for each Application in database.
Knowing this, i think that azure service bus queue is a good solution for this scenario.
However, the solution i can think of right now is to programmatically create a queue for each Application i have in database, and start listening to that queue.
Is possible to get messages from the queue based on a filter? (using FIFO principle) So i have to subscribe only to one queue? (instead of having a queue for each Application - which is very hard to maintain)
What you want is Azure Service Bus Topics/Subscriptions.
Subscriptions allow you to filter messages that are published to a topic using a SqlFilter on the message headers.
The article linked above should provide enough examples to meet your needs.
I think u can solve this by using Sessions.
I just came across this very clear article: https://dev.to/azure/ordered-queue-processing-in-azure-functions-4h6c which explains in to detail how Azure Service Bus Queue sessions work.
In short: by defining a SessionId on the messages you can force the ordering of the processing within a session, the downside is that there will be no parallelization for all messages in a session between multiple consumers of the queue.