I am building an application that needs to be able to create events, publish them, and have multiple consumers. The tutorials I have found so far suggest that Azure Topics are the right thing to use for this (multiple publishers and multiple subscribers), but I noticed an option in my Azure portal for EventHub and it seems like a highly scalable solution that may be a newer implementation of Pub/Sub for Azure. I have been searching for documentation comparing the two and haven't really found anything. Can someone explain why I would choose one of these solutions over the other.
The scenario I have is many clients in a Multi-Tenant application may create events at any time, those events need to be published to "n" subscribers for consumption. The subscribers need to be able to change without any change to the application (i.e. subscribers should be able to subscribe themselves to events without modifying publisher code).
Thanks for the help.
Event Hubs have below advantage as compared with Azure ServiceBus:
Its client-side cursor: Developers can use a client-side pointer, or
offset, to retrieve messages from a specific point in the event
stream
Partitioned consumer support: Throughput Unit and inbound messages
can be targeted at a specific partition through a user-defined
partition key value.
Significant time-based retention options
Related
Using Azure Service Bus - Topics, I want to implement a solution wherein messages are sent/notified to end consumers once the producer sends the message to Topic (like Queues).
I understand that Topics work as Pub/Sub model wherein subscribers need to read messages from subscriptions. But I'm looking for a workaround that works some what similar to Queue (where it triggers a web job / service when any message is received).
I have few thoughts like
1. Using Auto-Forwarding in subscriptions to forward messages to Queues but again I think if this kills the purpose of Topics
2. Schedule a job to process these requests but again I think if I'm delaying the process
First, I want to know if Service Bus Topic is right option to go with? Next, If possible to implement a workaround what is the best/better way?
PS: I have to send messages which has information - I guess I can't use Relays
Just to be clear, Queues and Topics in Service Bus are different. As you noted, Topics are useful in publish/subscribe scenarios.
Since you are looking for something that gets triggered, Azure functions might be what you need.
Azure Functions supports trigger and output bindings for Service Bus
queues and topics
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus
I think that #William is right, you can use/attach other process to the subscription to make what you trying to do.
he mentioned Azure Functions which is a good tool and I want to suggest Azure Logic Apps as well in case you want to take some decisions based in the message that you received.
With Azure Logic Apps you can create a logic Workflow and integrate many services using connectors provided by this tool.
You will find more in:
https://learn.microsoft.com/en-us/azure/connectors/connectors-create-api-servicebus
And for answer your question
First, I want to know if Service Bus Topic is right option to go with?
The quick answer is yes, using messaging patterns is the best way to create reliable solutions.
In your case you want as well notify another system after receiving a message.
The only thing that you need to be aware is, whenever you did not receive the notification what you'll do? you need to think about this scenario.
From discussion above.
Azure functions with Queues/Topics
Regardless of queues or topics. you can trigger azure function with both. This function will process the message . Now you can create two methods in same function SendEmail(), sendPhoneNotifcation() and parrellize the tasks using C# task parallel library. So same function will do both tasks in parallel.
Every time you get a message , your function is triggered. Process the message and notify user. The benefit is this function will scale automatically if you have large number of message coming through queue.
I have an online service hosted on Azure, that asynchronously sends data to on-premise clients.
Each client is identified by an unique code.
Actually there is a single topic, with a subscription for each client which has a filter on the unique code, that is sent as a parameter in the message. No message will ever be broadcasted to all the clients.
I feel that using topic this way is wrong.
The alternative that comes to my mind is to use a dedicated queue for each client, that is created on first contact
Could this be a better approach?
Thanks
In my opinion using Topics and Subscriptions is the right way to go. Here's the reason why:
Currently the routing logic (which message needs to go to which subscription) is handled by Azure Service Bus based on the rules you have configured. If you go with queues, the routing logic will need to come to your hosted service. You'll need to ensure that the queue exists before sending each message. I think it will increase the complexity at your service level somehow.
Furthermore, topics and subscriptions would enable you to do build an audit trail kind of functionality (not sure if you're looking for this kind of functionality). You can create a separate subscription that has a rule to deliver all messages (True SQL Rule) to that subscription along with client specific subscription.
Creating a separate Queue for each client is not advisable. This is the problem solved by Topics.
If you have separate Queue for each client, then you need to send messages to multiple Queues from Server. This will become tedious when the number of clients increases.
Having a single Topic and multiple Subscriptions is easy to manage as the message will be sent only to a single Topic from Server.
I want to create a Service Bus Topic with a couple of subscriptions using filters for different message types. However I need to guarantee that all messages sent to the Topic will be received and successfully processed by at least one subscription, even if all of the subscribing processes go offline.
Is there a better way than auto-forwarding to queues for each filter, and a way to capture messages ignored by all filtering subscribers without capturing all messages?
Edit: my motivation is to provide a queue-like mechanism with prioritisation without creating a queue for each message type/priority level, or at least manage the complexity of multiple queues on the listening side. A queue generally guarantees a consumer. Rather than have the publisher have to push to different queues I would like to use a topic and use filters to manage priority.
Based on my current knowledge of the SB I suspect that I just need to make sure the subscriptions are in place for a topic including an inverse catch-all filter subscription before exposing the topic for use. I don't know whether subscriptions are completely reliable.
However I need to guarantee that all messages sent to the Topic will be received and successfully processed by at least one subscription, even if all of the subscribing processes go offline.
There's a problem in that statement. Topics and subscriptions are there to implement pub/sub and decouple publishers from subscribers. The broker itself does not guarantee there will be subscribers.
While topics support EnableFilteringMessagesBeforePublishing (TopicDescription.EnableFilteringMessagesBeforePublishing) it is not recommended for production use.
Update
Based on the updated question, the general answer remains the same. Topics/subscriptions are for pub/sub and decoupling. If you want to ensure that no message is lost once subscriber is coming online, you will need to ensure that subscription is created first.
I don't know whether subscriptions are completely reliable.
Yes, subscriptions are reliable. Behind the scenes subscription is a queue.
In case you want to route your messages to different processors based on message type, publishing that message to a topic and having forwarding subscriptions is a good approach. You do need to be mindful of the quotas (how many subscriptions per topic you can create), but those are fairly high. And if you get to that point, it's possible to reduce number of subscriptions when a given processor handles multiple message types by having more complex SQL filtering rules.
My Azure based SaaS system publishes events and I have customers who wish to subscribe to them - webhooks seem undeniably the right architecture (And I'm currently a happy consumer of webhooks). I've found lots of great documentation and case studies on best practices (e.g. http://resthooks.org) however I've not managed to find an existing architecture, framework, project, sample or solution that implements the best practices.
I could build my own solution however I don't want to reinvent the wheel. I was expecting to find an existing framework (e.g. on Github) created by people much smarter than I but haven't had any success.
I currently use a number of Azure services (such as Service Bus, Cosmos, Table Storage) internally and consume using Azure Functions but what I don't have is an architecture for allowing my customers to subscribe to these events.
Specifically I'm looking for best practices and code samples on how to manage potentially millions of subscribers (who are external customers) and the approach to distribute the webhooks out to each of them.
I already understand how to publish and consume webhooks where I am an individual subscriber and there are already some great samples available - https://github.com/aspnet/AspLabs/tree/master/src/WebHooks
Can anyone point me in the right direction? (Preferably to a .NET / C# based solution)
Not sure if this is the 'right' direction but here are my current thoughts on a solution.
We are currently using CosmosDb and are leveraging the change feed to trigger an Azure function execution. The code within the function does a specific task for all tenants in our system. This code will be changed to simply send a new event to the Event Grid topic. A 'in-house' subscription will then be added that will handle what the function code is currently doing today.
We will then follow the subscription management guidance Zapier offers. In a nutshell it is to expose the capability to our customers to subscribe to the events that we publish via a few endpoints. In addition to standard CRUD stuff when a tenant adds/removes a subscription the code will leverage the Event Grid Management SDK to add/remove subscriptions to the appropriate topics within Event Grid (samples here). The subscriptions that get added will have filters set to ensure each tenant only receives their own events.
There are limitations to the number of subscriptions and topics with in Azure (details here). These limitations are acceptable in our case but is something you might need to look into more if you need to reach 1mm subscribers.
Here is how I visualize it:
Not 100% we'll build this but if we do I'll post back here any gotchas we uncover.
Cheers!
I have potentially tens or even hudnreds of thousands of clients who need to communicate with a central server.
Communication is in the form of:
receive command from central servers (process it on the client)
respond with a status to central servers
I would like to avoid having the client machines talk to any intermediate web/API servers and instead, I want them to go directly to ASB
No client can see each other's messages. Whatsoever. I understand I can use SAS tokens to provide temporary privilges to clients and renew them on a scheduled basis and that's great and works within my architecture. However, I'm not sure if I can utilize the same ASB topic and have each client have their own topic inside?
Is ASB even the right technology for this? Can I somehow maintain only two queues/service-bus subscriptions for this (request/reply) or must I create an individual queue for each indivdiual client?
TIA
It’s difficult to tell without knowing more about the nature of the messages you are sending – e.g. how many are being sent. However, with this many clients you may be coming up against the quotas which are shown here:
https://msdn.microsoft.com/en-us/library/azure/ee732538.aspx
The salient limitations are:
100 concurrent connections per entity (i.e. topic, queue or subscription)
2,000 subscriptions per topic
10,000 queues or topics per service bus namespace
100,000 correlation filters per topic
It’s worth taking a look at the Azure scalability scenarios described in the second half of this document:
https://msdn.microsoft.com/en-us/library/azure/hh528527.aspx
It may be possible to get the broadcast side of things going by getting clients to connect with correlation filters though I have not tried using them on this scale.
If you want to have lots of senders going to a single queue then you should consider using the Service Bus REST API for message sending.
Otherwise, I'm afraid you may want to consider a proxy...