What is the best practices around Azure EventGrid topics and Events?
Is it a bad idea to publish different event-types to the same Azure EventGrid topic?
e.g. multiple different domain events
When do we need different topics?
A single shared Topic for an entire application?
One Topic per Aggregate Root type?
One Topic per Event Type?
Any suggestions would be welcome as there are no clear answers
Part2.
What if I want to integrate with various Azure Logic Apps?
if multiple logic apps react to the same topic, would they steal messages from each-other?
Do each logic app create some invisible subscription?
No, it is not a bad idea to publish different event-types to the same Azure EventGrid topic: if the events are related to the same resource, it does make sense to publish them to the same EventGrid topic. Taking the example of a HR application, you can have EmployeeAdded and EmployeeRemoved events published on the same "employee" topic.
On the question on when different topics would be needed, I think it depends on a few factors such as how you are modelling the resources in your application, the events of interest on those resources, security model around which parts of the system should be able to publish to the topic / creation of event subscriptions on the topic. Ideally, all types of events for the same resource type (such as the "employee" resource type in the example above) can be on the same topic. When your system has more types of resources, you may want to create separate topics for each of them. Also, the desired security model has to be taken into account as well (e.g. let's say you want to restrict access to who can receive certain types of events).
Regarding the question about logic apps, if you create multiple logic apps that handle events from the same topic, each of them creates an event subscription on the same topic, and Event Grid would deliver the events on that topic to each of the event subscriptions. So, each logic app would receive the same event individually, and can process it independent of the other apps.
The Azure Event Grid (AEG) is not a generic Pub/Sub model. This model is based on the source of the events, where each event source (topicType) is handling own interest.
Subscriber subscribes an interest to the event source (topic) using a subscription. Note, that the AEG allows to subscribe only one topic in the subscription. There is a limit 500 subscriptions per topic.
In other words, if there is a multiple interest for event source (topic) by the same subscriber, this model requires to create a multiple subscriptions (one per topic) per subscriber. The filtering of the interest is possible only within the same topic.
The source of events in the AEG can be extended by custom topics (maximum 100 per Azure Subscription).
Based on the above, I do recommend for custom topics use the same model like is built-in for azure event sources (topicTypes) with a multiple eventTypes, which can be simplified a continuously deployment over environments.
To the Part2: The AEG doesn't use an 'invisible' subscription as a part of the integration. Every subscription created to the topic is visible and accessable, for example using a REST API
Update:
Azure Event Grid recently release (in preview - version 2018-09-15-preview) a new feature which can be helped for your solution using an Event Domain and Domain Topics, more details here.
You can use an updated tool Azure Event Grid Tester for testing all new preview release features, which they are not yet implemented in the Azure portal UI.
Related
I want to monitor for newly created Azure Subscriptions. Ideally, I would like to subscribe to events at the management group level and ingest them on event grid. What would be the best way to accomplish this?
On my research there is no native way for event grid to subscribe to events on the Management Group or Tenant Level.
As you mentioned, there is no native way to add an event grid at management group level.
However, you can use a polling technique (Azure TimerTrigger Function) with a REST GET request to obtain a list of all subscriptions, comparing to previously state and publishing its diference to the AEG custom topic.
My question is in related Azure logic apps / Event Grid.
I have set up an Event Grid Domain with multiple topics. I would like the create a logic app that will subscribe to topics within my Event Grid Domain. The flow trigger for the event grid domain appears to be missing the Topic field to allow it only to subscribe to particular topics. am I missing something?
How can I have a logic app subscribe specifically to one topic within an event grid domain in Azure?
Looks like this is a limitation with the UI. If you were to use custom values for both the Resource Type and Resource Name like below, I can confirm it works as expected
Also, feel free to raise a feature request on UserVoice to support this in the UI.
My Azure based SaaS system publishes events and I have customers who wish to subscribe to them - webhooks seem undeniably the right architecture (And I'm currently a happy consumer of webhooks). I've found lots of great documentation and case studies on best practices (e.g. http://resthooks.org) however I've not managed to find an existing architecture, framework, project, sample or solution that implements the best practices.
I could build my own solution however I don't want to reinvent the wheel. I was expecting to find an existing framework (e.g. on Github) created by people much smarter than I but haven't had any success.
I currently use a number of Azure services (such as Service Bus, Cosmos, Table Storage) internally and consume using Azure Functions but what I don't have is an architecture for allowing my customers to subscribe to these events.
Specifically I'm looking for best practices and code samples on how to manage potentially millions of subscribers (who are external customers) and the approach to distribute the webhooks out to each of them.
I already understand how to publish and consume webhooks where I am an individual subscriber and there are already some great samples available - https://github.com/aspnet/AspLabs/tree/master/src/WebHooks
Can anyone point me in the right direction? (Preferably to a .NET / C# based solution)
Not sure if this is the 'right' direction but here are my current thoughts on a solution.
We are currently using CosmosDb and are leveraging the change feed to trigger an Azure function execution. The code within the function does a specific task for all tenants in our system. This code will be changed to simply send a new event to the Event Grid topic. A 'in-house' subscription will then be added that will handle what the function code is currently doing today.
We will then follow the subscription management guidance Zapier offers. In a nutshell it is to expose the capability to our customers to subscribe to the events that we publish via a few endpoints. In addition to standard CRUD stuff when a tenant adds/removes a subscription the code will leverage the Event Grid Management SDK to add/remove subscriptions to the appropriate topics within Event Grid (samples here). The subscriptions that get added will have filters set to ensure each tenant only receives their own events.
There are limitations to the number of subscriptions and topics with in Azure (details here). These limitations are acceptable in our case but is something you might need to look into more if you need to reach 1mm subscribers.
Here is how I visualize it:
Not 100% we'll build this but if we do I'll post back here any gotchas we uncover.
Cheers!
I'm looking into using Azure Service Bus for publishing/delivering messages between different services in our application, and are trying to find a decent design for the topic and subscriptions within a Service Bus namespace.
The intent of the system is for service-a to publish a message with type service-a.test-event to a bus, and have any service listening on that event type get the message delivered. It will be a fairly low volume
I'm a bit torn on which of the following designs to use:
The Service Bus namespace has one topic events where all messages from all services are delivered. Any service subscribing to events from any other service create a subscription in this topic using filters to get the message types they want - one subscription per message type (eg. service-b-service-a-test-event).
The Service Bus namespace has one topic per publisher (eg. events-service-a). Any subscribing service to events from this service create a subscription in the topic using filters to get the message types they want - one subscription per message type (eg. service-b-test-event).
Service Bus seems to have a limit of 2000 subscriptions per topic, which as far as I can tell will be sufficient for us. If I had suspicions otherwise, option #2 would probably be the best choice (as I can have 10.000 topics per namespace). None of the other limitations of Service Bus, as far as I can tell, really impacts which of these options I should go for.
One additional requirement is that I want to have a service subscribing to any event from any service for recording reasons. If I went for option #1, that'd be very simple to implement. For option #2 however, that service needs to somehow make sure it has a subscription within any event topic in the namespace - and reconfigure itself once new topics are added and old topics are removed. That's outside the scope of this question, but a requirement for the design none the less.
When deciding about a topology, take into consideration things that matter to you the most. One of the principles of pub/sub is to decouple between publishers and subscribers. With topology #2 (topic per publisher) this principle is violates since every subscriber would have to know the name of the publisher. And if publisher of an event would change, all your subscribers would have to get that knowledge in order to re-subscribe. Topology #1 solves that problem by abstracting publishers away by using a single topic.
Also, the 2nd requirement you had (to audit all of the events) would be much easier to implement with this topology as you only have to maintain a single wiretap subscription on that topic.
You're correct about limitation of 2000 subscriptions per topic. Saying that, 2000 subscriptions is quite a lot. Once your system get to that number of subscribers, you'd probably want to review your architecture/topology anyway.
These two topologies are sound extremely similar to the NServiceBus on Azure Service Bus as a transport topologies. You could have a look to see some of the other ideas you could use for your topology, including benefits of one over another.
Disclaimer: I am working on NServiceBus Azure Service Bus transport. You don't have to use NServiceBus, though a lot of questions like the one you're posting here become a non-issue when you do.
I am building an application that needs to be able to create events, publish them, and have multiple consumers. The tutorials I have found so far suggest that Azure Topics are the right thing to use for this (multiple publishers and multiple subscribers), but I noticed an option in my Azure portal for EventHub and it seems like a highly scalable solution that may be a newer implementation of Pub/Sub for Azure. I have been searching for documentation comparing the two and haven't really found anything. Can someone explain why I would choose one of these solutions over the other.
The scenario I have is many clients in a Multi-Tenant application may create events at any time, those events need to be published to "n" subscribers for consumption. The subscribers need to be able to change without any change to the application (i.e. subscribers should be able to subscribe themselves to events without modifying publisher code).
Thanks for the help.
Event Hubs have below advantage as compared with Azure ServiceBus:
Its client-side cursor: Developers can use a client-side pointer, or
offset, to retrieve messages from a specific point in the event
stream
Partitioned consumer support: Throughput Unit and inbound messages
can be targeted at a specific partition through a user-defined
partition key value.
Significant time-based retention options