My Azure based SaaS system publishes events and I have customers who wish to subscribe to them - webhooks seem undeniably the right architecture (And I'm currently a happy consumer of webhooks). I've found lots of great documentation and case studies on best practices (e.g. http://resthooks.org) however I've not managed to find an existing architecture, framework, project, sample or solution that implements the best practices.
I could build my own solution however I don't want to reinvent the wheel. I was expecting to find an existing framework (e.g. on Github) created by people much smarter than I but haven't had any success.
I currently use a number of Azure services (such as Service Bus, Cosmos, Table Storage) internally and consume using Azure Functions but what I don't have is an architecture for allowing my customers to subscribe to these events.
Specifically I'm looking for best practices and code samples on how to manage potentially millions of subscribers (who are external customers) and the approach to distribute the webhooks out to each of them.
I already understand how to publish and consume webhooks where I am an individual subscriber and there are already some great samples available - https://github.com/aspnet/AspLabs/tree/master/src/WebHooks
Can anyone point me in the right direction? (Preferably to a .NET / C# based solution)
Not sure if this is the 'right' direction but here are my current thoughts on a solution.
We are currently using CosmosDb and are leveraging the change feed to trigger an Azure function execution. The code within the function does a specific task for all tenants in our system. This code will be changed to simply send a new event to the Event Grid topic. A 'in-house' subscription will then be added that will handle what the function code is currently doing today.
We will then follow the subscription management guidance Zapier offers. In a nutshell it is to expose the capability to our customers to subscribe to the events that we publish via a few endpoints. In addition to standard CRUD stuff when a tenant adds/removes a subscription the code will leverage the Event Grid Management SDK to add/remove subscriptions to the appropriate topics within Event Grid (samples here). The subscriptions that get added will have filters set to ensure each tenant only receives their own events.
There are limitations to the number of subscriptions and topics with in Azure (details here). These limitations are acceptable in our case but is something you might need to look into more if you need to reach 1mm subscribers.
Here is how I visualize it:
Not 100% we'll build this but if we do I'll post back here any gotchas we uncover.
Cheers!
Related
Doing some learning around Azure Notification hubs and trying to understand one of the basic concepts but not able to find any documentation that states it explictly.
Is an azure notification hb able to send a push notification directly or does it have to have a third party set up in its settings (ie APNS, GCM.FCM, WNS, MPNS,ADM, Baidu)?
Azure Notification Hubs (ANH) must be configured with credentials for one or more of the push notification services (PNS) that you've enumerated.
Essentially, each of these providers have a reliable mechanism for delivering a specific notification to a specific device, some of them also provide mechanisms for delivering the same notification to many devices. But those solutions are usually limited to just their platform.
ANH seeks to provide a wrapper for each PNS that provides routing and scale to target many devices with a single request, so that you don't need to write that code separately for each platform. Using more advanced techniques, like templates, can even allow you to abstract away platform differences altogether.
However, ANH is not an alternative to the PNSs you've listed. I hope that helps.
What is the best practices around Azure EventGrid topics and Events?
Is it a bad idea to publish different event-types to the same Azure EventGrid topic?
e.g. multiple different domain events
When do we need different topics?
A single shared Topic for an entire application?
One Topic per Aggregate Root type?
One Topic per Event Type?
Any suggestions would be welcome as there are no clear answers
Part2.
What if I want to integrate with various Azure Logic Apps?
if multiple logic apps react to the same topic, would they steal messages from each-other?
Do each logic app create some invisible subscription?
No, it is not a bad idea to publish different event-types to the same Azure EventGrid topic: if the events are related to the same resource, it does make sense to publish them to the same EventGrid topic. Taking the example of a HR application, you can have EmployeeAdded and EmployeeRemoved events published on the same "employee" topic.
On the question on when different topics would be needed, I think it depends on a few factors such as how you are modelling the resources in your application, the events of interest on those resources, security model around which parts of the system should be able to publish to the topic / creation of event subscriptions on the topic. Ideally, all types of events for the same resource type (such as the "employee" resource type in the example above) can be on the same topic. When your system has more types of resources, you may want to create separate topics for each of them. Also, the desired security model has to be taken into account as well (e.g. let's say you want to restrict access to who can receive certain types of events).
Regarding the question about logic apps, if you create multiple logic apps that handle events from the same topic, each of them creates an event subscription on the same topic, and Event Grid would deliver the events on that topic to each of the event subscriptions. So, each logic app would receive the same event individually, and can process it independent of the other apps.
The Azure Event Grid (AEG) is not a generic Pub/Sub model. This model is based on the source of the events, where each event source (topicType) is handling own interest.
Subscriber subscribes an interest to the event source (topic) using a subscription. Note, that the AEG allows to subscribe only one topic in the subscription. There is a limit 500 subscriptions per topic.
In other words, if there is a multiple interest for event source (topic) by the same subscriber, this model requires to create a multiple subscriptions (one per topic) per subscriber. The filtering of the interest is possible only within the same topic.
The source of events in the AEG can be extended by custom topics (maximum 100 per Azure Subscription).
Based on the above, I do recommend for custom topics use the same model like is built-in for azure event sources (topicTypes) with a multiple eventTypes, which can be simplified a continuously deployment over environments.
To the Part2: The AEG doesn't use an 'invisible' subscription as a part of the integration. Every subscription created to the topic is visible and accessable, for example using a REST API
Update:
Azure Event Grid recently release (in preview - version 2018-09-15-preview) a new feature which can be helped for your solution using an Event Domain and Domain Topics, more details here.
You can use an updated tool Azure Event Grid Tester for testing all new preview release features, which they are not yet implemented in the Azure portal UI.
I understand that Microservices is about independent loosely coupled services. I have read https://en.wikipedia.org/wiki/Microservices.
When it comes to Azure, I understand there are many components like Azure Service Fabric, AKS and also have the option of deploying containers within Azure VMs using Docker or any other containerization tools. However, since Microservices is about developing atmoic individually scalable services, can this also be achieved by deploying each service as an Azure Web API APP within an App Service Plan and configure Auto-Scale based on Performance metrics (though each API APP may not be individually scalable, they can still be individually manageable in terms of deployment, configuration etc)?
Can someone please suggest if this thought process is correct?
Microservices aren't a platform or technology so if you can make small independently deployable services then they are microservices. Sure - some tech helps but it depends on your situation.
If you only need a few services you probably don't need anything complex. Make sure services are well modeled, own their own data and ideally have a good monitoring and deployment pipeline setup. Design for service failure where possible.
Do you need to scale each part independently? Ideally, you should be able to but do services have very different requirements? You could have many small App service plans but that comes at cost of unused resources so split when you need to.
This question and of course the answers are going to be opinion based, but generally when thinking in terms of micros services, think not in terms as things like loads of API's and VM's etc. Instead think in terms of. When i upload an image, its needs to be resized, and the table updated to give a url for the thumb. or when XXX record is updated in database, Run XXX in order to create a report, or update Azure search. and that each service, just knows how to do a single thing only. I.E Resize an image.
Now one could say. I have a system, A repo library, and some functions library. When an image is posted, I upload, then call this, and that etc.
With Micor services. You would instead just add the image to a queue. Create an azure function that has a queue trigger. that would resize and save both the large and the thumb to storage. this would then either update the database, or in true micro service, it would add a queue to store the new info, another function would watch that queue and insert into the database.
You can use the DB queue from anything. You can use the Blob queue from anything. Your main API, does not care how images are handled. You can change your functions one day to maybe save to dropbox, instead of azure blob. All really easy, with no re-build of the API, because the API does not care.
A good example I use it for is email and SMS. My systems dont know how to send an email, or an SMS. They only know how to add to a queue. My microservices. SendEmail and SendSMS do know how to do it, and I can change how and who i send that content with, really easy. I can tomorrow change from Twilio to send grid, without ever telling the API that i've done it.
On a more complex thing. I have approval, at the moment that approval sends an email or SMS to either user or admin, and that can change over time. So I have an SMS server, Email Service and and approvalService. when approval happens, it just adds a config to the queue, The rest is done by a logic app, that knows to send an email to XXX and an SMS to XXX and then update database. My api, is just a post, that creates a queue.
Basically what I am saying here is to get started, maybe porting an existing app. Start with the workflow stuff, like send an email, resize an image, create a report, create a PDF, email 50 subscribers etc. and take all that code out and put into there own micro service that just knows how to do one thing. Then when you grow with confidence, create a workflow from all of these services with Logic Apps, let azure take care of the rest, thats what they want to do.
I am building an application that needs to be able to create events, publish them, and have multiple consumers. The tutorials I have found so far suggest that Azure Topics are the right thing to use for this (multiple publishers and multiple subscribers), but I noticed an option in my Azure portal for EventHub and it seems like a highly scalable solution that may be a newer implementation of Pub/Sub for Azure. I have been searching for documentation comparing the two and haven't really found anything. Can someone explain why I would choose one of these solutions over the other.
The scenario I have is many clients in a Multi-Tenant application may create events at any time, those events need to be published to "n" subscribers for consumption. The subscribers need to be able to change without any change to the application (i.e. subscribers should be able to subscribe themselves to events without modifying publisher code).
Thanks for the help.
Event Hubs have below advantage as compared with Azure ServiceBus:
Its client-side cursor: Developers can use a client-side pointer, or
offset, to retrieve messages from a specific point in the event
stream
Partitioned consumer support: Throughput Unit and inbound messages
can be targeted at a specific partition through a user-defined
partition key value.
Significant time-based retention options
My problem is a simple one. I'm working on a project that uses the service bus from Microsoft Azure to send messages asynchronously between different modules on different virtual machines. And a lot of messages are sent through this bus, so we want to have some indicators about it's performance and other usage information. Why? Because when everything is working, users are happy. When the system is slow, we want to show the user some interesting graphs, statistics, meters and other gadgets to give them an indication if there's a problem within Azure or with something else. And to do this, I need data about the usage of the Azure service bus.
So, which Azure API's are available to display what kind of (diagnostic) information about the service bus?
(Users should have no access to Azure itself! They should just see some performance data to re-assure them Azure is working fine. Or else I could look at it and discover some problem with it, fix it and then make users happy again.)
To elaborate what I'm looking for, the Azure website has some nice chart when you click the Monitor of the Azure bus showing you overviews of the number of incoming messages, the number of errors and their types, size information and the number of succesful operations, all based on a specified period. It would be nice if I could receive this data within my project.
The entity metrics API will give you the exact data the portal is using:
http://msdn.microsoft.com/en-us/library/windowsazure/dn163589.aspx
Here's a Subscribe! episode I recorded with Rajat on the topic http://channel9.msdn.com/Blogs/Subscribe/Service-Bus-Namespace-Management-and-Analytics
I've spent quite some time to make the entity metrics API work, so I decided to share the results.
Here is a full C# code example of how to consume those API:
github repository.
It's a small library which wraps the HTTP request into strongly typed .NET classes. You can also grab it from NuGet.
Finally, here is my blog post with the walkthrough.