My question is in related Azure logic apps / Event Grid.
I have set up an Event Grid Domain with multiple topics. I would like the create a logic app that will subscribe to topics within my Event Grid Domain. The flow trigger for the event grid domain appears to be missing the Topic field to allow it only to subscribe to particular topics. am I missing something?
How can I have a logic app subscribe specifically to one topic within an event grid domain in Azure?
Looks like this is a limitation with the UI. If you were to use custom values for both the Resource Type and Resource Name like below, I can confirm it works as expected
Also, feel free to raise a feature request on UserVoice to support this in the UI.
Related
I want to monitor for newly created Azure Subscriptions. Ideally, I would like to subscribe to events at the management group level and ingest them on event grid. What would be the best way to accomplish this?
On my research there is no native way for event grid to subscribe to events on the Management Group or Tenant Level.
As you mentioned, there is no native way to add an event grid at management group level.
However, you can use a polling technique (Azure TimerTrigger Function) with a REST GET request to obtain a list of all subscriptions, comparing to previously state and publishing its diference to the AEG custom topic.
I am using the built in events tab under the IoT Hub to route data to an event topic. The subscriber is an Azure function (It has it's system identity enabled).
Data is routing to the event topic but the delivery is failing when trying to reach the Azure Function.
Diagnostic logs do not show anything useful:
Event Grid Subscription:
Is there a way to use this functionality even though the Azure Function has it's System Identity turned on?
I noticed this question had a similar problem and solved it using RBAC but I can't ask a question as I don't have enough reputation to comment. Event Grid Subscription event going to delivery failure
Any ideas?
I have provisioned Azure Event Grid domain in the portal and created a couple of Domain Topics under the hood. Also, a service bus queue has been configured as an endpoint to receive those events.
Now, I just want to send a few sample events to the Azure Event Grid Domain. How do I go about it? is there any tool available to send the events out of the box?
Any suggestion is much appreciated.
Basically, any REST Client tool can be used for posting an event to the AEG domain or you can try my Azure Event Grid Tester:
Basically what I understood from few Azure topics is as below:
Azure Event Hub - where data is received initially and converted into events
Service Bus- acting as a queue
Azure Event Grid - where events converted in hub are transferred here.
so the connection is like below:
Hub -> Service Bus -> Event Grid -> Pub Sub -> Storage
I understood this concept. My problem is I want data to be pushed from the event grid to GCP (subscription / topics). My question are:
How can I establish this using PUSH method?
What do I need to develop exactly?
How can I push things from grid to pubsub/subscriptions?
I found this link where data is getting published into Event Grid but I want to push data from the event grid to gcp. Can anybody explain me where am I going wrong or what exactly should I start with. I am new to this and its very confusing so I just need little bit of guidance over here.
I have below doubts:
Is there any direct subscriber option available with event grid listener? I mean can I directly link my google storage account with this listener so, whenever there is an event triggered it will be directly pushed to my GCP account(I don't have Azure account with me right now since access issue is in progress so I can't see it that's why I am asking here)
Suppose I have 20 columns in my data but I want only 16 columns to be pushed in GCP so is there any customization possible while sending data from event grid/event hub to pub/sub
If I write custom connectors code as per the links provided in the below answers then how can I run it? I mean where I can deploy those scripts on the cloud so that they will be triggered automatically whenever an event is triggered?
Can I implement webhooks in this scenario? (as an alternative to connectors), If yes then how can I do it and on which side do I need to create it?
Also, I read some articles and I came to know from a few guys that they experienced data loss in this entire process. So, what's the possibility over here and how can it be avoided
Can anybody explain me where am I going wrong or what exactly should I start with.
It's right here:
so the connection is like below:
Hub -> Service Bus -> Event Grid -> Pub Sub -> Storage
Although this might be the case, it sounds very much as if you're looking at one (very) specific scenario where data flows in this exact way.
Azure Event Hub, Azure Service Bus and Azure Event Grid can work together, but can also be used completely separate from each other.
Event Grid
The purpose of Event Grid is to enable Reactive programming. Use this when you want to react to (status) changes.
Event Hubs
Event Hubs facilitate a big data pipeline. Use this when you need telemetry and distributed data streaming.
Service Bus
The purpose of Service bus is to enable High-value enterprise messaging. Use this when you want to do something like Order processing and financial transactions.
In some cases, you use the services side by side to fulfill distinct roles. For example, an ecommerce site can use Service Bus to process the order, Event Hubs to capture site telemetry, and Event Grid to respond to events like an item was shipped.
In other cases, you link them together to form an event and data pipeline. You use Event Grid to respond to events in the other services. For an example of using Event Grid with Event Hubs to migrate data to a data warehouse, see Stream big data into a data warehouse.
Taken from the very interesting and important documentation article Choose between Azure messaging services - Event Grid, Event Hubs, and Service Bus
EDIT
My problem is I want data to be pushed from event grid to GCP (subscription / topics). So how can I establish this using PUSH method??
Possibly the simplest solution is to have an Event Grid Event trigger a webhook (which might run an Azure Function or a Google Cloud Function) which in turn puts the event/message on the GCP Topic.
Publishing messages is quite well documented. There are examples on how to do so with a REST call, command-line, C#, Go, JAVA, NodeJS, PHP, Python and Ruby.
EDIT 2
What you need to do is create an Event Grid Subscription to listen to and handle Event Grid Events.
Here's an example screenshot on how to listen for events for a specific Storage Account and call a WebHook whenever such an event occurs:
Pay attention to the "Endpoint Details": that's where you can specify to, for instance, call a webhook every time an event is triggered.
The easiest way to transfer the EventHub generated events would probably be to create an EventHub event receiver in Node.js (which you mentioned in your comments) as described here, which receives events and publishes them to Cloud Pub/Sub directly, as described in the Cloud Pub/Sub publisher documentation for Node.js.
What is the best practices around Azure EventGrid topics and Events?
Is it a bad idea to publish different event-types to the same Azure EventGrid topic?
e.g. multiple different domain events
When do we need different topics?
A single shared Topic for an entire application?
One Topic per Aggregate Root type?
One Topic per Event Type?
Any suggestions would be welcome as there are no clear answers
Part2.
What if I want to integrate with various Azure Logic Apps?
if multiple logic apps react to the same topic, would they steal messages from each-other?
Do each logic app create some invisible subscription?
No, it is not a bad idea to publish different event-types to the same Azure EventGrid topic: if the events are related to the same resource, it does make sense to publish them to the same EventGrid topic. Taking the example of a HR application, you can have EmployeeAdded and EmployeeRemoved events published on the same "employee" topic.
On the question on when different topics would be needed, I think it depends on a few factors such as how you are modelling the resources in your application, the events of interest on those resources, security model around which parts of the system should be able to publish to the topic / creation of event subscriptions on the topic. Ideally, all types of events for the same resource type (such as the "employee" resource type in the example above) can be on the same topic. When your system has more types of resources, you may want to create separate topics for each of them. Also, the desired security model has to be taken into account as well (e.g. let's say you want to restrict access to who can receive certain types of events).
Regarding the question about logic apps, if you create multiple logic apps that handle events from the same topic, each of them creates an event subscription on the same topic, and Event Grid would deliver the events on that topic to each of the event subscriptions. So, each logic app would receive the same event individually, and can process it independent of the other apps.
The Azure Event Grid (AEG) is not a generic Pub/Sub model. This model is based on the source of the events, where each event source (topicType) is handling own interest.
Subscriber subscribes an interest to the event source (topic) using a subscription. Note, that the AEG allows to subscribe only one topic in the subscription. There is a limit 500 subscriptions per topic.
In other words, if there is a multiple interest for event source (topic) by the same subscriber, this model requires to create a multiple subscriptions (one per topic) per subscriber. The filtering of the interest is possible only within the same topic.
The source of events in the AEG can be extended by custom topics (maximum 100 per Azure Subscription).
Based on the above, I do recommend for custom topics use the same model like is built-in for azure event sources (topicTypes) with a multiple eventTypes, which can be simplified a continuously deployment over environments.
To the Part2: The AEG doesn't use an 'invisible' subscription as a part of the integration. Every subscription created to the topic is visible and accessable, for example using a REST API
Update:
Azure Event Grid recently release (in preview - version 2018-09-15-preview) a new feature which can be helped for your solution using an Event Domain and Domain Topics, more details here.
You can use an updated tool Azure Event Grid Tester for testing all new preview release features, which they are not yet implemented in the Azure portal UI.