Azure Event Grid to Azure Relay, Hybrid Connection - azure

I'm testing out Azure Event Grid with a Azure Relay, Hybrid Connection handler. It's not working for me.
I can see messages being published to Azure Event Grid. So far so good.
I've set up a subscription to the Event Grid topic and I've configured it to send events to a handler specified as an Azure Relay with a Hybrid Connection endpoint.
Looking at the metrics for the subscription with the Hybrid Connection handler, I see the following telemetri:
Matched Events
Delivery Failed
Expired Events
… but I see no Delivery Succeeded event???
Also, the Hybrid Connection listener (just a simple Console App) connected to the Azure Relay receives nothing. I've tested the listener by sending some test messages directly to the Relay and that works fine.
The logical conclusion, is that the events published to the Event Grid are not being delivered probably to the Relay Hybrid Connection handler. But why? There are not that many parameters, so I not sure what I'm doing wrong. It seems rather straight forward to configure this.
I'm a the point where I'm beginning to believe that the Event Grid / Hybrid Connection scenario is currently not working. It is in preview after all, so that could explain it.
I know that there is not much to go on here, but I was hoping that others might have some experience with this?
The issue still remains. The issues seems to be related to the formating of the json being passed from the Event Grid Subscription to the Hybrid Connect.
Update
I finally had some time to look more carefully into this.
I set up the Event Grid Tester and every time the Event Grid recieves a message I can see this error in the log:
HybridConnection: Message processing failed - Unable to cast object of type 'Newtonsoft.Json.Linq.JValue' to type 'Newtonsoft.Json.Linq.JObject'.
I'm still not able to fix this, as I do not control the message. The message is generated by Azure Logic App and send to the Event Grid using the Event Grid connector. The Event Grid Connector in Azure Logic Apps is in preview, so that might explain the challenges I'm seeing.

I've successfully tested Event Grid subscriber locally using Azure Relay and documented it here. It is based on a sample Microsoft is providing and implemented using WCF variant of client, not the .NET Standard version.

you can simulate an Event Grid message to the HybridConnection url using a http POST request:
POST https://{myNamespace}.servicebus.windows.net/{myHybridConnectionName}
headers:
content-type: application/json
x-ms-version: 2015-07-08
Aeg-Event-Type: Notification
Authorization: SharedAccessSignature sr=xxx&sig=xxxx&se=11111111&skn=xxxxx
body:
{
"id": "123456",
"eventTime": "2018-07-22T13:09:07.5164877Z",
"eventType": "recordInserted",
"dataVersion": "1.0",
"metadataVersion": "1",
"subject": "/myapp/vehicles/motorcycles",
"data": {
"make": "Ducati",
"model": "Monster"
}
}
Note, that the sasToken for authorization header can be copied from the Azure Event Grid Tester log panel, when the HybridConnection has been opened.
the other option is to generate it by the following code:
using Microsoft.Azure.Relay;
// ...
var tokenProvider = TokenProvider.CreateSharedAccessSignatureTokenProvider("hybridconnectionPolicyName", "hybridconnectionPrimaryKey");
var token = tokenProvider.GetTokenAsync("https://{myNamespace}.servicebus.windows.net/{myHybridConnectionName}", TimeSpan.FromDays(10)).Result.TokenString;

Related

Not able to register Event Grid subscription with webhook delivery properties in Azure

I have a REST service hosted in Azure Web app. I registered a webhook on Azure Event Grid by pointing to REST service endpoint. I have followed below link and added endpoint validation with Event Grid events in REST service. I am able to register webhook successfully.
https://learn.microsoft.com/en-us/azure/event-grid/webhook-event-delivery
But I am facing issue(Not able subscribe webhook) if I configure any delivery properties in Event Grid like Authorization or content-type headers as shown below. Please refer below attachment for error details(Shown right side of pic) as well.
Event Grid subscription with webhook delivery properties failure
Could someone please help me on this.
Thanks in advance,
Ashok
First, we need to check how event delivery is authenticated with event handler.
Also, make sure that validation call is successful with event grid, Event grid supports two ways of validations.
Synchronous Validation
Asynchronous Validation
Subscription validation event example as below:
[
{
"id": "2d1781af-3a4c-4d7c-bd0c-e34b19da4e66",
"topic": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"subject": "",
"data": {
"validationCode": "512d38b6-c7b8-40c8-89fe-f46f9e9622b6",
"validationUrl": "https://rp-eastus2.eventgrid.azure.net:553/eventsubscriptions/myeventsub/validate?id=0000000000-0000-0000-0000-00000000000000&t=2021-09-01T20:30:54.4538837Z&apiVersion=2018-05-01-preview&token=1A1A1A1A"
},
"eventType": "Microsoft.EventGrid.SubscriptionValidationEvent",
"eventTime": "2021-00-01T22:12:19.4556811Z",
"metadataVersion": "1",
"dataVersion": "1"
}
]
Refer to Webhook event delivery from MS Docs
Also check this for troubleshooting validation issues

azure data factory BlobEventsTrigger : set "advanced filter" programatically

I am trying to set "advanced filter" of BlobEventsTrigger programmatically.
They are reset at each deployment. I need only 3 and having 7 of them are causing the job to start twice. It is super annoying to delete them manually after each deployment.
I have tried to add a field "advancedFilters" or "blobType" to the trigger json file without success.
"typeProperties": {
"blobPathBeginsWith": "/bingofile/blobs/",
"blobPathEndsWith": "/_SUCCESS",
"ignoreEmptyBlobs": false,
"scope": "/subscriptions/bingofilesup/resourceGroups/bingofilesup/providers/Microsoft.Storage/storageAccounts/bingofilesup",
"events": [
"Microsoft.Storage.BlobCreated"
]
I've also tried az eventgrid system-topic event-subscription update but this library does not work when it comes to updating advanced filter. It asks an endpoint (which is normally a facultative argument) , and when provided the existing data factory endpoint, it fails reaching it.
I have checked the documentation about this endpoint and it is said to be the webhook endpoint .
Endpoint where EventGrid should deliver events matching this event
subscription. For webhook endpoint type, this should be the
corresponding webhook URL. For other endpoint types, this should be
the Azure resource identifier of the endpoint. It is expected that the
destination endpoint to be already created and available for use
before executing any Event Grid command.
But it does not work .
Deployment failed. Correlation ID:
95e4fab5-163e-48ab-8cb2-b23432516e53. Webhook validation handshake
failed for [webwook end point provided in the topic]. Http POST
request failed with response code Unknown. For troublehooting, visit
https://aka.ms/esvalidation.
Any observation or suggestion would be great, thanks in advance !
According to my test, the endpoint https://pmeastasia.svc.datafactory.azure.com:4443/triggerevent/BlobEventsTrigger/<> is juts a base URL. When the events are sent to data factory or update subscription, azure will generate an endpoint with the base URL to do auth. So if you want to update the subscription with other tools, I think you need to use fildder to catch the request to get the whole endpoint at first.

Is there a way to get the status of an Event Grid trigger for azure function (Complete /Pending or Running)

by Httptrigger Azure function, if you send a POST request you receive something like this as a response:
{
"id": "66ee5d08196874aeb99c9e62ddc7b190",
"statusQueryGetUri": "https://asynchttpfunction.azurewebsites.net/runtime/webhooks/durabletask/instances/66ee5d08196945aeb44c9e62ddc7b190?taskHub=Orchestration&connection=Storage&code=FSVfJyGODSeKHPO0cM8Po9e1jMT7MghVMGuJqTaGTN56E1RUHnlVJg==",
"sendEventPostUri": "https://asynchttpfunction.azurewebsites.net/runtime/webhooks/durabletask/instances/66ee5d08196945aeb44c9e62ddc7b190/raiseEvent/{eventName}?taskHub=Orchestration&connection=Storage&code=FSVfJyGODSeKHPO0cM8Po9e1jMT7MghVMGuJqTaGTN56E1RUHnlVJg==",
"terminatePostUri": "https://asynchttpfunction.azurewebsites.net/runtime/webhooks/durabletask/instances/66ee5d08196945aeb44c9e62ddc7b190/terminate?reason={text}&taskHub=Orchestration&connection=Storage&code=FSVfJyGODSeKHPO0cM8Po9e1jMT7MghVMGuJqTaGTN56E1RUHnlVJg==",
"rewindPostUri": "https://asynchttpfunction.azurewebsites.net/runtime/webhooks/durabletask/instances/66ee5d08196945aeb44c9e62ddc7b190/rewind?reason={text}&taskHub=Orchestration&connection=Storage&code=FSVfJyGODSeKHPO0cM8Po9e1jMT7MghVMGuJqTaGTN56E1RUHnlVJg==",
"purgeHistoryDeleteUri": "https://asynchttpfunction.azurewebsites.net/runtime/webhooks/durabletask/instances/66ee5d08196945aeb44c9e62ddc7b190?taskHub=Orchestration&connection=Storage&code=FSVfJyGODSeKHPO0cM8Po9e1jMT7MghVMGuJqTaGTN56E1RUHnlVJg=="
}
The statusQueryGetUri provides information of the long running orchestration instance. If you follow this link you will receive a suitable runtimeStatus that describes the status of the orchestration instance along with some other useful information.here
My question is now:
actually we don't send a POST request to an Event grid Azure function trigger, Is there any way to get the status of the Azure function? Complete or is still running?
The Azure Event Grid is an eventing Pub/Sub model where the interest of source is distributed to the subscribed event handler endpoint or resource in the reliable manner with a retry policy and dead-lettering option. The AEG is waiting for delivery response processing max. 60 seconds.
There is no built-in the features what you are asking in the AEG, however you can use the REST API for metrics of the specific subscription to obtain its counters value:
MatchedEventCount,
DeliveryAttemptFailCount,
DeliverySuccessCount,
DroppedEventCount,
DeadLetteredCount
The following GET is an example for getting a subscription metrics:
https://management.azure.com/subscriptions/mysubId/resourceGroups/mygroup/providers/Microsoft.EventGrid/topics/mytester/providers/Microsoft.EventGrid/eventSubscriptions/mysubscription/providers/Microsoft.Insights/metrics?api-version=2018-01-01&interval=PT5M&metricnames=MatchedEventCount,DeliveryAttemptFailCount,DeliverySuccessCount,DroppedEventCount,DeadLetteredCount
Note, that the authorization header with a bearer token is required for this call.
More details about the monitoring an event message delivery can be found here.

Streaming data consumed from third-party WebHooks to application using an event broker in Azure

I'm currently working on a project where I need to consume a third-party Webhook into my application. The issue is, this third-party service doesn't allow me to pick which events to push to my application through this Webhook, so I'll have to respond to all of them even if they are mostly useless, and if I need to expand on my application or divide it into microservices, I would be streaming the same data to all services even if they have different needs. Also, I would be facing data loss in case of issue with my application server.
The solution would be to use an Event Broker, which would collect all events from the Webhook, respond to the provider with a 200 OK status code, push the event to a specific topic which will be stored until all the concerned subscribed services receive that data.
I'm looking for a fully managed service in Azure, so far I've come across Azure Event Grid, Azure Event Hub and Azure Service Bus.
I wanted to know about the feasibility of this scenario, and if I can stream directly from a Webhook to one of these Azure services.
No, afaik you cannot stream directly into those service. You will need to setup something that accepts the webhook and sends it to one of those listened service.
However, what I would do is create an http triggered azure function. You should be able to configure the webhook to post to the function.
Once you got your function setup you can create some logic there to route the message to the proper channels based on its content. Now that could be an application of yours, a Service Bus Queue, Azure Storage Queue or Event Grid. I would not recommend an Event Hub as it is less suited for this specific purpose.
In the case of consuming a third-party events without guarantee their in order processing and the webhook payload is an array, the Azure Event Grid can be consumed your third-party webhook directly.
The following screen snippet shows this example:
The above integration is based on the Custom Topic Endpoint with a CustomInputSchema.
The Custom Topic Endpoint sends a response back to the webhook with the following HTTP response code:
Success 200 OK
Event data has incorrect format 400 Bad Request
Invalid access key 401 Unauthorized
Incorrect endpoint 404 Not Found
Array or event exceeds size limits 413 Payload Too Large
The AEG model distributing an event in the loosely decoupled Pub/Sub manner with a reliable and retry delivery to the subscriber based on its subscriptions. The AEG subscription represents a logical connectivity between the source of the interest and consumer. It is a set of metadata describing by consumer what, where and how.
Basically there are two delivery patterns such as:
Push-PushAck where the event is pushed to the subscriber handler for its business processing and the result is back to the AEG e.g. Web Hook (Azure Fuction) and Hybrid Connection.
Push-PullAck where the event is reliable delivered to the subscriber and the delivery response is returned back to the AEG. The event must be pulled out from this delivery target for its business post-processing, e.g. Service Bus Queue, Storage Queue and Event Hubs.
UPDATE:
For creating a custom topic endpoint with a CustomInputSchema can be used for example the REST API
The following is an example of the payload PUT request:
{
"location": "westus",
"properties": {
"inputSchema": "CustomEventSchema",
"inputSchemaMapping": {
"properties": {
"id": {
"sourceField": null
},
"topic": {
"sourceField": null
},
"eventTime": {
"sourceField": null
},
"eventType": {
"sourceField": null,
"defaultValue": "notification"
},
"subject": {
"sourceField": null,
"defaultValue": "/webhook/events"
},
"dataVersion": {
"sourceField": null,
"defaultValue": "1.0"
}
},
"inputSchemaMappingType": "Json"
}
}
}
The above CustomInputSchema enables to use any input event schema for this Custom Topic endpoint. That's very nice feature of the AEG. The "bad news" is that the events must be in the array included also a single event. I hope, that the AEG team will make an improvement for custom and domain topics when the single event can be published also as a JObject (no inside of the array).
For bypassing an input event schema via AEG eventing model, the subscriber (consumer of the source event interest) must declared a DeliverySchema = CustomInputSchema. The default output event schema is the EventGridSchema.
The following examples show an event message published to the custom topic with above CustomInputSchema and delivered to the subscribers using a CustomInptutSchema and the other one with the EventGridSchema.
Fire Event to the Custom Topic Endpoint (array of event(s)):
[
{
"abcd": 12345
}
]
Subscriber with DeliverySchema = CustomInputSchema:
{
"abcd": 12345
}
Subscriber with DeliverySchema = EventGridSchema (default schema):
{
"id": "f92a5dbf-d206-4e61-ac1e-7498c543039a",
"eventTime": "2019-07-14T07:19:00.3969337Z",
"eventType": "notification",
"dataVersion": "1.0",
"metadataVersion": "1",
"topic": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rk2012/providers/Microsoft.EventGrid/topics/testerTopic8",
"subject": "/webhook/events",
"data":
{
"abcd": 12345
}
}
Note, that the events can be filtered (selected) for each subscriber based on its subscription properties in the loosely decoupled Pub/Sub manner. In other words, subscriber can subscribed to the AEG eventing model any time via its subscription where is declared a specific source interest, e.g. topic, subject, event data, etc., mechanism of the delivery, retrying, deadlettering, etc.

Building SOAP Listener with Azure Functions

I am using Azure Functions to build some integrations between various systems. I new requirement is to respond to record updates in Salesforce. Some quick research yielded what seems like a good solution from the Salesforce side. Use Outbound messaging which can send SOAP requests on record modifications.
How to create Salesforce application that will send record to external web service when record created/changed(https://salesforce.stackexchange.com/questions/73425/how-to-create-salesforce-application-that-will-send-record-to-external-web-servi)
The challenge now is to be able create a SOAP listener in Azure Function. I have created basic HTTP Triggers for my other listeners. Is there anything "built-in" to Azure Functions that would allow me to easily consume the incoming SOAP request?
Salesforce has the basics for a solution based on a more traditional web service and an ASMX file but I am not sure if or how that can be applied in Azure Functions. (https://developer.salesforce.com/docs/atlas.en-us.api.meta/api/sforce_api_om_outboundmessaging_listener.htm)
That notification is just a SOAP request that is made over HTTP, so really not too different than a regular HTTP trigger request.
Although you could just treat that as a plain request and parse the contents yourself, Azure Functions does expose the great WebHook support we get from ASP.NET WebHooks, and luckily, there is a Salesforce receiver that significantly simplifies this task.
DISCLAIMER: It's worth noting that although the receiver is technically enabled in Azure Functions, there's no official support for it yet, so you won't find a lot of documentation and help will be limited to what you get on SO and Forums. Official support to this and other receivers will hopefully be coming soon, which means documentation, templates and UI support will become available.
To get started, you need the following:
Create a new function, selecting the GenericWebHook - CSharp template (this works for node as well, but I'll focus on C# here.
Follow the steps outlined on the ASP.NET WebHooks integration with Salesforce post in order to create the outbound message. Here you want to use the Function Url given to you by the portal WITHOUT THE CODE QUERY STRING (having the code there wouldn't hurt, but the receiver does not use that information).
IMPORTANT: Get your Salesforce Organization ID, which will be used for authentication and is located under Administer > Company Profile > Company Information > Salesforce.com Organization ID and back in the Azure Functions portal, open the Keys panel, delete your default function key (not host key) and create a new key, named default (this name is important) using the Organization ID value you got from Salesforce.
Go to Integrate
On the integration page, select Advanced Editor on the upper right (as mentioned, there's no official support, so the UI does not expose this. We're putting our explorer hats on and venturing into a more advanced workflow here :) )
Change the webHookType property value to sfsoap and save the configuration. Your function.json config should look like the following:
function.json:
{
"bindings": [
{
"type": "httpTrigger",
"direction": "in",
"webHookType": "sfsoap",
"name": "req"
},
{
"type": "http",
"direction": "out",
"name": "res"
}
],
"disabled": false
}
Switch to the Develop tab. We're ready to write our code.
This is where the ASP.NET WebHooks receiver shines! It will parse the notification for you, exposing strong typed objects you can work with. All you need to do is modify the method/function signature you get withe template to use the SalesforceNotifications type, making sure you're referencing the required assembly (Microsoft.AspNet.WebHooks.Receivers.Salesforce, which is made available to you, so no need for package reference) and namespace reference (Microsoft.AspNet.WebHooks).
Here is a full sample of a function that will receive the request and log the Organization ID, Action ID, grab the first notification and log all of its properties:
#r "Microsoft.AspNet.WebHooks.Receivers.Salesforce"
using Microsoft.AspNet.WebHooks;
public static void Run(SalesforceNotifications req, TraceWriter log)
{
log.Info($"Salesforce webhook was triggered!");
log.Info(req.OrganizationId);
log.Info(req.ActionId);
var notification = req.Notifications.First();
foreach (var p in notification.Keys)
{
log.Info($"{p} = {notification[p]}");
}
}
This process will be a lot smoother when the receiver is officially supported, but even with the added steps, this still beats having to parse the SOAP messages yourself :)
I hope this helps!

Resources