I have a REST service hosted in Azure Web app. I registered a webhook on Azure Event Grid by pointing to REST service endpoint. I have followed below link and added endpoint validation with Event Grid events in REST service. I am able to register webhook successfully.
https://learn.microsoft.com/en-us/azure/event-grid/webhook-event-delivery
But I am facing issue(Not able subscribe webhook) if I configure any delivery properties in Event Grid like Authorization or content-type headers as shown below. Please refer below attachment for error details(Shown right side of pic) as well.
Event Grid subscription with webhook delivery properties failure
Could someone please help me on this.
Thanks in advance,
Ashok
First, we need to check how event delivery is authenticated with event handler.
Also, make sure that validation call is successful with event grid, Event grid supports two ways of validations.
Synchronous Validation
Asynchronous Validation
Subscription validation event example as below:
[
{
"id": "2d1781af-3a4c-4d7c-bd0c-e34b19da4e66",
"topic": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"subject": "",
"data": {
"validationCode": "512d38b6-c7b8-40c8-89fe-f46f9e9622b6",
"validationUrl": "https://rp-eastus2.eventgrid.azure.net:553/eventsubscriptions/myeventsub/validate?id=0000000000-0000-0000-0000-00000000000000&t=2021-09-01T20:30:54.4538837Z&apiVersion=2018-05-01-preview&token=1A1A1A1A"
},
"eventType": "Microsoft.EventGrid.SubscriptionValidationEvent",
"eventTime": "2021-00-01T22:12:19.4556811Z",
"metadataVersion": "1",
"dataVersion": "1"
}
]
Refer to Webhook event delivery from MS Docs
Also check this for troubleshooting validation issues
Related
I am trying to set "advanced filter" of BlobEventsTrigger programmatically.
They are reset at each deployment. I need only 3 and having 7 of them are causing the job to start twice. It is super annoying to delete them manually after each deployment.
I have tried to add a field "advancedFilters" or "blobType" to the trigger json file without success.
"typeProperties": {
"blobPathBeginsWith": "/bingofile/blobs/",
"blobPathEndsWith": "/_SUCCESS",
"ignoreEmptyBlobs": false,
"scope": "/subscriptions/bingofilesup/resourceGroups/bingofilesup/providers/Microsoft.Storage/storageAccounts/bingofilesup",
"events": [
"Microsoft.Storage.BlobCreated"
]
I've also tried az eventgrid system-topic event-subscription update but this library does not work when it comes to updating advanced filter. It asks an endpoint (which is normally a facultative argument) , and when provided the existing data factory endpoint, it fails reaching it.
I have checked the documentation about this endpoint and it is said to be the webhook endpoint .
Endpoint where EventGrid should deliver events matching this event
subscription. For webhook endpoint type, this should be the
corresponding webhook URL. For other endpoint types, this should be
the Azure resource identifier of the endpoint. It is expected that the
destination endpoint to be already created and available for use
before executing any Event Grid command.
But it does not work .
Deployment failed. Correlation ID:
95e4fab5-163e-48ab-8cb2-b23432516e53. Webhook validation handshake
failed for [webwook end point provided in the topic]. Http POST
request failed with response code Unknown. For troublehooting, visit
https://aka.ms/esvalidation.
Any observation or suggestion would be great, thanks in advance !
According to my test, the endpoint https://pmeastasia.svc.datafactory.azure.com:4443/triggerevent/BlobEventsTrigger/<> is juts a base URL. When the events are sent to data factory or update subscription, azure will generate an endpoint with the base URL to do auth. So if you want to update the subscription with other tools, I think you need to use fildder to catch the request to get the whole endpoint at first.
I'm currently working on a project where I need to consume a third-party Webhook into my application. The issue is, this third-party service doesn't allow me to pick which events to push to my application through this Webhook, so I'll have to respond to all of them even if they are mostly useless, and if I need to expand on my application or divide it into microservices, I would be streaming the same data to all services even if they have different needs. Also, I would be facing data loss in case of issue with my application server.
The solution would be to use an Event Broker, which would collect all events from the Webhook, respond to the provider with a 200 OK status code, push the event to a specific topic which will be stored until all the concerned subscribed services receive that data.
I'm looking for a fully managed service in Azure, so far I've come across Azure Event Grid, Azure Event Hub and Azure Service Bus.
I wanted to know about the feasibility of this scenario, and if I can stream directly from a Webhook to one of these Azure services.
No, afaik you cannot stream directly into those service. You will need to setup something that accepts the webhook and sends it to one of those listened service.
However, what I would do is create an http triggered azure function. You should be able to configure the webhook to post to the function.
Once you got your function setup you can create some logic there to route the message to the proper channels based on its content. Now that could be an application of yours, a Service Bus Queue, Azure Storage Queue or Event Grid. I would not recommend an Event Hub as it is less suited for this specific purpose.
In the case of consuming a third-party events without guarantee their in order processing and the webhook payload is an array, the Azure Event Grid can be consumed your third-party webhook directly.
The following screen snippet shows this example:
The above integration is based on the Custom Topic Endpoint with a CustomInputSchema.
The Custom Topic Endpoint sends a response back to the webhook with the following HTTP response code:
Success 200 OK
Event data has incorrect format 400 Bad Request
Invalid access key 401 Unauthorized
Incorrect endpoint 404 Not Found
Array or event exceeds size limits 413 Payload Too Large
The AEG model distributing an event in the loosely decoupled Pub/Sub manner with a reliable and retry delivery to the subscriber based on its subscriptions. The AEG subscription represents a logical connectivity between the source of the interest and consumer. It is a set of metadata describing by consumer what, where and how.
Basically there are two delivery patterns such as:
Push-PushAck where the event is pushed to the subscriber handler for its business processing and the result is back to the AEG e.g. Web Hook (Azure Fuction) and Hybrid Connection.
Push-PullAck where the event is reliable delivered to the subscriber and the delivery response is returned back to the AEG. The event must be pulled out from this delivery target for its business post-processing, e.g. Service Bus Queue, Storage Queue and Event Hubs.
UPDATE:
For creating a custom topic endpoint with a CustomInputSchema can be used for example the REST API
The following is an example of the payload PUT request:
{
"location": "westus",
"properties": {
"inputSchema": "CustomEventSchema",
"inputSchemaMapping": {
"properties": {
"id": {
"sourceField": null
},
"topic": {
"sourceField": null
},
"eventTime": {
"sourceField": null
},
"eventType": {
"sourceField": null,
"defaultValue": "notification"
},
"subject": {
"sourceField": null,
"defaultValue": "/webhook/events"
},
"dataVersion": {
"sourceField": null,
"defaultValue": "1.0"
}
},
"inputSchemaMappingType": "Json"
}
}
}
The above CustomInputSchema enables to use any input event schema for this Custom Topic endpoint. That's very nice feature of the AEG. The "bad news" is that the events must be in the array included also a single event. I hope, that the AEG team will make an improvement for custom and domain topics when the single event can be published also as a JObject (no inside of the array).
For bypassing an input event schema via AEG eventing model, the subscriber (consumer of the source event interest) must declared a DeliverySchema = CustomInputSchema. The default output event schema is the EventGridSchema.
The following examples show an event message published to the custom topic with above CustomInputSchema and delivered to the subscribers using a CustomInptutSchema and the other one with the EventGridSchema.
Fire Event to the Custom Topic Endpoint (array of event(s)):
[
{
"abcd": 12345
}
]
Subscriber with DeliverySchema = CustomInputSchema:
{
"abcd": 12345
}
Subscriber with DeliverySchema = EventGridSchema (default schema):
{
"id": "f92a5dbf-d206-4e61-ac1e-7498c543039a",
"eventTime": "2019-07-14T07:19:00.3969337Z",
"eventType": "notification",
"dataVersion": "1.0",
"metadataVersion": "1",
"topic": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rk2012/providers/Microsoft.EventGrid/topics/testerTopic8",
"subject": "/webhook/events",
"data":
{
"abcd": 12345
}
}
Note, that the events can be filtered (selected) for each subscriber based on its subscription properties in the loosely decoupled Pub/Sub manner. In other words, subscriber can subscribed to the AEG eventing model any time via its subscription where is declared a specific source interest, e.g. topic, subject, event data, etc., mechanism of the delivery, retrying, deadlettering, etc.
Created new azure event grid domain with cloud event schema using portal.
Created new web hook endpoint using azure function that can receive both subscription validation event as well as event notifications.
Created new azure event grid topic for the above domain (as part of following subscription) using portal.
Created new azure event grid subscription with cloud event schema with the above web hook endpoint.
When the subscription is Created, the endpoint was invoked by the grid infrastructure with subscription validaiton event to verify web hook endpoint.
To my surprise, the validation event structure (shown below) seemed to conform to native event grid schema and not cloud event schema:
[{
"id": "6309ef83-117f-47aa-a07c-50f6e71a8ca5",
"topic": "/subscriptions/13ad1203-e6d5-4076-bf2b-73465865f9f0/resourceGroups/xxxx-sandbox-rg/providers/Microsoft.EventGrid/domains/eg-xxx-test-cloud-domain/topics/eg-xxx-test-cloud-topic",
"subject": "",
"data": {
"validationCode": "391889BB-FCC3-4269-A2BD-0918B5BAB0AE",
"validationUrl": "https://rp-westus.eventgrid.azure.net/eventsubscriptions/xxxx-subscription-3/validate?id=391889BB-FCC3-4269-A2BD-0918B5BAB0AE&t=2019-01-30T15:45:37.0521594Z&apiVersion=2018-09-15-preview&[Hidden Credential]"
},
"eventType": "Microsoft.EventGrid.SubscriptionValidationEvent",
"eventTime": "2019-01-30T15:45:37.0521594Z",
"metadataVersion": "1",
"dataVersion": "2"
}]
I expected following subscription validation event that conforms to cloud event schema (based on the 0.1 version of cloud event schema at https://learn.microsoft.com/en-us/azure/event-grid/cloudevents-schema#cloudevent-schema):
{
"eventID" : "6309ef83-117f-47aa-a07c-50f6e71a8ca5",
"source" : "/subscriptions/13ad1203-e6d5-4076-bf2b-73465865f9f0/resourceGroups/xxxx-sandbox-rg/providers/Microsoft.EventGrid/domains/eg-xxx-test-cloud-domain/topics/eg-xxx-test-cloud-topic",
"data": {
"validationCode": "391889BB-FCC3-4269-A2BD-0918B5BAB0AE",
"validationUrl": "https://rp-westus.eventgrid.azure.net/eventsubscriptions/xxxx-subscription-3/validate?id=391889BB-FCC3-4269-A2BD-0918B5BAB0AE&t=2019-01-30T15:45:37.0521594Z&apiVersion=2018-09-15-preview&[Hidden Credential]"
},
"eventType" : "Microsoft.EventGrid.SubscriptionValidationEvent",
"eventTime" : "2019-01-30T15:45:37.0521594Z",
"cloudEventsVersion" : "0.1",
"eventTypeVersion" : "2",
}
What am I missing thing?
Basically, the webhook subscriber is handling the following two groups of the events. The specific event type is stored in the http header 'aeg-event-type'.
Internal events of the Event Grid model such as the eventTypes SubscriptionValidation and SubscriptionDeletion. The schema for these event types are always the same as a default schema such as an EventGridSchema. In other words, it's not depended on the EventDeliverySchema. IMO, having the default schema for internal events is making a strong event types specially when we have a CustomInputSchema.
Interest source events (topics) are events defined by input schema and presently the Event Grid model supports 3 types such as EventGridSchema (default), CloudEventSchema and CustomInputSchema.
The AEG supports the following schema input/output mappings:
EventGridSchema to delivery schemas EventGridSchema and CloudEventSchema
CloudEventSchema to delivery schema only CloudSchemaSchema
CustomInputSchema to delivery schema EventGridSchema and CloudEventSchema and CustomInputSchema
The event type in the header is: aeg-event-type=Notification and the schema is based on subscribed EventDeliverySchema (see the following mappings).
Based on the above, for your scenario you should have a separate strong type objects for Internal events (default schema is EventGridSchema) and for Notification events based on the subscribed EventDeliverySchema.
The following is an example of the http headers:
aeg-subscription-name=EVENTGRIDSCHEMA
aeg-delivery-count=0
aeg-data-version=
aeg-metadata-version=0
aeg-event-type=SubscriptionValidation
Note, there is only a subscription name to figure out which an EventDeliverySchema has been subscribed. It will be nice to have an additional aeg header for example: aeg-subscription-labels to pass some subscription metadata to the subscriber handler.
As a workaround, we can pass to the subscriber webhook handler some values via the url query parameters, for instance: &eds=CustomInputSchema
This is a known issue / expected behavior in the Azure Event Grid implementation of Cloud Event V0.1 spec. At the time Cloud Events v0.1 spec was implemented in Azure Event Grid, there was no validation handshake / abuse protection model defined in the Cloud Events standard and hence Event Grid's existing validation handshake model/schema was used for Cloud Event subscribers as well.
I have implemented calendar syncing with our internal CRM using the Outlook REST (v2) API. I am using Push Notifications (Webhooks) to push Office 365 calendar changes back into our CRM.
I am seeing a large number of "Missed" notifications. eg
{
"value": [
{
"#odata.type": "#Microsoft.OutlookServices.Notification",
"Id": null,
"SubscriptionId": "xxxxxxx",
"SubscriptionExpirationDateTime": "2018-11-21T11:12:10Z",
"SequenceNumber": 1,
"ChangeType": "Missed"
}
]
}
This is the basic flow of events:
• User creates a Calendar event in our CRM
• This event is created in the Users Outlook calendar
• A subscription event (https://outlook.office.com/api/v2.0/users/{User}/subscriptions) is created for this users calendar to “monitor” any Updated or Deleted actions for these Calendar event. Any events that have not been created by the CRM are ignored and are not monitored. I am also not monitoring any Created events.
• When the user Updates or Deletes a monitored event outlook will call our webhook
• When the webhook receives the event it will sync the changed event
However, the "Missed" notifications are generally more than the "Updated" notifications. I am seeing (on average) 150 Missed events and only 50 Updated Events.
Why do we have so many missed events?
How can I debug this to identify why we are getting Missed events?
Thanks.
I'm testing out Azure Event Grid with a Azure Relay, Hybrid Connection handler. It's not working for me.
I can see messages being published to Azure Event Grid. So far so good.
I've set up a subscription to the Event Grid topic and I've configured it to send events to a handler specified as an Azure Relay with a Hybrid Connection endpoint.
Looking at the metrics for the subscription with the Hybrid Connection handler, I see the following telemetri:
Matched Events
Delivery Failed
Expired Events
… but I see no Delivery Succeeded event???
Also, the Hybrid Connection listener (just a simple Console App) connected to the Azure Relay receives nothing. I've tested the listener by sending some test messages directly to the Relay and that works fine.
The logical conclusion, is that the events published to the Event Grid are not being delivered probably to the Relay Hybrid Connection handler. But why? There are not that many parameters, so I not sure what I'm doing wrong. It seems rather straight forward to configure this.
I'm a the point where I'm beginning to believe that the Event Grid / Hybrid Connection scenario is currently not working. It is in preview after all, so that could explain it.
I know that there is not much to go on here, but I was hoping that others might have some experience with this?
The issue still remains. The issues seems to be related to the formating of the json being passed from the Event Grid Subscription to the Hybrid Connect.
Update
I finally had some time to look more carefully into this.
I set up the Event Grid Tester and every time the Event Grid recieves a message I can see this error in the log:
HybridConnection: Message processing failed - Unable to cast object of type 'Newtonsoft.Json.Linq.JValue' to type 'Newtonsoft.Json.Linq.JObject'.
I'm still not able to fix this, as I do not control the message. The message is generated by Azure Logic App and send to the Event Grid using the Event Grid connector. The Event Grid Connector in Azure Logic Apps is in preview, so that might explain the challenges I'm seeing.
I've successfully tested Event Grid subscriber locally using Azure Relay and documented it here. It is based on a sample Microsoft is providing and implemented using WCF variant of client, not the .NET Standard version.
you can simulate an Event Grid message to the HybridConnection url using a http POST request:
POST https://{myNamespace}.servicebus.windows.net/{myHybridConnectionName}
headers:
content-type: application/json
x-ms-version: 2015-07-08
Aeg-Event-Type: Notification
Authorization: SharedAccessSignature sr=xxx&sig=xxxx&se=11111111&skn=xxxxx
body:
{
"id": "123456",
"eventTime": "2018-07-22T13:09:07.5164877Z",
"eventType": "recordInserted",
"dataVersion": "1.0",
"metadataVersion": "1",
"subject": "/myapp/vehicles/motorcycles",
"data": {
"make": "Ducati",
"model": "Monster"
}
}
Note, that the sasToken for authorization header can be copied from the Azure Event Grid Tester log panel, when the HybridConnection has been opened.
the other option is to generate it by the following code:
using Microsoft.Azure.Relay;
// ...
var tokenProvider = TokenProvider.CreateSharedAccessSignatureTokenProvider("hybridconnectionPolicyName", "hybridconnectionPrimaryKey");
var token = tokenProvider.GetTokenAsync("https://{myNamespace}.servicebus.windows.net/{myHybridConnectionName}", TimeSpan.FromDays(10)).Result.TokenString;