Is there a webhook that notifies that a model publish operation has been completed (succeeded/failed)?
I found a webhook that notifies that the operation has started. However, it is not useful for me because I need to copy the model to another directory after the operation is completed and the model is updated.
As per the documentation of the model.publish event, the payload.state property in the webhook callback represents:
Reason why the notification was triggered. Possible values are PUBLISHING_PENDING and PUBLISHING_IN_PROGRESS.
Note: There is no PUBLISHING_COMPLETE state. Use the dm.version.added event to receieve notifications when publishing is finished.
So, consider using the dm.version.added event.
Related
Context:
In Azure function with EventHubTrigger, I save data mapped from handled event to database (through the Entity framework). This action performs synchronously
Trigger a new event about successful data insertion using event hub producer. This action is async
Handle that triggered event at some other place
I guess it might happen that something fails during saving data, so I am wondering how to prevent inconsistency and secure that event is not sent if it should not.
As far as I know Azure Event Hub has no outbox pattern implemented yet, so I guess I would need to mimic it somehow.
I am also thinking about alternative and a bit smelly solution to make this publish event method synchronous in step 2 (even if nature of the event-driven is to be async) and to add an addition check between step 1 and step 2 - to make sure that everything is saved in db. Only if that condition is fulfilled, event is going to be triggered (step 3).
Any advice?
There's nothing in the SDK that would manage distributed transactions on your behalf. The simplest approach would likely be having a column in your database that allows you to mark when the event was published, and then have your function flow:
Write to the database with the "event published" flag unset; on failure abort.
Publish the event; on failure abort. (the data stays in written)
Write to the database to set the "event published" flag.
You'd need a second Function running on a timer that could scan your database for rows older than XX minutes ago that still need an event, which then do steps 2 and 3 from your initial flow. In failure scenarios, you will have some potential latency between the data being written and the event published or may see duplicate events. (Event Hubs has an at least once guarantee, so you'll need to be able to handle duplicates regardless.)
I am using Twilio Taskrouter for making reservations for a video call. What I want is to notify the user that all workers are currently busy, with a message displayed for the user on the browser. So I wanted to know how can one catch the task.canceled event in the browser so that I can notify the user about the task getting canceled due to the task timeout. I have seen the JS SDK for Twilio but I only saw events for Workers, Workspace and Task Queues and couldn't find any such events for Tasks, so is there anyway I can listen to the task's events on the browser side?
I think there is a way to get the assignment status of the task, more details can be found here: TaskRouter Task Resource. On this page look for assignmentStatus resource property. The current status of the Task's assignment. Can be: pending, reserved, assigned, canceled, wrapping, or completed
let me know what you think.
I've started thinking through a prototype architecture for a system I want to build based on Azure Functions and Event Grid.
What I would like to achieve is to have a single point of entry (Function) which a variety of external vendors will send Webhook (GET) HTTP requests to. The purpose of the Function is to add some metadata to the payload, and publish the package (metadata + original payload from vendor) to an Event Grid. The Event Grid will then trigger another Function, whose purpose is to respond to the original Webhook HTTP request with e.g. a status 204 HTTP code.
The diagram below is a simplified version of the architecture, the Event Grid will of course publish events also to other Functions, but for the sake of simplicity…
The challenge I'm facing at the moment is that the context of the original Webhook HTTP request from external vendor is lost after the first Function is triggered. Trying to send the context as part of the event payload to Event Grid feels like an anti-pattern, and regardless I cannot get it working (the .done() function is lost somewhere in the event). Trying to just use context.res = {} and context.done() in the last Function won't respond to the vendor's original HTTP request.
Any ideas here? Is the whole architecture just one big anti-pattern -- will it even work? Or do I have to immediately send the HTTP response in the first Function triggered by the vendor's request?
Thank you!
You are mixing two difference patterns such as a message-driven and event-driven.
The Azure Event Grid is a distributed Pub/Sub eventing Push model, where the subscriber subscribing an interest on the source in the loosely decoupled manner.
In your scenario, you want to use an eventing model within the message exchange request-response pattern in the sync manner. The request message exchange context can not flow via the Pub/Sub eventing model and back to the anonymous endpoint such as actually a point for response message.
However, there are a several options how to "logical" integrate these two different patterns, the following is showing some of them:
using a request - replyTo message exchange pattern, such as a full duplex communication, one for request and the other one for replyTo.
using a request - response message exchange pattern with a polling state. Basically, your first function will wait for a subscriber state and then return back to the caller. In the distributed internet architecture, we can use an azure lease blob storage for sharing a state between the sync part and async eventing part.
In your scenario, the first AF will create this lease blob, then firing an event to the AEG and then it will periodically polling the state in the lease blob for end of aggregation process (multiple subscribers, etc.).
Also, for this kind of pattern, you can use Azure Durable Function to simplify an integration to the event-driven AEG model.
The following screen snippet shows a sequence diagram using an Azure Lease Blob for sharing a "Request State" in the distributed model. Note, that this pseudo sync/async pattern is suitable for cases when the Request-Response is processing within a short time less than 60 seconds.
For more details about using a Lease Blob within the Azure Function, see my answer here.
Pardon if my terminology is a little off; I'm new to this.
I have created an Azure Event Grid subscription which triggers an event whenever I upload a file to blob storage. I have an Azure Function which responds to this event. I've got this all working finally, but I have a slough of left-over messages from previous (bad) uploads that are failing periodically (as viewed from the Logs window in the Azure portal for the associated Azure Function). It's as if they're stored in a queue somewhere and retried periodically, though I'm not sure if that's how it works.
In any case, what I want to be able to do is purge any in-transit or queued events, but I don't know where to find them to do this. As far as I know they're just floating about in the ether.
How can I purge these events so they don't keep triggering my Azure Function at random times?
Event Grid will automatically retry delivery of the message if anything other then a 200 or 202 (OK/Accepted) is returned when a delivery attempt is made. By default it will try again for 24 hours and it uses an exponential backup that adds additional time in between each request until it gives up. What you're seeing is that default process running. (You can also configure a dead letter handling with a storage account so the undelivered messages get stored somewhere if it eventually fails).
What you are likely looking for is the Retry Policy you can create when creating a subscription. Pretty sure you can set the number of maximum delivery attempts to 1 so it won't retry (and without dead letter support turned on the message would essentially be dropped). More details on this can be found at https://learn.microsoft.com/en-us/azure/event-grid/manage-event-delivery#set-retry-policy
I'm not aware of any way to "dequeue" already submitted messages without that retry policy already in place - you may have to delete and recreate the subscription to that event grid topic.
In addition to #JoshCarlisle's answer and more clear to the Event Grid message delivery and retry document:
The dead-lettering enables a special case in the retry policy logic.
In the case of the dead-lettering is turn-on and subscriber failed with a HttpStatusCode.BadRequest, the Event Grid will stop a retrying process and the event is sent to the dead-letter endpoint. This error code indicates, that the delivery will never succeed.
the following code snippet shows some properties in the dead-letter message:
"deadLetterReason": "UndeliverableDueToHttpBadRequest",
"deliveryAttempts": 1,
"lastDeliveryOutcome": "BadRequest",
"lastHttpStatusCode": 400,
the following list shows some of the status codes where the Event Grid will continue in the retrying process:
HttpStatusCode.ServiceUnavailable
HttpStatusCode.InternalServerError
HttpStatusCode.RequestTimeout
HttpStatusCode.NotFound
HttpStatusCode.Conflict
HttpStatusCode.Forbidden
HttpStatusCode.Unauthorized
HttpStatusCode.NotImplemented
HttpStatusCode.Gone
Example of the some dead-letter properties, when the HttpStatusCode.RequestTimeout:
"deadLetterReason":"MaxDeliveryAttemptsExceeded",
"deliveryAttempts":3,
"lastDeliveryOutcome":"TimedOut",
"lastHttpStatusCode":408,
Now, you can see the above two difference cases described in the deadLetterReason property such as "UndeliverableDueToHttpBadRequest" vs "MaxDeliveryAttemptsExceeded"
One more thing:
When the dead-lettering is turn-on, the Event Grid will NOT deliver a dead-letter message to the dead-letter endpoint immediately, but after ~300 seconds. I hope this is a bug and it will be fix soon.
Practically, if the subscriber failed for instance HttpStatusCode.BadRequest, we can not wait for 5 minutes the event from the container storage, it must be an event driven close to the real-time.
I have a Logic App that sends an event to a specified Event Hub using the Send Event action.
It seems that regardless of whether or not the event is accepted by the specified Event Hub, the Logic App continues on regardless. Unlike the Azure Functions action, there appears to be no automatically generated StatusCode property available for Send Event action.
Is it possible to check the response from Event Hubs so that I may determain whether or not to halt execution?
Update
After a completed run, it seems that there is a status code returned by Event Hubs, although unusually it seems to be 200 where as typically when sending events it's 201.
However, when editing the Logig App, there doesn't seem to be any way of accessing that status code in order to check the success/failure of the send event action.
You should be able to use #outputs('Send_event')?['statusCode'] to access the status code.