Azure EventGrid moving the Events to Dead-letter queue without retry.
But I am seeing the following reason:
"deadLetterReason": "UndeliverableDueToClientError"
"deliveryAttempts": 1
"lastDeliveryOutcome": "Probation"
"lastHttpStatusCode": 0
Please help me to understand the dead letter reason attributes.
deliveryAttempts = 1
Why EventGrid is not retying when Destination is put in probation? Can we do any config settings to ask the EventGrid to retry before moving the Events to Dead-letter queue.
lastHttpStatusCode = 0 What's this means?
The concept of Event Grid dead letter explained as it moves event to the dead letter queue when the destination like storage account is unable to send the Save events that cannot be delivered to storage event in a certain time period or exceeded the number of limits.
As shown in the below image In Create Event Subscription select Additional features and Enable dead lettering to catch any events that are not delivered successfully to the subscribers . This will send any undelivered events to the specified storage account
lastHttpStatusCode = 0
*As mentioned in the Document The HTTP response code of 0 indicates that the request was cancelled. It can happen because of timeout, abortion on the request. It indicates the request failed to execute within a specified time. Here 0 indicates the no status code was returned by destination.
You can also refer the same issue Status code 0 of Http Request.
delivery attempts = 1
Here delivery attempts 1 indicates the event grid has attempted to deliver the events only once.
Undeliverable due To Client error
It indicates mainly client-side network error in destination.
You can set Dead letter retry policy by following MS Document.
As shown in below image set configure retry polices ok and the max delivery attempts must be 1 to 30 and even you can set EVENT SUBSCRIPTION EXPIRATION TIME
Dead letter MS document for reference
Related
We have a problem with the Azure Portal for a specific tenant. It shows the following error message as if there were more than 12k requests to the portal. We make like 15k requests but to the Cosmos DB. Something new we did was to create an Azure Container Registry and 5 containers in the Resource Group that have FFmpeg running and streaming. Any clues about how to find the origin of this error?
Number of tenant-level requests exceeded the limit of '12000' for time
interval '01:00:00'. Please try again after '300' seconds. (Code:
TenantRequestsThrottled)
You receive the HTTP status code 429 Too many requests. The response includes a Retry-After value, which specifies the number of seconds your application should wait (or sleep) before sending the next request. If you send a request before the retry value has elapsed, your request isn't processed and a new retry value is returned.
Please follow the link for solution: URL
I am using Azure CDN for Akamai and noticing that it return 400 Error code with following message - "We can only accept 100 paths for purging concurrently. Please try again in a few minutes."
I have Logic App which is triggered when a new asset is uploaded in storage account. This error happens only for 50% of changes.
How do I handle this ? The retry policy on logic app does not trigger because it's 400 status code.
As per documentation it says 50 concurrent request at profile level. but my workload sends only 10 request and REST api process 5 request successfully.
by Httptrigger Azure function, if you send a POST request you receive something like this as a response:
{
"id": "66ee5d08196874aeb99c9e62ddc7b190",
"statusQueryGetUri": "https://asynchttpfunction.azurewebsites.net/runtime/webhooks/durabletask/instances/66ee5d08196945aeb44c9e62ddc7b190?taskHub=Orchestration&connection=Storage&code=FSVfJyGODSeKHPO0cM8Po9e1jMT7MghVMGuJqTaGTN56E1RUHnlVJg==",
"sendEventPostUri": "https://asynchttpfunction.azurewebsites.net/runtime/webhooks/durabletask/instances/66ee5d08196945aeb44c9e62ddc7b190/raiseEvent/{eventName}?taskHub=Orchestration&connection=Storage&code=FSVfJyGODSeKHPO0cM8Po9e1jMT7MghVMGuJqTaGTN56E1RUHnlVJg==",
"terminatePostUri": "https://asynchttpfunction.azurewebsites.net/runtime/webhooks/durabletask/instances/66ee5d08196945aeb44c9e62ddc7b190/terminate?reason={text}&taskHub=Orchestration&connection=Storage&code=FSVfJyGODSeKHPO0cM8Po9e1jMT7MghVMGuJqTaGTN56E1RUHnlVJg==",
"rewindPostUri": "https://asynchttpfunction.azurewebsites.net/runtime/webhooks/durabletask/instances/66ee5d08196945aeb44c9e62ddc7b190/rewind?reason={text}&taskHub=Orchestration&connection=Storage&code=FSVfJyGODSeKHPO0cM8Po9e1jMT7MghVMGuJqTaGTN56E1RUHnlVJg==",
"purgeHistoryDeleteUri": "https://asynchttpfunction.azurewebsites.net/runtime/webhooks/durabletask/instances/66ee5d08196945aeb44c9e62ddc7b190?taskHub=Orchestration&connection=Storage&code=FSVfJyGODSeKHPO0cM8Po9e1jMT7MghVMGuJqTaGTN56E1RUHnlVJg=="
}
The statusQueryGetUri provides information of the long running orchestration instance. If you follow this link you will receive a suitable runtimeStatus that describes the status of the orchestration instance along with some other useful information.here
My question is now:
actually we don't send a POST request to an Event grid Azure function trigger, Is there any way to get the status of the Azure function? Complete or is still running?
The Azure Event Grid is an eventing Pub/Sub model where the interest of source is distributed to the subscribed event handler endpoint or resource in the reliable manner with a retry policy and dead-lettering option. The AEG is waiting for delivery response processing max. 60 seconds.
There is no built-in the features what you are asking in the AEG, however you can use the REST API for metrics of the specific subscription to obtain its counters value:
MatchedEventCount,
DeliveryAttemptFailCount,
DeliverySuccessCount,
DroppedEventCount,
DeadLetteredCount
The following GET is an example for getting a subscription metrics:
https://management.azure.com/subscriptions/mysubId/resourceGroups/mygroup/providers/Microsoft.EventGrid/topics/mytester/providers/Microsoft.EventGrid/eventSubscriptions/mysubscription/providers/Microsoft.Insights/metrics?api-version=2018-01-01&interval=PT5M&metricnames=MatchedEventCount,DeliveryAttemptFailCount,DeliverySuccessCount,DroppedEventCount,DeadLetteredCount
Note, that the authorization header with a bearer token is required for this call.
More details about the monitoring an event message delivery can be found here.
I'm using the code to implement Azure service bus topics that can be found here:
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-how-to-use-topics-subscriptions
I've tried to run two instances of the subscriber program, which holds these methods:
private static void RegisterOnMessageHandlerAndReceiveMessages()
{
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandler)
{
MaxConcurrentCalls = 1,
AutoComplete = false
};
_subscriptionClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
}
private static async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
Console.WriteLine($"Received #{message.SystemProperties.SequenceNumber} message: {Encoding.UTF8.GetString(message.Body)}");
await _subscriptionClient.CompleteAsync(message.SystemProperties.LockToken);
}
However, this doesn't allow both subscribers to recieve the message, it is recieved one message each split across both subscribers.
How does I ensure both subscribers get all the messages coming from the bus?
Azure Service Bus is a broker with Competing Consumer pattern when it comes to retrieving messages. This is expected behaviour. What you have here is a scaled-out processing endpoint, where both are trying to process messages from the same subscription. The messages are distributed between these two competing consumers. If you need to distribute the same message to more than a single subscriber, you should have different subscription entities created and listened to.
It's worth stating - as it's not obvious from any of the documentation - that each topic should be setup with a subscription for each subscriber.
I.e. if I have 3 subscribers:
/myservicebus/mytopic/mysubscription1
/myservicebus/mytopic/mysubscription2
/myservicebus/mytopic/mysubscription3
I was expecting - probably same as OP - that topics and subscribers were automatically configured as one:many.
Essentially, if multiple subscribers are configured to the same subscription, then this becomes a 'first sub wins' scenario - the same as if it were a queue.
I also ran into this.
I resolved it by making an extra micro service just for forwarding the messages. The forwarder service itself must subscribe to the messages that each of your subscribers must receive. Each Subscriber registers itself (with a unique id) with the event forwarder service (by sending a message to it on startup). The subscribers must then use the message name + the unique id to subscribe to the messages. The message forwarder service will receive the messages and forward them to all the registered subscribers by changing the subject/label of the message (the unique id is added to the subject/label).
This workaround also has it's limitations, there can only be one instance of the message forwarder service or otherwise you have the same problem again. And the message forwarder service must also be running before the other subscribers otherwise it doesn't receive the registrations.
I have a server which will need to send messages to several clients to let the clients know that something needs to be done.
I am trying to achieve this by using Azure Event Hub.
I use the following code to send the message:
await eventHubClient.SendAsync(
new EventData(Encoding.UTF8.GetBytes(String.Format("Message {0}, {1}", i, sMessage))),
"1")
.ConfigureAwait(continueOnCapturedContext: false);
await eventHubClient.CloseAsync();
I use two WPF application as listeners which will create the listener at startup and will save the EventProcessorHost in a private variable.
When I send a message it's random which of the listeners will process the message.
Is is possible to send messages to multiple recipients with Azure Event Hub?
You need to put each listener to a separate Consumer Group.
Listeners of the same consumer group are "Competing Consumers", i.e. the first one who takes a lock on an event hub partition wins.