Problem: I have a Azure Topic Topic1 with Subscriptions Subscription-A, Subscription-B, Subscription-C
Each Subscription is listen to by an Azure function and it uses the message to perform unique action like charging customer or Updating another subsystem etc.
If the execution in one of the functions results in error , I would like to requeue the message into that specific subscription for execution after a delay say 1 min. not right way. [ Currently, If the function results in error, message is retried right away couple of times before pushing to dead letter. Looks like default behavior by Azure service bus and functions]
I can not resend a message to Topic1 as that message would be read by all the 3 subscriptions. I don't want this as the other message were processed successfully by the respective subscription/functions. This requeued message should be filtered out by other subscriptions on this topic.
Any ideas or suggestion would be of great help.
Related
I have created a Azure service bus topic subscription which receives a message form another http trigger function app and insert information into a database.
Every message I receive has an ID and based on the ID, I decide to either add a new record or updating existing one.
Problem is happening when 2 messages with the same ID are received at the same time and ends up creating 2 database records.
Is there any way to configure topic subscription function to run only 1 instance at a time? I don't have control on the function app which send
Is there any way to configure topic subscription function to run only 1 instance at a time?
You can configure everything like number of instances to be running, number of messages to be processing, number of calls and sessions, etc., in host.json file. All the attributes of host.json related to Azure Functions Service bus trigger bindings are available in this MS Doc reference.
For your requirements such as processing 1 message at a time, you can define the following attributes to 1 in the host.json file:
"maxConcurrentSessions": 1,
"maxMessageBatchSize": 1,
"maxConcurrentCalls": 1,
"messageHandlerOptions": {
"maxConcurrentCalls": 1
},
Normally, Azure Functions will process the messages in multiple and in parallel. So, the attributes like maxConcurrentSessions, maxConcurrentCalls plays the major role to define number of sessions and calls to be processed for every instance.
"batchOptions": {
"maxMessageCount": 1
}
Above complete configuration of host.json will do processing of 2nd message only after the execution of 1st message in the Service bus topic subscription azure function.
I am using azure service bus topic and subscription mechanism and want to process the messages which are all in the dead letter queue.
Moreover i want to process the messages via azure web job in C# and send them back to queue. So i want to know how I can process the messages on the deadletter queue through my application?
When a message is deadlettered it goes onto the dead letter queue for the subscription from which it was read. You access that just like you'd access the original subscription except that you append /$DeadLetterQueue to the subscription name.
Moreover i want to process the messages via azure web job in C# and send them back to queue.
As spodger pointed that the path of your deadletter subscription would be:
{topic-path}/Subscriptions/{subcription-name}/$DeadLetterQueue
You could use the WebJobs SDK for Service Bus and leverage the ServiceBusTrigger to access your dead letter queue message(s) as follows:
public void ProcessDeadletterQueue(
[ServiceBusTrigger("topicName", "subscriptionName/$DeadLetterQueue")] BrokeredMessage message)
{
//TODO:
}
For more details, you could refer to here.
When a message is dead-lettered from a Service Bus Entity(Queue or Topic Subscription), it will be moved to the dead-letter path of the same entity. The reason for dead-lettering will be available in the message's custom properties DeadLetterReason and DeadLetterErrorDescription.
In order to receive the dead-letter messages,
string path = Microsoft.ServiceBus.Messaging.SubscriptionClient.FormatDeadLetterPath(topicPath, subscriptionName);
var subscriptionClient = SubscriptionClient.CreateFromConnectionString(connectionString, path);
BrokeredMessage message = subscriptionClient.Receive();
In a pervious project I have managed to setup a Alert rule that looks at poison queue message count and alerts using a webhook into slack when something is in the queue (once per day).
I was trying to find where this exists in Azure as it looks like things have moved around. If this is not a feature provided by azure could you give guidance on what's the best route in implementing something similar.
Thanks
I was trying to find where this exists in Azure as it looks like things have moved around. If this is not a feature provided by azure could you give guidance on what's the best route in implementing something similar.
As far as I know, currently there is no a feature provided by azure to send the a alert rule to check the poison queue count.
You need write you own logic to achieve this requirement.
I suggest you could considerusing webjob/azure function timer trigger or queue trigger.
If you want to check the poison count every 5 minutes(for example), you could choose timer trigger.
Then in the timer trigger method, you could use ApproximateMessageCount method to get the queue messages' count.
At last, you could use sendgrid to send the notification email to special account.
Codes:
//get the storage account from the connection string
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
//instantiate the client
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
CloudQueue q = queueClient.GetQueueReference("queue-poison");
q.FetchAttributes();
var qCnt = q.ApproximateMessageCount;
If you want to get the count when the new queue message has added into the poison queue. You could choose queue trigger. The codes is as same as the timer trigger, just change the parameters.
I have a multi instance application, i'm using service bus event hubs to put into it some messages and it broadcast to all other instances, the condition to get the message to all instances is that every instance need to be in a separate consumers group otherwise an instance is going to get the message and delete it so other instance won't get the message, so my solution was that first every instance creates it's own consumers group and then listen to the event hub, but the problem here is that i will have a lot of consumers groups not in use after a while due to instances crashes,
my question is : is it possible to detect and get all consumers groups that are not in use to delete theme ?
P.S : i tried with topic/subscription also, it works well, but i have the same problem just replace above consumer group by subscription :) .
I got the answer : no you can't detect consumers groups witch are not in use but in the other hand you can detect that a subscription is not in use because it has a property "LastAccessDate" so using this field i can see if a subscription is in use or not and when was the last accessedDate, so i change from my service bus "event hub" to "topic/subscription" and it works :).
I send a message to ASB, some data (a long task) will start being processed, but the processing fails miserably (someone turns off the computer/out of memory/whatever). How should I handle this situation? Like putting the queue back on the bus? Do I need to create my own monitoring/requeueing unit?
Azure ServiceBus has an internal retry policy. If a message fails to deliver, it will be send back to the queue automatically.
When creating a queue or a topic/subscription you can specify the MaxDeliveryCount.
QueueDescription.MaxDeliveryCount
SubscriptionDescription.MaxDeliveryCount
Default value is 10. A message is automatically deadlettered after this number of deliveries.