I've relatively new to Azure Service Bus Queues and am building a project that needs to process messages in a queue in the order in which they arrive (FIFO).
Using Microsoft's documentation, I was able to figure that part out. From what I understand I need to turn sessions on for the queue?
What I'm struggling with is determining what would be the best approach / service stack to perform the following set of ordered tasks against the queue.
First let's assume that we have a FIFO-based Service Bus Queue in place that has n number of messages. How might I:
Pick up the first message from the queue.
Process the message using an Azure Function.
Send a payload to IoT Hub that will deliver to an external device (C2D).
This is the part that I can't figure out...Wait for either a Completed indicator to return from IoT Hub or wait until the TTL expires for the outgoing IoT Hub Message.
Now complete the item in the queue.
Start back at 1.
I would imagine that perhaps a Logic App might help me achieve what I'm attempting to do. Everything seems straight forward up until #4. I can't figure out how to have the logic app 1. Wait until IoT Hub acknowledges that the Cloud-to-Device message was sent or expired and 2. Don't process the next message in the queue until the IoT Hub acknowledgment has been received and I've marked the current queue message as complete.
Please be advised: The reason why I'm so specific about this is that the devices receiving the outgoing IoT Hub C2D messages care about order. If they receive messages out of order, it throws the process off.
Any suggestions is greatly appreciated.
Azure Service bus Queues with Session can be used for achieving ordered processing of the message.
From the question, I can understand that the message from the Service Bus Queue should be removed only after the acknowledgement is received from IOT Hub.
I can sense a problem that may happen with this flow.
First of all, you need to understand about lock duration property in Service Bus Queue. Based on the value set to this property, the messages will be locked for x minutes or seconds. Whenever a message is received from a Queue, a lock will be applied for the message. So that the message will not be available for any other receivers for x minutes or seconds. Complete operation should happen before the lock gets expired. Once the lock is expired, the message will be available for other receivers to process. In your case, there is a chance for the same Azure Function to receive the message once again (resulting in duplication)
The maximum value that can be set for the lock duration property is 5 minutes.
So, if you are sure that the IOT Hub responds with in 5 minutes, you can proceed with this implementation. Still, there will be no option to let the Function know when to process the next message.
Related
I know that the messages in an event hub expires after a certain period of time depending on how we configure it but is there any way we can delete the events received in an event hub through code or through configuration in the Azure portal as soon as we receive them?
please go through this documentation for sending and receiving messages in eventhub;
https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-java-get-started-send
https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-dotnet-framework-getstarted-send
At this time, there is no mechanism to delete all messages. Messages expire automatically beyond their 24 hour retention. If you care about messages only from the time you subscribe, you can do a one time subscription with SubscribeRecency of newest (check the java sdk for exact value).
Subscriptions are durable and if you disconnect and reconnect you will see newest messages only the first time you subscribe and not each time i.e. message delivery will commence from newest messages upon first subscription and subsequent reconnects will resume delivery of messages published since you last connected
If you want messages deleted once you received them, you might want to consider Azure Service Bus Queues - they support exactly that.
Event Hubs provide an immutable append-only log, in other words, events are not supposed to be changed once they are created. Think an EH message as an event at a point in time therefore you cannot go past in time and change an event.
If you need mutable messaging then consider Azure Service Bus.
I am using Azure Service Bus Queues in one of my .net core application to receive the messages from the Queue in the FIFO order. Once received the message from the Queue, then I have processed the business logic with each message. Once processed the business logic with each message then I am trying to delete the message from Queue with following line of code:
await _messageReceiver.CompleteAsync(message.SystemProperties.LockToken);
But for the few messages, I am unable to delete the messages from Queue. Because lock on the message expired even though I configured the “Lock Duration” 3 minutes on Service Bus Queue.
I am getting the following error while deleting/completing the message from Service Bus Queue:
Exception Type: System.InvalidOperationException
Exception Message: Operation is not valid due to the current state of the object.
These are properties I configured on the Azure Service Bus Queue:
Max Delivery Count =1
Lock Duration=3 minutes
There are a few reasons why you are not able to delete the messages in Azure Service Bus Queue.
Challenge with Lock Duration
The first challenge is Time, specifically lock duration for every message once they have been retrieved. The problem is that the messages must be displayed in the UI and then provide time for the customer to choose the messages to be resubmitted or deleted. The solution would be to ask the customer to increase the lock duration, but this is not possible due to several reasons. Firstly, increasing lock duration might not be possible due to customer integration, so we cannot depend on that.
Read more on this blog post on how to overcome the challenge.
Referring to https://github.com/Azure/azure-service-bus/tree/master/samples/DotNet/GettingStarted/Microsoft.Azure.ServiceBus/BasicSendReceiveUsingTopicSubscriptionClient, I understand how Azure Service Bus Topics work in general, my question is more about how it actually works.
When a MesageHandler is registered (subscriptionClient.RegisterMessageHandler), it starts receiving messages as I see in
Console.WriteLine($"Received message: SequenceNumber:{message.SystemProperties.SequenceNumber} Body:{Encoding.UTF8.GetString(message.Body)}");
However, my question is whether the client actually receives the messages using a pull mode or it is a push from the Service Bus? Is there a continuous polling done by the Client to receive the messages - how does this work internally?
The client is performing a long-poll. I.e. it will ask for a message and wait for it. If after a timeout period of one minute nothing is returned, it will poll again. In case a message is available before timeout expires, the message will be given to the message handler and polling will start again. Azure Service Bus does not push messages to the clients.
In Azure, we have two separate messaging technologies and it's not very well documented when to use what? While EventGrid is really cool, I did not come across when to use EventGrid(scenarios) vs the Storage/ServiceBus queue? Can someone help?
E.g. if I have the following scenario :
A status of a flag changes and based on that, I want to trigger an algorithm that would do recalculations, few inserts/updates etc. in the database.
For implementing this - I can either use EventGrid or Storage Queue. How do we figure what to use in such scenario? I was looking for some kind of guidance.
Basically, Azure Event Grid handles events and Azure ServiceBus handles messages.A message is raw data produced by a service to be consumed or stored. Events are also messages (lightweigth), but they don’t generally convey a publisher intent, other than to inform.
1) If the purpose is to just to store the information ServiceBus can be used.
2) If the information received is used to trigger another service Azure Event Grid can be used.
Find more info here
https://learn.microsoft.com/en-us/azure/event-grid/compare-messaging-services
https://azure.microsoft.com/en-us/blog/events-data-points-and-messages-choosing-the-right-azure-messaging-service-for-your-data/
Events are like notifications from a service to inform the world that something happened in the domain of the publisher (similar to an email notification). There is no expectations from the publisher to have any actions taken. A message is a command you send to a specific receiver with the expectation of the message to be processed (like an asynchronous post request).
Events will work in pub/sub pattern and multiple subscribers could be configured to the events. The service that needs to react to an event will get notified by the event grid when an event occurs (http call from event grid to the receiver). The event will remain in the event grid until deletion (cleanup) and there is no garantie of keeping the original order (no FIFO).
In the other hand, messages will be added to a queue and will be deleted once the “message processor” is done with it. The messages in the queue will keep the original order (FIFO). The message processor has to pull messages from the queue.
In your scenario, you could use a combination of both. Service A sends an event “StatusChanged”, then you can configure a subscription to that event and send a message to a queue, then have your logic to process that message. This will end up with a fully async communication pattern. This is ideal to support scenarios where you processor is down or too busy. The incoming messages will simply get accumulated in the queue and eventually being processed once the service is back up and running. And without affecting the original service that sent the “StatusChanged” event..
I am using Azure Service Bus to implement message communication between separate bounded contexts. I am curious about what techniques people use to ensure that domain events raised in one bc are guaranteed to be received by another consuming bc.
For example, say the "orders" bc raises an "orderPlaced" event, how can I ensure that this event is received by a "shipping" bc. I understand that 2 phase commit is not advisable in cloud, so what is the alternative? How do I mitigate against the order being placed, but the message failing to be sent to the service bus in the event of a network failure?
Thoughts would be welcomed. Thanks.
If you send a BrokeredMessage to a Service Bus Queue and receive an acknowledgement, the message has been successfully stored in the queue. You don't have to worry about the message dying in transit due to a network error after you've been told it is persisted.
What you can worry about is a Service Bus Queue falling offline for a period of time and being unavailable. During an outage, your orderPlaced message wouldn't be able to get into the queue in the first place, and your shipping logic wouldn't be able to receive orders that are already persisted in your queue.
Note that Service Bus Queue outages are transient and the Queue recovers and returns to normal service. At that time, your shipping app could drain the queue of existing messages, and your ordering app could once again insert orderPlaced messages. I don't actually recall the last time I've seen one of my Service Bus Queues go down - it's a rare event.
If you are super-concerned about never ever ever EVER dropping a message, look at paired namespaces. Basically, this allows for failover to standby queues so that you can insert messages while your primary is down. Automatic detection checks to see when your primary queue comes back online. And a siphon process sucks messages that were inserted into the failover queue during the outage back into the primary once the primary comes back online.
Edit: When sending, there is still the chance that even though you had a valid Service Bus Queue connection in your QueueClient or MessagingFactory, the underlying Service Bus Queue just went down like a glass-jawed prizefighter. The vast majority of the time, these errors are transient. To handle them, set the RetryPolicy property of your MessagingFactory or QueueClient. Off the top of my head, I think that the only policy currently available is the RetryExponential policy. This will perform a back-off that will retry sending the message until the specified number of attempts are exhausted. This is the easy-peasy way to handle transient errors that pop up in your Service Bus Queue connection.