I know that the messages in an event hub expires after a certain period of time depending on how we configure it but is there any way we can delete the events received in an event hub through code or through configuration in the Azure portal as soon as we receive them?
please go through this documentation for sending and receiving messages in eventhub;
https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-java-get-started-send
https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-dotnet-framework-getstarted-send
At this time, there is no mechanism to delete all messages. Messages expire automatically beyond their 24 hour retention. If you care about messages only from the time you subscribe, you can do a one time subscription with SubscribeRecency of newest (check the java sdk for exact value).
Subscriptions are durable and if you disconnect and reconnect you will see newest messages only the first time you subscribe and not each time i.e. message delivery will commence from newest messages upon first subscription and subsequent reconnects will resume delivery of messages published since you last connected
If you want messages deleted once you received them, you might want to consider Azure Service Bus Queues - they support exactly that.
Event Hubs provide an immutable append-only log, in other words, events are not supposed to be changed once they are created. Think an EH message as an event at a point in time therefore you cannot go past in time and change an event.
If you need mutable messaging then consider Azure Service Bus.
Related
I have an Azure Service Bus topic back2gq. I've configured EnableDeadLetteringOnMessageExpiration and forwarding to a queue called dlq. The messages on dlq are consumed by a monitoring service. My intention was to detect when a client has stopped processing messages (due to crash or disconnect).
My messages have a TTL of four seconds. The default TTL for the subscription is ten seconds. However, the messages don't get forwarded to the dlq after ten seconds.
I've used the Azure Service Bus Explorer to see what is going on:
The messages start piling up if there are no clients actively listening on the topic. All messages stay marked as 'active' and don't expire. After approximately 120 seconds all messages are flushed to the dlq in one go...
I had expected to see an incremental increase either on the DeadLetter count of back2gw or the message count on dlq.
The Service Bus documentation states (emphasis mine):
The expiration for any individual message can be controlled by setting the TimeToLive system property, which specifies a relative duration. The expiration becomes an absolute instant when the message is enqueued into the entity. At that time, the ExpiresAtUtc property takes on the value (EnqueuedTimeUtc + TimeToLive). The time-to-live (TTL) setting on a brokered message is not enforced when there are no clients actively listening.
In the next paragraph it goes on to say:
All messages sent into a queue or topic are subject to a default expiration that is set at the entity level with the defaultMessageTimeToLive property and which can also be set in the portal during creation and adjusted later.
I interpret this to mean that messages should still expire after defaultMessageTimeToLive is exceeded even if there aren't any active clients. Is the observed behavior correct? Did I misunderstand the docs? I'm on the Standard plan, is this perhaps Premium feature ;)
Expired messages will eventually be moved to the DeadLetter queue (and subsequently forwarded if configured accordingly), but it you cannot depend on when exactly that happens.
The chapter about message browsing mentions (emphasis mine):
Consumed and expired messages are cleaned up by an asynchronous
"garbage collection" run and not necessarily exactly when messages
expire, and therefore Peek may indeed return messages that have
already expired and will be removed or dead-lettered when a receive
operation is next invoked on the queue or subscription.
This periodic cleanup behavior matches my observation.
I've relatively new to Azure Service Bus Queues and am building a project that needs to process messages in a queue in the order in which they arrive (FIFO).
Using Microsoft's documentation, I was able to figure that part out. From what I understand I need to turn sessions on for the queue?
What I'm struggling with is determining what would be the best approach / service stack to perform the following set of ordered tasks against the queue.
First let's assume that we have a FIFO-based Service Bus Queue in place that has n number of messages. How might I:
Pick up the first message from the queue.
Process the message using an Azure Function.
Send a payload to IoT Hub that will deliver to an external device (C2D).
This is the part that I can't figure out...Wait for either a Completed indicator to return from IoT Hub or wait until the TTL expires for the outgoing IoT Hub Message.
Now complete the item in the queue.
Start back at 1.
I would imagine that perhaps a Logic App might help me achieve what I'm attempting to do. Everything seems straight forward up until #4. I can't figure out how to have the logic app 1. Wait until IoT Hub acknowledges that the Cloud-to-Device message was sent or expired and 2. Don't process the next message in the queue until the IoT Hub acknowledgment has been received and I've marked the current queue message as complete.
Please be advised: The reason why I'm so specific about this is that the devices receiving the outgoing IoT Hub C2D messages care about order. If they receive messages out of order, it throws the process off.
Any suggestions is greatly appreciated.
Azure Service bus Queues with Session can be used for achieving ordered processing of the message.
From the question, I can understand that the message from the Service Bus Queue should be removed only after the acknowledgement is received from IOT Hub.
I can sense a problem that may happen with this flow.
First of all, you need to understand about lock duration property in Service Bus Queue. Based on the value set to this property, the messages will be locked for x minutes or seconds. Whenever a message is received from a Queue, a lock will be applied for the message. So that the message will not be available for any other receivers for x minutes or seconds. Complete operation should happen before the lock gets expired. Once the lock is expired, the message will be available for other receivers to process. In your case, there is a chance for the same Azure Function to receive the message once again (resulting in duplication)
The maximum value that can be set for the lock duration property is 5 minutes.
So, if you are sure that the IOT Hub responds with in 5 minutes, you can proceed with this implementation. Still, there will be no option to let the Function know when to process the next message.
In Azure, we have two separate messaging technologies and it's not very well documented when to use what? While EventGrid is really cool, I did not come across when to use EventGrid(scenarios) vs the Storage/ServiceBus queue? Can someone help?
E.g. if I have the following scenario :
A status of a flag changes and based on that, I want to trigger an algorithm that would do recalculations, few inserts/updates etc. in the database.
For implementing this - I can either use EventGrid or Storage Queue. How do we figure what to use in such scenario? I was looking for some kind of guidance.
Basically, Azure Event Grid handles events and Azure ServiceBus handles messages.A message is raw data produced by a service to be consumed or stored. Events are also messages (lightweigth), but they don’t generally convey a publisher intent, other than to inform.
1) If the purpose is to just to store the information ServiceBus can be used.
2) If the information received is used to trigger another service Azure Event Grid can be used.
Find more info here
https://learn.microsoft.com/en-us/azure/event-grid/compare-messaging-services
https://azure.microsoft.com/en-us/blog/events-data-points-and-messages-choosing-the-right-azure-messaging-service-for-your-data/
Events are like notifications from a service to inform the world that something happened in the domain of the publisher (similar to an email notification). There is no expectations from the publisher to have any actions taken. A message is a command you send to a specific receiver with the expectation of the message to be processed (like an asynchronous post request).
Events will work in pub/sub pattern and multiple subscribers could be configured to the events. The service that needs to react to an event will get notified by the event grid when an event occurs (http call from event grid to the receiver). The event will remain in the event grid until deletion (cleanup) and there is no garantie of keeping the original order (no FIFO).
In the other hand, messages will be added to a queue and will be deleted once the “message processor” is done with it. The messages in the queue will keep the original order (FIFO). The message processor has to pull messages from the queue.
In your scenario, you could use a combination of both. Service A sends an event “StatusChanged”, then you can configure a subscription to that event and send a message to a queue, then have your logic to process that message. This will end up with a fully async communication pattern. This is ideal to support scenarios where you processor is down or too busy. The incoming messages will simply get accumulated in the queue and eventually being processed once the service is back up and running. And without affecting the original service that sent the “StatusChanged” event..
I am using Azure Service Bus to implement message communication between separate bounded contexts. I am curious about what techniques people use to ensure that domain events raised in one bc are guaranteed to be received by another consuming bc.
For example, say the "orders" bc raises an "orderPlaced" event, how can I ensure that this event is received by a "shipping" bc. I understand that 2 phase commit is not advisable in cloud, so what is the alternative? How do I mitigate against the order being placed, but the message failing to be sent to the service bus in the event of a network failure?
Thoughts would be welcomed. Thanks.
If you send a BrokeredMessage to a Service Bus Queue and receive an acknowledgement, the message has been successfully stored in the queue. You don't have to worry about the message dying in transit due to a network error after you've been told it is persisted.
What you can worry about is a Service Bus Queue falling offline for a period of time and being unavailable. During an outage, your orderPlaced message wouldn't be able to get into the queue in the first place, and your shipping logic wouldn't be able to receive orders that are already persisted in your queue.
Note that Service Bus Queue outages are transient and the Queue recovers and returns to normal service. At that time, your shipping app could drain the queue of existing messages, and your ordering app could once again insert orderPlaced messages. I don't actually recall the last time I've seen one of my Service Bus Queues go down - it's a rare event.
If you are super-concerned about never ever ever EVER dropping a message, look at paired namespaces. Basically, this allows for failover to standby queues so that you can insert messages while your primary is down. Automatic detection checks to see when your primary queue comes back online. And a siphon process sucks messages that were inserted into the failover queue during the outage back into the primary once the primary comes back online.
Edit: When sending, there is still the chance that even though you had a valid Service Bus Queue connection in your QueueClient or MessagingFactory, the underlying Service Bus Queue just went down like a glass-jawed prizefighter. The vast majority of the time, these errors are transient. To handle them, set the RetryPolicy property of your MessagingFactory or QueueClient. Off the top of my head, I think that the only policy currently available is the RetryExponential policy. This will perform a back-off that will retry sending the message until the specified number of attempts are exhausted. This is the easy-peasy way to handle transient errors that pop up in your Service Bus Queue connection.
We are trying to use Azure Topic/Subscriptions to distribute changes in one domain to others services that need to update their local cache. So we have one guy publishing a message and a bunch, not knowing of eachother, listening to this topic queue.
I might have missunderstod the idea of the azure TopicDescription.DefaultMessageTimeToLive but I thought it indicated that as long as the message is still within this timeout, it will be delivered, regardless if the subscriber is "online" at the time of publishing.
But this does not seem do be the case?
What I want to accomplish is that if I have a DefaultMessageTimeToLive set to 10 minutes, all subscribers are guaranteed to get all published messages if they have a downtime lower than 10 minutes.
When I try it, I do not receive messages unless I am listening at the time of publishing. (Added remark: Each receiving queue has its own unique name)
Have I got it wrong or is there a configuration I missed?
If you want a topic available to N subscribers (listeners), then you need N subscriptions. Subscriptions are "virtual" FIFO queues. If a subscription has more than one listener, they "compete" for the next message. This pattern is often termed the “competing consumer” pattern. Read more at Service Bus Queues, Topics, and Subscriptions and How to Use Service Bus Topics/Subscriptions.
The TopicDescription.DefaultMessageTimeToLive just defines how long a message is available before it is moved to the dead-letter queue.