On Azure (or from VS) the dialog for creating new messages inside a queue says the message expires in 7 days, yet it is gone in less than a few seconds. Why? (I created a continuous running WebJob as described in this article)
The message disapear because it has been consummed by your Web job.
The retention delay means you have X days to consume the message (in your case, 7 days). After the delay expired, the message is automatically deleted.
If you want multiple consumer for your messages, instead of a queue, you can use Service Bus with subscription or topics, or Event Hub with consumer groups.
The messages stay in the Service Bus Queue or Topic subscription until they are processed i.e received in receive and delete mode by the receiver.
The message will not be removed from the queue if it is received in peek lock mode.
In your case as the message is processed by the webjob it was not available in the queue.
The messages also have default time to live property which can be set after which the message will be moved to the dead letter path of the same messaging entity(queue or topic subscription).The messages after the given time duration in time to live after scheduled enwueued time utc will be moved to the dead letter path with dead letter reason TTLExpiration
Related
Azure Service Bus entities (queues/topics) support a Time to Live (TTL). When the TTL passes the message expires. On expiry, the system deletes the message OR moves it to the Dead-Letter Queue (DLQ). Does Service Bus have another setting to delete messages from the DLQ after a specified period? For instance, to avoid passing size quotas, we might like to delete messages from the DLQ after six months.
See also:
Do messages in dead letter queues in Azure Service Bus expire?
https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-expiration?WT.mc_id=Portal-Microsoft_Azure_ServiceBus
Azure Service Bus doesn't have an expiration option on the dead-letter queues. This is likely intentional, as the system shouldn't just lose those messages but rather do something about them.
Sometimes, monitoring all dead-letter queues for total size and whatnot is inconvenient. One option is to create a centralized DLQ. That will allow the following:
Monitoring a single "dead-letter" queue.
Receive messages from a single entity for processing.
Keep the size under control by specifying a TTL on the queue.
For example, let's say you've got two queues, test-dlq and test-dlq2. You'd configure those to auto-forward dead-lettered messages to a 3rd queue, test-dlq-all. With that, when you have messages that are received by test-dlq or test-dlq2 and dead-lettering,
Those messages will end up in the centralized "DLQ" queue (test-dlq-all).
The nice part is whenever you have messages auto-forwarded, you'll always know where they originally dead-lettered.
For example, let's say you've got two messages, each from a different queue, ending up in test-dlq-all, the centralized "DLQ".
Inspecting its messages will reveal a system property, DeadLetterSource, stamped with the name of the queue it was dead-lettered initially in.
This solution lets you set TTL on the test-dlq-all queue and have messages auto-perged.
Also, worth mentioning that it's possible to either set up dead-lettering with the centralized "DLQ" or get messaged dead-lettered as a result of failing processing that exceeds MaxDeliveryCount. For that reason, it is worth wither monitoring test-dlq-alls DLQ.
I am working on Azure Service Bus. My service bus queue is processing one message 3 times. My lock time of message is 5 minutes. Every message is processing max of 2 mins but I don't know why the queue is picking same message and sending to processing and the duplicate messages are picking after 5 mins only.
How can I resolve this?
With Azure Service Bus messages will be re-processed when a message is not actioned by the receiving party. An action would be completing, deferring, dead-lettering. If you don't have any of those, once LockDuration on the broker side expires, the message will be re-delivered. Additional situation when a message would be re-delivered without waiting for LockDuration to time out would be to abandon a message. Then a message is picked up right away by the next request for new messages.
You should share your code to provide enough context. Messages can be received manually using MessageReceiver.ReceiveAsync() or using user-callback API. For the first option you have to action messages (complete for example). For the other option, there's a configuration API where you could opt-out of auto-completion and would be required manually complete message passed into user-callback.
I have an azure function monitoring an Azure service bus queue with default lock duration of 30 seconds. This azure function sends out email notifications based on the user information in the message coming from the queue.
I noticed duplicate emails going out and thus checked the trace logs to find out that azure function has been invoked twice for the same user. Log entries are as follows:
2018-08-09T14:38:05.1249371Z - Executing 'AzureFunction' (Reason='New ServiceBus message detected on 'servicebusqueue'.', Id=4657012a-94ac-4b22-a628-2e94285aeeb7)
2018-08-09T14:38:33.3335833Z - Executing 'AzureFunction' (Reason='New ServiceBus message detected on 'servicebusqueue'.', Id=3ff8eea3-9b9b-43ae-a797-5acf01c2ae6c)
The message was added only once to the queue and I am trying to understand what could generate the other one. Can it be because of the lock duration?
Yes it could be due to the lock duration. The message will be completed (received and deleted) from the Queue, only after the execution of the function is complete. If the execution time exceeds 30 seconds, the message will be unlocked, making it available for any other receivers.
In your case, the receiver would be the same Azure Function, which reads the message another time, that's why you are seeing duplicated processing of the message.
The maximum value of Lock Duration is 5 minutes. If the Azure Function just sends an email notification when the message is received, you can increase the lock duration to 5 minutes. The email transmission shouldn't take that long, so that the message will not be available for another receiver.
If you have plans to add something to the Azure Function in addition to sending the notification, you can set auto renewal for the lock in Azure Function, check here for more details. This will keep the message locked, leaving no option for duplication.
Messages added only once, but it's at-least-once delivery guarantee with PeekLock.
If your message was not successfully completed within 30 seconds from receiving, it will be unlocked and recieved again. Increasing lock duration or reducing processing time should address it.
IN Azure Service bus topic how the messages are moved from dead letter queue to topic?
whether it will automatically moved to topic or we need to configure the properties of topic in portal or whether there is any other way to do it?(i prefer not to use any code here i wish to do only configuration changes)
We had a batch of around 60k messages, which need to be reprocessed from the dead letter queue. Peeking and send the messages back via Service Bus Explorer took around 6 minutes per 1k messages from my machine. I solved the issue by setting a forward rule for DLQ messages to another queue and from there auto forward it to the original queue. This solution took around 30 seconds for all 60k messages. This works well for queues and topics.
We are facing a problem with Service Bus.
We have a topic, with two subscriptions.
We have enabled Duplicate Detection on those, with 1 minutes window (tried with 2 seconds first). We are using Duplicate Detection to avoid multiple messages processed in short interval (to maintain the interval between the messages)
We are using the message scheduling (ScheduledEnqueueTimeUtc) to repeat the messages to appear after 5 minutes, with same message ID (every time a new message is created with schedule, and old message is completed)
The workflow is as follows (problem):
First time a message is published (without scheduling)
This message is immediately consumed by the message pump, and a new message with same details and a schedule time of 5 minutes is send to the topic (UTC), expecting it to appear after 5 minutes
The message is not appearing in the subscription
When debugged, this issue doesn’t come up
When we send the First message with at least 30 second delay (scheduled), then it is working fine
If we recreate the topic and subscription with Duplicate Detection turned off, we are able to get the message using the above workflow
Since we have no clue on what is happening to the published message, we need help to identify the root cause of the issue.
This is an expected behavior of the ASB.
When a message is scheduled, it's actually enqueued on the broker with delayed appearance. ASB on the server side de-duplicates messages upon arrival and uses message ID for de-dup.
In your case, if you delay dispatch of the second message and the original message is processed, there will be nothing to de-dup and the second message will be enqueued. If you don't delay, then the broker will see an identical ID to the previously sent message that has not been completed or DLQed yet, and it will be de-duped.
Possible way to go about is not reuse the same transport message ID (ID used for the BrokeredMessage). In case you need to associate messages, you can use Properties for that.