I am working on Azure Service Bus. My service bus queue is processing one message 3 times. My lock time of message is 5 minutes. Every message is processing max of 2 mins but I don't know why the queue is picking same message and sending to processing and the duplicate messages are picking after 5 mins only.
How can I resolve this?
With Azure Service Bus messages will be re-processed when a message is not actioned by the receiving party. An action would be completing, deferring, dead-lettering. If you don't have any of those, once LockDuration on the broker side expires, the message will be re-delivered. Additional situation when a message would be re-delivered without waiting for LockDuration to time out would be to abandon a message. Then a message is picked up right away by the next request for new messages.
You should share your code to provide enough context. Messages can be received manually using MessageReceiver.ReceiveAsync() or using user-callback API. For the first option you have to action messages (complete for example). For the other option, there's a configuration API where you could opt-out of auto-completion and would be required manually complete message passed into user-callback.
Related
I am looking into setting up a web job trigger to read message from service bus queue. What would be the best practice to implement a retry logic in case of any errors handling the downstream systems.
Would we be able to throw an exception so that the message will not be deleted from the queue and will be retried after certain time period?
Appreciate your feedback.
You don't need to define retry logic explicitly. When the message is de-queued from service bus , it gets invisible from queue for certain time period (lock time default 30secs , you can configure it). You try to process the message , if it gets successful you simply call BrokeredMessage.CompleteAsync which means i am done and mark this message as completed. If you have some problem in down stream you can abandon the message by calling BrokeredMessage.AbandonAsync . This will unlock the message and the message appears back in the queue. The message will be picked up by the worker again and process it. Until you get successful or reach the max retry limit after which the message is send to dead letter queue.
On Azure (or from VS) the dialog for creating new messages inside a queue says the message expires in 7 days, yet it is gone in less than a few seconds. Why? (I created a continuous running WebJob as described in this article)
The message disapear because it has been consummed by your Web job.
The retention delay means you have X days to consume the message (in your case, 7 days). After the delay expired, the message is automatically deleted.
If you want multiple consumer for your messages, instead of a queue, you can use Service Bus with subscription or topics, or Event Hub with consumer groups.
The messages stay in the Service Bus Queue or Topic subscription until they are processed i.e received in receive and delete mode by the receiver.
The message will not be removed from the queue if it is received in peek lock mode.
In your case as the message is processed by the webjob it was not available in the queue.
The messages also have default time to live property which can be set after which the message will be moved to the dead letter path of the same messaging entity(queue or topic subscription).The messages after the given time duration in time to live after scheduled enwueued time utc will be moved to the dead letter path with dead letter reason TTLExpiration
Sometimes there are some messages in Azure Queues that are not taken in charge by Azure Functions and also are not visible from StorageExplorer.
These messages are created without any visibility delay.
Is there any way to know what do those messages contain, and why are they not processed by our Azure Functions?
In the image you can see that we have a message in queue but it is not visible in the list and it is there from hours.
The Azure Queue API currently has no way to check invisible messages.
There are several situations in which a message will become invisible:
The message was added with an VisibilityTimeout in the Put Message request. The message will be invisible until this initial timeout expires.
The message has been retrieved (dequeued). Whenever a message is retrieved it will be invisible for the duration of the VisibilityTimeout specified by the Get Messages request, or 30 seconds by default.
The message has expired. Messages expire after 7 days by default, or after the MessageTTL specified in the Put Message request. Note: after a while these messages are automatically deleted, but until that point they are there as invisible message.
Use cases
Initial VisibilityTimeout
Messages are created with an initial VisibilityTimeout so that the message can be created now, but processed later (after the timeout expires), for whatever reason the creator has for wanting to delay this processing.
VisibilityTimeout on retrieving
The intended process for processing queue messages is:
The application dequeues one or more messages, optionally specifying the next VisibilityTimeout. This timeout should be bigger than the time it takes to process the message(s).
The application processes the message(s).
The application deletes the messages. When the processing fails the message(s) are not deleted.
Message(s) for which the process failed will become visible again as soon as their VisibilityTimeout expires, so that they can be re-tried. To prevent endless retries step 2. should start by checking the DequeueCount of the message: if it is bigger than the desired retry-count it should be deleted, instead of processed. It is good practice to copy such messages to a deadletter / poison queue (for example a queue with the original queue name plus a -poison suffix).
MessageTTL
By default messages have a time-to-live of 7 days. If the application processing cannot handle the amount of messages being added, a backlog could build up. Adjusting the TTL will determine what happens to such backlog.
Alternatively the application could crash, so that the backlog would build up until the application would be started again.
It seems that the message is expired. The following steps could reproduce the issue, you could test it.
Add message with a short TTL
After the message has been expired
We are facing a problem with Service Bus.
We have a topic, with two subscriptions.
We have enabled Duplicate Detection on those, with 1 minutes window (tried with 2 seconds first). We are using Duplicate Detection to avoid multiple messages processed in short interval (to maintain the interval between the messages)
We are using the message scheduling (ScheduledEnqueueTimeUtc) to repeat the messages to appear after 5 minutes, with same message ID (every time a new message is created with schedule, and old message is completed)
The workflow is as follows (problem):
First time a message is published (without scheduling)
This message is immediately consumed by the message pump, and a new message with same details and a schedule time of 5 minutes is send to the topic (UTC), expecting it to appear after 5 minutes
The message is not appearing in the subscription
When debugged, this issue doesn’t come up
When we send the First message with at least 30 second delay (scheduled), then it is working fine
If we recreate the topic and subscription with Duplicate Detection turned off, we are able to get the message using the above workflow
Since we have no clue on what is happening to the published message, we need help to identify the root cause of the issue.
This is an expected behavior of the ASB.
When a message is scheduled, it's actually enqueued on the broker with delayed appearance. ASB on the server side de-duplicates messages upon arrival and uses message ID for de-dup.
In your case, if you delay dispatch of the second message and the original message is processed, there will be nothing to de-dup and the second message will be enqueued. If you don't delay, then the broker will see an identical ID to the previously sent message that has not been completed or DLQed yet, and it will be de-duped.
Possible way to go about is not reuse the same transport message ID (ID used for the BrokeredMessage). In case you need to associate messages, you can use Properties for that.
What is the difference between Receive and Peek in the context of Azure Service Bus Topics?
Peek: This method enables you to view messages without locking or receiving them.
Receive: Can work in 2 modes: PeekLock (receives the message but keeps it peek-locked until the receiver abandons the message. The maximum timeout is 5 minutes before message expiration) and ReceiveAndDelete (deletes the message after it is received).
So this means you'll use the Peek when your goal is to look at the messages without actually consuming them (maybe you're building a "Queue browser", ... or your process needs to decided if it wants/can consume the message). And you'll use Receive when you're really planning to consume the message and do whatever you need to do with it.
Here is documentation on the PeakLock and ReceiveAndDelete values for ReceiveMode. Note that PeakLock is the default.