I am using Azure Service Bus REST API to receive messages.
The requirement is to have a scheduled job to read messages from Azure Service Bus Queues and forward them for processing. If processed successfully, then delete them from the Queue or keep them in the Queue to be processed in the next scheduled job. I am using Peek-Lock Message (Non-Destructive Read) method(https://learn.microsoft.com/en-us/rest/api/servicebus/peek-lock-message-non-destructive-read).
The problem i am facing is inside my loop, how to know that i have read the queue fully so that i do not re-read the same queue again.
Your requirement is somewhat problematic.
If processed successfully, then delete them from the Queue or keep them in the Queue to be processed in the next scheduled job.
Successful processing should always result in message completion. Otherwise, you're asking for trouble. When processing messages in peek-lock mode, the message is locked for up to 5 minutes. It's your responsibility to complete it if the processing is successful. If it wasn't completed, that's a sign the processing wasn't successful and it should be read again given your requirement. Do not leave successfully processed messages in the queue.
The problem i am facing is inside my loop, how to know that i have read the queue fully so that i do not re-read the same queue again.
You shouldn't be concerned about this. Read messages and process. If failed to process, the message will reappear. Otherwise, a message should be removed. If you want to handle idempotency, i.e. ensure that if for some reason the message is not processed more than once, upon successful processing and prior to completion store the message ID (assuming it's unique) in a data store and validate any new message against that data store.
Related
I am looking into setting up a web job trigger to read message from service bus queue. What would be the best practice to implement a retry logic in case of any errors handling the downstream systems.
Would we be able to throw an exception so that the message will not be deleted from the queue and will be retried after certain time period?
Appreciate your feedback.
You don't need to define retry logic explicitly. When the message is de-queued from service bus , it gets invisible from queue for certain time period (lock time default 30secs , you can configure it). You try to process the message , if it gets successful you simply call BrokeredMessage.CompleteAsync which means i am done and mark this message as completed. If you have some problem in down stream you can abandon the message by calling BrokeredMessage.AbandonAsync . This will unlock the message and the message appears back in the queue. The message will be picked up by the worker again and process it. Until you get successful or reach the max retry limit after which the message is send to dead letter queue.
I'm wondering if it is possible to send a brokered message to a queue/topic where the message is already in a deferred state?
I'm asking this because I currently have a process that does the following ...
The process starts and a brokered message is sent to a queue (this triggers a function that records the message body as an entity in table storage with a 'Processing' status).
Additional work is done in the process
If we get to the end of the process without any issues, another brokered message is sent to the queue with a completion message (this triggers the same function that updates the entity in table storage with a 'Complete' status).
While this method is mostly working, it feels clunky and fragile. I would really like to be able to send a message to the queue and then have the final step make the message visible on the queue so it can be consumed by the function (Durable Function).
I thought about setting the ScheduledEnqueueTimeUtc, but I can't guarantee when the process will finish (I'm thinking worst case scenario here) so I'm not sure how long to set it.
I also looked at the Defer option for a BrokeredMessage but it seems this can only be set from the receiver and not be in a deferred state initially.
Is what I'm trying to do possible with Service Bus brokered messages? Could I set the Scheduled Enqueue time so some ridiculously long time (e.g. 2 hours) and if it reaches that time it is automatically expired and moved to the Dead Letter queue? Should I send the initial message to the Dead Letter queue and then once the process is complete, retrieve it and resubmit it?
Has anyone had any experience with implementing a process like this ... send a start message and only process the message once a completion notification has been received? I need this to be as robust as possible as I'm dealing with financial transactions in this process.
Hopefully my explanation makes sense.
I'm wondering if it is possible to send a brokered message to a queue/topic where the message is already in a deferred state?
That's not possible. You can only delay a brand new message, not defer it. Deferring required a message to be received first for it to have a SequenceNumber.
Using ScheduledEnqueueTimeUtc has its challenges as you will be sending it in the future, but cannot cancel once processing is over. Instead, you could leverage QueueClient.ScheduleMessageAsync() that returns back SequenceNumber immediately. This way you can set the message far into future, but also cancel it if processing is finished earlier.
I ended up solving this issue by keeping the process of sending two messages, but refactoring my durable function to record the messages in Table Storage, check that both messages have been received and if they have, add a new message to Azure Queue Storage. A second function listens to the queue which starts its process.
After much testing, this appears to be quite a robust solution. It then doesn't matter what order the two messages arrive, or how long they take ... as long as both of them have arrive, that is when the second function will kick off.
I have a spring integration flow where a processor (scheduled) sequentially reads messages from a queue (jms) and attempts processing. If the processor finds that the message can't be processed until another event finishes, it sends the message back to the original queue and attempts processing later.
If it just keep sending messages that can't be processed, back to the queue, it creates an infinite loop.
So I need to hold onto them until I finish reading all messages in the queue that already exist. And trigger a release when all existing messages are read, before sending them to the queue. How do I go about this?
Note that I don't want to aggregate the messages, but just temporarily hold them, and somehow. Also note that my processor is scheduled to read messages (not message driven).
In this case you have to acknowledge those messages in the queue anyway and re-send them back to it using JmsTemplate (or JmsSendingMessageHandler).
The feature with dequeue that the failed message is returned to the head of the queue. That's how you see it again and again and don't reach other messages (also you can do that with the concurrency).
With the resending messages in case of failure back to the queue, you place them in the tail of the queue. So, "bad" messages will be available later, after processing other, existing messages.
I am using the Microsoft Azure ServiceBus for Queue messages using WCF for the subscriptions. I am trying to implement retry logic. I use Peak/Lock to view the message and then have to do some local processing on the message. If that processing fails, I unlock the message so I can try processing it again. The problem is I need to build a have a delay in-between processing tries. Currently it is popped back into the queue and then is processed almost immediately. There needs to be about 2 minutes between attempts.
If you always have to wait 2 minutes before re-processing the message of that particular queue, you could try to configure the lock-timeout on the queue to be 2 minutes (plus the time you expect it will take you to process the message) and then just let the lock expire, instead of unlocking it. This has the downside that you would need to keep an eye on your processing time, and extend the lock's timeout if needed.
Another option could be to receive and complete the message, set a scheduled delivery of 2 minutes into the future, and re send the message. This has the downside that you need to consume it and ack it, which involves certain risks (e.g. your process dies before you get a chance to re-send the message).
"If the message is Peeked in Peek Lock mode from a Queue then you don't have the receive context in the message. You can receive the message in Peek Lock mode, which will lock the message for the interval specified for the 'lock duration' property of the queue. Locked messages cannot be received until its lock expires. Thus, by setting the lock duration to 2 minutes and Receiving messages in Peek Lock mode will solve this issue.
You can either write custom code to update the Lock Duration property. Tools like Service Bus Explorer, Serverless360 etc provides options to update property using graphical user interface."
I'd like to use Azure Queues as a state machine for a high-load/high-scale web service.
The client would submit a request to a web service endpoint, at which point i'd return a request id.
I'd then submit the message to a queue so that a worker role can process it, but no database activity occurs during the submission process. Instead, I want to use the queue that the message lives in to represent it's current state.
My problem is that if a worker role grabs the message off the queue to process it, it becomes invisible on that queue. If I want to check the status of the processing of that message, I have an ambiguous message state. Either the message was lost/never received, or it's in the queue but invisible because it's being processed.
Ideally, I'd like to be able to peak at the invisible message. If I find one that matches the request id, I know it's being processed if it's invisible, or it's waiting to be processed if it's visible. Obviously, I know when it's completed processing because that operation will result in a database write.
So is this possible, or is the fact that I can't peek at invisible messages in an Azure queue make this a no?
Windows Azure Storage Queues are for message-passing. They're not going to help you for state-machine processing, especially since each message can be processed at least once (since an app can run into an unexpected exception case while processing a message, the vm instance could crash, etc., and then the queue message re-appears after timeout (and now potentially out of order with the rest of your messages.
You're better off using an Azure Table row (or SQL table row).
In this case, I'd recommend using a blob to store the status of the message. Whenever a worker picks up a message, the blobID could be included and the worker can update the status blob. Your out-of-band process/website/whatever can query the blob to gather status information.