I am using Azure service bus topic/subscriptions with sessions.
I am setting a hardcoded value for the sessionId as I want to control processing of incoming messages.
So even if there are more than one eligible subscribers which can process this message only one will get the sessionlock and process it. I release the lock by closing the session once it is done processing so that new messages can be picked up by any of the processors. My only concern here is, is there any downside of closing the session after completion of message processing?If I close session, will the processor with closed session be able to open session and process the message at later?
TIA
Sessions are usually intended for groups of messages and not a single message. When you want those messages to be processed in order (FIFO) a single subscriber should handle that subscription until messages in the session are all processed. And then, what you do, closing a session, is fine as it will expire eventually. Not to mention that your subscriber could start processing another session rather than waiting for an "empty" session to time out.
In case you have a single message per session, I would advise against using sessions to begin with.
Related
I have a session enabled subscription on a service bus topic and have four subscription clients running against this subcription.
I publish 10000 messages across 100000 random sessions to the topic and looking at my output I can see multiple subscription clients processing messages for the same session id within seconds of each other, i.e.
SessionId
Client A
ClientB
1234
processing at 10:30:04
1234
processing at 10:30:29
1234
processing at 10:31:00
This was done with the session idle timeout default of 60 seconds.
I then set the session idle timeout to a 2 second timespan. Each SessionProcessor has 100 MaxConcurrentSessions and has AutoCompleteMessages set to false.
I am also observing a lot of errors for "The session lock was lost".
When I receive a session lock exception that session is then started on a different client and the same message is processed on the second client that had already been processed on the client with the session lock exception.
My question is, do I need to record processed message id's before attempting to call CompleteMessageAsync on the ProcessSessionMessageEventArgs so I can then check it for every single message being consumed to avoid processing it again or is there a reason I'm unaware of for my session to lose the session lock - does this happen when the session idle time has been surpassed before calling CompleteMessageAsync?
This is all in test code so my processor is simply writing to the console and doing non long running tasks.
When using a topic, messages are copied and forwarded to all subscriptions, in which the session would be visible for receivers.
If you don't want all receivers to process each sessions, then you should just use a single queue with all receivers listening on it. Then each session and its set of messages would be processed only once, by one of the receivers.
We have a Topic with Subscriptions with a default LockDuration of 1min, and multiple SubscriptionClients listening to each subscription. For our test purposes, there are 3 clients listening to a single subscription.
SubscriptionClients are created as:
Client = new SubscriptionClient(endPoint, topicName, subscriptionName);
We put one message on the Topic, which is filtered into the Subscription.
We would expect one of the SubscriptionClients to pick up the message, and the other two clients cannot because it is locked.
What is actually happening, is all three clients are simultaneously picking up the same message, with different DeliveryCounts, and all within the 1minute lock duration.
Is there something wrong with the way we're creating the SubscriptionClient such that the lock is shared between them rather than being exclusive?
There are possibly two things that could be wrong. And none of those would be the broker but likely the client-side code.
MaxLockDuration is too short and while one client is still working on the message, the other client(s) receives that same message. You should be able to confirm by looking at the duration of the message processing. If it exceeds MaxLockDuration set on the queue, that's it.
You're using a message handler with automatic lock renewal and that one is failing to extend the lock. In that case, you would have a message handler error callback raised with the details.
Either way, you could log the errors and share the logs if possible to help with pinpointing what the issue is.
My setup:
I've created an Azure Service Bus Queue with sessions enabled.
Lock duration = 1 minute
I've created an Azure Logic App with the trigger "When a message is received in a queue (peek-lock)
Session id = Next Available
Interval = 15, Frequency = Second
If the steps in my Azure Logic App succeed, I Complete the message in a queue, otherwise I Dead-letter the message in a queue
My results:
The Azure Logic App picks up one queue message at a time and processes it, but waits the entire Lock duration (in my case, 1 minute) before picking up the next message.
Note: When I set Session id to be a specific session id, it does not wait, but processes messages immediately
My question:
I'd like the Azure Logic App to process more queue messages immediately after completion when the Session id = Next Available. Is this possible and is there a setting I'm missing to tell the Azure Logic App to not wait the entire lock period?
Do your messages have the same SessionId?
In that case I think your Logic App is holding the session lock, so you need to use the "Close a session in a queue" task.
If you have multiple messages with same SessionId, it seems to hold the session lock for one minute before it takes the next message with the same SessionId based on my simple test. Closing the session resulted in immediate processing of the next message in same session.
I have a similar case in production use where I have to wait for a while and pull all messages with the same SessionId together, but it runs nicely concurrent Logic Apps if there are multiple sessions going on, and that was the same in my test setup.
I am using the Microsoft Azure ServiceBus for Queue messages using WCF for the subscriptions. I am trying to implement retry logic. I use Peak/Lock to view the message and then have to do some local processing on the message. If that processing fails, I unlock the message so I can try processing it again. The problem is I need to build a have a delay in-between processing tries. Currently it is popped back into the queue and then is processed almost immediately. There needs to be about 2 minutes between attempts.
If you always have to wait 2 minutes before re-processing the message of that particular queue, you could try to configure the lock-timeout on the queue to be 2 minutes (plus the time you expect it will take you to process the message) and then just let the lock expire, instead of unlocking it. This has the downside that you would need to keep an eye on your processing time, and extend the lock's timeout if needed.
Another option could be to receive and complete the message, set a scheduled delivery of 2 minutes into the future, and re send the message. This has the downside that you need to consume it and ack it, which involves certain risks (e.g. your process dies before you get a chance to re-send the message).
"If the message is Peeked in Peek Lock mode from a Queue then you don't have the receive context in the message. You can receive the message in Peek Lock mode, which will lock the message for the interval specified for the 'lock duration' property of the queue. Locked messages cannot be received until its lock expires. Thus, by setting the lock duration to 2 minutes and Receiving messages in Peek Lock mode will solve this issue.
You can either write custom code to update the Lock Duration property. Tools like Service Bus Explorer, Serverless360 etc provides options to update property using graphical user interface."
I am working with Windows Azure Message queues. I want know if is there a method to lock messages in the queue when i get them ?
When you retrieve a message from the queue, it's marked as invisible until you delete it (or the timeout period is reached). When it's marked as invisible, nobody else sees the message. I guess that's as closed to "locked" as you're going to get.
If, while processing, you feel you need more time, you can modify the message and extend the invisibility timeout.
You do need to focus on idempotent operations with Windows Azure queues: Assume that any given message may be processed more than once:
Processing goes beyond invisibility timeout, so some other worker gets the message
VM instance crashes while processing message, causing it to re-appear in the queue and get processed again