I am using Azure Service Bus with .NET core
In our application, we are sending Session Messages to Service Bus. Whenever we receive a cancellation request with the session-Id, we need to remove/delete/complete messages with that specific sessionId without any further processing
I tried to access the MessageSession from another receiver to cancel messages from the queue
But I got error - The requested session 'session-name' cannot be accepted. It may be locked by another receiver.
Is there any other possible way to delete messages from service bus queue with a specific sessionId (session may be locked in few cases)
The only workaround, I'm able to get is
Update in database stating sessionId is cancelled & checking this for each message.
But this is not efficient for a large number of messages, as we need to extra DB hit for each message.
Is there any other possible way to delete messages from service bus queue with a specific sessionId?
It is possible to delete the message with SessionId using a search feature in Serverless360 where you can specify the query as SessionId = "1" and retrieve the message and delete it. But, it is obvious that you cannot retrieve session messages when it is locked and hence your second test case is not possible anyhow unless you could manage to use some action like renew lock in your orchestration with some delay in processing the messages.
Below is the screenshot of sample orchestration where each time the lock gets renewed and hence you can retrieve the message once you have the Session-Id
Related
Is it possible to implement the following pseudo scenario with an Azure Service Bus?
I have a function that can scale out to 50 instances, it uses a service bus trigger. I would like to guarantee that related messages are only processed if an existing related message is NOT currently being processed.
Let's say I have a message (Message A) being processed by a function instance that's associated with UserID 1234. Another message (Message B) appears on the queue which is also associated with UserID 1234, the service bus should "ignore" it because a related message is already being processed. Another message (Message C) with UserID 9876 appears on the queue, this gets handled straight away because there is no in action message with UserID 9876.
Message A finishes processing and Message B is now picked up.
Currently I have a routing function which consumes the initial service bus trigger and then routes it to one of 10 functions each of which is responsible for messages where the last digit of the UserID is 0-9.
This means that if function "4" is busy with a request it won't be able to process any other requests where the UserID ends with 4, thus guaranteeing the system cannot process a related message at the same time. It does it's job but doesn't scale.
There's no conditional locking. From the description, sounds like you want to process messages associated with the same user ID, one at a time. For that, Azure Service Bus has a feature called Message Sessions.
As far as I know, azure service bus has a locking mechanism built into it. So no matter the message, if you have a single queue you are reading from, when a function picks that message up the rest of your functions would not pick up that same message. To solve the issue of not processing a duplicate message by userID I would recommend using something table storage to validate if that userID has been processed already (So in your function when you pick up a message you insert that into azure table storage before doing any processing and also have a check to see if it exists before processing).
service bus - https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-transfers-locks-settlement
table storage - https://learn.microsoft.com/en-us/azure/cosmos-db/table/quickstart-dotnet?toc=https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fazure%2Fstorage%2Ftables%2Ftoc.json&bc=https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fazure%2Fbread%2Ftoc.json&tabs=azure-cli%2Cwindows
In summary I think the solution here would be using multiple technologies and use a central point your scaled out functions could validate against whats processed by other functions, whatever that central database is
Is there a way to have multiple listening clients on one Azure Topic Subscription, and they all recieve ALL messages?
My understanding is that the only implementation of a Subscription is that the Published message is only delivered to ONE client on that subscription, as it is like a queue.
Can these messages be copied to multiple clients using the same Subscription?
EDIT: Potential use case example
A server notifies all of its clients (web clients via browser, or application), that are subscribed to the topic, of an object that has changed its value
More simply, multiple PCs are able to see a data value change
EDIT 2: My setup/what I'm looking for
The issue that I am running into is that a message is marked as consumed by one client, and not delivered to the other client. I have 3 PCs in a test environment:(1 PC publishing messages (we'll call this the Publisher) to the topic, and 2 other PCs subscribed to the topic using the same SubscriptionName (We'll call these Client 1 and Client 2)).
So we have this setup:
Publisher - Publishes to topic
Client 1 - Subscibed using SubscriptionName = Test1
Client 2 - Subscribed using SubscriptionName = Test1
The Publisher publishes 10 messages to the topic.
Client 1 gets Message 0
Client 2 gets Message 1
Client 1 gets Message 2
... And so on (Not all 10 messages are recieved by both Client 1 and Client 2)
I want the Clients to recieve ALL messages, like this:
Client 1 AND Client 2 get Message 0
Client 1 AND Client 2 get Message 1
Client 1 AND Client 2 get Message 2
... And so on.
Service Bus is a one-to-one or end-to-end messaging system.
What you need is Azure Event Hub or Event Grid.
It is not possible for both the client1 and client2 to get the same messsage.
To put it straight, when a message is received by client1 from a subscription and processed successfully, the message is removed from the subscription, so the client2 will not be able to receive the same message again.
Hope this clarifies.
Yes, its a one-to-one implementation, but, if you have real concern about message processing completing in sequential order then it depends on the Receive mode.
You can specify two different modes in which Service Bus receives messages.
Receive and delete.
In this mode, when Service Bus receives the request from the consumer, it marks the message as being consumed and returns it to the consumer application. This mode is the simplest model. It works best for scenarios in which the application can tolerate not processing a message if a failure occurs. To understand this scenario, consider a scenario in which the consumer issues the receive request and then crashes before processing it. As Service Bus marks the message as being consumed, the application begins consuming messages upon restart. It will miss the message that it consumed before the crash.
Peek lock.
In this mode, the receive operation becomes two-stage, which makes it possible to support applications that can't tolerate missing messages.
Finds the next message to be consumed, locks it to prevent other consumers from receiving it, and then, return the message to the application.
After the application finishes processing the message, it requests the Service Bus service to complete the second stage of the receive process. Then, the service marks the message as being consumed.
If the application is unable to process the message for some reason, it can request the Service Bus service to abandon the message. Service Bus unlocks the message and makes it available to be received again, either by the same consumer or by another competing consumer. Secondly, there's a timeout associated with the lock. If the application fails to process the message before the lock timeout expires, Service Bus unlocks the message and makes it available to be received again.
If the application crashes after it processes the message, but before it requests the Service Bus service to complete the message, Service Bus redelivers the message to the application when it restarts. This process is often called at-least once processing. That is, each message is processed at least once. However, in certain situations the same message may be redelivered. If your scenario can't tolerate duplicate processing, add additional logic in your application to detect duplicates. For more information, see Duplicate detection. This feature is known as exactly once processing.
Check this link for more details.
Scenario:
We send messages (of same type) to a queue from multiple senders. Each sender will send groups of messages that are tied together by a unique session id for a given group. There is not limit to number of groups. On the consuming side, we have multiple receivers.
Question:
If receivers do not call await session.CloseAsync() on the last message in a group, will there be any performance implications down the road in our instance of azure service bus?
With message sessions, closing a session indicates that any other receiver can pick up the session and continue processing (message sessions can be only processed by a single receiver to ensure FIFO semantics). You should strive to close the session when it's no longer needed or is over. That way your client can start processing messages for another session that is pending processing.
There is a good sample with explanation how sessions work from Clemens you might have a look at as well.
I am using windows azure service bus for messaging. We have created web api to pull the messages from service bus. Mobile app is using this web api for getting messages. We are using PeekLock mode in subscription client and the lockduration is 5 minutes.
On client side if messages takes more than 5 minute to process then message will unlock. So before it unlock we want to renew the Lock. So we have created another web api to renew the lock. In web api we are passing the LockToken. But when we use subscriptionClient.RenewMessageLockAsync(new Guid(lockToken)), it throws error "The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue" before the Lock expiration time. We are initiating the renew lock before 1.5 min of message LockUntilUtc time.
My question here is can we renew the message lock before it expirytime? And if yes then why azure service bus throwing lock expired error? Please help me to understand this renew lock things.
You can extend the lock manually, but you need to make sure it happens before the server is unlocking it.
Another option, which I personally prefer more, is to use OnMessage API. It allows you to specify OnMessageOptions.AutoRenewTimeout which will be an extension time in case your callback that handles the message is not done. OnMessage API will also ensure that extension is invoked if you're about to exceed lock duration. This way you also won't need to have another web API, which sounds a little off.
Have a look at the recent post I wrote on this API.
I've looked everywhere for information on this but can't seem to find what i'm looking for.
I've got an azure topic, with one subscription.
The handler for the subscription failed some messages, they've been put on the deadletter queue.
I can access the messages but i'm at a loss for how to restore them.
I don't want to create a copy of the message and send it to the topic. I specifically want to move it to the subscription queue it came from.
Is there any way to do this?
I don't know if this is possible.
What we mostly do to handle & resume messages, is to use the pattern of defering messages. If a handler fails a message, he can then defer the message and keep track of the messagesequence number (typically in a management / tracking database or component).
Whenever the message needs to be reprocessed (retried), a receiver can then specifically get that message (using the same defered message id) and process it again.