We have a Topic with Subscriptions with a default LockDuration of 1min, and multiple SubscriptionClients listening to each subscription. For our test purposes, there are 3 clients listening to a single subscription.
SubscriptionClients are created as:
Client = new SubscriptionClient(endPoint, topicName, subscriptionName);
We put one message on the Topic, which is filtered into the Subscription.
We would expect one of the SubscriptionClients to pick up the message, and the other two clients cannot because it is locked.
What is actually happening, is all three clients are simultaneously picking up the same message, with different DeliveryCounts, and all within the 1minute lock duration.
Is there something wrong with the way we're creating the SubscriptionClient such that the lock is shared between them rather than being exclusive?
There are possibly two things that could be wrong. And none of those would be the broker but likely the client-side code.
MaxLockDuration is too short and while one client is still working on the message, the other client(s) receives that same message. You should be able to confirm by looking at the duration of the message processing. If it exceeds MaxLockDuration set on the queue, that's it.
You're using a message handler with automatic lock renewal and that one is failing to extend the lock. In that case, you would have a message handler error callback raised with the details.
Either way, you could log the errors and share the logs if possible to help with pinpointing what the issue is.
Related
I have a request to implement a dashboard with the information about which message in Azure Service Bus queue was completed when (with some info about message parameters). Unfortunately we do not have an access to the reciever's code and cannot change the code to log the time of the message delivery. So, we need to subscribe somehow to a moment when reciever takes away the message.
I have already investigated Azure portal API in order to find something, but there is no such a possibility, I have tried to find something on stackoverflow and in Google, but no results.
There is 1 idea: use 2 queues and azure function between them. Put all messages to the first queue, azure function recieves a message, logs the info about the message and puts it to the second queue and waits until other services takes the message away from the second queue. Second queue will always have only 1 message and this way we will be able to understand what message was for sure delivered and when.
However what I do not like is the second message queue executes not the role of the real queue (it means something is wrong here and I need to use something else), performance of such a system can be not high enough...
Any help is appreciated (articles, videos, ideas). Thank you.
My service consumes messages from an Azure Service Bus subscription. A dependency of my service was down for a while, which caused a lot of messages to end up in the deadletter queue (DLQ). Now that the service is back up, I want to reprocess all messages from the DLQ. How can I move/resubmit all messages from the DLQ back in to the main queue.
Restrictions:
It's thousands of messages, so manually handling them isn't feasible.
The topic has about ten subscriptions. I don't want to resubmit the messages to the topic, because then all subscriptions would receive the messages, leading to double-processing.
I don't want to run the service against the DLQ directly, because some messages are broken and cause permanent errors, i.e. they would end up in the DLQ again, which would lead to an infinite loop. Moreover, the broken messages are put back at the front of the queue, effectively starving healthy messages that come after the broken ones.
I realize this is a while after the original post but if anyone else stumbles on this problem, there is a fairly handy solution baked into the Service Bus Explorer (which I have found to be incredibly handy with ASB development).
After connecting to your Service Bus and finding the needed namespace, find the desired topic and subscription with the deadletters in it. From there Right Click and Receive Deadletter Queue Messages and hit OK.
From there, highlight which you would like to send back to the main queue and hit Resubmit Selected Messages in Batch Mode.
Thomas, you probably already found your answer since this is quite awhile ago. think of DLQ (or any existing queue that you have) as just another collection variable like in a PC app, but residing on the cloud. just like a PC-app or in-memory collection variable from your tool-kit, you have many ways of utilising it. off course there are limitations and differences between these 2 types of collection variables, but that's how you design your solution as though the DLQ is just another collection variable by knowing those limitations and differences.
For some queuing implementations, one of the solutions would be to have another instance of the same app pointing to the DLQ, but with a fairly long visibility timeout (e.g. 6 or 12 or even 24 hours depending on your SLA), since you don't want to repeat them too often. However, this is not applicable to Azure service bus, as it limits the visibility timeout to at most 5 minutes.
if the DLQ contains broken un-recoverable jobs, you should fix the app to delete them based on the error messages when the unknown exception occurred. once the fix is deployed, such broken un-recoverable jobs would have been removed by your app and never get sent to the DLQ in the first place. and those already in the DLQ will be removed by the fixed app.
The only option to replay DLQ messages is to receive them from DLQ, create new message with same content and send it again to the topic. They will end up at the end of subscription queue.
You can't send messages directly to the subscription. There is a trick to add a metadata property to the message, and then adjust all except one subscription to filter out such messages. It's up to you to decide if it's going to help in your scenario.
As for tooling, we always did that with custom code, because we always needed some extra work to be done, like logging each replayed message for further analysis.
The quick answer is that you cannot directly move messages back to the main queue of a subscription. This is by design with how Microsoft implemented their topics and subscriptions.
Option #1
There is the option to use Azure Service Bus topic filters https://learn.microsoft.com/en-us/azure/service-bus-messaging/topic-filters and define/tag your messages in a manner that would only allow them to be received on the targeted subscription.
Option #2
The other option would be to change your current implementation. You would set up "delivery queues" (regular service bus queues) and configure each corresponding subscription to auto forward its messages to these delivery queues. Your message processing logic would then listen on these "delivery queues" vs the subscription. Any failures would then result in DLQ messages on these associated "delivery queues" which could then be handled outside of the topic/subscriptions.
I would appreciate your thoughts on this.
I have a node app which subscribes to a RabbitMQ queue. When it receives a message, it checks it for something and then saves it to a database.
However, if the message is missing some information or some other criteria is not yet met, I would like the subscriber to publish the message back onto the RabbitMQ queue.
I understand logically this is just connecting to the queue and publishing the message, but is it really this simple or is this a bad practice or potentially dangerous?
Thanks for your help.
As I point out in the comment, When you create connection with queue, and set autoAck = true, to enable message acknowledge. The message in the queue will be deleted until receive acknowledge.
When the received message meets requirement, then send ack message to this queue, and this message will be deleted from queue. Otherwise, no ack message is sent to queue, this message will stay in the queue.
As for you mentioned in comment, the valid process may take 5 minutes, just set the send ack message as callback function of validation function.
In your question, you describe two criterion for when a message may not be processed:
if the message is missing some information or
some other criteria is not yet met
The first of these appears to be an issue with the message, and it doesn't seem that it makes much sense to re-queue a message that has a problem. The appropriate action is to log an error and drop the message (or invoke whatever error-handling logic your application contains).
The second of these is rather vague, but for the purposes of this answer, we will assume that the problem is not with the message but with some other component in the system (e.g. perhaps a network connection issue). In this case, the consuming application can send a Nack (negative acknowldegement) which can optionally requeue the message.
Keep in mind that in the second case, it will be necessary to shut down the consumer until the error condition has resolved, or the message will be redelivered and erroneously processed ad infinitum until the system is back up, thus wasting resources on an unprocessable message.
Why use a nack instead of simply re-publishing?
This will set the "redelivered" flag on the message so that you know it was delivered once already. There are other options as well for handling bad messages.
I don't get it.
Say I have one queve, one topic, one subscroption. And three clients that subscribe on that.
I send a message.
First client recieve a message and call complete() action.
Will second client recieve a message?
What if there will forth client, who subscribe on it?
Question is - when will message completely remove from queve/topic/subscription??
P.S. Case when "one-to-one" - is clear.
I would recommend you check out the Competing Consumers pattern. (https://msdn.microsoft.com/en-us/library/dn568101.aspx?f=255&MSPPError=-2147217396).
You can have many roles (i.e. Azure Worker Role) checking for messages in a queue (competing consumers), locking them exclusively while processing. Each role is fighting for a message, and the first guy who grabs it (by chance), has it "exclusive" for the moment. If the consumer that gets and processes the message succeeds, run the Complete() method, otherwise the Abandon(message). Complete() finishes it for good, and Abandon throws it back into the frenzy of competing consumers. You can even grab it again if you're healthy!
You can set the dead message parameter in the Azure Management Portal which determines how many times it can be reintroduced for other consumers. At some point, things just aren't working, so kill the message so other messages can resume unimpeded.
Let me know if you have more specific needs. Would be happy to help. This pattern works extremely well.
Kindest regards...
I am using an Azure queue and have several different processes reading from the queue.
My system is built in a way that assumes each message is read only once.
This Microsoft article claims Azure queues have an at least once delivery guarantee which potentially means two processes can read the same message from the queue.
This StackOverflow thread claims that if I use GetMessage then the message becomes invisible to all other processes for the invisibility timeout.
Assuming I use GetMessage() and never exceed the message invisibility time before I DeleteMessage, can I assume I will get each message only once?
I think there is a property in queue message named DequeueCount, which is the number of times this message has been dequeued. And it's maintained by queue service. I think you can use this property to identify whether your message had been read before.
https://learn.microsoft.com/en-us/dotnet/api/azure.storage.queues.models.queuemessage.dequeuecount?view=azure-dotnet
No. The following can happen:
GetMessage()
Add some records in a database...
Generate some files...
DeleteMessage() -> Unexpected failure (process that crashes, instance that reboots, network connectivity issues, ...)
In this case your logic was executed without calling DeleteMessage. This means, once the invisibility timeout expires, the message will appear in the queue and be processed once again. You will need to make sure that your process is idempotent:
Idempotence is the property of certain operations in mathematics and
computer science, that they can be applied multiple times without
changing the result beyond the initial application.
An alternative solution would be to use Service Bus Queues with the ReceiveAndDelete mode (see this page under How to Receive Messages from a Queue). If you receive the message it will be marked as consumed and never appear again. This way you can be sure it is delivered At-Most-Once (see the comparison with Storage Queues here). But then again, if something happens while your are processing the message (ie: server crashes, ...), you could loose valuable information.
Update:
This will simulate an At-Most-Once in storage queues. The message can arrive multiple times via GetMessage, but will only be processed once by your business logic (with the risk that some of your business logic will never execute).
GetMessage()
DeleteMessage()
AddRecordsToDatabase()
GenerateFiles()