I'm creating a consumer of an Azure Service Bus topic (subscription) that does nothing but store some statistics. The messages sent to the topic contains a rather large body, that is handled by another consumer on a second subscription on the same topic.
Since the statistics consumer can handle a large number of messages in one go, I was wondering if it is possible to receive a lot of messages but leave out the body to improve performance when communicating with Service Bus and to receive even more messages in one go.
I'm currently doing this:
this.messageReceiver = new MessageReceiver(conn, path);
...
await messageReceiver.ReceiveAsync(10, TimeSpan.FromSeconds(5));
It works pretty sweet but it would be nice to be able to receive 100 or more messages, without having to worry about moving large messages over the network.
Before anyone suggests it, I already know that I can ask for a count, etc. on a topic subscription. I still need the Message object since that contains an entry in the UserProperties dictionary that is used to calculate the stats.
Not possible. You can peek, but that brings the whole payload and headers w/o incrementing the DeliveryCount of the message. You could request it as a broker feature here.
Related
We have a Topic with Subscriptions with a default LockDuration of 1min, and multiple SubscriptionClients listening to each subscription. For our test purposes, there are 3 clients listening to a single subscription.
SubscriptionClients are created as:
Client = new SubscriptionClient(endPoint, topicName, subscriptionName);
We put one message on the Topic, which is filtered into the Subscription.
We would expect one of the SubscriptionClients to pick up the message, and the other two clients cannot because it is locked.
What is actually happening, is all three clients are simultaneously picking up the same message, with different DeliveryCounts, and all within the 1minute lock duration.
Is there something wrong with the way we're creating the SubscriptionClient such that the lock is shared between them rather than being exclusive?
There are possibly two things that could be wrong. And none of those would be the broker but likely the client-side code.
MaxLockDuration is too short and while one client is still working on the message, the other client(s) receives that same message. You should be able to confirm by looking at the duration of the message processing. If it exceeds MaxLockDuration set on the queue, that's it.
You're using a message handler with automatic lock renewal and that one is failing to extend the lock. In that case, you would have a message handler error callback raised with the details.
Either way, you could log the errors and share the logs if possible to help with pinpointing what the issue is.
I have 65k records in Azure Service Bus Topic, while testing, whenever my test application is started, it reads all the 65k records. Can you please help me how can we avoid reading messages that have already read or How can we read only the messages that are send after executing test application?
From the question, it's unclear what exactly you're after. Here are a few things for consideration.
Queues/subscriptions are intended to be read by the consumers, not to store messages and access conditionally. To avoid consuming messages, you should consume those either by using ReceiveAndDelete receiving more, or PeekLock and completing the received messages.
If these messages are test messages and are not intended for the production, do not mix the environments and use different namespaces.
Alternatively, set a short TimeToLive on your test messages to get rid of those. You could also drop the entity and recreate it, but I try to avoid this if your performing testing quite often.
My service consumes messages from an Azure Service Bus subscription. A dependency of my service was down for a while, which caused a lot of messages to end up in the deadletter queue (DLQ). Now that the service is back up, I want to reprocess all messages from the DLQ. How can I move/resubmit all messages from the DLQ back in to the main queue.
Restrictions:
It's thousands of messages, so manually handling them isn't feasible.
The topic has about ten subscriptions. I don't want to resubmit the messages to the topic, because then all subscriptions would receive the messages, leading to double-processing.
I don't want to run the service against the DLQ directly, because some messages are broken and cause permanent errors, i.e. they would end up in the DLQ again, which would lead to an infinite loop. Moreover, the broken messages are put back at the front of the queue, effectively starving healthy messages that come after the broken ones.
I realize this is a while after the original post but if anyone else stumbles on this problem, there is a fairly handy solution baked into the Service Bus Explorer (which I have found to be incredibly handy with ASB development).
After connecting to your Service Bus and finding the needed namespace, find the desired topic and subscription with the deadletters in it. From there Right Click and Receive Deadletter Queue Messages and hit OK.
From there, highlight which you would like to send back to the main queue and hit Resubmit Selected Messages in Batch Mode.
Thomas, you probably already found your answer since this is quite awhile ago. think of DLQ (or any existing queue that you have) as just another collection variable like in a PC app, but residing on the cloud. just like a PC-app or in-memory collection variable from your tool-kit, you have many ways of utilising it. off course there are limitations and differences between these 2 types of collection variables, but that's how you design your solution as though the DLQ is just another collection variable by knowing those limitations and differences.
For some queuing implementations, one of the solutions would be to have another instance of the same app pointing to the DLQ, but with a fairly long visibility timeout (e.g. 6 or 12 or even 24 hours depending on your SLA), since you don't want to repeat them too often. However, this is not applicable to Azure service bus, as it limits the visibility timeout to at most 5 minutes.
if the DLQ contains broken un-recoverable jobs, you should fix the app to delete them based on the error messages when the unknown exception occurred. once the fix is deployed, such broken un-recoverable jobs would have been removed by your app and never get sent to the DLQ in the first place. and those already in the DLQ will be removed by the fixed app.
The only option to replay DLQ messages is to receive them from DLQ, create new message with same content and send it again to the topic. They will end up at the end of subscription queue.
You can't send messages directly to the subscription. There is a trick to add a metadata property to the message, and then adjust all except one subscription to filter out such messages. It's up to you to decide if it's going to help in your scenario.
As for tooling, we always did that with custom code, because we always needed some extra work to be done, like logging each replayed message for further analysis.
The quick answer is that you cannot directly move messages back to the main queue of a subscription. This is by design with how Microsoft implemented their topics and subscriptions.
Option #1
There is the option to use Azure Service Bus topic filters https://learn.microsoft.com/en-us/azure/service-bus-messaging/topic-filters and define/tag your messages in a manner that would only allow them to be received on the targeted subscription.
Option #2
The other option would be to change your current implementation. You would set up "delivery queues" (regular service bus queues) and configure each corresponding subscription to auto forward its messages to these delivery queues. Your message processing logic would then listen on these "delivery queues" vs the subscription. Any failures would then result in DLQ messages on these associated "delivery queues" which could then be handled outside of the topic/subscriptions.
We are working on a scenario where user requests for generating some documents (through a web app) and these requests ( a message) are sent to azure service bus queue which when processed will result in generating a document.A user can make multiple requests for same document with updated data (which is part of the message) .What we need is that if there is a latest message from the user for the same document we take that message for generating document and other older messages can be ignored. User request can be identified using a unique identifier which potentially can be message id.
Is there a way currently in azure service bus queues to implement this.
To me this looks something exactly opposite of duplicate detection in queues.
Technically, you can use Peek / PeekBatch methods to read the entire queue (on each receive of the document request message), and if a newer message with the same request / document is detected, skip the older (current) one. However depending on an average number of messages in your queue, this can be very inefficient. In my experience PeekBatch is somewhat fast (seconds, up to tens of seconds for hundreds messages) - but scanning the entire queue again and again does not feel right. Alternatively, if you can, instead of processing document requests right away. Persist the requests somewhere else externally, outside of ASB. Then periodically scan that table, and execute only the latest one, and purge the rest.
When a Web Role places a message onto a Storage Queue, how can it poll for a specific, correlated response? I would like the back-end Worker Role to place a message onto a response queue, with the intent being that the caller would pick the response up and go from there.
Our intent is to leverage the Queue in order to offload some heavy processing onto the back-end Worker Roles in order to ensure high performance on the Web Roles. However, we do not wish to respond to the HTTP requests until the back-end Workers are finished and have responded.
I am actually in the middle of making a similar decision. In my case i have a WCF service running in a web role which should off-load calculations to worker-roles. When the result has been computed, the web role will return the answer to the client.
My basic data structure knowledge tells me that i should avoid using something that is designed as a queue in a non-queue way. That means a queue should always be serviced in a FIFO like manner. So basically if using queues for both requests and response, the threads awaiting to return data to the client will have to wait untill the calculation message is at the "top" of the response queue, which is not optimal. If storing the responses by using Azure tables, the threads poll for messages creating unnecessary overhead
What i belive is a possible solution to this problem is using a queue for the requests. This enables use of the competeing consumers pattern and thereby load-balancing. On messages sent into this queue you set the correlationId property on the message. For reply the pub/sub part ("topics") part of Azure service bus is used togehter with a correlation filter. When your back-end has processed the request, it published a result to a "responseSubject" with the correlationId given in the original request. Now this response ca be retrieved by your client by calling CreateSubscribtion (Sorry, i can't post more than two links apparently, google it) using that correlation filter, and it should get notified when the answer is published. Notice that the CreateSubscribtion part should just be done one time in the OnStart method. Then you can do an async BeginRecieve on that subscribtion and the role will be notified in the given callback when a response for one of it's request is available. The correlationId will tell you which request the response is for. So your last challenge is giving this response back to the thread holding the client connection.
This could be achieved by creating Dictionary with the correlationId's (probably GUID's) as key and responses as value. When your web role gets a request it creates the guid, set it as correlationId, add it the hashset, fire the message to the queue and then call Monitor.Wait() on the Guid object. Then have the recieve method invoked by the topic subscribition add the response to the dictionary and then call Monitor.Notify() on that same guid object. This awakens your original request-thread and you can now return the answer to your client (Or something. Basically you just want your thread to sleep and not consume any ressources while waiting)
The queues on the Azure Service Bus have a lot more capabilities and paradigms including pub / sub capabilities which can address issues dealing with queue servicing across multiple instance.
One approach with pub / sub, is to have one queue for requests and one for the responses. Each requesting instance would also subscribe to the response queue with a filter on the header such that it would only receive the responses targeted for it. The request message would, of course contain the value to the placed in the response header to drive the filter.
For the Service Bus based solution there are samples available for implementing Request/Response pattern with Queues and Topics (pub-sub)
Let worker role keep polling and processing the message. As soon as the message is processed add an entry in Table storage with the required corelationId(RowKey) and the processing result, before deleting the processed message from the queue.
Then WebRoles just need to do a look up of the Table with the desired correlationId(RowKey) & PartitionKey
Have a look at using SignalR between the worker role and the browser client. So your web role puts a message on the queue and returns a result to the browser (something simple like 'waiting...') and hook it up to the worker role with SignalR. That way your web role carries on doing other stuff and doesn't have to wait for a result from asynchronous processing, only the browser needs to.
There is nothing intrinsic to Windows Azure queues that does what you are asking. However, you could build this yourself fairly easily. Include a message ID (GUID) in your push to the queue and when processing is complete, have the worker push a new message with that message ID into a response channel queue. Your web app can poll this queue to determine when processing is completed for a given command.
We have done something similar and are looking to use something like SignalR to help reply back to the client when commands are completed.