I noticed CloudQueue.ApproximateMessageCount would return the number of messages including the expired ones. This is probably a bug. Is there any way to see how many messages are in the queue?
So after doing some digging around I think I found the behavior you're talking about. From what I can tell the messages are still in the queue when they expire but aren't retrievable. They seem to remain in there for a short period of time and then are cleared out.
If I had to guess it may be similar to a storage bus queue in that expired messages are moved to some sort of dead letter queue. Except with storage queues you can't access the dead letter queue and the dead letter queue is automatically cleared after some period of time.
I'll update this answer if I find more.
Edit
I confirmed the behavior. It seems expired messages remain in the queue but you can't interact with them. They disappear eventually without intervention it appears.
Related
It is possible to set up TTL for messages in Azure Service Bus. I am wondering if there is a possibility to do the same for Dead Letter Queue?
What I want to achieve is "auto-cleaning" of the DLQ of the old messages that are probably not relevant anymore anyway, so that we don't need to do this manually (which is not supported out of the box either).
What I want to achieve is "auto-cleaning" of the DLQ of the old messages that are probably not relevant anymore anyway so that we don't need to do this manually (which is not supported out of the box either).
You can receive and delete messages from the dead-letter queue, but you cannot set up a TTL on the dead-lettered messages as those are created and moved into the sub-queue by the service. While the dead-letter queue mimics the regular queue in many concepts, it is not.
One of the semi-automated would be to create a process that peeks messages and completes based on the criteria you define, such as message age. Unfortunately, there's no good way to peek at messages in general. Not much can be done for the dead-lettered messages, other than peeking all and then filtering out those that need to be actioned.
Another alternative is to transition those dead-lettered messages into a database and then have a process to retire based on the defined criteria w/o the need to peek at all of the messages constantly.
I have configured service bus and I am sending messages to a topic in it.
I am observing a strange behavior that my messages are going to dead letter queue and not the active queue.
I have checked the properties for my topic like the auto delete on idle, default time to live but not able to figure out the reason.
I tried turning off my listener on this topic hoping some code failure causing the messages to go to dead letter. But still not able to figure out the reason.
Inspect queue's MaxDeliverCount. If dead-lettered messages exceed that value, it's an indication your code was failing to process the messages and they were dead-lettered for that reason. The reason is stated in the DeadLetterReason header. If that's the case, as suggested in the comments, log in your code the reason of failure to understand what's happening.
Additional angle to check if your message is getting aborted. This could happen when you use some library or abstraction on top of Azure Service Bus client. If it is, it will eventually get dead-lettered as well. Just like in the first scenario, you'll need some logs to understand why this is happening.
I am using rabbitmq 3.7.0 with nodejs using this library in typescript. All consumers are {noAck: false, exclusive: false}, all the queues are {prefetch:1, expires: 14400000, durable:false} (4 hours).
I have a lot of queues that does not have any pending ack but still holds hundreds of messages. Beside this, the queues do not expire when it should, what will generate performance issues after a couple of days. I haven't found any justification on why the message still in the queue after the ack process.
Thanks
Edit (1) - missing information I've perceived from the answers
As far as I could assess, there is no active consumer for such number of queues. Is it possible to confirm this information in the management tool?
What complicates the scenario is that I can't simply delete the queue after the last consumer because I could use it to recover a broken session, and if I don't delete de queue after the last consumer, the queue continues to be updated. But the expires parameter should suffice for this.
Some snapshots from the management tool:
I suggest you read the documentation for how expires works: https://www.rabbitmq.com/ttl.html#queue-ttl
If your queue is not being expired it means you most likely have a consumer registered on it. There are a couple other cases where queues won't be expired.
As for the messages that are supposedly acknowledged but remain in the queue the only explanation I have is that perhaps they have not been delivered at all.
Finally, without more information or a reliable reproducer there's not much further assistance that can be given with the limited amount of information provided.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
It seems that new azure SDK extends the visibilitytimeout to <= 7 days. I know by default, when I add a message to an azure queue, the live time is 7days. When I get message out, and set the visibilitytimeout to 7days. Does that mean I don't need to delete this message if I don't care about message reliable? the message will disappear later 7 days.
I want to take this way because DeleteMessage is very slow. If I don't delete message, doesn't it have any impact on performance of GetMessage?
Based on the documentation for Get Messages, I believe it is certainly possible to set the VisibilityTimeout period to 7 days so that messages are fetched only once. However I see some issues with this approach instead of just deleting the message once the process is done:
What happens when you get the message and start processing it and somehow the process fails? If you set the visibility timeout to be 7 days, then the message would never appear in the queue again and thus the process it was supposed to do never gets done.
Even though the message is hidden, it is still there in the queue thus you keep on incurring storage charges for that message. Even though the cost is trivial but why keep the message when you don't really need it.
A lot of systems rely on Approximate Messages Count property of a queue to check on the health of processes which are performed by messages in a queue. Please note that even though you make the message hidden, it is still there in the queue and thus will be included in total messages count in the queue. So if you're building a system which relies on this for health check, you will always find your system to be unhealthy because you're never deleting the messages.
I'm curious to know why you find deleting messages to be very slow. In my experience this is quite fast. How are you monitoring message deletion?
Rather than hacking around the problem I think you should drill into understanding why the deletes are slow. Have you enabled logs and looked at the e2elatency and serverlatency numbers across all your queue operations. Ideally you shouldn't be seeing a large difference between the two for all your queue operations. If you do see a large difference then it implies something is happening on the client that you should investigate further.
For more information on logging take a look at the following articles:
http://blogs.msdn.com/b/windowsazurestorage/archive/tags/analytics+2d00+logging+_2600_amp_3b00_+metrics/
http://msdn.microsoft.com/en-us/library/azure/hh343262.aspx
Information on client side logging can also be found in this post: e client side logging – which you can learn more about in this blog post. http://blogs.msdn.com/b/windowsazurestorage/archive/2013/09/07/announcing-storage-client-library-2-1-rtm.aspx
Please let me know what you find.
Jason
I have a simple work role in azure that does some data processing on an SQL azure database.
The worker basically adds data from a 3rd party datasource to my database every 2 minutes. When I have two instances of the role, this obviously doubles up unnecessarily. I would like to have 2 instances for redundancy and the 99.95 uptime, but do not want them both processing at the same time as they will just duplicate the same job. Is there a standard pattern for this that I am missing?
I know I could set flags in the database, but am hoping there is another easier or better way to manage this.
Thanks
As Mark suggested, you can use an Azure queue to post a message. You can have the worker role instance post a followup message to the queue as the last thing it does when processing the current message. That should deal with the issue Mark brought up regarding the need for a semaphore. In your queue message, you can embed a timestamp marking when the message can be processed. When creating a new message, just add two minutes to current time.
And... in case it's not obvious: in the event the worker role instance crashes before completing processing and fails to repost a new queue message, that's fine. In this case, the current queue message will simply reappear on the queue and another instance is then free to process it.
There is not a super easy way to do this, I dont think.
You can use a semaphore as Mark has mentioned, to basically record the start and the stop of processing. Then you can have any amount of instances running, each inspecting the semaphore record and only acting out if semaphore allows it.
However, the caveat here is that what happens if one of the instances crashes in the middle of processing and never releases the semaphore? You can implement a "timeout" value after which other instances will attempt to kick-start processing if there hasnt been an unlock for X amount of time.
Alternatively, you can use a third party monitoring service like AzureWatch to watch for unresponsive instances in Azure and start a new instance if the amount of "Ready" instances is under 1. This will save you can save some money by not having to have 2 instances up and running all the time, but there is a slight lag between when an instance fails and when a new one is started.
A Semaphor as suggested would be the way to go, although I'd probably go with a simple timestamp heartbeat in blob store.
The other thought is, how necessary is it? If your loads can sustain being down for a few minutes, maybe just let the role recycle?
Small catch on David's solution. Re-posting the message to the queue would happen as the last thing on the current execution so that if the machine crashes along the way the current message would expire and re-surface on the queue. That assumes that the message was originally peeked and requires a de-queue operation to remove from the queue. The de-queue must happen before inserting the new message to the queue. If the role crashes in between these 2 operations, then there will be no tokens left in the system and will come to a halt.
The ESB dup check sounds like a feasible approach, but it does not sound like it would be deterministic either since the bus can only check for identical messages currently existing in a queue. But if one of the messages comes in right after the previous one was de-queued, there is a chance to end up with 2 processes running in parallel.
An alternative solution, if you can afford it, would be to never de-queue and just lease the message via Peek operations. You would have to ensure that the invisibility timeout never goes beyond the processing time in your worker role. As far as creating the token in the first place, the same worker role startup strategy described before combined with ASB dup check should work (since messages would never move from the queue).