I would like to make sure that I don't insert a message to the queue multiple times. Is there any ID/Name I can use to enforce uniqueness?
vtortola pretty much covered it, but I wanted to add a bit more detail into why it's at least once delivery.
When you read a queue item, it's not removed from the queue; instead, it becomes invisible but stays in the queue. That invisibility period defaults to 30 seconds (max: 2 hours). During that time, the code that got the item off the queue has that much time to process whatever command was in the queue message and delete the queue item.
Assuming the queue item is deleted before the timeout period is reached, all is well. However: Once the timeout period is reached, the queue item becomes visible again, and the code holding the queue item may no longer delete it. In this case, someone else can read the same queue message and re-process that message.
Because of the fact a queue message can timeout, and can re-appear:
Your queue processing must be idempotent - operations on a queue message must result in the same outcome (such as rendering a thumbnail for a photo).
You need to think about timeout adjustments. You might find that commands are valid but processing is taking too long (maybe your 45-second thumbnail rendering code worked just fine until someone uploaded a 25MP image)
You need to think about poison messages - those that will never process correctly. Maybe they cause an exception to be thrown or have some invalid condition that causes the message processor to abort processing, which leads to the message eventually re-appearing in the queue. There's a property callded DequeueCount - consider viewing that property upon reading a queue item and, if equal to, say, 3, push the message into a table or blob and send yourself a notification to spend some time debugging that message offline.
More details on the get-queue low-level REST API is here. This will give you more insight into Windows Azure queue message handling.
Azure queues doesn't ensure message order and either message uniqueness. Messages will be processed "at least once", but nothing ensures it won't be processed twice, so it doesn't ensure "at most once".
You should get ready to receive the same message twice. You can put an ID in the body of the message as part of your data.
Related
When a message is retrieved from an azure queue but not deleted from it, the messages visibility timeout expires and the message is (re)added to the end of the queue.
Is there a way to return such messages to the head of the queue instead?
When Azure Queue messages re-appear, they don't necessarily get sent to the end of the queue. They just reappear, and at that point, no real guarantee of order. It doesn't even get moved from its current position; it's just visible again. Azure storage queues aren't set up for guaranteed order. So no, there's no way to force a message to appear at the head of the queue when it reappears after its invisibility timeout expires.
Also, check out this forum answer from Jai Haridas regarding queue message ordering. Specifically:
The messages in a queue today are sorted by its visibility time. So the ordering of messages purely depends on when they are made visible. However, it is important for an app to not assume FIFO order or any specific order as it may change in future. You can only rely that 1) a message will be eligible based on its visibility timeout and 2) Message processing should be made idempotent and use the new UpdateMessage to save state
UpdateMessage() allows you to modify the queue message (e.g.adding breadcrumbs), so the next time you start processing it, you can pick up at a point beyond "start." Note that you can also adjust the timeout value, while it's still in your possession and invisible, to allow you to keep working on the message.
I am using QueueDescription.MessageCount to get the number of messages remaining in one of my queue's how ever it is showing a positive number but i cannot receive any messages.
I have checked the active and deadletter queue but there is nothing to receive. I have tried to use Recieve(); RecieveBatch(); onMessage(); - All the same.
Has anyone else seen this? Are the messages stuck or is the count incorrect.
Thanks
Steve.
The only reason I can think of is if you have received() the message but did not call complete() or abandon(). In this case the message is "hidden" for a specific length of time (default is 30 seconds). During this time, your message count would be positive even though you cannot receive any messages.
When 30 seconds (the default) elapses, the messages become visible and you may receive() them again. Note that the .DeliveryCount property is incremented every time you receive the message until it reaches 10 (the default) and the message is dumped into the dead-letter sub queue.
Be sure to always call complete() on any message you have processed. Call abandon() if you want it to be immediately available in the queue for another worker to process. Also check the DeliveryCount property to be sure you are not processing a message twice. (This can happen if you take a long time processing a message before calling complete() and by then the lock has been released.)
I know that if a worker fails to process a message off of the queue that it will become visible again and you have to code against this (idempotent). But is it possible that a worker can dequeue a message twice? Based on my logging, I seem to be seeing this behavior and I'm not sure why. I'm even deleting the message in between going go get the next message and it seems like I got it again.
Yes, you can dequeue same message twice. This can happen for two reasons:
Worker A dequeues Message B and invisibility timeout expires. Message B becomes visible again and Worker C dequeues Message B, invalidating Worker A's pop receipt. Worker A finishes work, goes to delete Message B and error is thrown. This is most common.
In certain conditions (very frequent queue polling) you can get the same message twice on a GetMessage. This is a type of race condition that while rare does occur. Worker A and B are polling very quickly and hit the queue simultaneously and both get same message. This used to be much more common (SDK 1.0 time frame) under high polling scenarios, but it has become much more rare now in later storage updates (can't recall seeing this recently).
That being said - if you only have 1 worker popping messages, then you are queueing message twice. 1 and 2 only happen when you have more than 1 worker.
You shouldn't be able to dequeue it twice. And if I recall things properly, even deleting it twice shouldn't be possible because the pop receipt should change after the second dequeue and lock.
As SilverNinja suggests, I'd look to see if perhaps the message was inadvertantly queued twice.
Do you have more than one worker role?
It is possible (especially with processes that take a while) that the timeout on the queue item visibility could end before your role has finished processing whatever it is doing. In this case another identical role could pick up the same message (which is effectively what you need to allow for - you do not want it to be a problem if the same message is processed multiple times).
At this point the first role will finish and dequeue the message and then the other role that picked it up after the timeout will end and attempt to dequeue the message. Off the top of my head I don't recall what exactly happens when a role attempts to dequeue an already dequeued message.
I've got a scheduler and some workers in Azure. The scheduler puts messages into a queue and the workers pull those messages and work on them. I've now just come into a scenario where I will need to move some data from table storage to our database once a certain threshold has been reached. These items need to be processed in order, oldest first. Once that threshold is met all the other items are processed in order. The current message that triggered the transfer needs to be stuffed at the end of the line and be reprocessed.
So, to the meat of my question...
Is it fine to simply resend the message to the queue as is or is there a potential for that to cause problems?
queueProvider.SendMessage(message);
A co-worker mentioned that he "though he might have read something about needing to do something special." I haven't seen anything to confirm his suspicions yet however so I thought I would pose the question here just to be safe.
The short answer is that it is fine. If you have a CloudQueueMessage, you can just send it to any queue (it is just a REST request at the end of the day). Every time you AddMessage(), it creates a new ID (might be same pop receipt but that doesn't matter). That being said, there are some things you might want to take care of and or investigate:
If you push a message onto one queue, pop it, and push to another queue or same queue, you should probably delete the first message off the queue. Merely popping it means that you have set the invisibility time out, but that it will reappear soon (and you now have identical message content on each queue). So, if I pop a message and immediately push it again, I now have 2 messages in the queue with identical content.
You can now update messages. This might be appropriate for you if you need ordering. You can indicate on the message itself in metadata or content what stage of processing it is in and you get some ordering here with a thoughtful implementation.
It is recommended that all logic inside the consumer of the queue be idempotent since a message can actually be picked up more than once. We have to keep in mind that the queue service guarantees that a message will be delivered, AT LEAST ONCE - so you could end up duplicating messages with this approach.
Using azure table storage I read about Entity Group Transactions within the same partition. Now what happens if I use the Azure Queue together with table storage. Is it possible to get handle a message from the queue, insert into table storage. If something breaks, rollback and put the message on the queue again?
Or how should I handle such a scenario with Azure
Tables and queues don't have any associated transactions.
Here's some general queue usage guidance:
Make sure queue actions are idempotent - that is, you'd have the same results executing a queue message more than once, with repeatable side-effects
Set a reasonalbe queue message visibility timeout. If your task looks like it will take longer, you can extend a message's invisibility timeout. This prevents other threads / role instances from grabbing the same queue item while you're still working on it.
For long-running tasks (or those where you want to avoid consuming a resource multiple times if possible), modify your queue message along the way, giving yourself status hints. For example: you have a render-video queue message: 'RENDER|Source-URL'. you're rendering video, and it needs two passes. You've done pass 1 with results stored in a temporary blob. You could modify the message with something like 'RENDER|Source-URL|Pass1-URL'. Now, assume something goes wrong and your rendering task fails for some reason. Later, when you pick up this message again, you can start from pass 2, instead of from the very beginning.
You don't need to worry about putting messages back on the queue. Messages aren't actually removed from the queue until you explicitly delete them. They simply become invisible during the invisibility timeout period you choose. If you don't delete by the end of that period (or extend the period), the message becomes visible again for someone else to read. Note: At this point, once someone else reads the queue message, the original message-holder will no longer be able to delete the message.