I want to run a function when my azure queue reaches x length or has not been looked at for y minutes. ie:
I have a queue myQueue, which a webhook is adding messages to. I want to process the queue if it reaches X messages in length or if it hasn't been processed for Y minutes (this is incase there are less than X messages sat in queue).
Am I missing something obvious? I can see queue triggers, but that is for every message that enters queue (I think?) or timer triggers that will do my time part but not a mix.
Any ideas?
The Queue Trigger is to trigger a function whenever a message arrives into a queue. There's no 'batch' queue trigger, so you either work with a time trigger that runs every x seconds and check for the queue length and start processing, or you live with the original queue trigger.
Related
I have an application that lets users upload upto 200 documents/resumes.
The objective of the application is to return a parsed and scored result for each of these documents.
The front end splits these 200 documents into batches of 10. i.e 20 messages are put into a queue(RabbitMQ).
I have a 6 worker processes listening to the queue( scripts that are triggered by an entry_point).
The workers take the message and splits the resumes if it is a batch message. If it is not a batch message, the worker starts processing the message. ( the average time for the processing is around 8 secs).
The queue gets piled up with 200 resumes and the 6 workers get 5 messages. Processing each message sequentially.
Which means, if another user uploads even 1 resume,one of the workers needs to reach the end of the queue to pick that message and the user is left waiting till the processing of the 200 resumes.
I'm doing this using Rabbitmq,and python2.7.
I'm using a blockingconnection to connect to the queue and process the messages.
The only way to get to the last user's message is to complete the processing of all the message as fast as i can, which could mean more processes or more containers. When i fire up more proceatses using multiprocessing (pool of 6 workers), the cpu utilization is at the highest and cannot handle any more messages.
How can i prevent my users from waiting for the response. Is adding more workers to listen and consume from the queue the only way?
The consumer is just a plain consumer with no API. The tasks are directly picked from the queue and processed.
More workers i add, the faster the queue is consumed. But still the user that had uploaded probably last still has to wait for a long time.
I have tried using npm packages like sqs-queue-parallel & sqs-consumer for consuming messages of SQS in node
But lately I have mechanism where when error happens for a particular message while processing, it should be moved to dead letter queue
But as of now it keeps on retrying the message by maximum receive count times
Is it possible with some other npm package, were whenever an error happens it should be moved directly to dead letter queue?
Know this is a bit late but think OP is trying to ask for a dynamic policy. I.e.:
on normal errors -> retry as per redrive-policy.
However, for certain failures you might know you can't recover even if you try it a hundred items. In that case -> move message directly to dead letter queue.
How to do the latter if presumably what is asked.
Answer is probably to manually copy message to deadletter queue (it behaves just like any other queue in that regard) and remove message from source queue afterwards.
Don't believe there's a 'special' way to do this.
You can configure your SQS queue to move messages to your Dead Letter Queue after any number of failed message receives between 1 and 1000.
To have a message moved to the Dead Letter Queue after only one failed receive, then modify your queue's configuration and set the "Maximum Receives" value to 1. This would be part of your queue's "Redrive Policy".
See the following AWS documentation on configuring your queue:
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/SQSDeadLetterQueue.html
You dont need to use a new npm for that its automatically happens when you finish dealing with a message. for example if you been using node-consumer when you finish with the message you do:
done() //in order to remove from queue due to for success probably
or
done(err) //in order to keep in queue
so now in order to move message from queue to dead letter queue you dont need to do anything else in your code but only in you sqs console manager :
create a new queue
call it dead-messages (or whatever)
set the "Maximum Receives" value to 1 (that means after one call to
"done(error)" the message will removed from your queue and go to the dead queue.
refresh!!!!
go back to your source queue (original one)
go to configure queue
set retrieve policy
put the name that you gave to the dead letter queue
thats it! good luck and i have to say the sqs is great way to scale tasks.
I think OP is asking if there is a way to move messages to DLQ after "A SINGLE FAILURE" in processing the message. As per these 2 SQS documentations I see these 2 points:
if the source queue has a redrive policy with maxReceiveCount set to 5, and the consumer of the source queue receives a message 6 times without ever deleting it, Amazon SQS moves the message to the dead-letter queue (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html).
Maximum receives value must be between 1 and 100 (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue.html)
Which means that even if you set Maximum receives value to 1, your consumer would still receive the message "AT-LEAST TWICE"
I am not able to find any solution where you can move the failed message to DLQ after a single failure. Would love to hear other people's thoughts on this
I am using QueueDescription.MessageCount to get the number of messages remaining in one of my queue's how ever it is showing a positive number but i cannot receive any messages.
I have checked the active and deadletter queue but there is nothing to receive. I have tried to use Recieve(); RecieveBatch(); onMessage(); - All the same.
Has anyone else seen this? Are the messages stuck or is the count incorrect.
Thanks
Steve.
The only reason I can think of is if you have received() the message but did not call complete() or abandon(). In this case the message is "hidden" for a specific length of time (default is 30 seconds). During this time, your message count would be positive even though you cannot receive any messages.
When 30 seconds (the default) elapses, the messages become visible and you may receive() them again. Note that the .DeliveryCount property is incremented every time you receive the message until it reaches 10 (the default) and the message is dumped into the dead-letter sub queue.
Be sure to always call complete() on any message you have processed. Call abandon() if you want it to be immediately available in the queue for another worker to process. Also check the DeliveryCount property to be sure you are not processing a message twice. (This can happen if you take a long time processing a message before calling complete() and by then the lock has been released.)
I know that if a worker fails to process a message off of the queue that it will become visible again and you have to code against this (idempotent). But is it possible that a worker can dequeue a message twice? Based on my logging, I seem to be seeing this behavior and I'm not sure why. I'm even deleting the message in between going go get the next message and it seems like I got it again.
Yes, you can dequeue same message twice. This can happen for two reasons:
Worker A dequeues Message B and invisibility timeout expires. Message B becomes visible again and Worker C dequeues Message B, invalidating Worker A's pop receipt. Worker A finishes work, goes to delete Message B and error is thrown. This is most common.
In certain conditions (very frequent queue polling) you can get the same message twice on a GetMessage. This is a type of race condition that while rare does occur. Worker A and B are polling very quickly and hit the queue simultaneously and both get same message. This used to be much more common (SDK 1.0 time frame) under high polling scenarios, but it has become much more rare now in later storage updates (can't recall seeing this recently).
That being said - if you only have 1 worker popping messages, then you are queueing message twice. 1 and 2 only happen when you have more than 1 worker.
You shouldn't be able to dequeue it twice. And if I recall things properly, even deleting it twice shouldn't be possible because the pop receipt should change after the second dequeue and lock.
As SilverNinja suggests, I'd look to see if perhaps the message was inadvertantly queued twice.
Do you have more than one worker role?
It is possible (especially with processes that take a while) that the timeout on the queue item visibility could end before your role has finished processing whatever it is doing. In this case another identical role could pick up the same message (which is effectively what you need to allow for - you do not want it to be a problem if the same message is processed multiple times).
At this point the first role will finish and dequeue the message and then the other role that picked it up after the timeout will end and attempt to dequeue the message. Off the top of my head I don't recall what exactly happens when a role attempts to dequeue an already dequeued message.
I would like to make sure that I don't insert a message to the queue multiple times. Is there any ID/Name I can use to enforce uniqueness?
vtortola pretty much covered it, but I wanted to add a bit more detail into why it's at least once delivery.
When you read a queue item, it's not removed from the queue; instead, it becomes invisible but stays in the queue. That invisibility period defaults to 30 seconds (max: 2 hours). During that time, the code that got the item off the queue has that much time to process whatever command was in the queue message and delete the queue item.
Assuming the queue item is deleted before the timeout period is reached, all is well. However: Once the timeout period is reached, the queue item becomes visible again, and the code holding the queue item may no longer delete it. In this case, someone else can read the same queue message and re-process that message.
Because of the fact a queue message can timeout, and can re-appear:
Your queue processing must be idempotent - operations on a queue message must result in the same outcome (such as rendering a thumbnail for a photo).
You need to think about timeout adjustments. You might find that commands are valid but processing is taking too long (maybe your 45-second thumbnail rendering code worked just fine until someone uploaded a 25MP image)
You need to think about poison messages - those that will never process correctly. Maybe they cause an exception to be thrown or have some invalid condition that causes the message processor to abort processing, which leads to the message eventually re-appearing in the queue. There's a property callded DequeueCount - consider viewing that property upon reading a queue item and, if equal to, say, 3, push the message into a table or blob and send yourself a notification to spend some time debugging that message offline.
More details on the get-queue low-level REST API is here. This will give you more insight into Windows Azure queue message handling.
Azure queues doesn't ensure message order and either message uniqueness. Messages will be processed "at least once", but nothing ensures it won't be processed twice, so it doesn't ensure "at most once".
You should get ready to receive the same message twice. You can put an ID in the body of the message as part of your data.