I have a simple video encoding worker role that pulls messages from a queue encodes a video then uploads the video to storage. Everything seems to be working but occasionally when deleting the message after I am done encoding and uploading I get a "StorageClientException: The specified message does not exist." Although the video is processed, I believe the message is reappearing in the queue because it's not being deleted correctly. I have the message visablilty set to 5 mins, none of the videos have taken more than 2 to process.
Is it possible that another instance
of the Worker role is processing and
deleting the message?
Doesn't the GetMessage() prevent
other worker roles from picking up
the same message?
Am I doing something wrong in the
setup of my queue?
What could be causing this message to
not be found on delete?
some code...
//onStart() queue setup
var queueStorage = _storageAccount.CreateCloudQueueClient();
_queue = queueStorage.GetQueueReference(QueueReference);
queueStorage.RetryPolicy = RetryPolicies.Retry(5, new TimeSpan(0, 5, 0));
_queue.CreateIfNotExist();
public override void Run()
{
while (true)
{
try
{
var msg = _queue.GetMessage(new TimeSpan(0, 5, 0));
if (msg != null)
{
EncodeIt(msg);
PostIt(msg);
_queue.DeleteMessage(msg);
}
else
{
Thread.Sleep(WaitTime);
}
}
catch (StorageClientException exception)
{
BlobTrace.Write(exception.ToString());
Thread.Sleep(WaitTime);
}
}
}
If encoding process takes more time than the message invisibility timeout (5 minutes in your case), then the message will show up in the queue again. This will cause second worker to start processing it. However, chances are that by the time second worker finishes processing, first worker would already be done with the work, deleting it properly. This will cause the second worker to fail at the deletion phase, since the message no longer exists for him.
This happens due to the lightweight transactional model by Windows Azure Queues. It guarantees, that the message will be processed at least once (even if the worker fails silently), but does not guarantee "only once" processing.
Since your encoding process seems to be idempotent and lightweight (since error shows up infrequently), I'd just I advise to increase the invisibility timeout and explicitly capture this exception (by status codes) around DeleteMessages (optionally logging the process duration in order to be able to tweak invisibility timeouts further).
Is it possible it's taking longer than the five minutes you've set as a timeout?
I had my development, production and stage all pulling from the same queue this was causing some strange behavior. I believe this to be the culprit.
Related
I try to get 32 messages per request from Azure Queue.
queue.ApproximateMessageCount;
This code gives me the result of 1509. Telling me the connection is OK and it has records. Also I check in queue it really has 1509 records.
But when I try to retrieve records I don't get any record.
I do the following:
var messages = await queue.GetMessagesAsync(configuration.MessageBatchSize);
if (!messages.Any()) {
return;
}
It always goes in the if and returns.
What is going on here and what am I missing?
Do do that, receiving messages in batch mode, i use this kind of code :
var messages = await queueClient?.ReceiveBatchAsync(Max_Messages);
foreach (var message in messages)
{
await dispatcher.Dispatch(message); // do something with each message
}
But, for receiving messages with ReceiveBatchAsync, the queue have to be configured with the EnableBatchedOperations flag to true.
ApproximateMessageCount property represents the total number of messages available in queue at that particular moment. It does not represent that all messages (max #32 messages in a pull) are ready to be dequeued. You can use this property to infer that how many messages are in queue.
queue.ApproximateMessageCount;
If you could not retrieve the message by, GetMessagesAsync(numberOfMessages), then it says that all messages are not available or invisible for current QueueClient.
var cloudQueueMessages = await cloudQueue.GetMessagesAsync(numberOfMessages);
You could try polling the queue after sometime to see if messages came back to surface.
Note that, be advised of setting adequate visibility timeout for any message being dequeued to avoid indefinite starvation :)
Can someone offer some more guidance on the use of the Azure Service Bus OnMessageOptions.AutoRenewTimeout
http://msdn.microsoft.com/en-us/library/microsoft.servicebus.messaging.onmessageoptions.autorenewtimeout.aspx
as I haven't found much documentation on this option, and would like to know if this is the correct way to renew a message lock
My use case:
1) Message Processing Queue has a Lock Duration of 5 minutes (The maximum allowed)
2) Message Processor using the OnMessageAsync message pump to read from the queue (with a ReceiveMode.PeekLock) The long running processing may take up to 10 minutes to process the message before manually calling msg.CompleteAsync
3) I want the message processor to automatically renew it's lock up until the time it's expected to Complete processing (~10minutes). If after that period it hasn't been completed, the lock should be automatically released.
Thanks
-- UPDATE
I never did end up getting any more guidance on AutoRenewTimeout. I ended up using a custom MessageLock class that auto renews the Message Lock based on a timer.
See the gist -
https://gist.github.com/Soopster/dd0fbd754a65fc5edfa9
To handle long message processing you should set AutoRenewTimeout == 10 min (in your case). That means that lock will be renewed during these 10 minutes each time when LockDuration is expired.
So if for example your LockDuration is 3 minutes and AutoRenewTimeout is 10 minutes then every 3 minute lock will be automatically renewed (after 3 min, 6 min and 9 min) and lock will be automatically released after 12 minutes since message was consumed.
To my personal preference, OnMessageOptions.AutoRenewTimeout is a bit too rough of a lease renewal option. If one sets it to 10 minutes and for whatever reason the Message is .Complete() only after 10 minutes and 5 seconds, the Message will show up again in the Message Queue, will be consumed by the next stand-by Worker and the entire processing will execute again. That is wasteful and also keeps the Workers from executing other unprocessed Requests.
To work around this:
Change your Worker process to verify if the item it just received from the Message Queue had not been already processed. Look for Success/Failure result that is stored somewhere. If already process, call BrokeredMessage.Complete() and move on to wait for the next item to pop up.
Call periodically BrokeredMessage.RenewLock() - BEFORE the lock expires, like every 10 seconds - and set OnMessageOptions.AutoRenewTimeout to TimeSpan.Zero. Thus if the Worker that processes an item crashes, the Message will return into the MessageQueue sooner and will be picked up by the next stand-by Worker.
I have the very same problem with my workers. Even the message was successfully processing, due to long processing time, service bus removes the lock applied to it and the message become available for receiving again. Other available worker takes this message and start processing it again. Please, correct me if I'm wrong, but in your case, OnMessageAsync will be called many times with the same message and you will ended up with several tasks simultaneously processing it. At the end of the process MessageLockLost exception will be thrown because the message doesn't have a lock applied.
I solved this with the following code.
_requestQueueClient.OnMessage(
requestMessage =>
{
RenewMessageLock(requestMessage);
var messageLockTimer = new System.Timers.Timer(TimeSpan.FromSeconds(290));
messageLockTimer.Elapsed += (source, e) =>
{
RenewMessageLock(requestMessage);
};
messageLockTimer.AutoReset = false; // by deffault is true
messageLockTimer.Start();
/* ----- handle requestMessage ----- */
requestMessage.Complete();
messageLockTimer.Stop();
}
private void RenewMessageLock(BrokeredMessage requestMessage)
{
try
{
requestMessage.RenewLock();
}
catch (Exception exception)
{
}
}
There are a few mounts since your post and maybe you have solved this, so could you share your solution.
We were load testing an Azure queue using multiple processes. We noticed that all of the machines would simultaneously pause while sending messages to the queue.
The pauses are of different length. Most frequent are half second pauses, but we've seen pauses on all senders up to a hand-full of seconds.
Since I originally posted, we've seen the same behavior with a Java client using the HTTP bindings.
Additionally, we've seen pauses which affect every instance except one. While several of the executing programs will stop simultaneously, one will continue to send at the same rate.
In all cases, after the pause is over, we see all of the instances resume at the old rate. We start all the instances by hand over a period of several seconds.
The .NET code looks like:
using (var memoryStream = new MemoryStream())
{
using (var streamWriter = new StreamWriter(memoryStream))
{
streamWriter.Write(messageText);
streamWriter.Flush();
memoryStream.Position = 0;
var message = new BrokeredMessage(memoryStream, false);
message.Properties["Name"] = "DeviceStatusProbed";
message.Properties["MessageId"] = messageIdText;
sender.SendAsync(message);
if (messageNumber == 0)
{
stopwatch.Start();
}
messageNumber++;
Console.WriteLine("MPS: " + 1000 * messageNumber / (double)stopwatch.ElapsedMilliseconds);
Console.WriteLine("Sent message with MessageID = " + message.MessageId);
}
}
What are the potential causes in the Azure environment for such freezes? If the cause isn't expected, is there tooling or guidance available to help us determine the cause?
Thanks.
can you provide more information about how long is the "pause".. do you see a failure in the Send call ?? How long did you see the pause ?
There isn't any reason why you should see such pauses .
It is possible that the call times out after 60 seconds (default) or just fails if there is an upgrade in SERVICEBUS or any Azure infrastructure. But upon retry it should succeed.
It might be an issue with the stream as you are wrapping the MemoryStream in a using statement.
Since the SendAsync is called asynchronously the memory stream might have already been disposed before being sent on the service bus.
Can you try and remove the using statement on the memory stream and instead call:
var message = new BrokeredMessage(memoryStream, ownsStream: true);
I'm using the Azure Service Bus SubscriptionClient.OnMessage method; configured to process up to 5 messages concurrently.
Within the code I need to wait for all messages to finish processing before I can continue (to properly shutdown an Azure Worker Role). How do I do this?
Will SubscriptionClient.Close() block until all messages have finished processing?
Calling Close on SubscriptionClient or QueueClient will not block. Calling Close closes off the entity immediately as far as I can tell. I tested quickly just using the Worker Role With Service Bus Queue project template that shipped with Windows Azure SDK 2.0. I added a thread sleep for many seconds in the message process action and then shut down the role while it was running. I saw the Close method get called while the messages were processing in their thread sleep but it certainly did not wait for the for message processing to complete, the role simple closed down.
To handle this gracefully you'll need to do the same thing we did when dealing with any worker role that was processing messages (Service Bus, Azure Storage queue or anything else): keep track of what is being worked on and shut down when it is complete. There are several ways to deal with that but all of them are manual and made messy in this case because of the multiple threads involved.
Given the way that OnMessage works you'll need to add something in the action that looks to see if the role has been told to shutdown, and if so, to not do any processing. The problem is, when the OnMessage action is executed it HAS a message already. You'd probably need to abandon the message but not exit the OnMessage action, otherwise it will keep getting a message if there are ones in the queue. You can't simply abandon the message and let the execution leave the action because then the system will be handed another message (possibly the same one) and several threads doing this may cause messages to get too many dequeue counts and get dead lettered. Also, you can't call Close on the SubscriptionClient or QueueClient, which would stop the receive loop internally, because once you call close any of the outstanding message processing will throw an exception when .Complete, .Abandon, etc. is called on the message because the message entity is now closed. This means you can't stop the incoming messages easily.
The main issue here is because you are using the OnMessage and setting up the concurrent message handling by setting the MaxConcurrentCalls on the OnMessageOptions, which means the code that starts and manages the threads is buried in the QueueClient and SubscriptionClient and you don't have control over that. You don't have a way to reduce the count of threads, or stop the threads individually, etc. You'll need to create a way to put the OnMessage action threads into a state where they are aware that the system is being told to shut down and then complete their message and not exit the action in order for them to not continuously be assigned new messages. This means you'll likely need to also set the MessageOptions to not use autocomplete and manually call complete in your OnMessage action.
Having to do all of this may severely reduce the actual benefit of using the OnMessage helper. Behind the scenes OnMessage is simply setting up a loop calling receive with the default timeout and handing of messages to another thread to do the action (loose description). So what you get by using the OnMessage approach is away from having to write that handler on your own, but then the problem you are having is because you didn't write that handler on your own you don't have control over those threads. Catch-22. If you really need to stop gracefully you may want to step away from the OnMessage approach, write your own Receive loop with threading and within the main loop stop receiving new messages and wait for all the workers to end.
One option, especially if the messages are idempotent (which means processing them more than once yields the same results... which you should be mindful of anyway) then if they are stopped in mid processing they will simply reappear on the queue to be processed by another instance later. If the work itself isn't resource intensive and the operations are idempotent then this really can be an option. No different than when an instance might fail due to hardware failure or other issues. Sure, it's not graceful or elegant, but it certainly removes all the complexity I've mentioned and is still something that can happen anyway due to other failures.
Note that the OnStop is called when an instance is told to shut down. You've got 5 minutes you can delay this until the fabric just shuts it off, so if your messages take longer than five minutes to process it won't really matter if you attempt to shut down gracefully or not, some will be cut off during processing.
You can tweak OnMessageAsync to wait for processing of messages to complete, and block new messages from beginning to be processed:
Here is the implementation:
_subscriptionClient.OnMessageAsync(async message =>
{
if (_stopRequested)
{
// Block processing of new messages. We want to wait for old messages to complete and exit.
await Task.Delay(_waitForExecutionCompletionTimeout);
}
else
{
try
{
// Track executing messages
_activeTaskCollection[message.MessageId] = message;
await messageHandler(message);
await message.CompleteAsync();
}
catch (Exception e)
{
// handle error by disposing or doing nothing to force a retry
}
finally
{
BrokeredMessage savedMessage;
if (!_activeTaskCollection.TryRemove(message.MessageId, out savedMessage))
{
_logger.LogWarning("Attempt to remove message id {0} failed.", savedMessage.MessageId);
}
}
}
}, onMessageOptions);
And an implementation of Stop that waits for completion:
public async Task Stop()
{
_stopRequested = true;
DateTime startWaitTime = DateTime.UtcNow;
while (DateTime.UtcNow - startWaitTime < _waitForExecutionCompletionTimeout && _activeTaskCollection.Count > 0)
{
await Task.Delay(_waitForExecutionCompletionSleepBetweenIterations);
}
await _subscriptionClient.CloseAsync();
}
Note that _activeTaskCollection is a ConcurrentDictionary (we can also use a counter with interlock to count the number of in progress messages, but using a dictionary allows you to investigate what happend easily in case of errors.
Problem:
I am reading messages from a Azure Storage Queue and then inserting them into a Storage Table using a Worker Role.
I want to read in messages but only process them if there are at least 100 (this is to optimize the Storage Table batch insert which is occurring). If there are less than 100 messages, then I want to cancel the message processing and make them immediately visible on the queue again for the next queue read.
Question:
Is it possible to mark a message which has just been read by CloudQueue.GetMessages(...) as visible without having to wait for the timeout to expire?
Code: (in WorkerRole.cs)
public override void Run()
{
while (true)
{
var messages = queue.GetMessages(100);
if (messages.Count() >= 100)
{
// This will process, insert into a table, and delete from the queue
ProcessMessages(messages);
}
else
{
//!!! MARK MESSAGES AS VISIBLE ON THE QUEUE
System.Threading.Thread.Sleep(1000);
}
}
}
Thanks
You can check the queue's `ApproximateMessageCount' property (details here), which will give you a rough idea how many messages are waiting in the queue.
Also: you can set a message's invisibility timeout to something small (maybe 5-10 seconds?). After that period, the message becomes visible again. You can also modify invisibility timeout to something shorter after you read it.
Just remember that reading from the queue counts as a transaction, as does updating messages (e.g. updating invisibility timeout).
Waiting for 100 messages may be a non-optimal optimization. Oh, and GetMessages()(details here) is limited to 32 messages, so it doesn't make sense to wait for 100. Also: Transactions are really, really cheap (a penny per 100K transactions). I don't necessarily see the value here.
Reset the expire time to 0.0. That will hopefully do the trick.