I'm moving some background processing from an Azure web role to a worker role. My worker needs to do a task every minute or so, possibly spawning off tasks:
while(true){
//start some tasks
Thread.Sleep(60000);
}
Once I deploy, it will start running forever. So later, when I redeploy, how does Azure stop my process for redeployment?
Does it just kill it instantly? Is there a way to get a warning that it's shutting down? Do I just have to make sure everything is transactional?
When a role (either worker or web) is asked to gracefully shut down (because it is being scaled down or because you've asked for a redeployment) the OnStop method of the RoleEntryPoint class is called. This is the same class which has the Run method which likely either contains your loop or calls the code that contains that loop.
A couple of things to note here: The OnStop has 5 minutes to actually stop, after that the process is simply killed. If you have to call something else to shut down asynchronously, you'll need the thread in OnStop to be kept busy waiting until that other process is shut down. Once execution has left OnStop the platform assumes the machine can be shut down.
If you need to gracefully stop processing but it not require a shutdown of the machine then you can put a setting in the service config file that you can update to indicate work should be done or note. So for example a bool that says "ProcessQueues". Then in your onStart in RoleEntryPoint you hook the RoleEnvironmentChanging event. Your event handler then looks for a RoleEnvironmentConfigurationSettingChange to occur and then checks the ProcessQueues bool. If it is true it either starts up or continues processing, if it is false it stop the processing gracefully. You can then do a config change to control when things are running or not. This is one option of handling this and there are many more depending on how quickly you need to stop processing, etc.
Related
I'm using ServiceStack MQ (ServiceStack.Aws.Sqs.SqsMqServer v4.0.54).
I'm running MQ server inside a Windows Service.
My Goal:
When the Windows service is about to shutdown, I would like to
wait for all running workers to finish processing and then terminate
the MqServer.
Problem:
The ServiceStack MqServer (whether it's Redis/RabbitMq/Sqs) has a Stop() method. But it does not block until all workers complete their work. It merely
pulses the background thread to stop the workers and then it returns.
Then the Windows Service process stops, and existing workers get aborted.
This is the link to github source code -> https://github.com/ServiceStack/ServiceStack/blob/75847c737f9c0cd9f5dd4ea3ae1113dace56cbf2/src/ServiceStack.RabbitMq/RabbitMqServer.cs#L451
Temporary Workaround:
I subclass SqsMqServer, loop through the protected member 'workers' in the base class, and call Stop on each one. (in this case, this Stop() method is implemented correctly as a blocking call. It waits indefinitely until the worker is done with whatever it's currently working on).
Is my current understanding of how to shutdown the MqServer correct? Is this a bug or something I misunderstood.
The source code for SqsMqServer is maintained in the ServiceStack.Aws repository.
The Stop() method pulses the bg thread which StopWorkerThreads() and that goes through and stops all workers.
We have a number of tasks in a static queue on our server. When the server shuts down (or restarts) we'd prefer not to lose these tasks and therefore we will stash them in a DB structure. On boot this DB structure will be dumped back into the static queue and processing of these queued tasks will continue.
How is it possible to detect a shut down, halt that shutdown, and then continue the shutdown once the above DB storage function has been executed? from what context should this shutdown observation be made?
I'm not sure I understood your question, but if I got it right you want to run some code before your scripts exits to do some kind of cleanup.
You can use process.on(event, handler) to register an exit handler for your script for various events, including exit (the scripts exits), SIGINT (the user Ctrl + Cs the script) and uncaughtException (an exception thrown is not caught). Take a look at this answer.
We have some long running tasks on our roles and need to be sure to stop them ima controlled way. initially we tried to use On stop method but MSDN says that
Important
Code running in the OnStop method has a limited time to finish when it is called for reasons other than a user-initiated shutdown. After this time elapses, the process is terminated, so you must make sure that code in the OnStop method can run quickly or tolerates not running to completion. The OnStop method is called after the Stopping event is raised.
And timeout seems to be around 30 seconds and overall shutdown procedure should take no more than 5 minutes.
Does this limitation occurs also on Stopping event? I can't find a clear and direct answer anywhere.
Thanks
We have 1-2 worker,which spins 5 threads, each thread read messages from Azure queue and do long processing, each processing may take around 1-2 hrs. We would like to implement logic to Stop particular thread at particular worker role. User will submit request to cancel particular processing. We are saving worker role and thread information in our azure table. But we are stuck in implementing to stop particular worker role's thread which is processing. can any one give some idea/design to stop particular thread in particular worker. Can we make use of cancellation token of thread to stop thread. Please help us in stopping worker role's thread.
You will need a flag of some sort. So either a new queue which is monitored or a DB update.
Then have a new thread started in your worker role that monitors for these cancellation messages/flags, picks the right thread and stops it.
I wouldn't recommend doing anything within the thread that is processing because it would slow down your work, however if your thread has an OnStop method, you can use that to tidy up the thread before shutting it down.
Sometimes when IIS restarts the app pool it will start a new instance of my application before the previous instance is shut down completely. This causes me alot of problem so i wonder what i can do about it.
The course of action goes something like this. (spanning about 20 seconds)
Application is running, let's call this instance A.
Restart initializes
A new instance is started, let's call this B (Logged by Application_Start)
Incomming request is processed by instance B, this invalidates all data A has cached.
Timer on instance A is triggered, assumes its cache is valid and writes something invalid into the persistant storage.
Instance A is shut down (logged by Application_End)
Preferable i would like to disable the above behavior completely, IIS should only allow one instance. If not possible, can i in my code detect if other instances is alread running and then wait for it to quit inside application_start? If not possible, what is the best way to work around this?
Disable overlapped recycling:
"In an overlapped recycling scenario,
the process targeted for a recycle
continues to process all remaining
requests while a replacement worker
process is created simultaneously. The
new process is started before the old
worker process stops, and requests are
then directed to the new process. This
design prevents delays in service,
since the old process continues to
accept requests until the new process
has initialized successfully, and is
instructed to shut down only after the
new process is ready to handle
requests."
http://msdn.microsoft.com/en-us/library/ms525803(v=vs.90).aspx