According to the docs on task.revoke():
All worker nodes keeps a memory of revoked task ids, either in-memory or persistent on disk
And
Revoking tasks works by sending a broadcast message to all the workers, the workers then keep a list of revoked tasks in memory. When a worker starts up it will synchronize revoked tasks with other workers in the cluster.
This sounds like tasks are still around after you've revoked them. I don't understand why there's not a clear way to revoke the task and remove it from the queue.
The docs seem to imply you need to keep a list of revoked tasks indefinitely to ensure new workers don't pick them up in some circumstances.
I'm also aware that there's a function to completely purge the task queue, but that's not what I'm looking for.
Is there a way to revoke a task and purge it (and only it) from the task queue in Celery?
It is not possible to remove only one message in the queue other than removing them all with a purge or with a manual command in your broker.
However, you might not mind as a revoked task once processed by a worker is removed from the queue. So you don't have to maintain an eternal revoked id list.
You should keep an id on this list only while it has not been processed by a worker because the workers are busy or the task is scheduled for later.
The list should be persistent if all your workers could be stopped at the same time and you want to keep the flagged revoked tasks. Else, a new worker asks the already running workers about the tasks to flag as revoked.
Note: I analyzed a case with Redis as the broker and backend to get the answer. The task revoked was finally removed from the queue and visible as a result (marked as revoked).
Example:
The task with id 'A' is pushed in the queue and scheduled for in 1 hour
The task 'A' is revoke() so a message is sent to all workers to flag the task as revoked. The id is in the revoke list of each worker (cf in log Tasks flagged as revoked: A)
The task 'A' is still in the queue waiting for its ETA
After one hour, a worker executes the task. As the task is flagged as revoked, the worker does not execute the task but immediately writes the task result in the backend. The result says that the task is revoked (so not executed).
I don't know about the exact reason why you can't directly remove tasks from the queue. But my intuitions are:
All the brokers might not allow removing an element in the middle of the queue
Removing a task immediately and letting the task system consistent is maybe harder. And as the Celery team has a limited workforce, they don't want to support something complex if a simpler solution does the job
There is a tiny, nice section in the Celery documentation called "revoke: Revoking tasks" - please read it.
In short - default behaviour is to gracefully stop the task. Furthermore, the task may just be waiting in the queue, in which case revoke just removes it from the queue (simplest case). More complicated is when the task is already running... With terminate=True you tell Celery worker to send SIGINT to the worker process executing the task. But in some cases that may not work. - Just like you have "zombie processes" in Linux, you may have "zombie tasks" that are difficult to revoke (I know - it is not the best analogy, but you will get the point), in which case you revoke them with SIGKILL (by revoking with terminate=True, signal='SIGKILL'). Once revoke succeeds, you will not see the task in the queue.
Related
I want to implement something akin to work stealing or task migration in multiprocessor systems. Details below.
I am simulating a scheduling system with multiple worker nodes (resources, each with multiple capacity), and tasks (process) that arrive randomly and are queued by the scheduler at a specific worker node. This is working fine.
However, I want to trigger an event when a worker node has spare capacity, so that it steals the front task from the worker with the longest wait queue.
I can implement the functionality described above. The problem is that all the tasks waiting on the worker queue from which we are stealing work receive the event notification. I want to notify ONLY the task at the front of the queue (or only N tasks at the front of the queue).
The Bank reneging example is the closest example to what I want to implement. However, it (1) ALL the customers leave the queue when they are notified that the event was triggered, and (2) when event is triggered, the customers leave the system; in my example, I want to make the task wait at another worker (though it wouldn't wait, since the queue of that worker is empty).
Old question: Can this be done in SimPy?
New questions: How can I do this in SimPy?
1) How can I have many processes, waiting for a resource, listen for an event, but notify only the first one?
2) How can I make a process migrate to another resource?
Every 10 minutes several worker roles in Azure is set to process a set of jobs(100+). Some jobs are independent, but others are not. For (simple) example, a job A must be processed, send and acknowledged by a receiver before a job B can be sent.
Independent jobs can be put on queues to distribute to worker roles. I wonder if queues could work for dependent jobs in order to make a consistent solution.
Edit: I have used a too simplistic example. Jobs A and B both consist of several related messages. These messages will be distributed to n worker roles and will be sent separately, so Job A is finished when n worker roles get acks and then the messages (distributed to and processed by m worker roles) of job B can be sent.
I think in this case the only option would be to let a single worker role process both job A and B, otherwise a complex inter worker role synchronization mechanism is needed.
I think you can use queues to facilitate this. One possible solution would be to have the worker writing another message in same or other queue once Job A is finished. So worker will pick up the message for Job A, processes the job, writes another message that Job A is done and delete original message. Now another thread will pick up the message and start working on Job B. If the message is posted in the same queue, then the message needs to convey that it is part of a multi-job chain and what steps have been completed. If the message is posted in another queue (e.g. specific queue for Job B) then your code would know that this message is for Job B and should process it accordingly.
We have 1-2 worker,which spins 5 threads, each thread read messages from Azure queue and do long processing, each processing may take around 1-2 hrs. We would like to implement logic to Stop particular thread at particular worker role. User will submit request to cancel particular processing. We are saving worker role and thread information in our azure table. But we are stuck in implementing to stop particular worker role's thread which is processing. can any one give some idea/design to stop particular thread in particular worker. Can we make use of cancellation token of thread to stop thread. Please help us in stopping worker role's thread.
You will need a flag of some sort. So either a new queue which is monitored or a DB update.
Then have a new thread started in your worker role that monitors for these cancellation messages/flags, picks the right thread and stops it.
I wouldn't recommend doing anything within the thread that is processing because it would slow down your work, however if your thread has an OnStop method, you can use that to tidy up the thread before shutting it down.
I'm using gearman to queue a variety of different jobs, some which can always be serviced immediately, and some which can "fail", because they require an unreliable external service. (For example, sending email might require an SMTP server that's frequently unavailable.)
If an external service goes down, I'd like to keep all jobs which require that service on the queue, and retry one job occasionally (every few minutes, say) until the service becomes available again. (Perhaps optionally sending email if the service has not been available for hours.)
However I'd like jobs that don't require a failed service to be passed on to workers as soon as possible. How can this be achieved? (I'm happy to put some of the logic in the workers if necessary, although it seems to be a bit "late" to throttle on the worker side.)
Gearman should already be handle this. As long as you have some workers which specialise in handling jobs with unreliable dependancies and don't handle other jobs, along with some workers that either do all jobs, or just jobs without unreliable dependencies.
All you would need to do it add some code the unreliable dependancy workers so that they only accept jobs once that have checked that the dependent service is running, if the service is down then just have them wait a bit and retest the service (and continue ad infinitum), once the service is up then have them join the gearmand server, do job, return work, retest service, etc etc.
While the dependent service is down, the workers that don't handle jobs that need the service will keep on trundling through the job queue for the other jobs. Gearmand won't block an entire job queue (or worker) on one job type if there are workers available to handle other job types.
The key is to be sensible about how you define your job types and workers.
EDIT--
Ah-ha, I knew my thinking was a little out, (I wrote my gearman system about a year ago and haven't really touched it since). My solution to this type of issue was to have all the workers that normally handle dependent-job unregister their dependent job handling capability with the gearmand server once a failure was detected with the dependent service. (and any workers that are currently trying to complete that job should return a failure.) Once the service is backup - get those same workers to reregister their ability to handle that job. Do note this does require another channel of communications for the workers to be notified of the status of the dependent services.
Hope this helps
I have a simple work role in azure that does some data processing on an SQL azure database.
The worker basically adds data from a 3rd party datasource to my database every 2 minutes. When I have two instances of the role, this obviously doubles up unnecessarily. I would like to have 2 instances for redundancy and the 99.95 uptime, but do not want them both processing at the same time as they will just duplicate the same job. Is there a standard pattern for this that I am missing?
I know I could set flags in the database, but am hoping there is another easier or better way to manage this.
Thanks
As Mark suggested, you can use an Azure queue to post a message. You can have the worker role instance post a followup message to the queue as the last thing it does when processing the current message. That should deal with the issue Mark brought up regarding the need for a semaphore. In your queue message, you can embed a timestamp marking when the message can be processed. When creating a new message, just add two minutes to current time.
And... in case it's not obvious: in the event the worker role instance crashes before completing processing and fails to repost a new queue message, that's fine. In this case, the current queue message will simply reappear on the queue and another instance is then free to process it.
There is not a super easy way to do this, I dont think.
You can use a semaphore as Mark has mentioned, to basically record the start and the stop of processing. Then you can have any amount of instances running, each inspecting the semaphore record and only acting out if semaphore allows it.
However, the caveat here is that what happens if one of the instances crashes in the middle of processing and never releases the semaphore? You can implement a "timeout" value after which other instances will attempt to kick-start processing if there hasnt been an unlock for X amount of time.
Alternatively, you can use a third party monitoring service like AzureWatch to watch for unresponsive instances in Azure and start a new instance if the amount of "Ready" instances is under 1. This will save you can save some money by not having to have 2 instances up and running all the time, but there is a slight lag between when an instance fails and when a new one is started.
A Semaphor as suggested would be the way to go, although I'd probably go with a simple timestamp heartbeat in blob store.
The other thought is, how necessary is it? If your loads can sustain being down for a few minutes, maybe just let the role recycle?
Small catch on David's solution. Re-posting the message to the queue would happen as the last thing on the current execution so that if the machine crashes along the way the current message would expire and re-surface on the queue. That assumes that the message was originally peeked and requires a de-queue operation to remove from the queue. The de-queue must happen before inserting the new message to the queue. If the role crashes in between these 2 operations, then there will be no tokens left in the system and will come to a halt.
The ESB dup check sounds like a feasible approach, but it does not sound like it would be deterministic either since the bus can only check for identical messages currently existing in a queue. But if one of the messages comes in right after the previous one was de-queued, there is a chance to end up with 2 processes running in parallel.
An alternative solution, if you can afford it, would be to never de-queue and just lease the message via Peek operations. You would have to ensure that the invisibility timeout never goes beyond the processing time in your worker role. As far as creating the token in the first place, the same worker role startup strategy described before combined with ASB dup check should work (since messages would never move from the queue).