what are the setting needed or any other framework required to prevent users from killing each others spark job submitted to master and visible on localhost:8080. I want a situation where only admin or the user who submitted the job will be able to kill that job and no other user.
Related
I have a SERVICE that gets a request from a Webhook and this is currently deployed across seperate Cloud Run containers. These seperate containers are the exact same (image), however, each instance processes data seperately for each particular account.
This is due to a ~ 3-5 min processing of the request and if the user sends in more requests, it needs to wait for the existing process to be completed for that particular user before processing the next one to avoid racing conditions. The container can still receive webhooks though, however, the actual processing of the data itself needs to be done one by one for each account.
Is there no way to reduce the container count, as such for example, to use one container to process all the requests, while still ensuring it processes one task for each user at a time and waits for that to complete for that user, before processing the next request from the same user.
To explain it better, i.e.
Multiple tasks can be run across all the users
However, per user 1 task at a time processed; Once that is completed, the next task for that user can be processed
I was thinking of monitoring the tasks through a Redis Cache, however, with Cloud Run being stateless, I am not sure that is the right way to go.
Or seperating the requests and the actual work - Master / Worker - And having the worker report back to the master once a task is completed for the user across 2 images (Using the concurrency to process multiple tasks across the users), however that might mean that I would have to increase the timeout time for Cloud Run.
Good to hear any other suggestions.
Apologies if this doesn't seem clear, feel free to ask for more information.
According to the docs on task.revoke():
All worker nodes keeps a memory of revoked task ids, either in-memory or persistent on disk
And
Revoking tasks works by sending a broadcast message to all the workers, the workers then keep a list of revoked tasks in memory. When a worker starts up it will synchronize revoked tasks with other workers in the cluster.
This sounds like tasks are still around after you've revoked them. I don't understand why there's not a clear way to revoke the task and remove it from the queue.
The docs seem to imply you need to keep a list of revoked tasks indefinitely to ensure new workers don't pick them up in some circumstances.
I'm also aware that there's a function to completely purge the task queue, but that's not what I'm looking for.
Is there a way to revoke a task and purge it (and only it) from the task queue in Celery?
It is not possible to remove only one message in the queue other than removing them all with a purge or with a manual command in your broker.
However, you might not mind as a revoked task once processed by a worker is removed from the queue. So you don't have to maintain an eternal revoked id list.
You should keep an id on this list only while it has not been processed by a worker because the workers are busy or the task is scheduled for later.
The list should be persistent if all your workers could be stopped at the same time and you want to keep the flagged revoked tasks. Else, a new worker asks the already running workers about the tasks to flag as revoked.
Note: I analyzed a case with Redis as the broker and backend to get the answer. The task revoked was finally removed from the queue and visible as a result (marked as revoked).
Example:
The task with id 'A' is pushed in the queue and scheduled for in 1 hour
The task 'A' is revoke() so a message is sent to all workers to flag the task as revoked. The id is in the revoke list of each worker (cf in log Tasks flagged as revoked: A)
The task 'A' is still in the queue waiting for its ETA
After one hour, a worker executes the task. As the task is flagged as revoked, the worker does not execute the task but immediately writes the task result in the backend. The result says that the task is revoked (so not executed).
I don't know about the exact reason why you can't directly remove tasks from the queue. But my intuitions are:
All the brokers might not allow removing an element in the middle of the queue
Removing a task immediately and letting the task system consistent is maybe harder. And as the Celery team has a limited workforce, they don't want to support something complex if a simpler solution does the job
There is a tiny, nice section in the Celery documentation called "revoke: Revoking tasks" - please read it.
In short - default behaviour is to gracefully stop the task. Furthermore, the task may just be waiting in the queue, in which case revoke just removes it from the queue (simplest case). More complicated is when the task is already running... With terminate=True you tell Celery worker to send SIGINT to the worker process executing the task. But in some cases that may not work. - Just like you have "zombie processes" in Linux, you may have "zombie tasks" that are difficult to revoke (I know - it is not the best analogy, but you will get the point), in which case you revoke them with SIGKILL (by revoking with terminate=True, signal='SIGKILL'). Once revoke succeeds, you will not see the task in the queue.
We are using a hazelcast executor service to distribute tasks across our cluster of servers.
We want to shut down one of our servers and take it out of the cluster but allow it to continue working for a period to finish what it is doing but not accept any new tasks from the hazelcast executor service.
I don't want to shut down the hazelcast instance because the current tasks may need it to complete their work.
Shutting down the hazelcast executor service is not what I want. That shuts down the executor cluster-wide.
I would like to continue processing the tasks in the local queue until it is empty and then shut down.
Is there a way for me to let a node in the cluster continue to use hazelcast but tell it to stop accepting new tasks from the executor service?
Not that easily, however you have member attributes (Member::setX/::getX) and you could set an attribute to signal "no new tasks please" and when you submit a tasks you either preselect a member to execute on based on the attribute or you use the overload with the MemberSelector.
Is it possible to do that without writing my own daemon? I know slurm can send you and email for each job, but I would like a single email when I have no more pending or running jobs.
One option is to submit an empty job just to send the email, and ask Slurm to run that job the latest.
You can do that using the --dependency singleton option. From the documentation:
singleton This job can begin execution after any previously launched
jobs sharing the same job name and user have terminated.
So you need to name all your jobs the same name (--name=commonname), and you should request the minimum resources possible to make sure that this job is not delayed further when all your other jobs are finished.
I made an jsf application.This application has a menu containing start,stop buttons.When start is pressed , application starts to get data from web sites, and updates its database.The application also has progress bar for update process.However,this process takes a long time to finish.I want that when i close my browser , it should go on updating database.Besides, when i open it again, i should get previous state.However, this isn't happening.When i close browser the application closes too.How do i do that?
Thanks.
In my case, I would not extend the session life. Instead, create a task and add the object that performs the task into a Queue in an #ApplicationScoped bean and save in database (or any other place) the user that started the job and the status of the job.
When the user logs off (manually logging off or closing the web browser), the task will still be executed because is managed by the whole application (not by a request nor user session). When the user logs in again, he/she could ask to this application queue about the status of the task.
You will need (at least):
An #ApplicationScoped managed bean that will contain and handle the tasks.
A way to handle and execute one or more tasks at the same time. This could be achieved with a ExecutorService or similar technologies. Note: don't dare to manually start new threads by your own, this will only lead to kill your web application server.
An structure to map the user with the tasks he/she has started. It could be mapped with a Map<String, List<Task>> in order that a single user could have more than 1 task at the moment. It would be better to save this in a database (or similar) in order to have a log for these tasks that don't reside in memory. You can even design this in order that if you undeploy the web application or the server suddenly shut downs, you could restart the tasks at hand from a savestate point.