What does worker, secondaryWorker, web worker, infra worker, dynamic worker, and local executors stand for respectively in DolphinDB? Why the secondaryWorker and dynamic worker is introduced and what’s the usage?
worker: the thread of regular interactive jobs. It divides clients’ requests into subtasks once received. Depending on the task granularity, the worker will either execute the tasks or allocate them to local executor or remote executor. The number of workers can be set by specifying the configuration parameter workerNum and the default value is determined by the number of CPU cores.
secondary worker: the thread of secondary jobs. It is used to avoid job loops and solve the deadlocks caused by circular dependency between tasks. The upper limit can be set by specifying the configuration parameter secondaryWorkerNum and the default value is workerNum.
web worker: the thread that processes HTTP requests. DolphinDB provides a web interface for cluster management, allowing users to interact with DolphinDB nodes. The upper limit can be set by specifying the configuration parameter webWorkerNum and the default value is 1.
infra worker: the thread that reports heartbeat within clusters. It solves the problem that the heartbeat cannot be reported to the master in time when a cluster is under high pressure.
dynamic worker: the dynamic working thread as a supplemental to worker. If a new task is requested when all the worker threads are occupied, the system creates a dynamic worker thread to perform the task. The upper limit can be set by specifying the configuration parameter maxDynamicWorker and the default value is workerNum. The thread will be recycled by the system after being idle for 60 seconds to release memory resources.
local executor: the local thread that executes sub-tasks allocated by worker. Each local executor can only execute one task at a time. All worker threads share one local executor. The number of local executors can be set by specifying configuration parameter localExecutors and the default value is the number of CPU cores minus 1. The number of workers and local executors directly determines the system’s performance for concurrent computing.
Related
The doc is:
With this option you can configure the maximum number of tasks a worker can execute before it’s replaced by a new process.
In what condition will a worker be replaced by a new process ? Does this setting make a worker, even with multi processes, can only process one task at one time?
It means that when celery has executed tasks more than the limit on one worker (the "worker" is a process if you use the default process pool), it will restart the worker automatically.
Say if you use celery for database manipulation and you forget to close the database connection, the auto restart mechanism will help you close all pending connections.
We are using a hazelcast executor service to distribute tasks across our cluster of servers.
We want to shut down one of our servers and take it out of the cluster but allow it to continue working for a period to finish what it is doing but not accept any new tasks from the hazelcast executor service.
I don't want to shut down the hazelcast instance because the current tasks may need it to complete their work.
Shutting down the hazelcast executor service is not what I want. That shuts down the executor cluster-wide.
I would like to continue processing the tasks in the local queue until it is empty and then shut down.
Is there a way for me to let a node in the cluster continue to use hazelcast but tell it to stop accepting new tasks from the executor service?
Not that easily, however you have member attributes (Member::setX/::getX) and you could set an attribute to signal "no new tasks please" and when you submit a tasks you either preselect a member to execute on based on the attribute or you use the overload with the MemberSelector.
I am trying to fork worker clusters to a maximun of 10, and only if the working load increases. Can it be done?
I have tried with strong-cluster-control's setSize, but I can't find an easy way of forking automatically (if many requests are being done then fork, for example), or closing/"suiciding" forks (maybe with a timeOut if nothing is being done, like in this answer)
This is my repo's main file at GitHub
Thank you in advance!!
I assume that you already have some idea as to how you would like to spread your load so I will not include details about that and instead focus on the interprocess communication required for this.
Notifying the master
To send arbitrary data to the master, you can use process.send() from a worker. The way I would go about this is probably something along these steps:
The application is started
Minimum amount of workers are spawned
Each worker will send the master a request message every time it receives a new request, via process.send()
The master keeps track of all the request events from all workers
If the amount of request events increases above a predefined threshold (i.e. > 100 requests/s) it spawns a new worker
If the amount of request events decreases below a predefined threshold it asks one of the workers to stop processing new requests and close itself gracefully (note that it should not simply kill the process to avoid interrupting ongoing requests)
Main point is: Do not focus on time - focus on rate. In an application that is supposed to handle tens to thousands of requests per second, your setTimout() (the task of which might be to kill the worker if it has been idle for too long) will never fire because Node.js evenly distributes your load across your workers - you could start with one worker, but once you reach your maximum you will never drop to one worker again under continuous load even if there is only one request per second.
It should be noted that it is counterproductive to spawn more workers than the amount of CPU cores you have at your disposal. It might, however, be beneficial to start with a single worker and incrementally increase the amount to all cores as load increases.
I am new to multithreading. In a windows service I am using SemaphoreSlim class to initiate a Thread Governer. The SemaphoreSlim constructor takes in two arguments - the thread pool size and the maximum thread pool size. I set this as following -
int poolSize = 2;
SemaphoreSlim threadGoverner = new SemaphoreSlim(poolSize, poolSize);
So the threadGoverner is initialized when the WindowsService is started, that is, the OnStart event. Now these two threads are being used to process some kind of jobs. I have a requirement where I need to change the pool size dynamically. So if the pool size is changed to 3, the service should start processing 3 requests.
So, what I am trying to do is to restart the service using a batch command file. But the problem I am facing is if there are more than 1 threads running, another continues to process these jobs. And that is causing my service to behave abnormally.
What I want is that if I detect a change in the pool size, no thread should start to process a job, so that I can restart the service without any anomaly.
Can anyone help me with this?
.net 2.0 aspx app / IIS6 creating a silly number of threads in w3wp.exe process app pool.
The app has been isolated to its own app pool with the following settings:
RECYCLING
recycle worker processses (in minutes) : 870
recycle worker process (no of requests): (not ticked)
recycle worker processes at the following times: 00:00
max virtual memory: (not ticked)
max used memory (in mb): 1000mb (1gb)
PERFORMANCE
shutdown worker processes after being idle for (time in mins): 20
limit the kernal request queue (number of requests): 1000
enable cpu monitoring (%): 85
refresh cpu usage numbers (in mins): 5
action performed when cpu usage exceeds maximum cpu uses: NO ACTION (keeps
sessions)
max number of worker processes: 1
HEALTH
enable pinging (checked)
ping worker process every (seconds) : 30
enable rapid fail protection (checked)
failures: 5
time period (in mins):5
start time limit - worker process must startup within (seconds): 90
shutdown time limit - worker process must shutdown within (seconds): 90
Normal running would see the w3wp.exe process utilise 300MB ram and 50
therads. When my problem occurs the thread count slowly increases to 10,000
, ram to 1GB before the threads are knocked back to 0. The w3wp.exe process
is NOT shutdown and my users are not logged out (crucially), ie they keep
their session and dont have to log back in . Altough the standard 50 threads
are killed in amongst the 10, 000 rouge threads.
1) Can an expert offer any pros/cons on the above app pool settings ?
2) The "max used mem" setting appears to be doing the trick to
automatiaclly handle this issue (by killing the threads, keeping the session
alive , but can someone explain why ? ... i take it threads are unrelated to
the session).
The app uses server based sessions but we store a local cookie for authentication.
Threads
10k threads is insanely high, and your threads are spending more time hopping on and off the processor than doing actual work. aka thrashing.
EDIT: I'm assuming here that it's a .NET web application.
Does your application use a ThreadPool or BackgroundWorkers? It seems like you'd have to be using some mechanism other than IIS's standard thread entourage (which is only around 4 per processor) to reach 10k threads.
Memory
Each thread requires memory to keep track of in addition to the memory it utilizes for work, so by sheer volume of threads, you are probably reaching the 1G limit.
Session (I will survive!)
The application is probably setup to store Session State in persistent storage or the session state service. With this being the case, the worker process can safely be recycled without losing user state information. If the session state was configured (in the Web.config) as In-Proc, then session state would be lost when the worker process recycled.
Work process recycling
One other thing of note, before a worker process dies, another worker process is setup and started to take it's place. It's somewhere in this process that you're probably seeing the w3wp.exe process (either old or new) with 0 threads.
BackgroundWorkers are like rabbits
If your threads are performing work that lasts longer than 1 second (1/2 second really), don't use BackgroundWorkers. Unless you change the ThreadPool's max threads (which is NOT recommended, as this can screw up deeper functionality within .NET), there's not a hard (enough) limit on the number of BackgroundWorkers that can run concurrently. It would be better to use a Producer Consumer Queue model in that case.
Check out this site. It's an awesome resource on concurrent programming with lots of models and examples.