In a recent course at school about networking / operating systems I learned about thread pools. Now he basic functionality is pretty straight forward and I understand this.
However, what's not specified in my book is what happens when the thread pool is exhausted? For example you have a pool with 20 threads in it and you have 20 connected clients. Another client tries to connect but there's no threads left in the pool, what happens then? Does the client go in a queue? Does the system make another thread to put in the pool? Something else?
The answer depends highly on your language, your operation system, and your pool implementation.
what happens when the thread pool is exhausted? Another client tries to connect but there's no threads left in the pool, what happens then? Does the client go in a queue?
Typically in a server situation, it depends on the socket settings. Either the socket connection gets queued by the OS or the connection gets refused. This is usually not handled by the thread-pool. In ~unix operation systems, this queue or "backlog" is handled by the listen method.
Does the system make another thread to put in the pool?
This depends on the thread-pool. Some pools are fixed size so no more threads will be added. Other thread-pools are "cached" thread pools so it will reuse a free thread or will create a new one if none are available. Many web servers have max thread settings on their pools so remote users don't thrash the system by starting too many concurrent connections.
It depends on the policy used by the thread-pool:
the pool size can be static, and when a new thread is requested the caller will wait on a synchronization primitives like a semaphore, or the request can be pushed into a queue
the pool size can be unlimited but this may be dangerous because creating too much threads can greatly reduce the performance; more often than note it is ranged between a min and a max set by the pool user
the pool can use a dynamic policy depending on the context: hardware resources like CPU or RAM, OS resources like synchronization primitives and threads, current process resources (memory, threads, handles...)
An example of a smart thread-pool: http://www.codeproject.com/Articles/7933/Smart-Thread-Pool
It depends on the thread pool implementation. They might be put on a queue, they might get a new thread created for them, or they might even just get an error message saying come back later. Or if you are the one implementing the thread pool, you can do whatever you want.
Related
A COM application based on the 'free' threading model subscribes to events published from another COM application that runs out of process.
The application works normally. But in some cases (or configurations?) it burns through a lot of so called Tpp worker threads.
These threads apparently belong to a thread pool managed by Windows/COM. And they are at least used by COM to deliver incoming events to the application.
When the application receives events, that always happens in the context of one of these worker threads.
In the normal situation, updates are coming in from at most 2 or 3 unique worker threads.
But in the abnormal situation the application sees new & unique worker thread IDs appear every 3-8 minutes. Over the course of a week the application has seen about 1000 unique threads (!).
I highly suspect there is something wrong here. Because surely the thread pool doesn't need so many different threads, right?
What could be a reason for the thread pool behavior I'm seeing. Is it just normal that it creates different threads from time to time? Are the old threads still sticking around doing nothing? What action could trigger this while the application is running in the context of the worker thread?
Notes:
Our application is an OPC DA client (and the other application is the Siemens OPC-DA server)
The OS is Windows 10
I do not yet know if the worker threads have exited or that they stick around doing nothing
By way of an experiment I have tried several bad/illegal things to see if it is possible for our application to somehow break a worker thread
- which would then explain why the thread pool would have to create a new one - we might have destroyed the old one. But that seems more difficult than I had expected:
When running in the context of the worker thread, I have tried...
deliberately hanging with while (true) {}, result: event delivery process just stalls, no new worker thread is being created for us though
the deliberate uncaught c++ exception, no new worker thread is created
triggering a deliberate (read) access violation, no new thread either...
And that got me thinking, if our application can't kill that worker thread in an obvious way, what else could or why would the thread pool behave like this?
For a user base of 100,000 and 4 users per game session, should we create new threads for each request such as create_session, move_player, use_attack, etc. ?
I wanted to know what would be the optimal way to handle large connections because if we create large number of threads, context switching will eat up most of the cycles and if no threads are created each request has to wait for previous request to complete.
I would avoid thread-per-connection if your goal is scalability. It would be better to have a queue of events and a thread pool.
A game company would probably use a non-connection-based internet protocol like UDP. All requests can theoretically come in on the same socket, so you only need 1 thread to handle that. That thread can assign work to other threads.
You can have a larger threadpool where any thread can be assigned any job. Or you could further organize the work into specific jobs, each with a threadpool to process a queue of tasks. But I wouldn't launch a new thread for each request.
How you design your threadpools and task distribution system depends on the libraries for whatever language you're using and the application requirements.
I understand that NodeJS uses a thread pool for blocking I/O calls. What does it do if all the threads in the thread pool are busy with some work and another request comes in?
In a case where the thread pool is needed and no workers are available, the request would be queued until a worker is free. The thread pool is not the sole approach, though. There are three operation types that utilize the thread pool in libuv as documented at the bottom of the page here under the title File I/O.
These operations types are:
Filesystem operations
DNS functions (getaddinfo and getnameinfo)
User-specified code
While not a direct answer to your question, I believe this post by Jason does a wonderful job of explaining thread pools in Node.js. Without going extremely in-depth, it introduces you to the functionality provided the libuv library and has links to very informative literature on the subject of the thread pool.
In apache, we have a single thread for each incoming request. Each thread consumes a memory space. the memory spaces don't collide with each other because of which each request serves it purpose.
How does this happen in node.js as it has single thread execution. A single memory space is used by all incoming requests. Why don't the requests collide with each other. What differentiates them?
As you self noticed an event based model allows to share the given memory more efficiently as the overhead of reexecuting a stack again and again is minimized.
However to make an event or single threaded model non-blocking you have to get back to threads somewhere and this is where nodes "io-engine" libuv is working.
libuv supplies an API which underneath manages IO-tasks in a thread pool if an IO task is done async. Using a thread pool results in not blocking the main process however extensive javascript operations still can do (this is why there is the cluster module which allows spawning multiple worker processes).
I hope this answers you question if not feel free to comment!
we have a mule app with HTTP inbound endpoint and I'm trying to figure out how to control the thread count under load. As an experiment I have added the following configuration:
<core:configuration>
<core:default-threading-profile doThreading="false" maxThreadsActive="500" poolExhaustedAction="RUN"/>
</core:configuration>
Under load I'm seeing the thread count peak at over 1000 threads. Am not sure why this is the case give the maxThreadsActive setting and the doThreading="false". Reading about poolExhaustedAction="RUN", I would expect the listener thread to block while processing inbound requests rather than spawn new ones, and finally reject the connection if its backlog queue is full. I never see rejected client connections.
Does Mule maintain a separate thread pool for each inbound endpoint in the app (sorry if this is in the documentation)? Even if so, don't think it helps explain what I'm seeing.
Any help appreciated. We are running a number of mule apps in one container and I'd like to control the total number of threads.
Thanks, Alfie.
Clearly the doThreading attribute on default-threading-profile is not enough to control Mule threading as a whole nor limit with a global cap the specific threading behaviour of transports. I reckon you're getting 500 threads for the HTTP message receiver pool and 500 for the VM message dispatcher pool.
I strongly suggest you reading about tuning Mule: http://www.mulesoft.org/documentation/display/current/Tuning+Performance
My gut feel is that you need to
configure threading on each transport (VM, HTTP), strictly specifying the pool size for receivers and dispatchers,
select flow processing strategies that prevent Mule from spawning new threads (i.e. use synchronous to hog the receiver threads),
select exchange patterns that also prevent Mule from spawning new threads (i.e. use request-response to piggyback the current execution thread).