What is the best algorithm to use for scheduling an application that will support 10K concurrent threads with heavy I/O but low CPU usage? Links to papers are appreciated.
Why wouldn't you use SCHED_RR? You said it yourself: low CPU usage. You could even nice the process when you expect to do some heavy I/O so you're scheduled less often than other processes.
In general, though, why not let the OS do what it's best at, and just worry about writing efficient code? The OS will know you're doing a blocking I/O call and will put your thread/task in a waitqueue and select another task to run. You don't need to worry about those details.
Actually I believe no scheduling mechanism will handle this number of threads flawlesly as the management tables in the kernel will become quite large.
If possible, I'd suggest rewriting the app to use asynchronous I/O, select() or something similar on the OS of your choice.
You will likely want SCHED_RR for this. You might be interested in reading this question regarding the difference between SCHED_FIFO and SCHED_RR.
Your problem is more related to I/O scheduling than Thread scheduling. The linux kernel offers various I/O scheduler mplementations. You can find a good article on this subject in this edition of LWN.
As grover suggested, you could also use some Thread pooling mechanisms, which are less resource intensive and will solve your purpose at least to some reasonable extent, if not fully
Related
Do you think threadpooling design pattern is the way to go for the multi-core future?
A threadpooling library for instance, if used extensively, makes/force the application writer
(1) to break the problem into separate parallel jobs hence promoting (enforcing :) ) parallelism
(2) With abstraction from all the low level OS calls, synchronization etc etc makes programmer's life easier. (Especially for C programmers :) )
I have strong belief its the best way (Or One of the "best" ways :) ) for multi-core future...
So, my question is, am I write in thinking so or I am in some delusion :)
Regards,
Microkernel
Thread pooling is a technique that involves a queue and a number of threads taking jobs from the queue and process them. This is in contrast to the technique of just starting new threads whenever a new task arrives.
Benefits are that the maximum number of threads is limited to avoid too much threading and that there is less overhead involved with any new task (Thread is already running and takes task. No threat starting needed).
Whether this is a good design highly depends on your problem. If you have many short jobs that come to your program at a very fast rate, then this is a good idea because the lower overhead is really a benefit. If you have extremely large numbers of concurrent tasks this is a good idea to keep your scheduler from having to do too much work.
There are many areas where thread pooling is just not helpful. So you cannot generalize. Sometimes multi threading at all is not possible. Or not even desired, as multi threading adds an unpredictable element (race conditions) to your code which is extremely hard to debug.
A thread pooling library can hardly "force" you to use it. You still need to think stuff through and if you just start one thread... Won't help.
As almost every informatics topic the answer is: It Depends.
the pooling system is fine with Embarrassingly parallel http://en.wikipedia.org/wiki/Embarrassingly_parallel
For other task where you need more syncornization between threads it's not that good
For the Windows NT engine thread pools are usually much less efficient than I/O Completion Ports. These are covered extensively in numerous questions and answers here. IOCPs enable event-driven processing in that multiple threads can wait on the IOCP until an event occurs due to an IOC (read or write) on a socket or handle which is then queued to the IOCP. The IOCP in turn pairs a waiting thread with the id of the event and releases the thread for processing. After the thread has processed the event and initiated a new I/O it returns to the IOCP to wait for the next event (which may or may not be the completion of the I/O it just initiated).
IOCs may also be artificially signalled by explicit posting from a non-event.
Using IOCPs is not polling. The optimal IOCP implementation will have as many threads waiting on the IOCP as there are cores in the system. The threads may all execute the same physical code if that is deemed efficient. Since a thread processes from an IOC up until it issues an I/O it does nothing which forces it to wait for other resources except perhaps to compete for access to thread-safe areas. It is a natural choice to move away from the "one handle per thread" paradigm. IOCP-controlled threads are therefore as efficient as the programmer is able to construct them.
I like the answer by #yaankee a lot except I would argue that thread pool is almost always the right way to go. The reason: a thread pool can degenerate itself into a simple static work partitioning model for problems like matrix-matrix multiply. OpenMP guided is kind of along those lines.
As far as I'm concerned, the ideal amount of threads is 3: one for the UI, one for CPU resources, and one for IO resources.
But I'm probably wrong.
I'm just getting introduced to them, but I've always used one for the UI and one for everything else.
When should I use threads and how? How do I know if I should be using them?
Unfortunately, there are no hard and fast rules to using Threads. If you have too many threads the processor will spend all its time generating and switching between them. Use too few threads you will not get the throughput you want in your application. Additionally using threads is not easy. A language like C# makes it easier on you because you have tools like ThreadPool.QueueUserWorkItem. This allows the system to manage thread creation and destruction. This helps mitigate the overhead of creating a new thread to pass the work onto. You have to remember that the creation of a thread is not an operation that you get for "free." There are costs associated with starting a thread so that should always be taken into consideration.
Depending upon the language you are using to write your application you will dictate how much you need to worry about using threads.
The times I find most often that I need to consider creating threads explicitly are:
Asynchronous operations
Operations that can be parallelized
Continual running background operations
The answer totally depends on what you're planning on doing. However, one for CPU resources is a bad move - your CPU may have up to six cores, plus hyperthreading, in a retail CPU, and most CPUs will have two or more. In this case, you should have as many threads as CPU cores, plus a few more for scheduling mishaps. The whole CPU is not a single-threaded beast, it may have many cores and need many threads for 100% utilization.
You should use threads if and only if your target demographic will virtually all have multi-core (as is the case in current desktop/laptop markets), and you have determined that one core is not enough performance.
Herb Sutter wrote an article for Dr. Dobb's Journal in which he talks about the three pillars of concurrency. This article does a very good job of breaking down which problems are good candidates for being solved via threading constructs.
From the SQLite FAQ: "Threads are evil. Avoid Them." Only use them when you absolutely have to.
If you have to, then take steps to avoid the usual carnage. Use thread pools to execute fine-grained tasks with no interdependencies, using GUI-framework-provided facilities to dispatch outcomes back to the UI. Avoid sharing data between long-running threads; use message queues to pass information between them (and to synchronise).
A more exotic solution is to use languages such as Erlang that are explicit designed for fine-grained parallelism without sacrificing safety and comprehensibility. Concurrency itself is of fundamental importance to the future of computation; threads are simply a horrible, broken way to express it.
The "ideal number of threads" depends on your particular problem and how much parallelism you can exploit. If you have a problem that is "embarassingly parallel" in that it can be subdivided into independent problems with little to no communication between them required, and you have enough cores that you can actually get true parallelism, then how many threads you use depends on things like the problem size, the cache line size, the context switching and spawning overhead, and various other things that is really hard to compute before hand. For such situations, you really have to do some profiling in order to choose an optimal sharding/partitioning of your problem across threads. It typically doesn't make sense, though, to use more threads than you do cores. It is also true that if you have lots of synchronization, then you may, in fact, have a performance penalty for using threads. It's highly dependent on the particular problem as well as how interdependent the various steps are. As a guiding principle, you need to be aware that spawning threads and thread synchronization are expensive operations, but performing computations in parallel can increase throughput if communication and other forms of synchronization is minimal. You should also be aware that threading can lead to very poor cache performance if your threads end up invalidating a mutually shared cache line.
I am looking to get into operating system kernel development and figured my contribution would be to extend the SANOS operating system in order to support multiple core machines. I have been reading books on operating systems (Tannenbaum) as well as studying how BSD and Linux have tackled this challenge but still am stuck on several concepts.
Does SANOS need to have more sophisticated scheduling algorithms when it runs on multiple CPUs or will what is currently in place work fine?
I know that it is a good idea for threads to have affinity to a core that they were started on, but is this handled via scheduling or by changing the implementation of how threads are created?
What would need to be considered such that SANOS could run on a machine with hundreds of cores? From what I can tell, BSD and Linux at best only support a maximum of a dozen of cores.
Your reading material is good. SO no problems there. Also take a peek at the CS downloadable lectures on operating system design from Stanford.
The scheduling algorithm may need to be more sophisticated. This depends on the types of applications running and how greedy they are. Do they yield themselves or are they forced to. That kind of thing. This is more a question of what your processes want, or expect. A RTOS will have more complex scheduling than a desktop.
Threads should have an affinity to one core, because 2 threads in one process can execute in parallel ... but not at the same real-time on the same core. Putting them on different cores allows them to really-run-in-parallel. Also caching can be optimized for core affinity. This is really a mix of your thread implementation and scheduler. The sched may want to ensure threads are started at the same time on cores, rather than ad-hoc to reduce the amount of time threads wait on eachother and things. If your thread library is user-space, maybe it assigns core, or lets the scheduler decide based on capacity or recent deaths.
Scalability is often a kernel limit (which can be arbitrary). In Linux, if I recall, the limits are due to static sizing of arrays that hold CPU information structs in the scheduler. Hence they are a fixed size. This can be changed by recompiling the kernel. Most good scheduling algorithms will support a very large number of cores. As your core or processor count gets higher, you need to be careful that you don't fragment a processes execution too much. If a program has 2 threads, try and schedule them in close-time-proximity because causation may exist (through shared data) between them.
You also need to decide how your threads are implemented, and how a process is represented (be it heavy or lightweight) in the kernel. Are threads kernel managed? user-space managed? These things all have an impact on scheduler design. Look at how POSIX threads are implemented in various operating systems. There is just so much for you to think about :)
in short there are not really any straight-cut answers to where the logic does, or should reside. It is all down to design, application expectation, time-constraints (on the programs) and so on.
Hope this helps, I am not an expert here however.
Is there an advantage of the operating system understanding the characteristics of how a thread may be used? For example, what if there were a way in Java when creating a new thread to indicate that it would be used for intensive CPU calculations vs will block for I/O. Wouldn't thread scheduling improve if this were a capability?
I'm not sure what you're actually expecting the OS to do with the information that a thread is I/O or compute. The things which actually make the most difference to how threads get scheduled (ie thread priority and thread CPU affinity) are already exposed by APIs (and support for NUMA aspects are starting to appear in mainstream OS APIs too).
If by a "compute thread" you mean it's something doing background processing and less important than a GUI thread (from the point of view of maintaining app responsiveness) probably the most useful thing you can do is lower the priority of the compute threads a little.
That's what OS processes do. The OS has sophisticated scheduling for the processes. The OS tracks I/O use and CPU use and dynamically adjusts priorities so that CPU-intensive processing doesn't interfere with I/O.
If you want those features, use a proper OS process.
Is that even necessary? Threads blocking on I/O will cause CPU-intensive threads to run. The operating system decides how to schedule threads. AFAIK there's no way to give any hints with Java.
Yes, it is very important to understand them specially if you are one of those architects who like opening lot of threads, specially on windows.
Jeff Richter over at Wintellect has a library called PowerThreading. It is very useful if you are developing applications on .NET, but since you are talking about JAVA, it is still better to understand OS threads, kernel models and how the interrupts work.
Is there anyone here with experience in the linux thread scheduler running multithreading applications in the new Quad core processors?. If there is someone like that can you please write here your experience about how is the performance of the kernel managing the different threads, Have you experienced any thread starving or the starving of one of the cores?.
Thank you.
Given that kernel developers like Christoph Lameter (and Ingo Molnar on the scheduler) have tuned the kernel to work well on 4096 processors, and given the amount of optimizations invested by Intel itself in the issue, with multicore specific tuning both for performance and energy saving, I bet the kernel is by far more optimized than anything any of us can write in userspace.
Same about the threading library; there is currently only one thread library, NPTL for Linux 2.6. LinuxThreads was removed from glibc in the 2.4 release, and NPTL was produced before the 2.6 release. And it's really fast.
Just make sure to avoid using an old kernel, the last release of your distro, or kernel.org, is the best. Before deploying in production, make sure to measure the performance difference, and consider whether that is worth the additional support costs (if any).
Linux supports using many processors very well itself. If I remember correctly with SMP, Linux supports 4096-processors. What will really make a difference is whether your applications are written to take advantage of multiple processors.
It works very well on a twin quad-core system (V8) we have in production... bloody fast.
But be very wary of Linux's propensity for thread starvation when locks (mutexes) are heavily contested. Imagine a scenario when 10 threads are working with one lock where the lock is needed very frequently but for very short amounts of time and the work done outside of the lock at any given point is less than a timeslice. Linux will very much tend to deliver the lock almost always to one thread to the exclusion of all the others.
This depends on the specific threading package bound into the kernal, too - I believe there are several.
I have gotten absolutely stunning results on our Intel Q6600's, both for parallel make and for some other parallel applications, but I have been careful to avoid excessive parallelism: I generally fork between four and eight threads, so there's not too much contention. If you fork enough threads, you will have noticeable overhead, especially if they contend for the same semaphores. My guess is that thousands of threads are probably too many, and dozens of threads are probably OK. But that's just a guess; if you want to know, you'll have to find someone who has measured, or you'll have to do an experiment yourself.
But for a dozen threads our results have been fabulous.