I am studying Solaris and Linux and am viewing Kernel Level Threads (KLTs) as the fundamental entity that can be scheduled and dispatched by the OS. I know that a multi-threaded OS must store thread execution context and provide mechanisms to schedule and dispatch KLTs, and that kernel level threads handle interrupts, system calls, and provide an interface to the CPU as a resource at the user-kernel interface. I am not clear on what services are necessary to support KLTs in a multi-threaded OS.
I cannot determine if there is a core kernel process that is necessary to support all KLTs, or if KLTs run interdependently as the base-level of computing. I would like to understand what minimum set of operations (resource allocation, scheduling) is necessary to support an OS with KLTs.
I have looked at Tanenbaums Tanenbaum's discussion of threads in his distributed systems book, Understanding the Linux Kernel, and MultiThreading the SunOS kernel but I cannot find an answer to my question.
I believe that answering the question -- What Operating System services are necessary to support kernel-level threads? -- will help me understand how KLTs are implemented.
Related
I am using some QThread based worker threads in KDE Neon 18.04 (based on Ubuntu 18.04, Kernel 4.15.0-46-generic). The worker threads interfered with my desktop applications so I decided to reduce their priority.
The Qt documentation of QThread::start(priority) says:
The effect of the priority parameter is dependent on the operating
system's scheduling policy. In particular, the priority will be
ignored on systems that do not support thread priorities (such as on
Linux, see http://linux.die.net/man/2/sched_setscheduler for more
details).
After reading the above documentation I expected priorities would have no effect on my Linux system. Still I gave it a try. And guess what - it worked perfectly.
So, why does the Qt documentation state there would be no thread priorities on Linux? And why does it work anyway?
Depending on which flavour of Linux/Unix/*Nix you use the scheduler may or may not support it. As far as I'm aware the majority of Qt's priority levels are supported on most Linux systems now, but not all of the priority levels. I suspect the documentation says it's unsupported so they don't need to list every combination of OS variant and scheduler variant that do support priority levels and which levels are supported.
You can validate it has created it with the correct priority by using htop or top and processing with awk: https://unix.stackexchange.com/questions/19301/what-is-a-command-to-find-priority-of-process-in-linux
If, for example, there is a let's say embedded application which run on unicore CPU. And then that application would be ported on multi core CPU. Would that app run on single or multiple cores?
To be more specific I am interested in ARM CPU (but not only) and toolchain specifics e. g. standard C/C++ libraries.
The intention of this question is this: is it CPU's responsibility to "decide" to execute on multiple cores or compiler toolchain, developer and standard platfor specific libraries? And again, I am interested also in other systems' tendencies out there.
There are plenty of applications and RTOS (for example Linux) that run on different CPUs but the same architecture, so does that mean that they are compiled differently?
Generally speaking single-threaded code will always run on one core. To take advantage of multiple cores you need to have either multiple processes, multiple threads, or both.
There's nothing your compiler can do to help you here. This is an architectural consideration.
If you have multiple threads, for example, most multi-core systems will run them on whatever cores are available if the operating system you're running is properly compiled to support that. Running an OS that's been compiled single-core only will obviously limit your options here.
A single threaded program will run in one thread. It is theoretically possible for the thread to be scheduled to move to a different core, but the scheduler cannot turn a single thread into multiple threads and give you any parallel processing.
EDIT
I misunderstood your question. If there are multiple threads in the application, and that application is binary compatible with the new multicore CPU, the threads will indeed be scheduled to run on different CPUs, if the OS scheduler deems it appropriate.
Well it all depends on the software that if it wants to utilize other cores or not (if present). Lets take an example of Linux on ARM's cortexA53.
Initially a vendor provided boot loader runs on, FSBL (First state bootloader). It then passes control to Arm trusted firmware. ATF then runs uboot. All these run on a single core. Then uboot loads linux kernel and passes control to it. Linux then initializes some stuff and looks into some option, first in the bootargs for smp or nosmp flags. if smp it will get the number of CPUs assigned to it from dtb and then using SMC calls to ATF it will start other cores and then assign work to those cores to provide true feel of multiprocessing environment. This is normally called load balancing and in linux it is mostly done in fair.c file.
I have read that linux kernel is multi threaded and there can be multiple threads running concurrently in each core. In a SMP (symmetric multiprocessing) environment where a single OS manages all the processors/cores how is multithreading implemented?
Is that kernel threads are spawned and each dedicated to manage a core. If so when are these kernel threads created? Is it during bootup at kern_init() after the bootstrapping is complete and immediately after the Application processors are enabled by the bootstrap processor.
So does each core have its own scheduler(implemented by the core's kernel thread) that manages the tasks from a common pool shared by all kernel threads?
How does (direct) messaging between kernel threads residing on different cores happen when they need to intimate some events that another kernel thread might be interested in?
I also thought if one particular selected core with one kernel scheduler that on every system timer interrupt acquire a big kernel lock and decide/schedule what to run on each core?
So I would appreciate any clarity in the implementation details. Thanks in advance for your help.
Early in kernel startup, a thread is started for each core. It is set to the lowest possible priority and generally does nothing but reduce the CPU power and wait for an interrupt. When actual work needs to get done, it's either done by threads other than these threads or by hardware interrupts which interrupt either this thread or some other thread.
The scheduler is typically invoked either by a timer interrupt or by a thread transitioning from running to a state in which it's no longer ready to run. Kernel calls that transition a thread to a state in which it's no longer ready to run typically invoke the scheduler to let the core perform some other task.
I'm working on an embedded Linux project where we need to ACK a message from the serial port within 200ms. If I'm not using some real-time variant then won't it be impossible to guarantee Linux would respond within that time bound? The hardware will be a 200MHz ARM running Debian. The kernel version currently used is "2.6.32 #1 SMP PREEMPT". Would also like to know exactly what PREEMPT means here.
Preemptive means an arbritary task with the right priority can interrupt any task running in the scheduler of the OS at any time. With this feature you could guarantee that your task with the timing constraint can meet it's requirement.
The picture below illustrates exactly what scheduling is doing:
Nowadays, virtually every popular OS support preemptive scheduling in user space. However, in kernel space (drivers, other critical kernel tasks) are not supported by this kind of scheduling so there are some initiatives like RTLinux and also your OS, Debian SMP PREEMPT which try to support full preemptive scheduling in the OS (making it a hard realtime system).
So yes, your kernel would provide the timing constraints needed for your application.
It means that it's a real-time kernel. Preempt means to interrupt or stop an action. So if an irq request comes into the kernel, the kernel immediately stops what it's doing and processes irq. So you get a real time response to events happening on the serial bus or an audio input.
I use build real-time kernels when needed (usually audio applications) and it's a series of patches that can be added to the vanilla kernel to make it preemptive (real time).
I don't know without testing if you can respond to a serial request in 200ms but that definitely sounds possible.
Along with Debian, RedHat and other distros with preemptive kernels, there are whole distros dedicated to realtime Linux like RTLinux and I would guess most of them would have ARM versions.
I'm looking for a thread pool abstraction in Linux that provides the same level of kernel scheduler support that the Win32 thread pool provides. Specifically, I'm interested in finding a thread pool that maintains a certain number of running threads. When a running pool thread blocks on I/O, I want the thread pool to be smart enough to start another thread running.
Anyone know of anything like this for linux?
You really can't do this without OS support. There's no good way to tell that a thread is blocked on I/O. You wind up having to atomically increment a counter before each operation that might block and decrement it after. Then you need a thread to monitor that counter and create an additional thread if it's above zero. (Remove threads if they're idle more than a second or so.)
Generally speaking, it's not worth the effort. This only works so well on Windows because it's the "Windows way" and Windows is built from the ground up for it. For Linux, you should be using epoll or boost::asio. Use something that does things the "Linux way" rather than trying to make the Windows way work on non-Windows operating systems.
You can write your own wrappers that use IOCP on Windows, epoll on Linux, and so on. But these already exist, so you need not bother.