I am working on some real-time programs that require mlock and FIFO scheduling policy for fast paths.
I am running two processes on Ubuntu 16.04 with 12 CPU cores, and I assigned the fast paths of these processes to different cores.
Process 1 starts normally and pins its fast thread to a CPU and sets the scheduling policy to FIFO on this thread.
When process 2 starts, before its fast thread is created, it tries to call mlock.
Then, process 2 is stuck.
I attached gdb to process 2, and the call stack seems to be inside the mlock function.
If I remove the FIFO setting on process 1, both processes can run normally.
My suspicion is that mlock is trying to access some kernel resources that is acquired by the fast thread of process 1.
So it is blocked and put on wait indefinitely.
Does anyone know exactly what it is waiting for?
I have observed this problem on two similar IBM servers with Ubuntu.
But, on a Supermicro machine with a Redhat Linux, this issue didn't occur.
Thanks for any hint or solution!
If you have a SCHED_FIFO process that completely occupied a CPU such that non-sched-fifo threads never get scheduled on that CPU, some in-kernel algorithms may stop working, depending on kernel version/configuration.
Try booting with rcutree.kthread_prio=N where N is larger than SCHED_FIFO priority of your thread.
Try playing with /proc/sys/kernel/sched_rt_runtime_us
Try to get in-kernel backtrace of hanged mlock() to understand where it is waiting - this may get a hint. For that, use /proc/pid/stack (if your kernel is compiled with CONFIG_STACKTRACE) or maybe 'echo t > /proc/sysrq-trigger'
Related
After the Kernel schedules a process that has threads, How does said process schedule its own threads during its time splice?
For most modern kernels, the kernel only schedules threads, and processes are mostly just a container for the threads to execute inside (e.g. a container that contains a virtual address space, however many threads, and a few other scraps like file handles).
For some kernels (mostly very old unix kernels that existed before threads were invented) the kernel schedules processes and then a user-space library emulates threads. For this to work properly all of the "blocking" system calls (e.g. write()) have to be replaced by asynchronous system calls (e.g. aio_write()) so that other threads in the process can be given CPU time; however I wouldn't want to assume it works properly (e.g. if any thread blocks, then maybe all threads in the process block).
Also it may not work when there's multiple CPUs (kernel gives a process one CPU, but then from the kernel's perspective that process is running and can't use a second CPU). There are sophisticated work-arounds for this (to support "M:N threading") but it's just easier and better to fix the scheduler so it works with threads. Fortunately/unfortunately this didn't matter much in the early days because very few computers had more than one CPU anyway.
Lastly; it doesn't work for thread priorities - e.g. one process might keep CPU busy executing an unimportant/low priority thread while another process doesn't get that CPU time when it desperately needs it for an important/high priority thread. This occurs because no process knows about threads belonging to other processes and the kernel only knows about processes and not threads.
Of course these are also the reasons why every kernel adopted "kernel schedules threads and not processes" (and those that didn't died).
It's down to jargon definitions, but threads are simply a bunch of processes sharing an address space. Older Unixes even called them Light Weight Processes.
With that classical understanding of threads, the answer is that, these days, it's the OS that does the scheduling and each thread gets its own timeslices.
Extras
Some OSes do things to "the whole process" - e.g. Windows will give the process that has mouse focus a priority boost (all it's threads get dynamically notched up a few priority places), to make that application appear to be more sprightly (this goes back to Windows 3).
Other operating systems will increase the priority of a thread dynamically, to solve priority inversion situations. This is where a low priority thread that has control of a resource (I/O, or perhaps a semaphore) is blocking a higher priority thread from running (because the resource is not available. This is the priority inversion, and it's solved by the OS boosting the priority of the blocking thread until it gives up the required resource.
Either the kernel schedules the threads or the kernel schedules processes simulates thread by scheduling it own threads.
Usually, the process schedules its own threads using a library that sets timers. When the timer handler saves the current "thread's" registers then loads a new set of registers from another "thread."
I was studying Operating Systems and am stuck on a doubt that when a currently running process on the processor requests for some I/O, the CPU becomes idle and the scheduler then schedules another process to execute on the CPU. How does the kernel here come to know that the CPU has become idle. Is there some kind of hardware interrupt sent by the processor?
The OS 'knows' that a CPU needs to become idle when it performs a scheduling run and has fewer ready threads than cores.
If the scheduler runs and has only two ready threads that can use CPU, but has four actual cores available, then it will direct the 'surplus' cores to an 'idle' thread that is a loop around a 'HLT', or like instruction, that causes the core to stop fetching and executing instructions until an interrupt is received.
In my option, the kernel always running on the CPU, and the kernel knows schedule which process or interrupt handler.
Say I have a running process with 2 running threads, what happen to the threads if the process is terminated? do they get terminated as well?
Also, what happen to the threads if the process is 'losing' the CPU attention (Other process took the CPU's attention) and therefore the process is in waiting state / suspended. Do its threads keep running? or?
thanks
Yes, the threads get disposed of by the operating system if and when the process terminates and gets cleaned up.
The threads in a process are scheduled according to their priority. In most OSes that is fairly straight forward, it's a simple matter of the higher priority threads winning. Windows does something different, a process's priority is adjusted by Windows according to whether it is the foreground process. Thus the process's threads' priorities are also adjusted...
Could someone help clarify my understanding of kernel threads. I heard that, on Linux/Unix, kernel threads(such as those of system calls) get executed faster than user threads. But, aren't those user threads scheduled by kernel and executed using kernel threads? could someone please tell me what is the difference between a kernel thread and a user thread other than the fact that they have access to different address spaces. what are other difference between them? Is it true that on a single processor box, when user thread is running, kernel will be suspended?
Thanks in advance,
Alex
I heard that, on Linux/Unix, kernel threads(such as those of system calls) get executed faster than user threads.
This is a largely inaccurate statement.
Kernel threads are used for "background" tasks internal to the kernel, such as handling interrupts and flushing data to disk. The bulk of system calls are processed by the kernel within the context of the process that called them.
Kernel threads are scheduled more or less the same way as user processes. Some kernel threads have higher than default priority (up to realtime priority in some cases), but saying that they are "executed faster" is misleading.
Is it true that on a single processor box, when user thread is running, kernel will be suspended?
Of course. Only one process can be running at a time on a single CPU core.
That being said, there are a number of situations where the kernel can interrupt a running task and switch to another one (which may be a kernel thread):
When the timer interrupt fires. By default, this occurs 100 times every second.
When the task makes a blocking system call (such as select() or read()).
When a CPU exception occurs in the task (e.g, a memory access fault).
First off, take a single processor system with multiple processes running in pseudo-parallel. When a process triggers a page fault, will this force the CPU to stop executing all programs until the page is loaded from disk?
If so, does this change on a multi-core or multiprocessor system, or can the other processes continue to read and write to memory while the page fault is dealt with?
Thanks!
First, scheduling does not work for processes but for threads. A page fault only suspends the thread incurring the fault (on Linux and Windows). The thread is descheduled and the CPU is free to do other work.
At the level of the OS interfacing hardware there is no synchronous IO anyway. It does not exist (at least with modern hardware). The OS does not sit in a tight spin-loop waiting for the hardware to signal IO completion. Instead, the thread is descheduled until the IO completed (or the respective wait handle becomes signaled).
Yes, this is not a problem at all. Nobody in their right mind designs a multi-process OS that's unable to run multiple processes, nor would they arbitrarily block process A because B is waiting on a disk I/O.