This morning I read about Linux real time scheduling. As per the book 'Linux system programming by Robert Love', there are two main scheduling there. One is SCHED_FIFO, fifo and the second is SCHED_RR, the round robin. And I understood how a fifo and a rr algorithm works. But as we have the system call,
sched_setscheduler (pid_t pid, int policy, const struct sched_parem *sp)
we can explicitly set the scheduling policy for our process. So in some case, two process running by root, can have different scheduling policy. As one process having SCHED_FIFO and another having SCHED_RR and with same priority. In that case, which process will be selected first? the FIFO classed process or the RR classed process? Why?
Consider this case. There are three process A,B,C. All are having same priority. A and B are RR classed processes and C is FIFO classed one. A and B are runnable (so both are running alternatively in some time intervel). And currently A is running. Now C becomes runnable. In this case, whether
1. A will preempt for C, or
2. A will run until its timeslice goes zero and let C run. Or
3. A will run until its timeslice goes zero and let B run.
a) here after B runs till its timeslice becomes zero and let C run or
b) after B runs till its timeslice becomes zero and let A run again (then C will starve untill A and B finishes)
In realtime scheduling, FIFO and RR do not have exactly the same meaning they have in non-realtime scheduling. Processes are always selected in a FIFO- manner, however, the time quantum for SCHED_FIFO is not limited unlike the time quantum for SCHED_RR.
SCHED_FIFO processes do not preempt SCHED_RR processes of the same priority.
sched_setscheduler(2) - Linux man page
...
"A process's scheduling policy determines where it will be inserted into the list of processes with equal static priority and how it will move inside this list. All scheduling is preemptive: if a process with a higher static priority becomes ready to run, the currently running process will be preempted and returned to the wait list for its static priority level. The scheduling policy only determines the ordering within the list of runnable processes with equal static priority."
...
"A SCHED_FIFO process runs until either it is blocked by an I/O request, it is preempted by a higher priority process, or it calls sched_yield(2)."
...
"When a SCHED_FIFO process becomes runnable, it will be inserted at the end of the list for its priority."
...
"SCHED_RR: Round Robin scheduling
SCHED_RR is a simple enhancement of SCHED_FIFO. Everything described above for SCHED_FIFO also applies to SCHED_RR, except that each process is only allowed to run for a maximum time quantum. If a SCHED_RR process has been running for a time period equal to or longer than the time quantum, it will be put at the end of the list for its priority. A SCHED_RR process that has been preempted by a higher priority process and subsequently resumes execution as a running process will complete the unexpired portion of its round robin time quantum."
man sched_setscheduler explains these scheduling policies in detail.
In this particular case because the two real-time processes have the same priority none of them will preempt the other. A SCHED_FIFO process runs until it blocks itself, SCHED_RR process runs until it blocks itself or its time quantum expires.
According to the man page, I think 1 is the answer. A, B are RR policy, C is FIFO policy. Since RR is also an enhancement FIFO, all of them are FIFO class.
Since all of them have the same priority, and man page say " A call to sched_setscheduler() or sched_setparam(2) will put the SCHED_FIFO (or SCHED_RR) process identified by pid at the start of the list if it was runnable. As a consequence, it may preempt the currently running process if it has the same priority. (POSIX.1-2001 specifies that the process should go to the end of the list.)"
Once calling sched_setscheduler to set the policy of C as FIFO, C will preempt A.
My understanding of the two different classes is that a process SCHED_FIFO is never pre-empted by the kernel. Even if another "SCHED_FIFO" class process is waiting its turn...
While SCHED_RR policy shares the cpu ressources a little bit more. The scheduler will let the SCHED_RR process run for a quanta of time, then pre-empt it only to let turn another SCHED_RR process. That is exactly Round Robin.
SCHED_FIFO is "stronger" in the sense that if a SCHED_FIFO process never yield() to the kernel or invoke a system call on a single core device, then all your other Real time processes may never run.
Related
I'm currently learning pthreads and am struggling to understand the relationship between thread priority and policy. What I know so far:
The thread priority is an integer that indicates priority. The higher this number, the higher priority the thread is treated by the OS.
The thread policy determines how the thread is executed among processes with the shared priority number. SCHED_RR and SCHED_FIFO are real-time policies that continuously execute unless an explicit "sleep" command is issued. Thus a programmer must very carefully write code when using these policies. SCHED_OTHER is a round robin policy that is not executed in real-time.
However, let's say I have the following scenarios (assume each thread does not use a "sleep" command).
Thread 1: priority = 0, policy = SCHED_OTHER
Thread 2: priority = 1, policy = SCHED_OTHER
// would thread 1 run at all?
Thread 1: priority = 0, policy = SCHED_RR
Thread 2: priority = 1, policy = SCHED_RR
// would thread 1 run at all?
I'm confused as to whether or not the the thread policy affects the thread priority, or if the thread priority always trumps the policy.
Edit: Found a web page that cleared up most of my confusion: https://computing.llnl.gov/tutorials/pthreads/man/sched_setscheduler.txt
Documentation for SCHED_RR says that it is the same as SCHED_FIFO except in certain cases when two or more threads have the same static priority.
Documentation for SCHED_FIFO makes it clear that if a thread with higher static priority is ready-to-run but not running, and if one or more threads with lower static priority are running, then one of the lower priority threads will be preempted in favor of the higher priority thread.
would thread 1 run at all [in the SCHED_RR case]?
That depends. What is thread 0 doing? How many CPUs does the system have? If those were the only two threads on a system that had only one CPU, then thread 1 would be allowed to run whenever thread 0 did not want to run.
Generally speaking, when you use static priorities, you want the highest priority threads to do the least amount of work. A high priority thread should spend most of its time waiting for some event. Then when the event happens, the thread should promptly acknowledge it, and then possibly signal a lower-priority thread if some kind of follow-up computation is required.
would thread 1 run at all [in the SCHED_OTHER case]?
As mentioned in my comment, if you're talking about static priorities (i.e., as set by the sched_setattr() system call, then the question is meaningless because threads that are scheduled under the SCHED_OTHER policy are all required to have the same static priority--zero.
Do two SCHED_FIFO tasks with equal priority get processing time within each period in Linux, granted neither of the tasks finish before the period ends?
Linux documentation says SCHED_FIFO processes can get preempted only by processes with higher priority, but my understanding is that CFS operates on a higher layer, and assigns timeslots to each of the two tasks within each period.
Linux documentation says SCHED_FIFO processes can get preempted only by processes with higher priority
This is correct, in addition to this, they can also be preempted if you set RLIMIT_RTTIME (getrlimit(2)) and that limit is reached.
The only other reasons why another SCHED_FIFO process (with the same priority) can be scheduled is if the first sleeps or if it voluntary yields (voluntary preemption).
CFS has nothing to do with SCHED_FIFO, it only takes care of SCHED_NORMAL, SCHED_BATCH and SCHED_IDLE.
I am testing my multithreaded application in Linux RT multicore machine.
However during testing, we are observing that scheduling (created with SCHED_FIFO scheduling policy ) in Linux RT is not happening according to the SCHED_FIFO policy.
We could see in multiple places that the higher priority thread execution is getting preempted by a lower priority thread.
Based on some research we did on the internet, we found that the following kernel parameters need to be changed from
/proc/sys/kernel/sched_rt_period_us containing 1000000
/proc/sys/kernel/sched_rt_runtime_us containing 950000
to
/proc/sys/kernel/sched_rt_period_us containing 1000000
/proc/sys/kernel/sched_rt_runtime_us containing 1000000
or
/proc/sys/kernel/sched_rt_period_us containing -1
/proc/sys/kernel/sched_rt_runtime_us containing -1
We tried doing both but still we are facing the problem sometimes. We are facing the issue even when higher priority thread is not suspended by any system call.
It would be great if you could let us know if you are aware of such problems in Linux RT scheduling and/or have any solutions to make the Linux RT scheduling deterministic based on priority.
There are no printfs or any system calls in the higher priority thread but still the higher priority thread is getting preempted by the lower priority thread.
Also I have made sure all the threads in the process are running on a single core using taskset command.
There could be two reasons:
CPU throttling: the scheduler is designed to reserve some CPU time to non-RT tasks; you have already disabled it by acting on the /proc/sys/kernel/ entries
blocking: your high-priority task is blocking either
on some synchronization mechanism (e.g., mutex, semaphore) or
on some blocking call (e.g., malloc, printf, read, write, etc.)
this is from sched_setscheduler(2) - Linux man page:
"Processes scheduled under one of the real-time policies (SCHED_FIFO, SCHED_RR) have a sched_priority value in the range 1 (low) to 99 (high)."
"A SCHED_FIFO process runs until either it is blocked by an I/O request, it is preempted by a higher priority process, or it calls sched_yield(2)."
I have the following code:
struct sched_param sp;
memset( &sp, 0, sizeof(sp) );
sp.sched_priority = 99;
sched_setscheduler( 0, SCHED_FIFO, &sp );
Now the process should be running under the highest possible priority (99)
and should never be preempted.
So, when it starts running the following loop:
while ( 1 ) ;
it should be running forever and no other process should be allowed to run.
In spite of this, when I start such a process, I can use other processes too. Other processes run much slower, but they DO run.
My processor has 2 cores, so I started two copies of the process.
Usage of both cores jumped to 97%-100%. Both processes were running their infinite loop.
I could still type commands in a shell and watch their output. I could use GUI programs as well.
How's that possible since a SCHED_FIFO process with a priority of 99 should never be preempted?
If you haven't changed any other policy settings, then you're likely getting throttled. See this informative article on the real-time limits added to the scheduler a few years back.
The gist of it is: Unprivileged users can use SCHED_FIFO and try to soak the CPU, but the RT limits code will force a little bit of SCHED_OTHER in anyway so you don't wedge the system. From the article:
Kernels shipped since 2.6.25 have set the rt_bandwidth value for the
default group to be 0.95 out of every 1.0 seconds. In other words, the
group scheduler is configured, by default, to reserve 5% of the CPU
for non-SCHED_FIFO tasks.
SCHED_FIFO and SCHED_RR are both meant for real-time uses. I am aware that SCHED_RR can be preempted by time slicing. But say if I have one thread set to SCHED_FIFO, and another set to SCHED_RR, if both threads are ready to run, are they scheduled purely by priority? What if they have same priority?
Conceptually, there is a list of runnable processes associated with each static priority level. These lists can contain both SCHED_FIFO and SCHED_RR processes - the two scheduling policies share the same set of static priorities.
When selecting a process to run, the scheduler takes the process at the head of the non-empty list with the highest static priority, regardless of the scheduling policy of that process.
The scheduling policies affect how the processes move within those lists. For SCHED_FIFO, once a process reaches the head of the list for a given priority it will stay there until it blocks or yields. For SCHED_RR, a runnable process that has exceeded its maximum time quantum will be moved to the end of the list for its static priority.