Consider a case where we have multiple processor/cores and two threads. Is it possible to force the linux scheduler to always schedule the specific thread(both) to a specific processor at every instance of its execution. Is setting processor affinity to the threads, while creation, sufficient for this purpose
If you look at the man page for taskset you can see the following statement:
The Linux scheduler will honor the given CPU affinity and the process will not run on any other CPUs.
This means that setting the CPU affinity for a particular process will make sure that it's always run on that CPU.
There exist API's that allow you to set thread affinity for particular threads, and I would imagine that this too will be honored by the OS scheduler.
If you look at sched_setaffinity you'll see a line that says:
These restrictions on the actual set of CPUs on which the process will run are silently imposed by the kernel.
which means this will make sure your threads are only run on CPU's set by this function.
Related
I use os/exec pkg to have a process run. I want to check it cpu affinity and modify it to bind the process to a specific cpu set. I find
func SchedSetaffinity(pid int, set *CPUSet) error
This function is in golang.org/x/sys/unix package. However, it says it just bind a thread to a specific cpu. I don't know wheter it works on process. And I wonder how to get the CPUSet. Is it a value I need to define?
Taskset : To enable a process run on a specific CPU, you use the command 'taskset' in linux. Accordingly you can arrive on a logic based on "taskset -p [mask] [pid]" where the mask represents the cores in which the particular process shall run, provided the whole program runs with GOMAXPROCS=1.
pthread_setaffinity_np : You can use cgo and arrive on a logic that calls pthread_setaffinity_np, as Go uses pthreads in cgo mode. (The pthread_attr_setaffinity_np() function sets the CPU affinity mask attribute of the thread attributes object referred to by attr to the value specified in cpuset. )
Go helps in incorporation of affinity control via "SchedSetaffinity" that can be checked for confining a thread to specific cores. Accordingly , you can arrive on a logic for usage of "SchedSetaffinity(pid int, set *CPUSet)" that sets the CPU affinity mask of the thread specified by pid. If pid is 0 the calling thread is used.
It should be noted that GOMAXPROCS variable limits the number of operating system threads that can execute user-level Go code simultaneously. If it is > 1 then, you may use runtime.LockOSThread of Go that shall pin the current goroutine to the current thread that is is running on . The calling goroutine will always execute in that thread, and no other goroutine will execute in it, until the calling goroutine has made as many calls to UnlockOSThread as to LockOSThread.
cgroups : There is also option of using cgroups that helps in organizing the processes hierarchically and distribution of system resources along the hierarchy in a controlled and configurable manner. Here, there is subsystem termed as cpuset that enables assigning individual CPUs (on a multicore system) and memory nodes to process in a cgroup. The cpuset lists CPUs to be used by tasks within this cgroup. The CPU numbers are comma-separated numbers or ranges. For example:
#cat cpuset.cpus
0-4,6,8-10
A process is confined to run only on the CPUs in the cpuset it belongs to, and to allocate memory only on the memory nodes in that cpuset. It should be noted that all processes are put in the cgroup that the parent process belongs to at the time on creation and a process can be migrated to another cgroup. Migration of a process doesn't affect already existing descendant processes.
Linux includes a few privileged processes called kernel threads. Is there any scheduler which runs/suspends them? If yes, is this scheduler the same as the system scheduler (I mean the one to schedule the whole system processes)?
The Linux scheduler is scheduling tasks. These can be
kernel threads (e.g. kswapd), or
single-threaded processes (e.g. bash), or
individual threads of a multi-threaded process (e.g. some browsers or servers)
The many threads of a multi-threaded process are tasks sharing a common address space (and other things, e.g. file descriptors).
AFAIK, the scheduler does not separate kernel threads from other tasks. But the scheduler do take into account scheduling policies (sched_setscheduler(2)) and priorities (setpriority(2)) (For most kernel threads, the priority is often very high). See sched(7)
Yes ! Let me clarify the system scheduler part here.
Every task is associated with a task_struct which contains the details of each task say its pid, its name, when it recently started, priority etc etc.http://lxr.free-electrons.com/source/include/linux/sched.h#L1224
Typically depending on the priority of the task either Fair scheduler or Real time scheduler kicks in and these co exist. Just to keep it simple and not to go into details, these are different scheduler algorithms that cater to different type of tasks.
Now Kernel threads also have an associated task_struct and as #Basile Starynkevitch pointed a couple of KPI's, we can use sched_setparam KPI's to modify the sched params and change the scheduler to which the task belongs to depening on what they are about to do.
I have a number of kernel threads that I want to get off of a given core for performance reasons. Some of these I am able to move using taskset however there are others I cannot.
In particular I see processes like migration, watchdog, rcuc, etc. that do not respond to my attempt to rebind them.
For example, if I try to rebind the watchdog process, I get the following:
# taskset -pc 0 207
pid 207's current affinity list: 0
sched_setaffinity: Invalid argument
failed to set pid 207's affinity.
How can I get these off of the cores so I can properly isolate them for performance reasons?
I suspect these processes are interfering with my full dynticks mode.
Several kernel threads are tied to a specific core, in order to effect capabilities needed by the SMP infrastructure, such as synchronization, interrupt handling and so on. The kworker, migration and ksoftirqd threads, for example, usually have one instance per virtual processor (e.g. 8 threads on a 4-core 8-thread CPU).
You cannot (and should not be able to) move those threads - without them that processor would not be fully usable by the system any more.
Why exactly do you want to move those threads anyway?
So I was reading about Processes and Threads and I had a question. Following is the scenario.
Uniprocessor Environment
I understand that the OS rotates the processes over processor for a particular time period.(quantum) . Now I get it when the process is single threaded, ie just one path of execution. In that case, whenever it is assigned the processor, it continues with it's execution. Let's say the process forks and or just creates a new thread. Now how does the entire process works? Is it that the OS will say to process P "Go on, continue with execution" and the Process within itself will pick the new thread or the parent thread on rotation? So that if there are more than two threads, the rotation seems fair to each thread. Or does the OS actually interacts with the threads? (In that case I am not sure what happens).
Multiprocessor Environment
Now say I have a multiprocessor environment. Now in this case, if there was just uni-threaded process, then OS will assign either of the processors to it and on it will go with it's execution. Now say, there are multiple threads in the Process. Now if I assign one of the processor to the process, and ask it to continue it's execution, and the Process has to pick either of the thread for it's execution, then there never will be parallel processing going on in that specific process. Since the process will have to put either of it's threads on the processor.
So how does it happen in both the cases?
Cheers.
Process Scheduing
Operating Systems ultimately control these types of thread scheduling.
Windows systems are priority-based and so will allow a process to consume more resources that others. This is why your machine can 'hang', if a process has been escalated to a high priority. Priorities are ranged between 1-31 as far as I know.
Mac OS / Linux / Unix are time-based, allowing all processes to have equal amounts of CPU time. Therefore loading more processes will slow your system down as they all share a smaller slice of execution time.
Uniprocessor Environment
The OS is ultimately responsible for this but switching processes involves (I cannot guarantee accuracy here, but its just an indication):
Halting a process / thread
Storing the current stack (code location)
Storing the current registers of the CPU
Asking the kernel for the next process/thread to run
Kernel indicates which one has to be run
OS reloads the registers from the cache
OS reloads the current stack for the next application.
Resumes the process
Obviously the more threads and processes you have running, the slower it will become. The problem is that the time taken to switch processes can actually take longer than the time allowed to execute the process.
Threads are just child processes of a single process. For a single processor, it just looks like additional work.
Multi-processor Environment
Multi-processor environments work differently as the cache is shared amongst processors. I believe these are called L1 (Level) and L2 caches. So the difference is that processor A can reload the state stored by processor B without conflicts. 'Hyper-threading' also has the same approach, although this is processor specific. The difference here is that a processor could solely control a specific process - this is called 'CPU Affinity' Its not encouraged for every process, but it does allow an application to have a dedicated processor to work off.
This is OS-specific, of course, but most operating systems schedule at the thread level. A process is just a grouping of threads. For example, on Linux, threads are called "tasks" and each is scheduled independently. They are created with the clone call. What is typically called a thread is a task which shares its address space (and other resources such as file descriptors, mount points, etc.) with the creating task. Note that the clone call can also create what is typically called a process if the flags to enable sharing are not passed.
Considering the above, any thread may be scheduled at any time on any processor, no matter how many processors there are available. That said, most OSs also attempt to maintain some measure of processor affinity to avoid excessive cache misses, but usually if a thread is runnable and a different CPU is available, it will change CPUs. Often there is also a way to specify which CPUs a particular thread may execute upon.
Doesn't matter whether there is 1 or 128 processors. The OS manages access to resources to try an efficiently match up requests with availabilty, and that includes CPU execution. If a thread is running, it has already managed to get some CPU but, if it requests a resource that is not immediately available, it no longer needs any CPU until that other resource does become free, and so the OS will remove CPU execution from it and, if there is another thread that is waiting for CPU, it will hand it over. When the requested reource does become available, the thread will be made ready again. If there is a core free, it will be made running 'immediately', if not, the CPU scheduling algorithm makes a decision on whether to stop a currently-running thread to free up a core or to leave the newly-ready thrad waiting.
It's better to try and ignore things like 'time-slice, quantum, priority' - it causes much confusion and FUD. If a running thread wants something it cannot have yet, it doesn't need any more CPU cycles, and the OS will take them away and, if another thread needs it, apply them there. That is why preemptive multitaskers exist - to match up threads with resources in an attempt to maximize forward progress.
I have few queries related to threads and Process scheduling.
When my process goes into sleep and wakes back, is it always that it will be scheduled on the same CPU that it got scheduled before?
When i create a thread from the process, Will it also be executed on the same CPU always? Even if other CPU's are free and sleeping.
I would like to know the mechanism in Linux in specific. Also i am creating the threads through pthread library. I am facing a random hangup issue which is always not reproducible. Need this information to proceed in the right direction.
On single processor/core systems
Yes
Yes
on multi processor/core systems
No.
No.
use taskset to retrieve or set a processes’s CPU affinity on multicore systems. Setting the CPU affinity to a specific processor/core will change the answers to
Yes
Yes
also for multicore systems.
From within an application you may use sched_setaffinity and/or sched_getaffinity to adjust the CPU affinity.
Edit: Additional details about how/when CPU swaps are managed with respect to cache disadvantages:
The Linux/SMP Scheduler: "... In order to achieve good system performance, Linux/SMP (2.4 kernel) adopts an empirical rule to solve the dilemma ..." Read the details in the linked reference, section The Linux/SMP Scheduler.
For the newer CFS (Completely Fair Scheduler) you'd look at sched_migration_cost. "...if the real runtime of the task is smaller than the values of this parameter then the scheduler assumes that it is still in the cache and tries to avoid moving the task to another CPU during the load balancing procedure ..." (e.g.: Completely Fair Scheduler and its tuning).
when process goes in to sleep and when it wake up ,it is not necessary that it will schedule on same cpu.if u have multiprocessor environment then according to scheduler policy it will schedule on any cpu.When process goes to sleep there are different reason ,it goes to sleep beacause it is waiting for io,any resource.When event will occurs it goes from waiting state to ready state.At that time which cpu will be free scheduler will schedule that process on free cpu.It is not necessary it will schedule on same cpu.
for extra information about scheduler open source code of scheduler in linux release tree path.