I found that there is a function called task_running in the kernel, and its judgment logic is as follows
static inline int task_current(struct rq *rq, struct task_struct *p)
{
return rq->curr == p;
}
static inline int task_running(struct rq *rq, struct task_struct *p)
{
#ifdef CONFIG_SMP
return p->on_cpu;
#else
return task_current(rq, p);
#endif
}
Is there any difference between rq->curr and p->on_cpu? I think they both mean that the process is being scheduled by the current cpu. Why are separate judgments required under SMP?
On a multiprocessor system (CONFIG_SMP) the scheduler usually needs to continuously acquire and releases the spinlocks p->pi_lock (for tasks i.e. struct task_struct) and rq->lock (for runqueues i.e. struct rq). Since lock contention is one of the main factors of slowdown in the case of multiprocessor systems, the on_cpu field of task_struct was added to avoid acquiring rq->lock to look at rq->curr.
On a uniprocessor system (no CONFIG_SMP), there is only one CPU with one runqeueue, and there is no need to acquire runqueue locks when checking the running task. Since the scheduler will not do any locking anyway, we can also avoid having the on_cpu field in task_struct, saving some memory and also dumping all the code that deals with it. In fact, if you take a look at the code for struct task_struct, you can see:
struct task_struct {
/* ... */
#ifdef CONFIG_SMP
int on_cpu;
/* ... */
#endif
/* ... */
}
Here's the relevant commit that implemented this optimization. It was part of this patchwork by Peter Zijlstra to reduce lock contention in the scheduler (expand the "related" field to list everything).
Related
The main question is: How we can wait for a thread in Linux kernel to complete? I have seen a few post concerned about proper way of handling threads in Linux kernel but i'm not sure how we can wait for a single thread in the main thread to be completed (suppose we need the thread[3] be done then proceed):
#include <linux/kernel.h>
#include <linux/string.h>
#include <linux/errno.h>
#include <linux/sched.h>
#include <linux/kthread.h>
#include <linux/slab.h>
void *func(void *arg) {
// doing something
return NULL;
}
int init_module(void) {
struct task_struct* thread[5];
int i;
for(i=0; i<5; i++) {
thread[i] = kthread_run(func, (void*) arg, "Creating thread");
}
return 0;
}
void cleanup_module(void) {
printk("cleaning up!\n");
}
AFAIK there is no equivalent of pthread_join() in kernel. Also, I feel like your pattern (of starting bunch of threads and waiting only for one of them) is not really common in kernel. That being said, there kernel does have few synchronization mechanism that may be used to accomplish your goal.
Note that those mechanisms will not guarantee that the thread finished, they will only let main thread know that they finished doing the work they were supposed to do. It may still take some time to really stop this tread and free all resources.
Semaphores
You can create a locked semaphore, then call down in your main thread. This will put it to sleep. Then you will up this semaphore inside of your thread just before exiting. Something like:
struct semaphore sem;
int func(void *arg) {
struct semaphore *sem = (struct semaphore*)arg; // you could use global instead
// do something
up(sem);
return 0;
}
int init_module(void) {
// some initialization
init_MUTEX_LOCKED(&sem);
kthread_run(&func, (void*) &sem, "Creating thread");
down(&sem); // this will block until thread runs up()
}
This should work but is not the most optimal solution. I mention this as it's a known pattern that is also used in userspace. Semaphores in kernel are designed for cases where it's mostly available and this case has high contention. So a similar mechanism optimized for this case was created.
Completions
You can declare completions using:
struct completion comp;
init_completion(&comp);
or:
DECLARE_COMPLETION(comp);
Then you can use wait_for_completion(&comp); instead of down() to wait in main thread and complete(&comp); instead of up() in your thread.
Here's the full example:
DECLARE_COMPLETION(comp);
struct my_data {
int id;
struct completion *comp;
};
int func(void *arg) {
struct my_data *data = (struct my_data*)arg;
// doing something
if (data->id == 3)
complete(data->comp);
return 0;
}
int init_module(void) {
struct my_data *data[] = kmalloc(sizeof(struct my_data)*N, GFP_KERNEL);
// some initialization
for (int i=0; i<N; i++) {
data[i]->comp = ∁
data[i]->id = i;
kthread_run(func, (void*) data[i], "my_thread%d", i);
}
wait_for_completion(&comp); // this will block until some thread runs complete()
}
Multiple threads
I don't really see why you would start 5 identical threads and only want to wait for 3rd one but of course you could send different data to each thread, with a field describing it's id, and then call up or complete only if this id equals 3. That's shown in the completion example. There are other ways to do this, this is just one of them.
Word of caution
Go read some more about those mechanisms before using any of them. There are some important details I did not write about here. Also those examples are simplified and not tested, they are here just to show the overall idea.
kthread_stop() is a kernel's way for wait thread to end.
Aside from waiting, kthread_stop() also sets should_stop flag for waited thread and wake up it, if needed. It is usefull for threads which repeat some actions infinitely.
As for single-shot tasks, it is usually simpler to use works for them, instead of kthreads.
EDIT:
Note: kthread_stop() can be called only when kthread(task_struct) structure is not freed.
Either thread function should return only after it found kthread_should_stop() return true, or get_task_struct() should be called before start thread (and put_task_struct() should be called after kthread_stop()).
As I was going through the below chunk of Linux char driver code, I found the structure pointer current in printk.
I want to know what structure the current is pointing to and its complete elements.
What purpose does this structure serve?
ssize_t sleepy_read (struct file *filp, char __user *buf, size_t count, loff_t *pos)
{
printk(KERN_DEBUG "process %i (%s) going to sleep\n",
current->pid, current->comm);
wait_event_interruptible(wq, flag != 0);
flag = 0;
printk(KERN_DEBUG "awoken %i (%s)\n", current->pid, current->comm);
return 0;
}
It is a pointer to the current process ie, the process which has issued the system call.
From the docs:
The Current Process
Although kernel modules don't execute sequentially as applications do,
most actions performed by the kernel are related to a specific
process. Kernel code can know the current process driving it by
accessing the global item current, a pointer to struct task_struct,
which as of version 2.4 of the kernel is declared in
<asm/current.h>, included by <linux/sched.h>. The current pointer
refers to the user process currently executing. During the execution
of a system call, such as open or read, the current process is the one
that invoked the call. Kernel code can use process-specific
information by using current, if it needs to do so. An example of this
technique is presented in "Access Control on a Device File", in
Chapter 5, "Enhanced Char Driver Operations".
Actually, current is not properly a global variable any more, like it
was in the first Linux kernels. The developers optimized access to the
structure describing the current process by hiding it in the stack
page. You can look at the details of current in <asm/current.h>. While
the code you'll look at might seem hairy, we must keep in mind that
Linux is an SMP-compliant system, and a global variable simply won't
work when you are dealing with multiple CPUs. The details of the
implementation remain hidden to other kernel subsystems though, and a
device driver can just include and refer to the
current process.
From a module's point of view, current is just like the external
reference printk. A module can refer to current wherever it sees fit.
For example, the following statement prints the process ID and the
command name of the current process by accessing certain fields in
struct task_struct:
printk("The process is \"%s\" (pid %i)\n",
current->comm, current->pid);
The command name stored in current->comm is the base name of the
program file that is being executed by the current process.
Here is the complete structure the "current" is pointing to
task_struct
Each task_struct data structure describes a process or task in the system.
struct task_struct {
/* these are hardcoded - don't touch */
volatile long state; /* -1 unrunnable, 0 runnable, >0 stopped */
long counter;
long priority;
unsigned long signal;
unsigned long blocked; /* bitmap of masked signals */
unsigned long flags; /* per process flags, defined below */
int errno;
long debugreg[8]; /* Hardware debugging registers */
struct exec_domain *exec_domain;
/* various fields */
struct linux_binfmt *binfmt;
struct task_struct *next_task, *prev_task;
struct task_struct *next_run, *prev_run;
unsigned long saved_kernel_stack;
unsigned long kernel_stack_page;
int exit_code, exit_signal;
/* ??? */
unsigned long personality;
int dumpable:1;
int did_exec:1;
int pid;
int pgrp;
int tty_old_pgrp;
int session;
/* boolean value for session group leader */
int leader;
int groups[NGROUPS];
/*
* pointers to (original) parent process, youngest child, younger sibling,
* older sibling, respectively. (p->father can be replaced with
* p->p_pptr->pid)
*/
struct task_struct *p_opptr, *p_pptr, *p_cptr,
*p_ysptr, *p_osptr;
struct wait_queue *wait_chldexit;
unsigned short uid,euid,suid,fsuid;
unsigned short gid,egid,sgid,fsgid;
unsigned long timeout, policy, rt_priority;
unsigned long it_real_value, it_prof_value, it_virt_value;
unsigned long it_real_incr, it_prof_incr, it_virt_incr;
struct timer_list real_timer;
long utime, stime, cutime, cstime, start_time;
/* mm fault and swap info: this can arguably be seen as either
mm-specific or thread-specific */
unsigned long min_flt, maj_flt, nswap, cmin_flt, cmaj_flt, cnswap;
int swappable:1;
unsigned long swap_address;
unsigned long old_maj_flt; /* old value of maj_flt */
unsigned long dec_flt; /* page fault count of the last time */
unsigned long swap_cnt; /* number of pages to swap on next pass */
/* limits */
struct rlimit rlim[RLIM_NLIMITS];
unsigned short used_math;
char comm[16];
/* file system info */
int link_count;
struct tty_struct *tty; /* NULL if no tty */
/* ipc stuff */
struct sem_undo *semundo;
struct sem_queue *semsleeping;
/* ldt for this task - used by Wine. If NULL, default_ldt is used */
struct desc_struct *ldt;
/* tss for this task */
struct thread_struct tss;
/* filesystem information */
struct fs_struct *fs;
/* open file information */
struct files_struct *files;
/* memory management info */
struct mm_struct *mm;
/* signal handlers */
struct signal_struct *sig;
#ifdef __SMP__
int processor;
int last_processor;
int lock_depth; /* Lock depth.
We can context switch in and out
of holding a syscall kernel lock... */
#endif
};
I have a multi-threaded process where a file is shared (read and written) by multiple threads. Is there any way a thread can lock one file segment so that other threads cannot access it?
I have tried fcntl(fd, F_SETLKW, &flock), but this lock only works for processes, not threads (a lock is shared between all threads in an process).
Yes - but not with the same mechanism. You'll have to use something like pthread mutexes, and keep track of the bookkeeping yourself.
Possible outline for how to make this work
Wait on and claim a process-level mutex over a bookkeeping structure
make sure no other threads within your process are trying to use that segment
mark yourself as using the file segment
Release the process-level mutex
Grab fnctl lock for process (if necessary)
Do your writing
Release fnctl lock to allow other processes to use the segment (if necessary)
Wait again on process-levelbookkeeping structure mutex (may not be necessary if you can mark it unused atomically)
mark segment as unused within your process.
Release process-level mutex
No. The region-locking feature you're asking about has surprising semantics and it is not being further developed because it is controlled by POSIX. (In fact, it is Kirk McKusick's preferred example of what's wrong with POSIX.) If there is a non-POSIX byte-range lock facility in Linux, I can't find it.
There is discussion of the problems of POSIX byte-range locking in a multithreaded world here: http://www.samba.org/samba/news/articles/low_point/tale_two_stds_os2.html.
However, if you're concerned only with threads within one process, you can build your own region-locking using semaphores. For example:
#include <stdbool.h>
#include <pthread.h>
#include <sys/types.h>
// A record indicating an active lock.
struct threadlock {
int fd; // or -1 for unused entries.
off_t start;
off_t length;
};
// A table of all active locks (and the unused entries).
static struct threadlock all_locks[100];
// Mutex housekeeping.
static pthread_mutex_t mutex;
static pthread_cond_t some_lock_released;
static pthread_once_t once_control = PTHREAD_ONCE_INIT;
static void threadlock_init(void) {
for (int i = 0; i < sizeof(all_locks)/sizeof(all_locks[0]); ++i)
all_locks[i].fd = -1;
pthread_mutex_init(&mutex, (pthread_mutexattr_t *)0);
pthread_cond_init(&some_lock_released, (pthread_condattr_t *)0);
}
// True iff the given region overlaps one that is already locked.
static bool region_overlaps_lock(int fd, off_t start, off_t length) {
for (int i = 0; i < sizeof(all_locks)/sizeof(all_locks[0]); ++i) {
const struct threadlock *t = &all_locks[i];
if (t->fd == fd &&
t->start < start + length &&
start < t->start + t->length)
return true;
}
return false;
}
// Returns a pointer to an unused entry, or NULL if there isn't one.
static struct threadlock *find_unused_entry(void) {
for (int i = 0; i < sizeof(all_locks)/sizeof(all_locks[0]); ++i) {
if (-1 == all_locks[i].fd)
return &all_locks[i];
}
return 0;
}
// True iff the lock table is full.
static inline bool too_many_locks(void) {
return 0 == find_unused_entry();
}
// Wait until no thread has a lock for the given region
// [start, start+end) of the given file descriptor, and then lock
// the region. Keep the return value for threadunlock.
// Warning: if you open two file descriptors on the same file
// (including hard links to the same file), this function will fail
// to notice that they're the same file, and it will happily hand out
// two locks for the same region.
struct threadlock *threadlock(int fd, off_t start, off_t length) {
pthread_once(&once_control, &threadlock_init);
pthread_mutex_lock(&mutex);
while (region_overlaps_lock(fd, start, length) || too_many_locks())
pthread_cond_wait(&some_lock_released, &mutex);
struct threadlock *newlock = find_unused_entry();
newlock->fd = fd;
newlock->start = start;
newlock->length = length;
pthread_mutex_unlock(&mutex);
return newlock;
}
// Unlocks a region locked by threadlock.
void threadunlock(struct threadlock *what_threadlock_returned) {
pthread_mutex_lock(&mutex);
what_threadlock_returned->fd = -1;
pthread_cond_broadcast(&some_lock_released);
pthread_mutex_unlock(&mutex);
}
Caution: the code compiles but I haven't tested it even a little.
If you don't need file locks between different processes, avoid the file locks (which are one of the worst designed parts of the POSIX API) and just use mutexes or other shared memory concurrency primitives.
There are 2 ways you can do it:
Use Mutex to get a record's lock in a thread within the same process. Once the lock is acquired, any other thread in the process, mapping the file that tries to acquire the lock is blocked until the lock is released.(Preferable and only most straightforward solution available in Linux).
Semaphores and mutexes on a shared memory or a memory mapped file.
I have read a few things from which I can make out that instead of scheduling a task with a scheduling policy it is better that we schedule an entity with a scheduling policy. The advantages being that you can schedule many things with the same scheduling policy. So there are two entities defined for two scheduling policies(CFS and RT) namely as sched_entity and sched_rt_entity. The code for CFS entity is (from v3.5.4)
struct sched_entity {
struct load_weight load; /* for load-balancing */
struct rb_node run_node;
struct list_head group_node;
unsigned int on_rq;
u64 exec_start;
u64 sum_exec_runtime;
u64 vruntime;
u64 prev_sum_exec_runtime;
u64 nr_migrations;
#ifdef CONFIG_SCHEDSTATS
struct sched_statistics statistics;
#endif
#ifdef CONFIG_FAIR_GROUP_SCHED
struct sched_entity *parent;
/* rq on which this entity is (to be) queued: */
struct cfs_rq *cfs_rq;
/* rq "owned" by this entity/group: */
struct cfs_rq *my_q;
#endif
};
and for RT(real time) entity is
struct sched_rt_entity {
struct list_head run_list;
unsigned long timeout;
unsigned int time_slice;
struct sched_rt_entity *back;
#ifdef CONFIG_RT_GROUP_SCHED
struct sched_rt_entity *parent;
/* rq on which this entity is (to be) queued: */
struct rt_rq *rt_rq;
/* rq "owned" by this entity/group: */
struct rt_rq *my_q;
#endif
};
Both of these uses the list_head structures defined in ./include/linux/types.h
struct list_head {
struct list_head *next, *prev;
};
I honestly do not understand how any such thing is going to be scheduled. Can anyone explain how this is working.
P.S.:
Moreover, I am really having a hard time understanding the meaning of the names of data members. Can anyone suggest a good read for understanding kernel structures so that I can figure out these things a bit easily. Most of the time I spend is wasted in searching what a data member could mean.
Scheduling entities were introduced in order to implement group scheduling, so that CFS (or RT scheduler) will provide fair CPU time for individual tasks but also fair CPU time to groups of tasks. Scheduling entity may be either a task or group of tasks.
struct list_head is just Linux way to implement linked list. In the code you posted fields group_node and run_list allow to create lists of struct sched_entity and struct sched_rt_entity. More information can be found here.
Using these list_heads scheduling entities are stored in certain scheduler related data structures, for example cfs_rq.cfs_tasks if an entity is a task enqueued using account_entity_enqueue().
Always up to date documentation of Linux kernel can be found within its sources. In this case you should check this directory and especially this file which describes CFS. There is also an explanation of task groups.
EDIT: task_struct contains a field se of type struct sched_entity. Then, having an address to a sched_entity object using container_of macro it is possible to retrieve an address to the task_struct object, see task_of(). (address of sched_entity object - offset of se in task_struct = address of task_struct object) This is quite common trick used also in the implementation of lists I mentioned earlier in this answer.
What is the difference between calling mlock() on some memory and calling shmctl(SHM_LOCK) on that same memory?
These are the only differences I can ascertain:
mlock() guarantees that all locked pages are loaded when it returns. shmctl(SHM_LOCK) prevents swapping, but doesn't proactively load non-resident pages.
shmctl(SHM_LOCK) can only be used on shared memory segments.
shmctl(SHM_LOCK) sets an extra flag (SHM_LOCKED) on the permissions of the shared memory segment.
Are there other differences? In particular, is there any reason not to use mlock() on a shared memory segment?
First of all, mlock() is the syscall to lock process memory in the RAM, and shmctl(X,SHM_LOCK,Y) is used to do shared (IPC) memory, which requires more control from all the producers and consumers, so, that is why mlock() syscall is much easier as:
int mlock(const void *addr, size_t len);
While shmctl is much more complex to operate, as:
int shmctl(int shmid, SHM_LOCK, struct shmid_ds *buf);
Where:
The buf argument is a pointer to a shmid_ds structure, defined in as follows:
struct shmid_ds {
struct ipc_perm shm_perm; /* Ownership and permissions */
size_t shm_segsz; /* Size of segment (bytes) */
time_t shm_atime; /* Last attach time */
time_t shm_dtime; /* Last detach time */
time_t shm_ctime; /* Last change time */
pid_t shm_cpid; /* PID of creator */
pid_t shm_lpid; /* PID of last shmat(2)/shmdt(2) */
shmatt_t shm_nattch; /* No. of current attaches */
...
};