Does read system call implies a synchronization on the descriptor inside the kernel? I've seen some code use just a read call to synchronize and coordinate between multiple consumer threads, like this:
rfd,wfd = pipe() or socketpair(); // setup fd
// a single writer:
write(wfd, ...);
// multiple read threads, each blocks on read:
read(rfd, work); // my questions is here
// do the work assigned by writer
while I used to think that a explicit lock like pthread_mutex has to be used, like this:
pthread_mutex_t lock;
// work threads:
pthread_mutex_lock(&lock);
read(rfd, work);
pthread_mutex_unlock(&lock);
// do work
So my question is whether an explicit lock necessary in this situation? Does read call guarantee proper thread safety in this case?
It is safe for multiple threads to call read() on the same file descriptor at once, but the read() call does not synchronize memory (it is not listed in the functions specified by POSIX to do so).
This means that it is only safe to rely on just the read() if you transfer all information required over the file descriptor itself. If you want to use shared memory, you also need a memory synchronisation function - you don't have to hold the lock over the read() though, it would be safe to have a pattern like:
/* writer */
pthread_mutex_lock(&lock);
/* ...write to shared memory... */
pthread_mutex_unlock(&lock);
write(wfd, ...);
/* readers */
read(rfd, ...);
pthread_mutex_lock(&lock);
pthread_mutex_unlock(&lock);
/* ... read from shared memory ... */
However, this would be quite odd, because if you are using shared memory and mutexes, you might as well use part of the shared memory and a condition variable to implement the signalling from writer to reader, instead of involving file descriptors.
Related
In Linux, I have a scenario where two threads execute a critical section, one acquires the lock (thread A) and the other(thread B) will wait for the lock. Later threadA releases the mutex lock. I am trying to understand how threadB will be moved to the running state and acquire the lock? How threadB(or operating system) knows that the lock is released by threadA?
I have a theory, please correct if I am wrong. threadB enters TASK_INTERRUPTABLE (blocked at the mutex and so waiting) state and it receives signal when threadA unlocks the mutex so it comes back to the running queue(TASK_RUNNING).
The Linux mutex struct keeps track of the current owner of the mutex (if any):
struct mutex {
atomic_long_t owner;
// ...
There's also a struct to keep track of what other tasks are waiting on a mutex:
/*
* This is the control structure for tasks blocked on mutex,
* which resides on the blocked task's kernel stack:
*/
struct mutex_waiter {
struct list_head list;
struct task_struct *task;
struct ww_acquire_ctx *ww_ctx;
#ifdef CONFIG_DEBUG_MUTEXES
void *magic;
#endif
};
Simplifying quite a bit, when you unlock a mutex, the kernel looks at what other tasks are waiting on that mutex. It picks one of them to become the owner, sets the mutex's owner field to refer to the selected task, and removes that task from the list of tasks waiting for the mutex. At that point, there's at least a good chance that task has become un-blocked, in which case it'll be ready to run once it's unblocked. At that point, it's up to the scheduler to decide when to run it.
Optimization
Since mutexes are used a lot, and they get locked and unlocked quite a bit, they use some optimization to help speed. For example, consider the following:
/*
* #owner: contains: 'struct task_struct *' to the current lock owner,
* NULL means not owned. Since task_struct pointers are aligned at
* at least L1_CACHE_BYTES, we have low bits to store extra state.
*
* Bit0 indicates a non-empty waiter list; unlock must issue a wakeup.
* Bit1 indicates unlock needs to hand the lock to the top-waiter
* Bit2 indicates handoff has been done and we're waiting for pickup.
*/
#define MUTEX_FLAG_WAITERS 0x01
#define MUTEX_FLAG_HANDOFF 0x02
#define MUTEX_FLAG_PICKUP 0x04
#define MUTEX_FLAGS 0x07
So, when you ask the kernel to unlock a mutex, it can "glance" at one bit in the owner pointer to figure out whether this is a "simple" case (nobody's waiting on the mutex, so just mark it as unlocked, and off we go), or a more complex one (at least one task is waiting on the mutex, so a task needs to be selected to be unblocked, and be marked as the new owner of the mutex.
References
https://github.com/torvalds/linux/blob/master/include/linux/mutex.h
https://github.com/torvalds/linux/blob/master/kernel/locking/mutex.c
Disclaimer
The code extracts above are (I believe) current as I write this answer. But as noted above, mutexes get used a lot. If you look at the code for a mutex 5 or 10 years from now, chances are you'll find that somebody has done some work on optimizing the code, so it may not precisely match what I've quoted above. Most of the concepts are likely to remain similar, but changes in details (especially the optimizations) are to be expected.
I have a program where one thread creates a file and other threads are waiting for the file to be created. Once the file is created the other threads will read this file and continue their processing. I want these other threads to run in parallel. The thread that creates the file does not know how many threads are waiting for it. What synchronization model should I use?
If you're on Linux, you might look at the inotify interface. This allows a thread / threads to watch a filesystem for something happening (e.g. a file being created). inotify.
Or you could use a pub/sub pattern in ZeroMQ; the creating thread makes the file, then sends some sort of message into a PUB socket. Whichever SUBscribers are listening would all get that message (they'd be blocked waiting for the message in the meantime), and then read the file. This would probably be more efficient.
If it is not known upfront how many threads are going to be waiting, you can use a the 'completion' pattern - this is a mutex-protected flag, with an condition variable to signal changes:
int done = 0;
pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t cond = PTHREAD_COND_INITIALIZER;
To wait for the completion (ie. in the threads waiting for the file to be created), you use the condition variable to wait for the flag to be set:
pthread_mutex_lock(&lock);
while (!done)
pthread_cond_wait(&cond, &lock);
pthread_mutex_unlock(&lock);
To signal the completion (ie. in the thread that has created the file), you set the flag and signal the condition variable:
pthread_mutex_lock(&lock);
done = 1;
pthread_cond_signal(&cond);
pthread_mutex_unlock(&lock);
You could wrap this all up in a 'completion' data type if you wanted.
I am trying to find a way to do 1-to-many communication between processes in Linux. I found that linux has named pipes, but those won't work with multiple readers (I found on other SO answers that once a reader reads some data, others don't get it); could someones clarify this - I see mixed answers on SO - whether its possible or not.
Also, from my understanding, with sockets, multiple clients can connect to a server, but in my case, I need a process to send out data to multiple processes (kind-of reverse) - like a broadcast.
Could anyone suggest what options are available ?
Edit : Can shared memory be used some how ?
Named pipes indeed are just like a pipe, i. e. they have one input and one output. So you cannot use a single (named) pipe to send information to more than one other process.
But there is nothing against the idea of using one pipe for each other process and filling all pipes with the same information. This will work when using named pipes, (unnamed) pipes, and sockets. But it will in all cases mean that the sender keeps track of all the receivers and sends the information to each of them separately.
If you want to keep the sender uninformed about the list of receivers, then the only way I can think of right now is to use a plain file. Your sender can write it, and the receivers all can read it.
You can use shared memory with read-write semaphores to exchange data from one process to many process. The read-write semaphores can be used to synchronize the data between the processes.
The following is the pseudo code to synchronize communication between one writer thread and 20 reader threads. Write is blocked till all the reader threads read the shared memory.
if it is a writer thread
{
Take read-write mutex lock
increment writers++;
Release read-write mutex lock
Sem wait for the node-is-empty
Access the shared memory and fill the empty node
Take read-write mutext lock
Decrement writers--;
Release read-write mutex lock
Sem post for the node-is-filled
}
else /* want to read */
{
Take read-write mutex lock
if (writers > 0 || readers == 20) //reader default value is 20 reader threads
{
Release read-write mutex lock
sem wait for the node-is-filled
Take read-write mutex lock
}
Release read write mutex lock
printf("Shared Memory: %d\n", memory);
Take read-write mutex lock
readers--;
if (readers == 0) //wait for all the threads to read the buffer
sem post for the node-is-empty
Release read-write mutex lock
}
How does the current implementation of semaphores work? Does it use spinlocks or signals?
How does the scheduler know which one to invoke if signals are used?
Also how does it work in user space? Kernel locking recommends spinlocks but user space does not. So are the implementations different in user space and kernel space for semaphores?
Use the power of Open Source - just look at source code.
The kernel-space semaphore is defined as
struct semaphore {
raw_spinlock_t lock;
unsigned int count;
struct list_head wait_list;
};
lock is used to protect count and wait_list.
All tasks waiting on a semaphore reside in wait_list. When the semaphore is upped, one tasks is woken up.
User-space semaphores should rely on semaphore-related system calls, Kernel provides. The definition of user-space semaphores is:
/* One semaphore structure for each semaphore in the system. */
struct sem {
int semval; /* current value */
int sempid; /* pid of last operation */
spinlock_t lock; /* spinlock for fine-grained semtimedop */
struct list_head sem_pending; /* pending single-sop operations */
};
The kernel uses definition of the user-space semaphore similar to the kernel-space one. sem_pending is a list of waiting process plus some additional info.
I should highlight again that neither kernel-space semaphore, nor user-space one uses spinlock to wait on lock. Spinlock is included in both structures only to protect structure members from the concurrent access. After the structure is modified, spinlock is released and the task rests in list until woken.
Furthermore, spinlocks are unsuitable to wait on some event from another thread. Before acquiring a spinlock, kernel disables preemption. So, in this case, on uniprocessor machines, spinlock will never be released.
I should also notice that user-space semaphores, while serving on behalf of user-space, are executing in kernel-space.
P.S. Source code for the kernel-space semaphore resides in include/linux/semaphore.h and kernel/semaphore.c, for user-space one in ipc/sem.c
I've to write a thread-safe library that uses a POSIX semaphore (used as a mutex with initial value = 1) for sync. I found some problems to correctly manage async signals. I've an application that links against this static library and the application (multi-threaded) calls library's functions. Access to some internals structures are controlled by a posix semaphore (it's internal to the library):
void library_func1(lib_handler *h)
{
sem_wait(sem);
/* do some stuff with global data */
sem_post(sem);
}
void library_func2(lib_handler *h)
{
sem_wait(sem);
/* do some stuff with global data */
sem_post(sem);
}
void library_close(lib_handler *h)
{
...
}
What append if an async signal, let's say SIGINT, is raised when one thread is locking the semaphore? If i relaunch the application i'll have a deadlock because the semaphore exists and it's value is 0. There is a function library_close that could release the semaphore when the async signal is raised but which is the best way to do and check this (I think that that function would be signal-safe only if followed by exit)? In multi-threaded application usually is a good practice having a single thread manager for all signals: this thread should be in the library or is ok to launch it in the application?
Thank you all.
Linux futexes had the same problem. It is not fully solvable, but what you could do is write the pid of the process locking the semaphore somewhere in the same shared memory region. If another process tries to lock the semaphore and it is taking too long (for some value of 'too long'), it finds out what process has the semaphore locked by reading the pid from the shared memory. If that process no longer exists, you know you are in a deadlock (and you should probably just die since the library's internal data may be in an inconsistent state).
There's still a small race with this as the process taking the lock may die just after locking but before writing its pid. AFAIK there's no way to avoid this using semaphores. (It might work if you have a lock implementation where the pid is written to the lock variable atomically on aquire, but you would probably need to write this yourself.)
The state of a static library doesn't carry over between different runs of the app and isn't shared by other apps using it. it's part of the state of the application that's using it. So your semaphore won't be in a wonky state.