According to the POSIX standard, writes to a pipe are guaranteed to be atomic (if the data size is less than PIPE_BUF).
As far as I understand, this means that any thread trying to write to the pipe will never access the pipe in the middle of another thread's write. What's not clear to me is how this is achieved and whether this atomicity guarantee has other implications.
Does this simply mean that the writing thread acquires a lock somewhere inside the write function?
Is the thread that's writing to the pipe guaranteed to never be scheduled out of context during the write operation?
Pipe write are atomic upto the size of Pipe. Let assume that a pipe size is 4kb, then the write are atomic up to the data_size < 4kb. In the POSIX systems, kernel uses internal mutexes, and locks the file descriptors for the pipe. Then it allows the requesting thread to write. If any other thread requests write at this point, then it would have to wait for the first thread. After that the file descriptors are unlocked, so the other waiting threads can write to the pipe. So yes, kernel would not allow more than one thread to write to the pipe at the same time.
However, there is an edge case to think. If data of size close to 4kb has already been written, and the reading has not been done yet, then the pipe may not be thread safe. Because, at this point, it may be possible that the total bytes written to the pipe may exceed to the 4kb limit.
Related
I have a Linux process with two threads, both sharing the same file descriptor to write data of 400 bytes to the same pipe every 100ms. I'm wondering if POSIX guarantees that this is thread-safe or if I need to add additional synchronization mechanisms to serialize the writing to the pipe from multiple threads (not processes).
I'm also aware that POSIX guarantees that multiple writes to the same pipe from different processes that are less than PIPE_BUF bytes are atomically written. But I'm not sure if the same guarantee applies to writes from multiple threads within the same process.
Can anyone provide some insight on this? Are there any additional synchronization mechanisms that I should use to ensure thread safety when writing to the same pipe from multiple threads using the same file descriptor in Linux?
Thank you in advance for any help or advice!
In the posix standard, on general information we read:
2.9.1 Thread-Safety
All functions defined by this volume of POSIX.1-2008 shall be thread-safe, except that the following functions need not be thread-safe.
And neither read nor write are listed afterwards. And so indeed, it is safe to call them from multiple threads. This however only means that the syscall won't crash, it doesn't say anything about the exact behaviour of calling them in parallel. In particular, it doesn't say about atomicity.
However in docs regarding write syscall we read:
Atomic/non-atomic: A write is atomic if the whole amount written in one operation is not interleaved with data from any other process. This is useful when there are multiple writers sending data to a single reader. Applications need to know how large a write request can be expected to be performed atomically. This maximum is called {PIPE_BUF}. This volume of POSIX.1-2008 does not say whether write requests for more than {PIPE_BUF} bytes are atomic, but requires that writes of {PIPE_BUF} or fewer bytes shall be atomic.
And in the same doc we also read:
Write requests to a pipe or FIFO shall be handled in the same way as a regular file with the following exceptions:
and the guarantee about atomicity (when size below PIPE_BUF) is repeated.
man 2 write (Linux man-pages 6.02) says:
According to POSIX.1-2008/SUSv4 Section XSI 2.9.7 ("Thread Interactions
with Regular File Operations"):
All of the following functions shall be atomic with respect to each
other in the effects specified in POSIX.1-2008 when they operate on
regular files or symbolic links: ...
Among the APIs subsequently listed are write() and writev(2). And
among the effects that should be atomic across threads (and processes)
are updates of the file offset. However, before Linux 3.14, this was
not the case: if two processes that share an open file description (see
open(2)) perform a write() (or writev(2)) at the same time, then the
I/O operations were not atomic with respect to updating the file off-
set, with the result that the blocks of data output by the two pro-
cesses might (incorrectly) overlap. This problem was fixed in Linux
3.14.
So, it should be safe as long as you're running at least Linux 3.14 (which is almost 9 years old).
According to an answer on this question : Why doing I/O in Linux is uninterruptible? I/O on linux is uninterruptible (uninterruptible in sleep). But if I start a process ,say a large 'dd' on a file and while the process is going on I forcefully unmount the Filesystem (where the file is),the process gets killed . ideally it should be in a hung state because it is sleeping and is UN.
"Uninterruptible" applies to the low-level read/write operations handled by the kernel. In C programming, these correspond broadly to read() and write() calls on the C standard library. That a utility can be interrupted does not say much about whether I/O operations can be interrupted, because a specific file operation in a utility might correspond to many low-level I/O operations.
In the case of dd, the default transfer block size is 512 bytes, so copying a large file might consist of many I/O operations. dd can be interrupted between these operations. I would expect the same to apply to most utilities that operate on files. If you can force them to work with huge data blocks (e.g., specify a gigabyte-size argument for bs= in dd) then you might be able to see that low-level I/O operations are uninterruptible.
Imagine a situation in which several threads can write on a certain buffer (no restrictions on its size) and all of these constraints apply:
Before all the threads start, the buffer is initialized to zero.
A thread may write on the buffer or not.
If a thread writes on the buffer, it writes a certain string of bytes (let's call it V, with V being not only made of zeros and being the same for all threads).
No thread ever read from the buffer.
The fact that a thread writes on the buffer, and what it writes on the buffer, does not depend on the fact that other threads have written on the buffer, or not.
If a thread starts writing on the buffer, then it writes it completely.
The threads write on the buffer without following a precise order of the bytes.
Question is: after all the threads have stopped (and this happens only provided that they have finished writing on the buffer, if they started to), am I guaranteed, under any real-world or at least plausible architecture, that:
either the buffer contains all zeros,
or the buffer contains exactly V?
If not, is there any consistent argument stating that another string of bytes, call it V', may be written? If so, what could be the differences between V and V'? Why?
Number 6 appears to be the clincher, it basically states that a write to the buffer is atomic and non-interruptible. Hence the contents will either be 0 or a consistent v.
Of course, that's not the normal case with threads unless you use something like a mutex.
Yes this is guaranteed (if all threads really write the same values to the same locations). Stores of bytes are atomic. This means that no values can be "made up" by the system. The only possible values for each byte in the buffer are either zero (because it was pre-initialized) or the value that all threads write.
The question now is: can a zero slip though somehow? Can a location in the byte[] somehow end up appearing to not ever have been written to? Answer: no. It is unclear which write comes through but they are all the same. If at least one thread wrote to the buffer, it will contain exactly V. (If no one ever wrote it will be all zeros. This is trivial. So I'm assuming that at least one thread wrote to the buffer).
Of course your reading thread must wait for the writers to terminate in order to force a memory barrier. The barrier guarantees that all previous writes are visible to the reader.
I'm using pthreads on Linux, and one of my threads periodically calls the write function on a device file descriptor. If the write call takes a while to finish, will my thread be suspended so other threads can run? I didn't set any of the scheduling features of pthreads, so my question is about default thread behavior.
So long as nothing else is trying to write to the same resource, the other threads should run while the writing thread waits for its write to complete.
If a write() call blocks, only the calling thread is suspended. This is documented in the POSIX spec for write():
If there is enough space for all the
data requested to be written
immediately, the implementation should
do so. Otherwise, the calling thread
may block; that is, pause until enough
space is available for writing.
Note that it says calling thread, not calling process.
See if blocking behavior is explicitly defined here
http://www.akkadia.org/drepper/nptl-design.pdf
In principle, YES, other threads can run.
But be aware that some filesystems have locking mechnisms which permit only one concurrent IO operation on a single file. So if another thread does another IO on the same file (even if it's via a different file descriptor) it MAY block it for some of the duration of the write() system call.
There are also other in-kernel locks for other facililties. Most of them will not block other threads running unless they're doing closely related activities, however.
If your device file descriptor is a shared resource, you have to take care of locking. But once it's thread-safe, calls to such shared resource are serialized, thus if one thread writes, the rest are blocked. If locking is not implemented, the data may be garbled.
I write some data to a pipe - possibly lots of data and at random intervals. How to read the data from the pipe?
Is this ok:
in the main thread (current process) create two more threads (2, 3)
the second thread writes sometimes to the pipe (and flush-es the pipe?)
the 3rd thread has infinite loop which reads the pipe (and then sleeps for some time)
Is this so far correct?
Now, there are a few thing I don't understand:
do I have to lock (mutex?) the pipe on write?
IIRC, when writing to pipe and its buffer gets full, the write end will block until I read the already written data, right? How to check for read data in the pipe, not too often, not too rarely? So that the second thread wont block? Is there something like select for pipes?
It is possible to set the pipe to unbuffered more or I have to flush it regularly - which one is better?
Should I create one more thread, just for flushing the pipe after write? Because flush blocks as well, when the buffer is full, right? I just don't want the 1st and 2nd thread to block....
[Edit]
Sorry, I thought the question is platform agnostic but just in case: I'm looking at this from Win32 perspective, possibly MinGW C...
I'm not answering all of your questions here because there's a lot of them, but in answer to:
do I have to lock (mutex?) the pipe on write?
The answer to this question is platform specific, but in most cases I would guess yes.
It comes down to whether the write/read operations on the pipe are atomic. If either the read or write operation is non-atomic (most likely the write) then you will need to lock the pipe on writing and reading to prevent race conditions.
For example, lets say a write to the pipe compiles down to 2 instructions in machine code:
INSTRUCTION 1
INSTRUCTION 2
Let's say you get a thread context switch between these 2 instructions and your reading thread attempts to read the pipe which is in an intermediate state. This could result in a crash, or (worse) data corruption which can often manifest itself in a crash somewhere else in the code. This will often occur as a result of a race condition which are often non-deterministic and difficult to diagnose or reproduce.
In general, unless you can guarantee that all threads will be accessing the shared resource using an atomic instruction set, you must use mutexes or critical sections.