What if multiple threads epoll wait on the same socket?
In my own experiment, it showed that only one thread can invoke epoll_wait successfully, the other threads show Invalid arguments error. Could someone explain it?
You can call epoll_wait concurrently on multiple threads for the same epoll_fd.
event.events = EPOLLIN | EPOLLET | EPOLLONESHOT;
http://www.csh.rit.edu/~rossdylan/presentations/EpollMT/
You can epoll_wait concurrently on multiple threads for the same fd. But epoll doesn't handle thread synchronization like IOCP. It is possible that all the threads come out of epoll_wait call when an event occurs on one of the sockets. Usually only one thread is enough to wait on epoll_wait. You can then give the task of receiving or sending data to other threads from the epoll_wait thread (polling thread).
Related
I have a pool of threads all calling epoll_wait on an epoll instance that is watching a set of file descriptors. All of the descriptors have been added to the epoll instance with EPOLLONESHOT. Is it guaranteed that only one thread will wake up when a file descriptor is triggered? Or is it possible that multiple threads will wake up? To rephrase that question, EPOLLONESHOT guaranteed to be thread safe and only wake one thread?
My testing seems to indicate that only one thread will wake up, but I'm looking for either a specification that asserts this or citations to kernel code.
I have several threads, one of them calls epoll_wait in a loop, others can open connections that need to be epoll'ed by first thread. Is it possible to just add new sockets with epoll_ctl while another thread waits in epoll_wait?
What will happen in the following scenario:
Thread 1 calls epoll_wait.
Thread 2 creates a socket(A) and adds it to epoll instance using epoll_ctl.
Someone sends some data, socket A becomes ready for read() call.
Will epoll_wait return socket A?
Yes, it will. The whole point of an epoll socket is that you don't have to duplicate effort. No snapshotting or use of multiple wait queues is involved.
Under the hood, the epoll socket has its own wait queue. When you block on the epoll socket, you are added to that single wait queue. No state is saved or anything like that. The state is in the epoll socket itself.
This is a follow-on to question 14221339.
I have a thread pool that is running inside an epoll_wait() loop.
An external thread calls epoll_ctl() and adds a listener socket with
(EPOLLET | EPOLLONESHOT | EPOLLIN).
When the thread pool only has a single thread, it intermittently fails to receive an EPOLLIN event for the first (and only) connection attempt. If I increase the thread pool to two, it almost always fails to receive an EPOLLIN event.
My understanding is that the epoll API is thread-safe, but this observation would seem to indicate otherwise.
With edge-triggered semantics, an incorrect sequence of calls can result in a race condition. There are three system calls involved:
epoll_ctl() to activate notifications (and reactivate if
EPOLLONESHOT is used).
epoll_wait() to receive notifications.
system input: read()/recv()/accept() in a loop until error EAGAIN.
If executed (repeatedly) in this order, a race between #3 and #1 is possible: when the input event in the kernel occurs after EAGAIN has been returned but before epoll_ctl() can be acted on. In general, the re-activation needs to be done before the I/O.
epoll_ctl() to activate notifications (and reactivate if
EPOLLONESHOT is used).
system input: read()/recv()/accept() in a loop until error EAGAIN.
epoll_wait() to receive notifications.
(Obviously, the I/O needs to be non-blocking.)
I have a pthread that runs in a loop, calling accept() in a blocking manner. Is there any way to interrupt that call from another thread? Everything points to sending the thread a signal, but apparently you can only send a process a signal.
I can't just kill the thread because then it leaves the socket open. And that's not very clean anyway. Is there really no way to do this?
You can signal a thread using pthread_kill(3).
The pthread_kill() function sends the signal sig to thread, another
thread in the same process as the caller.
If a signal handler is installed, the handler will be invoked in the
thread thread.
Note, you don't have to kill the thread; you can send a signal that simply makes accept fail with EINTR.
Either use select(), or send the singal to the process (this will be a problem if you just want to interupt one of the threads).
For example on windows there is MsgWaitForMultipleObjects that lets you asynchronously wait for windows messages, socket events, asynchronous io (IOCompletionRoutine), AND mutex handles.
On Unix you have select/poll that gives you everything except possibility to break out when some pthread_mutex is unlocked.
The story:
I have application that has main thread that does something with multiple sockets, pipes or files. Now from time to time there is a side job (a db transaction) that might take a longer time and if done synchronously in main thread would disrupt normal servicing of sockets. So I want to do the db operation in separate thread. That thread would wait on some mutex when idle until main thread decides to give it some job and unlocks the mutex so db thread can grab it.
The problem is how the db thread can notify back the main thread that it has finished the job. Main thread has to process sockets, so it cannot afford sleeping in pthread_mutex_lock. Doing periodic pthread_mutex_trylock is the last I would want to do. Currently I consider using a pipe, but is this the better way?
Using a pipe is a good idea here. Make sure that no other process has the write end of the pipe open, and then select() or poll() in the main thread the read end for reading. Once your worker thread is done with the work, close() the write end. The select() in the main thread wakes up immediately.
I don't think waiting on a mutex and something else would be possible, because on Linux, mutexes are implemented with the futex(2) system call, which doesn't support file descriptors.
I don't know how well it applies to your specific problem, but posix has message queues.