Does Creating a New Thread Duplicate File Descriptors and Socket Descriptors in Linux? - linux

Everyone knows the classic model of a process listening for connections on a socket and forking a new process to handle each new connection. Normal practice is for the parent process to immediately call close on the newly created socket, decrementing the handle count so that only the child has a handle to the new socket.
I've read that the only difference between a process and a thread in Linux is that threads share the same memory. In this case I'm assuming spawning a new thread to handle a new connection also duplicates file descriptors and would also require the 'parent' thread to close it's copy of the socket?

No. Threads share the same memory, so they share the same variables. If you close socket in parent thread, it will be also closed in child thread.
EDIT:
man fork: The child inherits copies of the parent’s set of open file descriptors.
man pthreads: threads share a range of other attributes (i.e., these attributes are process-wide rather than per-thread): [...] open file descriptors
And some code:
#include <cstring>
#include <iostream>
using namespace std;
#include <errno.h>
#include <fcntl.h>
#include <pthread.h>
#include <unistd.h>
// global variable
int fd = -1;
void * threadProc(void * param) {
cout << "thread: begin" << endl;
sleep(2);
int rc = close(fd);
if (rc == -1) {
int errsv = errno;
cout << "thread: close() failed: " << strerror(errsv) << endl;
}
else {
cout << "thread: file is closed" << endl;
}
cout << "thread: end" << endl;
}
int main() {
int rc = open("/etc/passwd", O_RDONLY);
fd = rc;
pthread_t threadId;
rc = pthread_create(&threadId, NULL, &threadProc, NULL);
sleep(1);
rc = close(fd);
if (rc == -1) {
int errsv = errno;
cout << "main: close() failed: " << strerror(errsv) << endl;
return 0;
}
else {
cout << "main: file is closed" << endl;
}
sleep(2);
}
Output is:
thread: begin
main: file is closed
thread: close() failed: Bad file descriptor
thread: end

In principle, Linux clone() can implement not only a new process (like fork()), or a new thread (like pthread_create perhaps), but also anything in between.
In practice, it is only ever used for one or the other. Threads created with pthread_create share the file descriptors with all other threads in the process (not just the parent). This is non-negotiable.
Sharing a file descriptor and having a copy is different. If you have a copy (like fork()) then all copies must be closed before the file handle goes away. If you share the FD in a thread, once one closes it, it's gone.

On Linux threads are implemented via the clone syscall using the CLONE_FILES flag:
If CLONE_FILES is set, the calling
process and the child processes share
the same file descriptor table. Any
file descriptor created by the calling
process or by the child process is
also valid in the other process.
Similarly, if one of the processes
closes a file descriptor, or changes
its associated flags (using the
fcntl(2) F_SETFD operation), the other
process is also affected.
Also have a look at the glibc source code for the details of how it is used in createthread.c:
int clone_flags = (CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGNAL
| CLONE_SETTLS | CLONE_PARENT_SETTID
| CLONE_CHILD_CLEARTID | CLONE_SYSVSEM
#if __ASSUME_NO_CLONE_DETACHED == 0
| CLONE_DETACHED
#endif
| 0);

Related

when the epoll_wait(edge triggered) receive EPOLLIN event

The fd in epoll will set to non-blocking,and regist with EPOLLIN event. if i don't read until EAGAIN each time,and the epoll_wait can receive the read event?I find a mail (golang-nuts) with a description like this:
One of the harder considerations about using edge-triggered epoll is you have to read()/write() until EGAIN, otherwise you will not receive another read/write event for the socket when you call epoll_wait.
I write a demo epollserver.cpp,
the server listens on port 12439 and reads only 1 character at a time,like:
while(1)
{
int num = epoll_wait(epfd, events, BUFSIZE, -1);
char buf[BUFSIZE] = {0};
for(int i = 0; i < num; i++)
{
if(events[i].events & EPOLLIN){
std::cout << "Ready to read!\n";
int size;
std::cout << "size=" << (size = read(events[i].data.fd, buf, 1) ,buf[size] = 0,size)
<< "\t buf=[" << buf << ']' << std::endl;
}
}
}
The client use nc -4 localhost 12439 and sends 4 characters at a time
However,the server epoll_wait can receive read event every time.
So,i wonder if i misanderstand something?
Yes, epoll_wait() will wake you up repeatedly until you have read all the input. This is the default behavior, unless you use the EPOLLET flag to enable edge-triggered mode.

Socket incoming connections can not push_back elements concurrently to a globally defined std::vector

I am new in socket programming and at this moment I am confronted with a problem that I can not solve. I have read from several sources that the C++ standard template (STL) containers are not thread-safe, so that one as a programmer has to impose a mechanism that ensures that several threads do not modify the data of a container concurrently.
For instance, Thread safety std::vector push_back and reserve
I have used the std::mutex class to make sure that nobody writes data in the same container at the same time when programming threads. However, this is not working for me when I use sockets.
Suppose I have 4 clients, each one sending data (int) to the server in the following order:
client_0: 4
client_1: 8
client_2: 5
client_4: 7
Observe the following code for a simple server:
#define PORT 60000
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <vector>
#include <string>
#include <iostream>
#include <mutex>
using namespace std;
vector<int> inputQueue; //<--------!
mutex mtx; //<---------------------!
void printVector(vector<int> input) {
cout << "inputQueue: [";
for (unsigned int i = 0; i < input.size(); i++ ) {
if (i != input.size() - 1)
cout << input[i] << ", ";
else
cout << input[i];
}
cout << "]." << endl;
}
int main(int argc, char const *argv[])
{
int server_fd, client_fd;
struct sockaddr_in address;
int opt = 1;
int addrlen = sizeof(address);
if ((server_fd = socket(AF_INET, SOCK_STREAM, 0)) == 0) {
perror("socket failed");
exit(EXIT_FAILURE);
}
if (setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR | SO_REUSEPORT, &opt, sizeof(opt))) {
perror("setsockopt");
exit(EXIT_FAILURE);
}
address.sin_family = AF_INET;
address.sin_addr.s_addr = INADDR_ANY;
address.sin_port = htons( PORT );
if (bind(server_fd, (struct sockaddr *)&address, sizeof(address))<0) {
perror("bind failed");
exit(EXIT_FAILURE);
}
if (listen(server_fd, 10) < 0) {
perror("listen");
exit(EXIT_FAILURE);
}
while(1) {
char buffer[4];
if ((client_fd = accept(server_fd, (struct sockaddr *)&address, (socklen_t*)&addrlen))<0) {
perror("accept");
exit(EXIT_FAILURE);
}
if (!fork()) {
recv(client_fd, buffer, 4, MSG_WAITALL);
int receivedInt = int(
(unsigned char)(buffer[0]) << 24 |
(unsigned char)(buffer[1]) << 16 |
(unsigned char)(buffer[2]) << 8 |
(unsigned char)(buffer[3])
);
mtx.lock(); //<-------------------------------------!
inputQueue.push_back(receivedInt); //<--------------!
cout << "Client context. Integer registered: " << receivedInt << ": inputQueue length is " << inputQueue.size() << endl;
printVector(inputQueue); //<------------------------!
mtx.unlock(); //<-----------------------------------!
close(server_fd); close(client_fd);
}
cout << "Server context: inputQueue length is " << inputQueue.size() << endl;
printVector(inputQueue);
}
return 0;
}
The server must receive data making sure that they do so in the same order and registering their respective data in a vector of integers, that is, std::vector<int> inputQueue, using the push_back() method, so that inputQueue = {4, 8, 5, 7} at the end of the reception of all the data by the clients.
I must clarify that inputQueue is a global variable, which when starting the execution of the server, does not contain elements, but they are added as the clients register.
The problem is that none of the client registers elements in inputQueue. Notice in the following code that, depending on where you put the cout << instruction, you can see that the inputQueue size is different. This shows that within the context of the client, each client overwrites the first element of inputQueue, but outside it none of the clients is able to register a single element in inputQueue.
Apparently, each socket has its own copy of inputQueue, so when it is destroyed, the modified copy of inputQueue is also destroyed.
Output is the following:
Server context: inputQueue length is 0
inputQueue: [].
Client context. Integer registered: 4: inputQueue length is 1
inputQueue: [4].
Server context: inputQueue length is 1
inputQueue: [4].
Server context: inputQueue length is 0
inputQueue: [].
Client context. Integer registered: 8: inputQueue length is 1
inputQueue: [8].
Server context: inputQueue length is 0
inputQueue: [].
Server context: inputQueue length is 1
inputQueue: [8].
Client context. Integer registered: 5: inputQueue length is 1
inputQueue: [5].
Server context: inputQueue length is 1
inputQueue: [5].
Server context: inputQueue length is 0
inputQueue: [].
Client context. Integer registered: 7: inputQueue length is 1
inputQueue: [7].
Server context: inputQueue length is 1
inputQueue: [7].
Does anyone have any idea why this happens and how could they solve it? I hope you can help me. Thank you
if (!fork()) {
fork() creates a completely new, independent process with its own virtual memory address space. The shown code, apparently, expects both the child process and the original process to be interacting through the same object, namely a vector, locked by a mutex.
That's not what happens. You now have two completely independent processes. This is no different than running your program twice, at the same time or in quck succession. Do you expect both running copies of your program to somehow share the same vector and mutex? Of course not.
What you are looking to do, instead, is to use std::thread to create a new execution thread in the same process. Your C++ book should have more information how to create new execution threads with std::thread.
Furthermore, even if you replace the fork() with an analogous execution thread: that will still not solve all the problems here. You will also need to correctly handle synchronization between multiple execution threads. Specifically: there are no guarantees whatsoever that a new execution thread will insert something into the vector, before the other execution thread attempts to printVector its contents. The new execution thread could manage to do that, before the original execution thread enters printVector. Or it may not, and printVector finds a completely empty vector, because the other execution thread hasn't managed to push something into it, quickly enough. You now have two completely independent execution threads running at the same time, and you have no guarantees as to which thread does what, first.
You can even get a different result every time you run the multithreaded version of the shown program (and you probably will).
When you are ready to begin tackling this new problem, your C++ book will explain how to use condition variables, together with mutexes, to correctly implement multi-threaded synchronization. Unfortunately, this is not a topic that can be completely covered in a brief answer on stackoverflow.com, but it should have several dedicated chapters in your C++ book, where you will find more information.
P.S. The only reason your output shows anything in the input queue is because there's nothing to stop the child process to continue executing the program when it exits its if statement, and ends up, itself, calling printVector. It's not coming from the parent process. Each child process ends up printing the value it itself inserted into its own vector.
As noted by Miles Budnek, you are creating a new child process. Sockets are global OS objects, so are working as expected. Your vector, and the memory it's stored in, is local to the process and therefore cannot be accessed by your new proc.
Consider looking into std::thread:
https://en.cppreference.com/w/cpp/thread/thread
One of the most used methods of starting a thread is with a lambda.
#include <thread>
#include <iostream>
auto print_number(int number) -> void
{
std::cout << number << std::endl; // This runs in the new thread.
}
int main()
{
int num = 12;
auto t = std::thread([num](){print_number(num);}); // Spawn new thread that calls the lambda
t.join(); // Wait for thread to finish execution
return 0;
}

std::async performance on Windows and Solaris 10

I'm running a simple threaded test program on both a Windows machine (compiled using MSVS2015) and a server running Solaris 10 (compiled using GCC 4.9.3). On Windows I'm getting significant performance increases from increasing the threads from 1 to the amount of cores available; however, the very same code does not see any performance gains at all on Solaris 10.
The Windows machine has 4 cores (8 logical) and the Unix machine has 8 cores (16 logical).
What could be the cause for this? I'm compiling with -pthread, and it is creating threads since it prints all the "S"es before the first "F". I don't have root access on the Solaris machine, and from what I can see there's no installed tool which I can use to view a process' affinity.
Example code:
#include <iostream>
#include <vector>
#include <future>
#include <random>
#include <chrono>
std::default_random_engine gen(std::chrono::system_clock::now().time_since_epoch().count());
std::normal_distribution<double> randn(0.0, 1.0);
double generate_randn(uint64_t iterations)
{
// Print "S" when a thread starts
std::cout << "S";
std::cout.flush();
double rvalue = 0;
for (int i = 0; i < iterations; i++)
{
rvalue += randn(gen);
}
// Print "F" when a thread finishes
std::cout << "F";
std::cout.flush();
return rvalue/iterations;
}
int main(int argc, char *argv[])
{
if (argc < 2)
return 0;
uint64_t count = 100000000;
uint32_t threads = std::atoi(argv[1]);
double total = 0;
std::vector<std::future<double>> futures;
std::chrono::high_resolution_clock::time_point t1;
std::chrono::high_resolution_clock::time_point t2;
// Start timing
t1 = std::chrono::high_resolution_clock::now();
for (int i = 0; i < threads; i++)
{
// Start async tasks
futures.push_back(std::async(std::launch::async, generate_randn, count/threads));
}
for (auto &future : futures)
{
// Wait for tasks to finish
future.wait();
total += future.get();
}
// End timing
t2 = std::chrono::high_resolution_clock::now();
// Take the average of the threads' results
total /= threads;
std::cout << std::endl;
std::cout << total << std::endl;
std::cout << "Finished in " << std::chrono::duration_cast<std::chrono::milliseconds>(t2 - t1).count() << " ms" << std::endl;
}
As a general rule, classes defined by the C++ standard library do not have any internal locking. Modifying an instance of a standard library class from more than one thread, or reading it from one thread while writing it from another, is undefined behavior, unless "objects of that type are explicitly specified as being sharable without data races". (N3337, sections 17.6.4.10 and 17.6.5.9.) The RNG classes are not "explicitly specified as being sharable without data races". (cout is an example of a stdlib object that is "sharable with data races" — as long as you haven't done ios::sync_with_stdio(false).)
As such, your program is incorrect because it accesses a global RNG object from more than one thread simultaneously; every time you request another random number, the internal state of the generator is modified. On Solaris, this seems to result in serialization of accesses, whereas on Windows it is probably instead causing you not to get properly "random" numbers.
The cure is to create separate RNGs for each thread. Then each thread will operate independently, and they will neither slow each other down nor step on each other's toes. This is a special case of a very general principle: multithreading always works better the less shared data there is.
There's an additional wrinkle to worry about: each thread will call system_clock::now at very nearly the same time, so you may end up with some of the per-thread RNGs seeded with the same value. It would be better to seed them all from a random_device object. random_device requests random numbers from the operating system, and does not need to be seeded; but it can be very slow. The random_device should be created and used inside main, and seeds passed to each worker function, because a global random_device accessed from multiple threads (as in the previous edition of this answer) is just as undefined as a global default_random_engine.
All told, your program should look something like this:
#include <iostream>
#include <vector>
#include <future>
#include <random>
#include <chrono>
static double generate_randn(uint64_t iterations, unsigned int seed)
{
// Print "S" when a thread starts
std::cout << "S";
std::cout.flush();
std::default_random_engine gen(seed);
std::normal_distribution<double> randn(0.0, 1.0);
double rvalue = 0;
for (int i = 0; i < iterations; i++)
{
rvalue += randn(gen);
}
// Print "F" when a thread finishes
std::cout << "F";
std::cout.flush();
return rvalue/iterations;
}
int main(int argc, char *argv[])
{
if (argc < 2)
return 0;
uint64_t count = 100000000;
uint32_t threads = std::atoi(argv[1]);
double total = 0;
std::vector<std::future<double>> futures;
std::chrono::high_resolution_clock::time_point t1;
std::chrono::high_resolution_clock::time_point t2;
std::random_device make_seed;
// Start timing
t1 = std::chrono::high_resolution_clock::now();
for (int i = 0; i < threads; i++)
{
// Start async tasks
futures.push_back(std::async(std::launch::async,
generate_randn,
count/threads,
make_seed()));
}
for (auto &future : futures)
{
// Wait for tasks to finish
future.wait();
total += future.get();
}
// End timing
t2 = std::chrono::high_resolution_clock::now();
// Take the average of the threads' results
total /= threads;
std::cout << '\n' << total
<< "\nFinished in "
<< std::chrono::duration_cast<
std::chrono::milliseconds>(t2 - t1).count()
<< " ms\n";
}
(This isn't really an answer, but it won't fit into a comment, especially with the command formatting an links.)
You can profile your executable on Solaris using Solaris Studio's collect utility. On Solaris, that will be able to show you where your threads are contending.
collect -d /tmp -p high -s all app [app args]
Then view the results using the analyzer utility:
analyzer /tmp/test.1.er &
Replace /tmp/test.1.er with the path to the output generated by a collect profile run.
If your threads are contending over some resource(s) as #zwol posted in his answer, you will see it.
Oracle marketing brief for the toolset can be found here: http://www.oracle.com/technetwork/server-storage/solarisstudio/documentation/o11-151-perf-analyzer-brief-1405338.pdf
You can also try compiling your code with Solaris Studio for more data.

Linux ptrace memory reading & process management

I'm trying to read a processes memory on Linux (Xubuntu, to be precise). I'm pretty new to Linux, though I've done this same read using Win32API ReadProcessMemory() before in Windows. The general idea is that I'm trying to develop some software for a game which will get my stats and upload them to a server, which will track my progress and keep a log of it. The end goal is to make a bot which will automatically play and farm data about the game. In order to do this, I need to be able to access the processes memory. In Windows, that's dead easy. In Linux, it's proving a little more complex.
I've found a memory address which contains information I want to read. The information is an int32, and it is stored at 84a1bd8. I found it using GameConqueror 0.13. The address remains correct after restarting, so it appears there is no ASLR (as there was in Windows). I also know the ProcessID (I can find this using task manager for now, though if someone knows a simple way to get a PID by either ClassName, Exe name, or similar, that would be great too!) So, that looks like it should be all I really need to use PTRACE_PEEKDATA to read the memory, right? Well, that's the problem, it doesn't appear to be. My code looks like this:
#include <iostream>
#include <string>
#include <sys/ptrace.h>
#include <errno.h>
using namespace std;
int main()
{
pid_t pid = 4847;
int addr = 0x84a1bd8;
long ret = ptrace(PTRACE_TRACEME, pid, NULL, NULL);
cout << "ptrace Status: " << ret << endl;
cout << "Errno: " << errno << endl;
ret = ptrace(PTRACE_PEEKDATA, pid, (void*)addr, NULL);
cout << "ptrace Status: " << ret << endl;
cout << "Errno: " << errno << endl;
ret = ptrace(PTRACE_DETACH, pid, NULL, NULL);
cout << "ptrace Status: " << ret << endl;
cout << "Errno: " << errno << endl;
return 0;
}
The output looks like this:
ptrace Status: 0
Errno: 0
ptrace Status: -1
Errno: 3
ptrace Status: -1
Errno: 3
Being quite new to Linux, I don't know where I'm to find error codes and how I can work out what this error actually means, and nor do I know if I am even declaring the address correctly. Should I declare it as an int in it's decimal equivalent? Is there anything I'm missing?
Thanks for your time
Found the solution to be that when using ptrace() you must call in an order:
ptrace(PTRACE_ATTACH, pid, NULL, NULL)
ptrace(PTRACE_PEEKDATA, pid, addr, NULL)
ptrace(PTRACE_DETACH, pid, NULL, NULL)
So the simple answer: You need to attach and detach before and after reading the memory.
It may also be useful to know that between the attach and detach commands, the process will sleep, meaning this method isn't so good for my purpose, but may be useful to others :)
Thanks to #PeterL. for your help.

Why sleep() after acquiring a pthread_mutex_lock will block the whole program?

In my test program, I start two threads, each of them just do the following logic:
1) pthread_mutex_lock()
2) sleep(1)
3) pthread_mutex_unlock()
However, I find that after some time, one of the two threads will block on pthread_mutex_lock() forever, while the other thread works normal. This is a very strange behavior and I think maybe a potential serious issue. By Linux manual, sleep() is not prohibited when a pthread_mutex_t is acquired. So my question is: is this a real problem or is there any bug in my code ?
The following is the test program. In the code, the 1st thread's output is directed to stdout, while the 2nd's is directed to stderr. So we can check these two different output to see whether the thread is blocked.
I have tested it on linux kernel (2.6.31) and (2.6.9). Both results are the same.
//======================= Test Program ===========================
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <pthread.h>
#define THREAD_NUM 2
static int data[THREAD_NUM];
static int sleepFlag = 1;
static pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
static void * threadFunc(void *arg)
{
int* idx = (int*) arg;
FILE* fd = NULL;
if (*idx == 0)
fd = stdout;
else
fd = stderr;
while(1) {
fprintf(fd, "\n[%d]Before pthread_mutex_lock is called\n", *idx);
if (pthread_mutex_lock(&mutex) != 0) {
exit(1);
}
fprintf(fd, "[%d]pthread_mutex_lock is finisheded. Sleep some time\n", *idx);
if (sleepFlag == 1)
sleep(1);
fprintf(fd, "[%d]sleep done\n\n", *idx);
fprintf(fd, "[%d]Before pthread_mutex_unlock is called\n", *idx);
if (pthread_mutex_unlock(&mutex) != 0) {
exit(1);
}
fprintf(fd, "[%d]pthread_mutex_unlock is finisheded.\n", *idx);
}
}
// 1. compile
// gcc -o pthread pthread.c -lpthread
// 2. run
// 1) ./pthread sleep 2> /tmp/error.log # Each thread will sleep 1 second after it acquires pthread_mutex_lock
// ==> We can find that /tmp/error.log will not increase.
// or
// 2) ./pthread nosleep 2> /tmp/error.log # No sleep is done when each thread acquires pthread_mutex_lock
// ==> We can find that both stdout and /tmp/error.log increase.
int main(int argc, char *argv[]) {
if ((argc == 2) && (strcmp(argv[1], "nosleep") == 0))
{
sleepFlag = 0;
}
pthread_t t[THREAD_NUM];
int i;
for (i = 0; i < THREAD_NUM; i++) {
data[i] = i;
int ret = pthread_create(&t[i], NULL, threadFunc, &data[i]);
if (ret != 0) {
perror("pthread_create error\n");
exit(-1);
}
}
for (i = 0; i < THREAD_NUM; i++) {
int ret = pthread_join(t[i], (void*)0);
if (ret != 0) {
perror("pthread_join error\n");
exit(-1);
}
}
exit(0);
}
This is the output:
On the terminal where the program is started:
root#skyscribe:~# ./pthread sleep 2> /tmp/error.log
[0]Before pthread_mutex_lock is called
[0]pthread_mutex_lock is finisheded. Sleep some time
[0]sleep done
[0]Before pthread_mutex_unlock is called
[0]pthread_mutex_unlock is finisheded.
...
On another terminal to see the file /tmp/error.log
root#skyscribe:~# tail -f /tmp/error.log
[1]Before pthread_mutex_lock is called
And no new lines are outputed from /tmp/error.log
This is a wrong way to use mutexes. A thread should not hold a mutex for more time than it does not own it, particularly not if it sleeps while holding the mutex. There is no FIFO guarantee for locking a mutex (for efficiency reasons).
More specifically, if thread 1 unlocks the mutex while thread 2 is waiting for it, it makes thread 2 runnable but this does not force the scheduler to preempt thread 1 or make thread 2 run immediately. Most likely, it will not because thread 1 has recently slept. When thread 1 subsequently reaches the pthread_mutex_lock() call, it will generally be allowed to lock the mutex immediately, even though there is a thread waiting (and the implementation can know it). When thread 2 wakes up after that, it will find the mutex already locked and go back to sleep.
The best solution is not to hold a mutex for that long. If that is not possible, consider moving the lock-needing operations to a single thread (removing the need for the lock) or waking up the correct thread using condition variables.
There's neither a problem, nor a bug in your code, but a combination of buffering and scheduling effects. Add an fflush here:
fprintf (fd, "[%d]pthread_mutex_unlock is finisheded.\n", *idx);
fflush (fd);
and run
./a.out >1 1.log 2> 2.log &
and you'll see rather equal progress made by the two threads.
EDIT: and like #jilles above said, a mutex is supposed to be a short wait lock, as opposed to long waits like condition variable wait, waiting for I/O or sleeping. That's why a mutex is not a cancellation point too.

Resources