So I have some code that looks like this:
for (;;) {
errno=0;
epoll_event e = {};
auto wait_r = epoll_wait(g.epoll_fd, &e, 1, 0);
if (wait_r==0) break;
if(wait_r ==-1 && errno==EINTR) {
printf("got EINTR\n");
continue;
}
assert(wait_r == 1);
auto& c = *(Context*)e.data.ptr;
if(e.events & EPOLLERR ) {
int error = 0;
socklen_t errlen = sizeof(error);
auto r1 =getsockopt(c.socket, SOL_SOCKET, SO_ERROR, (void *)&error, &errlen);
assert(r1==0);
printf("Got EPOLLERR 2 %s\n", strerror(error));
}
if(e.events & EPOLLRDHUP || e.events & EPOLLHUP ) {
if (e.events & EPOLLRDHUP) {
printf("got to EPOLLRDHUP\n");
}
if (e.events & EPOLLHUP) {
printf("got to EPOLLHUP\n");
}
//continue; // keeps hitting this for same connections
break;
}
if (e.events & EPOLLIN) {
// Does a bunch of reads...
}
}
}
A single socket will getting stuck in the EPOLLRDHUP || EPOLLHUP case. The socket is likely closed, when I try to close it or do EPOLL_CTL_DEL I get a EBADFD. It was my understanding that epoll will automatically get rid of any dead sockets, but this doesn't seem to be the case... any ideas?
One other possible issue is that on the socket I am using recvmsg/sendmsg and I'm sending file descriptors between processes, over these sockets, which are unix domain stream sockets. I have tried to do a final recvmsg on it, but that fails as well... Any ideas?
For my issue, the solution was to change an a line like this:
c.socket = accept(g.server_socket, NULL, NULL);
to this:
c.socket = accept4(g.server_socket, NULL, NULL, SOCK_CLOEXEC);
If anyone else has this issue, look out for dup() and exec() calls. A dup() can cause epoll to act like it's not closed, even though you have already closed the fd that you added to epoll. epoll will only recognize that its closed once all copies of the fd are closed. exec() will essentially do the same thing as a dup() for every fd that you have that wasn't created with the SOCK_CLOEXEC flag...
Related
I have a non-blocking connected socket on Linux, and before my first send(), I do a select to see if the socket is ready for writing, but the select times out.
If I do not do the select, the send works fine.
If I do the select first, I never get to send anything.
Why is the select timing out before the send when the socket is obviously writable?
This is some of the code I am using:
fd_set writefds;
struct timeval timeout;//
FD_ZERO(&writefds);
FD_SET(s, &writefds);//s is my connected socket
timeout.tv_sec = 10;
timeout.tv_usec = 0;
ret = select(1, 0, &writefds, 0, &timeout);
switch(ret)
{
case 1://socket is ready for writing
bytesSent = send(s, (const char*)(buf+bytesSent), len-bytesSent, 0);
if(SOCKET_ERROR == bytesSent)
{
*error = errno;
if(EWOULDBLOCK == *error)
{
goto Exit;
}
sprintf(errmsg, ("send() failed %i\n"), *error);
}
else if(bytesSent < len)
{
sprintf(errmsg, ("send() incomplete\n"));
}
else
{
//sprintf(errmsg, ("sent %i bytes\n"), bytesSent);
}
break;
case 0://timeout --!!!! always comes here
bytesSent = -1;//treat as error
break;
default:
bytesSent = -1;
}
}
Blockquote
In your port from Windows select() to UNIX/Berkeley-derived select(), you preserved the dummy value you'd supplied as the first argument to select().
On UNIX, of course, this parameter (nfds) is meaningful and must be one greater than the highest fd in your sets. Thus, for you it should be s + 1.
I have 2 threads:
Thread A:
It's the select() loop. Perform socket handling for reading operations, such as accepting new connections, receiving data.
while (1) {
FD_ZERO(&fdReadSet);
numActiveSockets = 0;
for (std::unordered_map<SOCKET, TcpSocket*>::iterator it = m_sock_table.begin(); it != m_sock_table.end(); it++)
{
numActiveSockets++;
FD_SET(it->first, &fdReadSet);
}
int ret;
bool hasListen = false;
if (( ret = select(numActiveSockets, &fdReadSet, NULL, NULL, NULL)) == SOCKET_ERROR) {
printf("Select Failed, Error code = %d\n", WSAGetLastError());
return -1;
}
for (std::unordered_map<SOCKET, TcpSocket*>::iterator it = m_sock_table.begin(); it != m_sock_table.end(); it++)
{
if (FD_ISSET(it->first, &fdReadSet))
{
if (it->first == TcpSocket::m_listen_sock)
{
if (!hasListen)
{
sockaddr_in sock_addr;
int sockLength = sizeof(sock_addr);
SOCKET sock = accept(it->first, (sockaddr *) &sock_addr, &sockLength);
TcpSocket * socket = new TcpSocket();
socket->m_sock = sock;
m_sock_table[sock] = socket;
it = m_sock_table.begin();
hasListen = true;
}
}
else
{
char * buffer = it->second->GetWriteBuffer();
int numRead = recv(it->first, buffer, SOCKET_BUFFER_SIZE, 0);
if (numRead == SOCKET_ERROR)
{
int err = WSAGetLastError();
if (err == WSAECONNRESET)
{
printf("Connection [%i]: RESET Received. Closing Socket\n", it->first);
closesocket(it->first);
it = socketVector.erase(it->first); // iterator invalidated after erase
}
else
{
printf("Recv Failed. Error code = %d\n", err);
return -1;
}
}
else if (numRead == 0)//connection close
{
printf("Connection [%i]: Graceful exit. Closing Socket\n", it->first);
closesocket(it->first);
it = socketVector.erase(it->first); // iterator invalidated after erase
}
else {
/* Process received data */
}
}
}
}
}
Thread B:
Allow the application to perform connect() to establish new connections. If a connect() is successful, it will the add the returned socket to m_sock_table.
I have a socket table called m_sock_table which holds all the sockets. I use this m_sock_table to initialize the fdReadSet to be used in select().
-----------Problem-----------------
If thread A is blocked by select(), and at the same time thread B establish a new connection through connect(), the application wouldn't be able to receive data from the new connection, because fdReadset has not been updated withed the new connected socket.
What would be a good way to solve this problem? Or the design is just wrong from the start?
You could use a signal that doesn't do anything other than interrupting the system call:
#include <signal.h>
void do_nothing() { }
struct sigaction sa;
sa.sa_handler = do_nothing;
sigemptyset(sa.sa_mask);
#ifdef SA_INTERRUPT
sa.sa_flags = SA_INTERRUPT;
#else
sa.sa_flags = 0;
#endif
sigaction(SIGUSR1, &sa, 0);
Then, in thread B, after starting a new connection, send the signal, after making sure thread A will handle it:
/* need only be done once, but needed in every thread other than A */
sigset_t sigs;
sigemptyset(&sigs);
sigaddset(&sigs, SIGUSR1)
pthread_sigmask(SIG_BLOCK, &sigs, 0);
/* each time we create a new connection */
kill(getpid, SIGUSR1);
With the above, select will return with an EINTR error -- so check for that and loop (adding the new connection to the set).
I am using windows named pipe example. When I run sample programs to create pipe, write something and receive it in client program everything is just fine. When I move client code into dll, which runs in windows service it just won't receive sent bytes.
Server's code is as follows:
ThreadParams * params = reinterpret_cast<ThreadParams*>(args);
CString * connectionString = params->connectString;
HANDLE hPipe;
DWORD dwBytesRead;
TCHAR buf[1024];
int len;
hPipe = CreateNamedPipe(PIPE_NAME, // Name
PIPE_ACCESS_OUTBOUND | WRITE_OWNER, // OpenMode
PIPE_TYPE_BYTE | PIPE_READMODE_BYTE | PIPE_WAIT, // PipeMode
2, // MaxInstances
1024, // OutBufferSize
1024, // InBuffersize
2000, // TimeOut
NULL); // Security
if (hPipe == INVALID_HANDLE_VALUE)
{
Globals::WriteLog("Could not create the pipe",1);
exit(1);
}
Globals::WriteLog("connect...",1);
ConnectNamedPipe(hPipe, NULL);
Globals::WriteLog("...connected",1);
swprintf(buf, connectionString->GetBuffer());
len = wcslen(buf);
if (!WriteFile(hPipe, buf, len*sizeof(TCHAR), &dwBytesRead, NULL))
Globals::WriteLog("WriteFile failed",1);
else
wprintf(L"written %d bytes\n",dwBytesRead);
DisconnectNamedPipe(hPipe);
CloseHandle(hPipe);
And client:
CString finalResult = _T("");
HANDLE hOut = INVALID_HANDLE_VALUE;
TCHAR buf[1024];
DWORD len;
DWORD dwWritten;
Global::WriteLog("pwrite: waiting for the pipe...",1);
if (WaitNamedPipe(PIPE_NAME, NMPWAIT_WAIT_FOREVER) == 0)
{
Global::WriteLog("WaitNamedPipe failed. error=%d",1,GetLastError());
goto cleanup;
}
Global::WriteLog("the pipe is ready",1);
hOut = CreateFile(PIPE_NAME,
GENERIC_READ,
0,
NULL, OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL,
NULL);
if (hOut == INVALID_HANDLE_VALUE)
{
Global::WriteLog("CreateFile failed with error %d",1,GetLastError());
goto cleanup;
}
Global::WriteLog("Opened the pipe",1);
for (;;)
{
if (!ReadFile(hOut, buf, sizeof(buf), &dwWritten, NULL))
{
Global::WriteLog("ReadFile failed -- probably EOF. Read %d bytes.",1,dwWritten);
goto cleanup;
}
else
break;
}
finalResult = CString(buf);
Global::WriteLog("String from pipe:%S",1,buf);
cleanup:
if(hOut != INVALID_HANDLE_VALUE)
CloseHandle(hOut);
Server's code runs in a thread if that changes anything (I've tested it with this thread version in sample program and there was no problem with this).
Why does it not work?
Thanks in advance
Okay, it seems I figured it out. It seems I didn't understood documentation properly.
At server's side WriteFile function doesn't block until the string is read. My program simply wrote data and then closed handle - pipe. Client didn't catch the message and threw error stating that there is no process on the other side of pipe.
Also from client I removed for(;;) loop.
To wait for read operation on client side to complete I added
FlushFileBuffers(hPipe);
after successful write operation.
Hope that help somebody
I need async connect and disconnect for tcp client using epoll for Linux. There are ext. functions in Windows, such as ConnectEx, DisconnectEx, AcceptEx, etc...
In tcp server standard accept function is working, but in tcp client doesn't working connect and disconnect... All sockets are nonblocking.
How can I do this?
Thanks!
To do a non-blocking connect(), assuming the socket has already been made non-blocking:
int res = connect(fd, ...);
if (res < 0 && errno != EINPROGRESS) {
// error, fail somehow, close socket
return;
}
if (res == 0) {
// connection has succeeded immediately
} else {
// connection attempt is in progress
}
For the second case, where connect() failed with EINPROGRESS (and only in this case), you have to wait for the socket to be writable, e.g. for epoll specify that you're waiting for EPOLLOUT on this socket. Once you get notified that it's writable (with epoll, also expect to get an EPOLLERR or EPOLLHUP event), check the result of the connection attempt:
int result;
socklen_t result_len = sizeof(result);
if (getsockopt(fd, SOL_SOCKET, SO_ERROR, &result, &result_len) < 0) {
// error, fail somehow, close socket
return;
}
if (result != 0) {
// connection failed; error code is in 'result'
return;
}
// socket is ready for read()/write()
In my experience, on Linux, connect() never immediately succeeds and you always have to wait for writability. However, for example, on FreeBSD, I've seen non-blocking connect() to localhost succeeding right away.
From experience, when detect non-blocking connection , epoll is a little different from select and poll.
with epoll:
After connect() call is made, check return code.
If the connection can not be completed immediately, then register EPOLLOUT event with epoll.
Call epoll_wait().
if the connection failed, your events will be fill with EPOLLERR or EPOLLHUP, otherwise EPOLLOUT will be triggered.
I have a "complete" answer here in case anyone else is looking for this:
#include <sys/epoll.h>
#include <errno.h>
....
....
int retVal = -1;
socklen_t retValLen = sizeof (retVal);
int status = connect(socketFD, ...);
if (status == 0)
{
// OK -- socket is ready for IO
}
else if (errno == EINPROGRESS)
{
struct epoll_event newPeerConnectionEvent;
int epollFD = -1;
struct epoll_event processableEvents;
unsigned int numEvents = -1;
if ((epollFD = epoll_create (1)) == -1)
{
printf ("Could not create the epoll FD list. Aborting!");
exit (2);
}
newPeerConnectionEvent.data.fd = socketFD;
newPeerConnectionEvent.events = EPOLLOUT | EPOLLIN | EPOLLERR;
if (epoll_ctl (epollFD, EPOLL_CTL_ADD, socketFD, &newPeerConnectionEvent) == -1)
{
printf ("Could not add the socket FD to the epoll FD list. Aborting!");
exit (2);
}
numEvents = epoll_wait (epollFD, &processableEvents, 1, -1);
if (numEvents < 0)
{
printf ("Serious error in epoll setup: epoll_wait () returned < 0 status!");
exit (2);
}
if (getsockopt (socketFD, SOL_SOCKET, SO_ERROR, &retVal, &retValLen) < 0)
{
// ERROR, fail somehow, close socket
}
if (retVal != 0)
{
// ERROR: connect did not "go through"
}
}
else
{
// ERROR: connect did not "go through" for other non-recoverable reasons.
switch (errno)
{
...
}
}
I have tried the Sonny's solution and the epoll_ctl will return invalid argument. So i think maybe the right way to do this is as follow:
1.create socketfd and epollfd
2.use epoll_ctl to associate the socketfd and epollfd with epoll event.
3.do connect(socketfd,...)
4.check the return value or errno
5.if errno == EINPROGRESS, do epoll_wait
I'm working on an application that contains several server sockets that each run in a unique thread.
An external utility (script) is called by one of the threads. This script calls a utility (client) that sends a message to one of the server sockets.
Initially, I was using system() to execute this external script, but we couldn't use that because we had to make sure the server sockets were closed in the child that was forked to execute the external script.
I now call fork() and execvp() myself. I fork() and then in the child I close all the server sockets and then call execvp() to execute the script.
Now, all of that works fine. The problem is that at times the script reports errors to the server app. The script sends these errors by calling another application (client) which opens a TCP socket and sends the appropriate data. My issue is that the client app gets a value of 0 returned by the socket() system call.
NOTE: This ONLY occurs when the script/client app is called using my forkExec() function. If the script/client app is called manually the socket() call performs appropriately and things work fine.
Based on that information I suspect it's something in my fork() execvp() code below... Any ideas?
void forkExec()
{
int stat;
stat = fork();
if (stat < 0)
{
printf("Error forking child: %s", strerror(errno));
}
else if (stat == 0)
{
char *progArgs[3];
/*
* First, close the file descriptors that the child
* shouldn't keep open
*/
close(ServerFd);
close(XMLSocket);
close(ClientFd);
close(EventSocket);
close(monitorSocket);
/* build the arguments for script */
progArgs[0] = calloc(1, strlen("/path_to_script")+1);
strcpy(progArgs[0], "/path_to_script");
progArgs[1] = calloc(1, strlen(arg)+1);
strcpy(progArgs[1], arg);
progArgs[2] = NULL; /* Array of args must be NULL terminated for execvp() */
/* launch the script */
stat = execvp(progArgs[0], progArgs);
if (stat != 0)
{
printf("Error executing script: '%s' '%s' : %s", progArgs[0], progArgs[1], strerror(errno));
}
free(progArgs[0]);
free(progArgs[1]);
exit(0);
}
return;
}
Client app code:
static int connectToServer(void)
{
int socketFD = 0;
int status;
struct sockaddr_in address;
struct hostent* hostAddr = gethostbyname("localhost");
socketFD = socket(PF_INET, SOCK_STREAM, 0);
The above call returns 0.
if (socketFD < 0)
{
fprintf(stderr, "%s-%d: Failed to create socket: %s",
__func__, __LINE__, strerror(errno));
return (-1);
}
memset(&address, 0, sizeof(struct sockaddr));
address.sin_family = AF_INET;
memcpy(&(address.sin_addr.s_addr), hostAddr->h_addr, hostAddr->h_length);
address.sin_port = htons(POLLING_SERVER_PORT);
status = connect(socketFD, (struct sockaddr *)&address, sizeof(address));
if (status < 0)
{
if (errno != ECONNREFUSED)
{
fprintf(stderr, "%s-%d: Failed to connect to server socket: %s",
__func__, __LINE__, strerror(errno));
}
else
{
fprintf(stderr, "%s-%d: Server not yet available...%s",
__func__, __LINE__, strerror(errno));
close(socketFD);
socketFD = 0;
}
}
return socketFD;
}
FYI
OS: Linux
Arch: ARM32
Kernel: 2.6.26
socket() returns -1 on error.
A return of 0 means socket() succeeded and gave you file descriptor 0. I suspect that one of the file descriptors that you close has file descriptor 0 and once it's closed the next call to a function that allocated a file descriptor will return fd 0 as it's available.
A socket with value 0 is fine, it means stdin was closed which will make fd 0 available for reuse - such as by a socket.
chances are one of the filedescriptors you close in the forkExec() child path(XMLSocket/ServerFd) etc.) was fd 0 . That'll start the child with fd 0 closed, which won't happen when you run the app from a command line, as fd 0 will be already open as the stdin of the shell.
If you want your socket to not be 0,1 or 2 (stdin/out/err) call the following in your forkExec() function after all the close() calls
void reserve_tty()
{
int fd;
for(fd=0; fd < 3; fd++)
int nfd;
nfd = open("/dev/null", O_RDWR);
if(nfd<0) /* We're screwed. */
continue;
if(nfd==fd)
continue;
dup2(nfd, fd);
if(nfd > 2)
close(nfd);
}
Check for socket returning -1 which means an error occured.
Don't forget a call to
waitpid()
End of "obvious question mode". I'm assuming a bit here but you're not doing anything with the pid returned by the fork() call. (-:
As it is mentioned in another comment, you really should not close 0,1 or 2 (stdin/out/err), you can put a check to make sure you do not close those and so it will not be assigned as new fd`s when you request for a new socket