Multithreaded use of SOCK_STREAM sockets - multithreading

Question

I'm attempting to solve a multi-threaded socket use issue in my pthreaded Linux code, and I think it might be insoluble, given my requirements. Can you let me know whether you agree?
The problem is this:
I have multiple pthreads writing to a single SOCK_STREAM (TCP) socket (set for non-blocking IO). Each thread is attempting to write a single "message" at a time, and message data from different threads must not be interleaved.
I am prepared for send() to return EWOULDBLOCK/EAGAIN in a minority of cases (in which case I just defer the write of the message to an asynchronous thread using a mutex protected queuing structure)
I cannot get mutexes on these foreground writing threads (apart from the EWOULDBLOCK/EAGAIN cases), as the implied cache flushes give me an unacceptable CPU impact (this also means no use of other synchronization objects to work around the problem, such as atomic integers).
As far as I know, I cannot guarantee that send() won't decide to write some, but not all of any given message, thus resulting in (potentially) interleaved data
Is this problem insoluble? Is there any way to force send() to either buffer the entire message or none of it (when using non-blocking IO)?
Thanks!

Answer 1

I have multiple pthreads writing to a single SOCK_STREAM (TCP) socket (set for non-blocking IO). Each thread is attempting to write a single "message" at a time, and message data from different threads must not be interleaved.
TCP is a byte stream, it has no concept of "messages" like UDP does. You MUST synchronize access to a TCP socket, such as with a mutex, to avoid interweaving the "messages".
I am prepared for send() to return EWOULDBLOCK/EAGAIN in a minority of cases (in which case I just defer the write of the message to an asynchronous thread using a mutex protected queuing structure)
In which case, you will likely need to send MOST/ALL of your "messages" through that same queue, so it can ensure that they get sent in the proper order. If you start sending a "message" directly and it gets an EWOULDBLOCK/EAGAIN error, the remaining unsent data for that "message", and all subsequent "messages", MUST be queued until the queue is emptied. Whenever the socket is writable, send whatever is in the queue, removing only what is successfully sent, and repeating as needed until the queue is emptied. Only then can you safely send new "messages" directly until another EWOULDBLOCK/EAGAIN error is reported. But no matter what, you will still need to synchronize the sends so only 1 "message" is being sent at a time, whether directly or from the queue.
For example, you can do something like this (pseudo code):
void sendMsg(message msg)
{
lock(&mutex);
if (!queue.empty()) {
queue.add(&msg, sizeof(msg));
}
else
{
byte *ptr = &msg;
size_t size = sizeof(msg);
do {
int sent = send(sock, ptr, size, 0);
if (sent == -1) {
if (errno == EINTR) continue;
queue.add(ptr, size);
break;
}
ptr += sent;
size -= sent;
}
while (size > 0);
}
unlock(&mutex);
}
void socketIsWritable()
{
lock(&mutex);
while (!queue.empty()) {
int sent = send(sock, queue.data(), queue.size(), 0);
if (sent == -1) {
if (errno == EINTR) continue;
break;
}
queue.remove(sent);
}
unlock(&mutex);
}
I cannot get mutexes on these foreground writing threads (apart from the EWOULDBLOCK/EAGAIN cases), as the implied cache flushes give me an unacceptable CPU impact (this also means no use of other synchronization objects to work around the problem, such as atomic integers).
Synchronizing access to the socket is the only way to ensure the integrity of the "messages" you want to send. Which does mean there will be a small bottleneck in your communications. You can't begin sending a new "message" until a previous "message" has finished being sent in full.
As far as I know, I cannot guarantee that send() won't decide to write some, but not all of any given message, thus resulting in (potentially) interleaved data
Correct.
Is there any way to force send() to either buffer the entire message or none of it (when using non-blocking IO)?
No. And even if it could, that would still not prevent against interweaving when multiple threads try to call send() at the same time. You need to handle the necessary buffering in your own code.