Why does shutdown write in the client cause the connection to be closed? - linux

Question

NOTE: No error appears if the shutdown in client is deleted
// server.rs
use std::io::BufReader;
use std::io::Read;
use std::io::Write;
use std::net::Shutdown;
use std::net::TcpListener;
use std::thread;
fn main() {
let listener = TcpListener::bind("127.0.0.1:4000").unwrap();
for stream in listener.incoming() {
let mut stream = stream.unwrap();
thread::spawn(move || {
let mut reader = BufReader::new(&stream);
let mut buffer = [0; 1024];
let len = reader.read(&mut buffer).unwrap();
// no sleep no error, just simulating a time-consuming operation
thread::sleep(std::time::Duration::from_secs(1));
stream.write_all(&buffer[0..len]).unwrap();
stream.shutdown(Shutdown::Write).unwrap();
});
}
}
// client.rs
use std::io::{Read, Write};
use std::net::TcpStream;
use std::thread;
fn main() {
let mut clients = Vec::new();
for _ in 0..1000 {
clients.push(thread::spawn(move || {
let mut client = TcpStream::connect("127.0.0.1:4000").unwrap();
client.write_all("hello".as_bytes()).unwrap();
// no error if remove the following line
client.shutdown(std::net::Shutdown::Write).unwrap();
let mut buffer = Vec::new();
client.read_to_end(&mut buffer).unwrap();
println!("{}", std::str::from_utf8(&buffer).unwrap());
}));
}
for client in clients.into_iter() {
client.join().unwrap();
}
}
As I understand, shutdown the write operation will append FIN after sending the previous data, and then the peer (server) can still continue to write data. But among these 1000 clients, some error appeared:
// server
<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }', src/bin/server.rs:22:46
// client
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }', src/bin/client.rs:15:45
It seems that the connection is closed after the shutdown in client.
Update1:
I used Wireshark and this is one of the wrong connections:
No. Time Source Destination Protocol Length Info
1101 13.738139 127.0.0.1 127.0.0.1 TCP 56 10628 → 4000 [SYN] Seq=0 Win=65535 Len=0 MSS=65495 WS=256 SACK_PERM=1
1104 13.738157 127.0.0.1 127.0.0.1 TCP 44 4000 → 10628 [RST, ACK] Seq=409345761 Ack=1 Win=0 Len=0
1234 14.251615 127.0.0.1 127.0.0.1 TCP 56 [TCP Retransmission] [TCP Port numbers reused] 10628 → 4000 [SYN] Seq=0 Win=65535 Len=0 MSS=65495 WS=256 SACK_PERM=1
1250 14.251690 127.0.0.1 127.0.0.1 TCP 56 [TCP Port numbers reused] 4000 → 10628 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=65495 WS=256 SACK_PERM=1
1266 14.251726 127.0.0.1 127.0.0.1 TCP 44 10628 → 4000 [ACK] Seq=1 Ack=1 Win=2161152 Len=0
1376 14.251949 127.0.0.1 127.0.0.1 TCP 49 10628 → 4000 [PSH, ACK] Seq=1 Ack=1 Win=2161152 Len=5
1387 14.251970 127.0.0.1 127.0.0.1 TCP 44 4000 → 10628 [ACK] Seq=1 Ack=6 Win=2161152 Len=0
1402 14.251996 127.0.0.1 127.0.0.1 TCP 44 10628 → 4000 [FIN, ACK] Seq=6 Ack=1 Win=2161152 Len=0
1412 14.252013 127.0.0.1 127.0.0.1 TCP 44 4000 → 10628 [ACK] Seq=1 Ack=7 Win=2161152 Len=0
2092 15.261312 127.0.0.1 127.0.0.1 TCP 49 4000 → 10628 [PSH, ACK] Seq=1 Ack=7 Win=2161152 Len=5
2101 15.261384 127.0.0.1 127.0.0.1 TCP 44 10628 → 4000 [RST, ACK] Seq=7 Ack=6 Win=0 Len=0
Update2:
One of correct connections:
No. Time Source Destination Protocol Length Info
162 13.731960 127.0.0.1 127.0.0.1 TCP 56 10927 → 4000 [SYN] Seq=0 Win=65535 Len=0 MSS=65495 WS=256 SACK_PERM=1
166 13.731997 127.0.0.1 127.0.0.1 TCP 56 4000 → 10927 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=65495 WS=256 SACK_PERM=1
169 13.732013 127.0.0.1 127.0.0.1 TCP 44 10927 → 4000 [ACK] Seq=1 Ack=1 Win=2161152 Len=0
176 13.732035 127.0.0.1 127.0.0.1 TCP 49 10927 → 4000 [PSH, ACK] Seq=1 Ack=1 Win=2161152 Len=5
181 13.732046 127.0.0.1 127.0.0.1 TCP 44 4000 → 10927 [ACK] Seq=1 Ack=6 Win=2161152 Len=0
187 13.732059 127.0.0.1 127.0.0.1 TCP 44 10927 → 4000 [FIN, ACK] Seq=6 Ack=1 Win=2161152 Len=0
191 13.732074 127.0.0.1 127.0.0.1 TCP 44 4000 → 10927 [ACK] Seq=1 Ack=7 Win=2161152 Len=0
1495 14.746260 127.0.0.1 127.0.0.1 TCP 49 4000 → 10927 [PSH, ACK] Seq=1 Ack=7 Win=2161152 Len=5
1502 14.746369 127.0.0.1 127.0.0.1 TCP 44 10927 → 4000 [ACK] Seq=7 Ack=6 Win=2161152 Len=0
1505 14.746423 127.0.0.1 127.0.0.1 TCP 44 4000 → 10927 [FIN, ACK] Seq=6 Ack=7 Win=2161152 Len=0
1512 14.746529 127.0.0.1 127.0.0.1 TCP 44 10927 → 4000 [ACK] Seq=7 Ack=7 Win=2161152 Len=0

Answer 1

I strongly suspect it has to do with the backlog, the number of connections accepted by the OS TCP stack, but not yet handled by your application.
std doesn't allow you to control this number and defaults it to 128.
To control the number, I've re-implemented your server in tokio (well, it's actually just the main example on the tokio README), and set the backlog to 3000.
use std::time::Duration;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::{TcpListener, TcpSocket};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let addr = "127.0.0.1:4000".parse().unwrap();
let socket = TcpSocket::new_v4()?;
socket.bind(addr)?;
let listener: TcpListener = socket.listen(3000)?;
loop {
let (mut socket, _) = listener.accept().await?;
tokio::spawn(async move {
let mut buf = [0; 1024];
loop {
let n = match socket.read(&mut buf).await {
Ok(n) if n == 0 => return,
Ok(n) => n,
Err(e) => {
eprintln!("failed to read from socket; err = {:?}", e);
return;
}
};
tokio::time::sleep(Duration::from_secs(1)).await;
if let Err(e) = socket.write_all(&buf[0..n]).await {
eprintln!("failed to write to socket; err = {:?}", e);
return;
}
}
});
}
}
This makes the problem disappear. Reducing it to socket.listen(128) makes it reappear. (Disclaimer: I do not suggest that 3000 is a sane number for the backlog.)
I said I strongly suspect that this is the cause because I can't fully explain how the sleep causes the problem. It may be that the many sleeping threads slow down the scheduler, and thus the speed at which your server can accept connections. But that is speculation.
(Side note: My default ulimit for open files was fairly low. I had to increase it with ulimit -nS 10000 to not get in the way when testing this.)