Why Command's stdin write_all never terminate? - rust

Question

I have the following code. It runs python3 solution.py whose stdin is supplied from variable input
let mut cmd = Command::new("python3");
let mut child = cmd
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.arg("solution.py")
.spawn()
.expect("Failed to execute solution");
println!("Writing to stdin");
child
.stdin
.as_mut()
.unwrap()
.write_all(input.as_bytes())
.unwrap();
//never printed
println!("Finish writing to stdin");
child.stdin.as_mut().unwrap().flush().unwrap()
It always runs ok (and terminates fast) when the input variable is small. But when it is ~3MB (large String), it never terminates. I guess there is deadlock somewhere since the CPU usage is small.
Could you suggest how to make it run with large input? And why does it seem to suffer from deadlock?
Update
Simplified version of solution.py where this problem still happens (it just prints whitespace strings):
t = int(input())
for tt in range(t):
input()
res = [' '] * 1000
result = ''.join(res)
print("Case #{}: {}".format(tt+1, result))
Interestingly, if I change line 4 to the below, the program terminates
res = [' '] * 100
It is probably due to large output size?

Answer 1

Your Rust program is writing to a pipe connected to the stdin of the child process. The child process is writing to its stdout, which is again a pipe. The operating system is buffering some amount data for both pipes, but when the buffer is full it will wait for the process on the reading end to consume the data before accepting any further writes.
The write_all() call is constantly writing data into the stdin pipe, which is read by the child process. The child process is writing data to its stdout pipe, but nobody is consuming that data. once the stdout pipe buffer fills up, the child process blocks trying to write further data, so it stops reading data from stdin. That means the stdin pipe buffer will also fill up, at which point the parent process also blocks while trying to write further data.
The easiest way to resolve this is moving writing to the stdin pipe to a thread, and adding code in the main thread to read from stdout. This way, your Rust program reads and writes data in parallel, and no deadlock will occur. The documentation of std::process has an example demonstrating this approach.

Answer 2

The accepted answer explains the underlying issue and gives the directions how to solve it using the standard library.
In addition to that, you can use the subprocess crate whose communicate() method (modeled after Python's Popen.communicate()) is designed to prevent deadlocks in situations like yours:
let (out, _) = subprocess::Exec::cmd("python3")
.arg("solution.py")
.stdin("some data")
.communicate()?
.read_string()?;
// out is a String
Disclaimer: I'm the author of subprocess.