Does select() behave differently on python2 and python3? - linux

Question

I want to read stdout and stderr from a subprocess in the same thread as described in this post. While running the code inside Python2.7 works as expected, the select() call in Python3.3 seems to do not what it should.
Have a look - here is a script that would print two lines on both stdout and stderr, then wait and then repeat this a couple of times:
import time, sys
for i in range(5):
sys.stdout.write("std: %d\n" % i)
sys.stdout.write("std: %d\n" % i)
sys.stderr.write("err: %d\n" % i)
sys.stderr.write("err: %d\n" % i)
time.sleep(2)
The problematic script will start the script above in a subprocess and read it's stdout and stderr as described in the posted link:
import subprocess
import select
p = subprocess.Popen(['/usr/bin/env', 'python', '-u', 'test-output.py'],
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
r = [p.stdout.fileno(), p.stderr.fileno()]
while p.poll() is None:
print("select")
ret = select.select(r, [], [])
for fd in ret[0]:
if fd == p.stdout.fileno():
print("readline std")
print("stdout: " + p.stdout.readline().decode().strip())
if fd == p.stderr.fileno():
print("readline err")
print("stderr: " + p.stderr.readline().decode().strip())
Note that I start the Python subprocess using the -u option which causes Python to not buffer stdout and stderr. Also I print some text before calling select() and readline() to see, where the script blocks.
And here is the problem: running the script in Python3, after each cycle the output blocks for 2 seconds despite the fact, that two more lines are waiting to be read. And as indicated by the text before each call of select() you can see that it's select() which is blocking (not readline()).
My first thought was that select() only resumes on a flush on Python3 while Python2 it returns always when there's data available but in this case only one line would be read each 2 seconds (which is not the case!)
So my question is: is this a bug in Python3-select()? Did I misunderstand the behavior of select()? And is there a way to workaround this behavior without having to start a thread for each pipe?
Output when running Python3:
select
readline std
stdout: std: 0
readline err
stderr: err: 0
select <--- here the script blocks for 2 seconds
readline std
stdout: std: 0
select
readline std
stdout: std: 1
readline err
stderr: err: 0
select <--- here the script should block (but doesn't)
readline err
stderr: err: 1
select <--- here the script blocks for 2 seconds
readline std
stdout: std: 1
readline err
stderr: err: 1
select <--- here the script should block (but doesn't)
readline std
stdout: std: 2
readline err
stderr: err: 2
select
.
.
Edit: Please note that it has no influence whether the child process is a Python script. The following C++ program has the same effect:
int main() {
for (int i = 0; i < 4; ++i) {
std::cout << "out: " << i << std::endl;
std::cout << "out: " << i << std::endl;
std::cerr << "err: " << i << std::endl;
std::cerr << "err: " << i << std::endl;
fflush(stdout);
fflush(stderr);
usleep(2000000);
}}

Answer 1

It seems that the reason is buffering in subprocess.PIPE and first readline() call reads all available data (i.e., two lines) and returns first one.
After that, there is no unread data in pipe, so select() is not returning immediately. You can check this by doubling readline calls:
print("stdout: " + p.stdout.readline().decode().strip())
print("stdout: " + p.stdout.readline().decode().strip())
and ensuring that second readline() call doesn't block.
One solution is to disable buffering using bufsize=0:
p = subprocess.Popen(['/usr/bin/env', 'python', '-u', 'test-output.py'],
stdout=subprocess.PIPE, stderr=subprocess.PIPE, bufsize=0)
Another possible solution is to do a non-blocking readline() or to ask pipe file object its read buffer size, but I don't know is it possible.
You can also read directly from p.stdout.fileno() to implement non-blocking readline().
Update: Python2 vs. Python3
The reason why Python3 differs from Python2 here is likely to be in new I/O module (PEP 31136). See this note:
The BufferedIOBase methods signatures are mostly identical to that of RawIOBase (exceptions: write() returns None , read() 's argument is optional), but may have different semantics. In particular, BufferedIOBase implementations may read more data than requested or delay writing data using buffers.