I have a python subprocess that runs an arbitrary C++ program (student assignments if it matters) via POpen. The structure is such that i write a series of inputs to stdin, at the end i read all of stdout and parse for responses to each output.
Of course given that these are student assignments, they may crash after certain inputs. What i require is to know after which specific input their program crashed.
So far i know that when a runtime exception is thrown in the C++ program, its printed to stderr. So right not i can read the stderr after the fact and see that it did in face crash. But i haven't found a way to read stderr while the program is still running, so that i can infer that the error is in response to the latest input. Every SO question or article that i have run into seems to make use of subprocess.communicate(), but communicate seems to block until the subprocess returns, this hasn't been working for me because i need to continue sending inputs to the program after the fact if it hasn't crashed.
What i require is to know after which specific input their program crashed.
Call process.stdin.flush() after process.stdin.write(b'your input'). If the process is already dead then either .write() or .flush() will raise an exception (specific exception may depend on the system e.g, BrokenPipeError on POSIX).
Unrelated: If you are redirecting all three standard streams (stdin=PIPE, stdout=PIPE, stderr=PIPE) then make sure to consume stdout, stderr pipes concurrently while you are writing the input otherwise the child process may hang if it generates enough output to fill the OS pipe buffer. You could use threads, async. IO to do it -- code examples.
Related
I've written a log-to-stdout program which produces logs, and another exe read-from-stdin (for example filebeat) to collect logs from stdin. My problem is that my log-to-stdout speed may burst in a short period which exceeds read-from-stdin can accept, that will blocking log-to-stdout process, I'd like to know if there is a Linux API to tell if the stdout file descriptor can be written to (up to N bytes) without being blocked?
I've found some comments in nodejs process.stdout
In the case they refer to pipes:
They are blocking in Linux/Unix.
They are non-blocking like other streams in Windows.
Does that mean under Linux it's impossible to do non-blocking write on stdout? Some documents reference non-blocking file operate mode (https://www.linuxtoday.com/blog/blocking-and-non-blocking-i-0/), does it apply to stdout too? Because I'm using third-party logging (which expect stdout working at blocking mode), can I check stdout writable in non-blocking mode (before calling logging library), and then switch stdout back to blocking mode, so from logging library perspective, stdout fd still works as previously? (if I can tell stdout will be blocking, I'll throw output, since not being block is more important than output complete logging in my usage)
(Or if there is a auto-drop-pipe command, which can auto drop lines if pipeline will block, so I can call
log-to-stdout | auto-drop-pipe --max-lines=100 --drop-head-if-full | read-from-stdin)
I have been programing a Python discord bot (you don't need to know anything about discord for this question). I currently have it able to take in messages from discord, and want to be able to send messages by typing them into the console. Unfortunately, using input() completely stops the program, which is kind of a deal breaker. Is there a way to take input from the console without stopping the whole program?
In asynchronous mode you can create a worker that reads the standard input and pushes the result into a fifo queue and a main program that reads the output from the fifo when it has time.
The python documentation shows an example that uses this principle (but of course the actions of the worker and the main program are not the ones you want): https://docs.python.org/3/library/asyncio-queue.html#examples
I've got the below statement gets one sentence from stdin and print the sentence (stdout).
process.stdin.pipe(process.stdout);
But what does the below statement means?
The below statement acts like the first statement.
(Receive data from the user and print it on the screen.)
What does the sentence below mean and why does this happen?
process.stdout.pipe(process.stdout);
what does it mean to pipe data from stdout, stderr?
Conceptually you can pipe stdout to stdin. But it's not useful because there's no way to put data into stdout, and no way to get it out of stdin.
And, in some systems trying to read from stdout and / or write to stdin may throw an error.
stdin and stdout have been part of the UNIX / FreeBSD / Linux operating system for half a century now. Nodejs's process object simply exposes them. It's worth a bit of your time to learn how these fundamental OS building blocks work.
I'm building a GTK-3 application in Python (3.6 to be precise) which launches an external binary in a Gio.Subprocess and reads that's output from a Gio.DataInputStream. I'm following this recipe to read the output line by line by asynchronous operations. This happens in a loop of queue_read() and the _on_data() callback, which is interrupted only by self.cancellable triggered from _on_finished() or an eventual exception during reading from the stream.
Depending on the parameters used to call the external process, there is some arbitrary delay between each line of the output (each line is a unit of measured values, which should be displayed as soon as it is available). It also can happen that my spawned process terminates way before everything is read from the stream. Therefor, I cannot cancel reading in _on_finished() as in the linked code - I need to read until I get the stream's end-of-file and then omit to schedule reading another line.
How can I test for eof on a Gio.DataInputStream?
In C++ io streams for example there's a flag indicating this condition. The Gio.DataInputStream also raises GLib.Error in certain situations. But neither some flag seems to be available here nor an error is raised in this case. The only condition I experienced is that the async read operation (the _on_data() callback) completes immediately with line set to None after the subprocess terminated and no more data is available from the stream. If this is the way to go, is it documented somewhere?
I'm writing a terminal logging program - think the script command but a bit more featureful. One of the differences is that, whereas script captures stdout, stdin and stderr as one big character stream, I would like to keep them separate and record them as such.
In order to do this, I use the standard approach of running a child shell connected to a pty, but instead of using a single pty with stdin, stdout and stderr all connected to it, I use two ptys - with stdin and stderr connected to one pty, and stdout on the other. This way, the master process can tell what is coming from stdout and what from stderr.
This has, so far, worked fine. However, I'm starting to run into a few issues. For example, when trying to set the number of columns, I get the following:
$stty cols 169
stty: stdout appears redirected, but stdin is the control descriptor
This seems to be a result of this piece of code, which seems to check whether stdout and stderr are both ttys, but complains if they are not the same.
My question, therefore, is this: am I violating any fundamental assumptions about how Posix processes behave by acting in this way? If not, any idea why I'm seeing errors such as this? If so, is there any way I can get around this and still manage to separate stdout and stderr nicely?
One idea I had about this is to use a process directly on the pty which then runs the target program, e.g.
(wrapper) -> pty -> (controller) -> script
The controller would be responsible for running the script and capturing the stdout and stderr separately, feeding them back to the wrapper, perhaps by some non-std fd, or alternatively, serialising the data before shipping it back, e.g. prefixing output from stderr with stderr: and stdout with stdout: - then in the wrapper deserialize this and feed it back upstream or whatever you want to do with it.