I'm writing a terminal logging program - think the script command but a bit more featureful. One of the differences is that, whereas script captures stdout, stdin and stderr as one big character stream, I would like to keep them separate and record them as such.
In order to do this, I use the standard approach of running a child shell connected to a pty, but instead of using a single pty with stdin, stdout and stderr all connected to it, I use two ptys - with stdin and stderr connected to one pty, and stdout on the other. This way, the master process can tell what is coming from stdout and what from stderr.
This has, so far, worked fine. However, I'm starting to run into a few issues. For example, when trying to set the number of columns, I get the following:
$stty cols 169
stty: stdout appears redirected, but stdin is the control descriptor
This seems to be a result of this piece of code, which seems to check whether stdout and stderr are both ttys, but complains if they are not the same.
My question, therefore, is this: am I violating any fundamental assumptions about how Posix processes behave by acting in this way? If not, any idea why I'm seeing errors such as this? If so, is there any way I can get around this and still manage to separate stdout and stderr nicely?
One idea I had about this is to use a process directly on the pty which then runs the target program, e.g.
(wrapper) -> pty -> (controller) -> script
The controller would be responsible for running the script and capturing the stdout and stderr separately, feeding them back to the wrapper, perhaps by some non-std fd, or alternatively, serialising the data before shipping it back, e.g. prefixing output from stderr with stderr: and stdout with stdout: - then in the wrapper deserialize this and feed it back upstream or whatever you want to do with it.
Related
I've written a log-to-stdout program which produces logs, and another exe read-from-stdin (for example filebeat) to collect logs from stdin. My problem is that my log-to-stdout speed may burst in a short period which exceeds read-from-stdin can accept, that will blocking log-to-stdout process, I'd like to know if there is a Linux API to tell if the stdout file descriptor can be written to (up to N bytes) without being blocked?
I've found some comments in nodejs process.stdout
In the case they refer to pipes:
They are blocking in Linux/Unix.
They are non-blocking like other streams in Windows.
Does that mean under Linux it's impossible to do non-blocking write on stdout? Some documents reference non-blocking file operate mode (https://www.linuxtoday.com/blog/blocking-and-non-blocking-i-0/), does it apply to stdout too? Because I'm using third-party logging (which expect stdout working at blocking mode), can I check stdout writable in non-blocking mode (before calling logging library), and then switch stdout back to blocking mode, so from logging library perspective, stdout fd still works as previously? (if I can tell stdout will be blocking, I'll throw output, since not being block is more important than output complete logging in my usage)
(Or if there is a auto-drop-pipe command, which can auto drop lines if pipeline will block, so I can call
log-to-stdout | auto-drop-pipe --max-lines=100 --drop-head-if-full | read-from-stdin)
I've got the below statement gets one sentence from stdin and print the sentence (stdout).
process.stdin.pipe(process.stdout);
But what does the below statement means?
The below statement acts like the first statement.
(Receive data from the user and print it on the screen.)
What does the sentence below mean and why does this happen?
process.stdout.pipe(process.stdout);
what does it mean to pipe data from stdout, stderr?
Conceptually you can pipe stdout to stdin. But it's not useful because there's no way to put data into stdout, and no way to get it out of stdin.
And, in some systems trying to read from stdout and / or write to stdin may throw an error.
stdin and stdout have been part of the UNIX / FreeBSD / Linux operating system for half a century now. Nodejs's process object simply exposes them. It's worth a bit of your time to learn how these fundamental OS building blocks work.
In Node JS's child process documentation. The following is mentioned.
By default, pipes for stdin, stdout, and stderr are established between the parent Node.js process and the spawned child. These pipes have limited (and platform-specific) capacity.If the child process writes to stdout in excess of that limit without the output being captured, the child process will block waiting for the pipe buffer to accept more data.
I've been trying to spawn (child_process.spawn) a console application from node. The application is interactive and outputs a lot of data on initiation. Apparently, when you pipe, if the process outputting data is getting a head of the process getting input, the OS pause the process outputting. For e.g, when you run ps ax | cat, if ps process is getting ahead of cat process, ps will pause until cat can accept input again. But it should eventually pipe all data.
In my case, this console application (written in C) seems to be pausing completely when this happens. I could be wrong but this seems to be the case because the piped output stops mid way no matter what I do. This happens even when I'm piping from bash, so there's no problem with nodejs here. May be this application is not very compatible with how piping works in linux.
There's nothing I can do regarding this console application but is there some other way in node to get these data chunks without piping?
Edited:
The main code
const js = spawn('julius', ['-C', 'julius.jconf', '-dnnconf', 'dnn.jconf']);
I tried
js.stdout.on("data", (data) => { console.log(`stdout: ${js.stdout}`); });
and
js.stdout.pipe(process.stdout);
to capture the output. Same result, the output is being cut off.
I have a python subprocess that runs an arbitrary C++ program (student assignments if it matters) via POpen. The structure is such that i write a series of inputs to stdin, at the end i read all of stdout and parse for responses to each output.
Of course given that these are student assignments, they may crash after certain inputs. What i require is to know after which specific input their program crashed.
So far i know that when a runtime exception is thrown in the C++ program, its printed to stderr. So right not i can read the stderr after the fact and see that it did in face crash. But i haven't found a way to read stderr while the program is still running, so that i can infer that the error is in response to the latest input. Every SO question or article that i have run into seems to make use of subprocess.communicate(), but communicate seems to block until the subprocess returns, this hasn't been working for me because i need to continue sending inputs to the program after the fact if it hasn't crashed.
What i require is to know after which specific input their program crashed.
Call process.stdin.flush() after process.stdin.write(b'your input'). If the process is already dead then either .write() or .flush() will raise an exception (specific exception may depend on the system e.g, BrokenPipeError on POSIX).
Unrelated: If you are redirecting all three standard streams (stdin=PIPE, stdout=PIPE, stderr=PIPE) then make sure to consume stdout, stderr pipes concurrently while you are writing the input otherwise the child process may hang if it generates enough output to fill the OS pipe buffer. You could use threads, async. IO to do it -- code examples.
I have an embedded application that I want a simple-minded logger for.
The system starts from a script file, which in turn runs the application. There could be various reasons that the script fails to run the application, or the application itself could fail to start. To diagnose this remotely, I need to view the stdout from the script and the application.
I tried writing a tee-like logger that would repeat its stdin to stdout, and save the text in a FIFO for later retrieval via the network. Then I naively tried
./script | ./logger
I ended up with only the script stdout going to the logger, and the application stdout disappearing. I had similar results trying tee.
The system is running kernel 2.4.26, and busybox.
What is going on, and how can I accomplish my desired ends?
It turns out it was working exactly as I thought it should work, with one minor gotcha. stdout was being buffered, and without any fflush(stdout) commands, I never saw it. Had I been really patient, I would have suddenly seen a big gush of output when the stdout buffer filled up. A call to setlinebuf(3) fixed my problem.
Apparently, the application output doesn't end up on stdout...
The output is actually on stderr (which is usually also connected to the terminal)
./script.sh 2>&1 | ./logger
should then work
The application actively disconnects from stdin/stdout (e.g. by closing/reopening file descriptors 0,1(,2) or, using nohup, exec or similar utilities)
the script daemonizes (which also detaches from all standard streams)