What are the ways to flush a Linux command's stdout as a Node.js child process?
Ending the stdin stream (child.stdin.end) will work. As will unbuffering the command with stdbuf.
But I imagine there's a proper way to stream results from external commands.
How do we tell a command we're ready to consume while we are still providing data?
Example:
const { spawn } = require('child_process');
const child = spawn('uniq');
child.stdout.pipe(process.stdout);
child.stdin.write('a\nb\nb\nc\n', 'utf8');
// No output, child is still running.
(uniq is just an example here. It's the same with most Linux commands.)
Related
I want to be able to spawn two Node.js child processes but have the stdout from one be piped to the stdin of another. The equivalent of:
curl "https://someurl.com" | jq .
My Node's stdout will go to either a terminal or to a file, depending on whether the user pipes the output or not.
You can spawn a child process with Node.js's child_process built-in module. We need to processes, so we'll call it twice:
const cp = require('child_process')
const curl = cp.spawn('curl', ['https://someurl.com'], { stdio: ['inherit', 'pipe', 'inherit'] })
const jq = cp.spawn('jq', ['.'], { stdio: ['pipe', 'inherit', 'pipe'] })
The first parameter is the executable to run, the second is the array of parameters to pass it and the third is options. We need to tell it where the process's stdin, stdout and stderr are to be routed: 'inherit' means "use the host Node.js application's stdio", and 'pipe' means "we'll handle it programmatically.
So in this case curl's output and jq's input are left to be dealt with programmatically which we do with an additional line of code:
curl.stdout.pipe(jq.stdin)
which means "plumb curl's stdout into jq's stdin".
It's as simple as that.
As I read from the child_process module documentation of Node.js, I understand the difference between exec and spawn. The most crucial difference that is also highlighted in a similar StackOverflow question about the Spawn vs Exec:
The main difference is that spawn is more suitable for long-running processes with huge output. That's because spawn streams input/output with a child process. On the other hand, exec buffers output in a small (by default 200K) buffer.
However, I noticed, thanks to TS Intellisense that both exec and spawn function return similar object of type ChildProcess. So, I could technically write this for the exec function using stdout as stream and it works:
function cmdAsync (cmd, options) {
return new Promise((resolve) => {
const proc = exec(cmd, options);
proc.stdout?.pipe(process.stdout);
proc.on('exit', resolve);
});
}
cmdAsync('node server/main.mjs');
And without any buffer time/delay, I could see the logs generated by server/main.mjs file being piped into the parent process's stdout stream.
So, my question is exactly where the buffering is happening and how the streaming behavior is different for exec than spawn function! Also, can I rely of this feature even if it is undocumented?
We have a VS Code extension built in node that runs like so:
Our extension is a node process, it calls child_process.spawn(...), with pipe option for the stdio. The child process' stdin and stdout are used to pass messages between the two processes.
Due to a bug in the child process binary, we see an issue where SIGSEGV error code is thrown by the binary. When we call into the process on the commandline and hit this error, we see the stack trace is dumped to stderr.
In the VS Code extension/node process, we see the handlers for the child process exiting are hit, however the handlers for the stderr output do not.
In other non-crashing scenarios, we see stderr output is correctly transmitted.
The implementation of spawning and handling the child process is reasonably complex, however it boils down to something like this:
child_process.ChildProcess childProcess;
startChildProcess() {
this.childProcess = child_process.spawn(binary, args, {
cwd: root,
env: {...process.env, ...environment},
// pipe stdin and stdout, and inherit stderr (unless parent know better)
stdio: ['pipe', 'pipe', 'pipe'],
});
// set up stdin and stdout pipes here...
this.childProcess.on('exit', (code, signal) => {
// do something interesting when the process exits
// THIS CODE BLOCK IS HIT
}
this.childProcess.stderr?.addListener('data', (errOutput) => {
const errOutputStr = errOutput.toString();
process.stderr.write(errOutputStr);
// THIS CODE BLOCK IS NOT HIT WHEN THE ABOVE IS HIT
})
}
As annotated in the example, the stderr output is not hit in the case of a SIGSEGV from child process.
What can I do to ensure the errors are output? VS Code, the extension process and the child process are all running on Mac OS
My node process gets some PDF file via HTTP Request, then uses the request's onData event to pass the incoming data on to a properly configured lpr, spawned via child_process.exec. I write to stdin using process.stdin.write(...), followed by process.stdin.end() when done. This allows me to print those files immediately.
Now I have a situation where I don't want the data to be piped to lpr, but to some bash script. The script uses cat to process its stdin.
myscript.sh < somefile.pdf works as expected, as does cat somefile.pdf | myscript.sh.
However, when I spawn /path/to/script.sh from node (by simply replacing lpr with the script path in the source), the process exits with
events.js:183
throw er; // Unhandled 'error' event
^
Error: write EPIPE
at WriteWrap.afterWrite [as oncomplete] (net.js:868:14)
Subsequently, the whole node process crashes, the error sneaking around all try...catch blocks. Logging at the beginning of the bash script shows, it does not even get started.
When I target anything that's not a shell script but some compiled executable, like cat, echo,... everything works just fine.
Adding epipebomb module would not change anything.
I also tried piping to process.exec("bash", ["-c cat | myscript.sh"]), with the same errors.
An example bash script, just to test for execution:
#!/usr/bin/env bash
date > logfile.txt
cat > /dev/null
EDIT:
I think I maybe need to signal to keep the stdin stream open somehow.
The process-spawning part of the script, leaving promisification and output processing away:
const process = require("child_process")
// inputObservable being an rxjs Observable
execstuff(inputObervable) {
const task = process.spawn("/path/to/script.sh");
inputObservable.subscribe(
chunk => task.stdin.write(chunk),
error => console.error(error),
finished => task.stdin.end()
);
}
There is an example at child_process.spawn how you can write the following lines ps ax | grep ssh as node.js script, maybe it will be helpful for you:
const { spawn } = require('child_process');
const ps = spawn('ps', ['ax']);
const grep = spawn('grep', ['ssh']);
ps.stdout.on('data', (data) => {
grep.stdin.write(data);
});
ps.stderr.on('data', (data) => {
console.log(`ps stderr: ${data}`);
});
The first impression is that you are doing the same stuff, the problem may be in the chunk data, maybe one of the chunks is null, and it is closing the stream, and you want to close it by running task.stdin.end().
The other thing you can try is to run the node.js script with the NODE_DEBUG=stream node script.js
Will log the node.js internals how the stream, behaves, also may be helpful for you.
How to execute system command in node-webkit (or node.js) external process parallel of current script.
I'm trying to use child_process. After interruption of my script subprocess is exit. However i need a simple way execute bash command without output or with output but without program stop when my script will be interrupted.
I need a correct simple way.
Thanks all.
Use detached option in spawn/execute arguments:
If the detached option is set, the child process will be made the
leader of a new process group. This makes it possible for the child to
continue running after the parent exits.
By default, the parent will wait for the detached child to exit. To
prevent the parent from waiting for a given child, use the
child.unref() method, and the parent's event loop will not include the
child in its reference count.
Example of detaching a long-running process and redirecting its output
to a file:
var fs = require('fs'),
spawn = require('child_process').spawn,
out = fs.openSync('./out.log', 'a'),
err = fs.openSync('./out.log', 'a');
var child = spawn('prg', [], {
detached: true,
stdio: [ 'ignore', out, err ]
});
child.unref();