Node.js child processes and pipes - OSX vs Ubuntu - node.js

I am trying to get two long running node.js processes to communicate - a parent and a child - using pipes and Node's child-process module. I want the child to be able to send data back to the parent asynchronously, and I was hoping to use a pipe to do so.
Here's a simplified version of my code:
Parent:
cp = require('child_process')
es = require('event-stream')
child = cp.spawn('coffee', ['child.coffee'], {stdio: [null, null, null, 'pipe']})
so = child.stdout.pipe(es.split())
p3 = child.stdio[3].pipe(es.split())
so.on 'data', (data) ->
console.log('stdout: ' + data)
child.stderr.on 'data', (data) ->
console.log('stderr: ' + data);
p3.on 'data', (data) ->
console.log('stdio3: ' + data);
child.on 'close', (code) ->
console.log('child process exited with code ' + code)
child.stdin.write "a message from your parent", "utf8"
Child:
fs = require('fs')
p3 = fs.createWriteStream('/dev/fd/3', {encoding: 'utf8'})
process.stdin.on 'data', (data) ->
p3.write "hello #{process.pid} - #{data}\n", 'utf8'
process.stdout.write "world #{process.pid} - #{data}\n", 'utf8'
p3.end()
process.exit(0)
process.stdin.on 'end', (data) ->
console.log "end of stdin"
p3.end()
process.exit(0)
process.stdin.setEncoding('utf8')
process.stdin.resume()
The code works on OSX 10.9, but fails to run on a Ubuntu box. I have tried running it under both Ubuntu 12.04 and 14.04. I am running Node 10.2x.
/dev/fd/ under Ubuntu is symbolically linked to /proc/self/fd/ so I believe my child process is opening the right file.
The output from running the parent on Ubuntu is as follows:
$ coffee parent.coffee
stderr:
stderr: events.js:72
stderr: throw er; // Unhandled 'error' event
stderr:
stderr:
stderr:
stderr:
stderr: ^
stderr: Error: UNKNOWN, open '/dev/fd/3'
events.js:72
throw er; // Unhandled 'error' event
^
Error: read ECONNRESET
at errnoException (net.js:901:11)
at Pipe.onread (net.js:556:19)
I would expect to see (and do on a OSX box):
$ coffee parent.coffee
stdio3: hello 21101 - a message from your parent
stdout: world 21101 - a message from your parent
stdio3:
stdout:
child process exited with code 0
It is possible to communicate with the child using the command line also on Ubuntu, so the problem is likely in the parent when spawning the child process:
$ echo foo | coffee child.coffee 3>&1
hello 3077 - foo
world 3077 - foo
I have tried to investigate the kernel calls that node makes using strace, but couldn't make much sense of the output.

I figured it out myself. The error was in the child. Ubuntu linux is more strict when it comes to opening files that are already open, the line:
p3 = fs.createWriteStream('/dev/fd/3', {encoding: 'utf8'})
was throwing an error. The file descriptor 3 is already open when the child runs, so the code should look as follows:
Child:
fs = require('fs')
# parent opens the file descriptor 3 when spawning the child (and closes it when the child returns)
fd3write = (s) ->
b = new Buffer(s)
fs.writeSync(3,b,0,b.length)
process.stdin.on 'data', (data) ->
fd3write "p3 #{process.pid} - #{data}\n"
process.stdout.write "so #{process.pid} - #{data}\n", 'utf8'
process.exit(0)
process.stdin.on 'end', (data) ->
console.log "end of stdin"
process.exit(0)
process.stdin.setEncoding('utf8')
process.stdin.resume()
I hope this will be of help to someone else.
To use a pipe instead of stdin to send messages from the parent to the child this link might be of use: child-process-multiple-file-descriptors.

Related

Forked child process keeps terminated with code 1

I wrapped a module using Electron Packager. Because it has heavy computation, i put it in a sub process that would be forked from renderer.js when user clicks a button on index.html.
Pseudo-code renderer.js from :
let cp = require('child_process');
let subprocess;
function log(msg) {
// A function to log messages sent from subprocess
}
document.querySelector('#create').addEventListener('click', ev => {
subprocess = cp.fork('./subprocess.js');
log('A subprocess has been created with pid: ' + subprocess.pid + ' with exexPath = ' + process.execPath);
subprocess.on('exit', (code, signal) => log(`child process terminated: signal = ${signal} ; code = ${code}`));
subprocess.on('error', log);
subprocess.on('message', log);
});
The real problem is: this subprocess runs smoothly when i call electron ./ from console in working directory, but the build generated by Electron Packager wouldn't.
The subprocess does not show up in Task Manager, or rather, it is terminated as soon as it appears. The log says child process terminated: signal = null ; code = 1.
Although i guarded at the beginning of subprocess.js with this to catch uncaughtException
process.on('uncaughtException', (err) => {
process.send(`Caught exception: ${err}`);
});
Nothing is recorded in log. What should i do to overcome this situation?
System specs:
Window 10
Node 8.6
Electron 1.7.12
Electron Packager 10.1.2
I have experienced this too. One reason i came up with was because the child process will be the child process of electron itself.
In my case, it will not recognize the node modules i defined.
I suggest using spawn with the spawn process being node.exe. But that will not be practical once you build your app.

Child process output to socket.io

Forgive the silly question if so, I'm relatively new to Node.
I'm spawning a child process on my node server to import a dataset to a database. The child process, executing osm2pgsl with parameters, has its own internal output that displays the currently processed data and a count of what's been processed.
I have a simple node script to spawn this process, and log information from this process as and when it arrives. The main info that I need access to isn't polled through stdout, stderr or on, which is problematic.
Node script
var util = require('util'),
spawn = require('child_process').spawn,
file = process.argv[2],
ls = spawn('osm2pgsql', ['--slim', '-d', 'gis', '-U', 'postgres', '--number-processes', '3', file]);
ls.stdout.on('data', function (data) {
process.stdout.write('Currently processing: ' + data.toString() + '\r');
});
ls.stderr.on('data', function (data) {
console.log('stderr: ' + data.toString());
});
ls.on('exit', function (code) {
console.log('child process exited with code ' + code.toString());
});
Output
Mid: pgsql, scale=100 cache=800
Setting up table: planet_osm_nodes
stderr: NOTICE: table "planet_osm_nodes" does not exist, skipping
stderr: Setting up table: planet_osm_ways
stderr: NOTICE: table "planet_osm_ways" does not exist, skipping
stderr: Setting up table: planet_osm_rels
stderr: NOTICE: table "planet_osm_rels" does not exist, skipping
stderr:
Reading in file: /OSMDATA/great-britain-latest.osm.pbf
Processing: Node(10k 10.0k/s) Way(0k 0.00k/s) Relation(0 0.00/s)
Processing: Node(20k 20.0k/s) Way(0k 0.00k/s) Relation(0 0.00/s)
Processing: Node(30k 30.0k/s) Way(0k 0.00k/s) Relation(0 0.00/s)
From the stderr: line, you can see that I'm able to access that stream, but the Processing: ... is what I need access to above all else. This is being printed from within the child process and I'm not sure how to access it directly.
Is there any way of accessing the output (above) from within my Nodejs server?
EDIT: I'm intending on piping this output to Socket.io but I need access to it first, hence the title.
Figured it out but it may be useful to people in the future.
Because Processing: ... was being recursively updated on a single line using \r in the osm2pgsql source code, it was actually coming out of the stderr in the same manner as everything else.
The output for the Processing: ... is actually the following line:
stderr:
Reading in file: /OSMDATA/great-britain-latest.osm.pbf
Processing: Node(10k 10.0k/s) Way(0k 0.00k/s) Relation(0 0.00/s)
It didn't occur to me that the output may be multiple lines long.
I'm able to access the output via ls.stderr.on('data', function(data) {} );

Open apps using node.js spawn

I'm trying to do a little application with node.js that would run on mac and execute some commands.
I've successfully used spawn to run command lines such as xcodebuild, but xcrun doesn't seems to work when I try to open the iOS Simulator.
I can open on terminal by typing:
xcrun instruments -w 'iPhone 5s (9.2)' -t <template>
But if I use node and try to use spawn like this:
var args = ['instruments', '-w', `iPhone 5s (9.2)`, '-t', 'noTemp'];
var xcrun = spawn('xcrun', args);
So it got me thinking that maybe it had some limitation opening apps? I tried to run:
var args = ['/Applications/Spotify.app'];
var xcrun = spawn('open', args);
And nothing happens. I couldn't find anything related to that. My question is: is there anyway to open apps using node.js spawn? If there is, does someone know what's the problem with my code?
Here's the full code if needed:
var args = ['instruments', '-w', `${fullDevice}`, '-t', 'noTemp'];
var xcrun = spawn('xcrun', args);
xcrun.stdout.on('data', (data)=>{
console.log(data.toString('utf8'));
})
xcrun.on('close', (code) => {
socket.emit({
time: commands.getCurrentTime(),
type: 'success',
log: 'Device booted...'
});
callback();
if (code !== 0) {
console.log(`open process exited with code ${code}`);
}
});
OBS: if I run this piece of code the application doesn't terminate, the program doesn't continue and nothing happens.
EDIT: Changed:
xcrun.on('data', (data)=>{
To:
xcrun.stdout.on('data', (data)=>{
Spawned processes have two separate streams for stdout and stderr, so you will need to listen for data on those objects and not the spawned process object itself:
xcrun.stdout.on('data', function(data) {
console.log('stdout: ' + data.toString());
});
xcrun.stderr.on('data', function(data) {
console.log('stderr: ' + data.toString());
});
The problem was one line above. Not sure why, but there's a socket.emit call that is wrong and actually hold the program's execution.

node.js python child-process won't print in realtime

I'm trying to make a python program's stdout trigger an event for my node.js app to detect it and send that stdout's data to a client's browser.
The problem is the program won't deliver it's stdout until it terminates... and closes the pipe i guess.
I need it to be so as a line prints node gets the stdout after each print statement.
here's an example python program run with python program.py
from __future__ import print_function
import time
import os
import sys
for num in range(0,8):
print('my []-- ', num)
time.sleep(0.2)
sys.stdout.flush()
Node which spawns a child process with
proc = spawn('python3 program.py -u')
var pid = proc.pid
echoIO( pid )
toggleState()
processDriver(proc)
below listens for the stdout and sends it to a browser with socket io
function processDriver(proc) {
proc.stdout.setEncoding('utf-8');
proc.stdout.on('data', function (data) {
echoIO(data)
})
proc.stderr.on('data', function (data) {
console.log(data)
var res = data.toString()
echoIO(res)
})
proc.on('exit', function (code) {
console.log(code)
stateOff()
echoIO("exit " + code)
sendProcState(JSON.parse('{"status":"not_running"}'))
})
}
Also i've tested this problem in raspbian, ubuntu, and crunchbang, and the issue persists, no other solutions i've found on stackoverflow have worked yet. Sorry if this is about the 20th post.
EDIT:
I've also run into this now
stderr - execvp(): No such file or directory
events.js:72
throw er; // Unhandled 'error' event
^
Error: spawn ENOENT
...
proc = spawn('python', ['program.py'], '-u')
it turns out my method of spawning the child was the error. It's a miracle that it ever spawned a child at all, playing with the code generated the error above, that led me to the post below.
But once I spawned it with this line, instead of a string with 'python program.py -u' it worked.
Node.js - spawned process is generating error "execvp(): No such file or directory"

NodeJS: throw er; //Unhandled 'error' event (events.js:72) when using child_process spawn method

I've made an node.js app to list all .txt files from a directory recursively and, for each one, do some stuff.
Here's my app.js:
var spawn = require('child_process').spawn,
dir = spawn('dir', ['*.txt', '/b']);
dir.stdout.on('data', function (data) {
//do some stuff with each stdout line...
console.log('stdout: ' + data);
});
dir.stderr.on('data', function (data) {
//throw errors
console.log('stderr: ' + data);
});
dir.on('close', function (code) {
console.log('child process exited with code ' + code);
});
When I run node app.js via console, I get the error message below:
events.js:72
throw er; // Unhandled 'error' event
^
Error: spawn ENOENT
at errnoException (child_process.js:980:11)
at Process.ChildProcess._handle.onexit (child_process.js:771:34)
I'm using node v0.10.13 at win32 environment.
I do this way (spawn) because I want to handle stdout line by line (the exec method release entire stdout as one string).
* UPDATE *
By the way, using spawn for child_process does not guarantee that the output for cmd dir will be line by line. I've created a question for that too.
That happen because dir is not a executable in Windows. It's a command from the shell.
The solution for your problem is the following:
var dir = spawn('cmd', ['/c', 'dir']);
dir.stdout.on("data", function() {
// do things
})
This exact problem was here also.
Several things:
dir is not a real executable in windows, so node.js cannot find the program you want to run.
You didn't bind an 'error' handler to your child process and the event was turned into an exception that crashed your node instance. Do this:
dir.on('error', function (err) {
console.log('dir error', err);
});
Use fs.readdir instead. It's a standard node.js API that does the same thing.

Resources