I spawn a child process like this:
var child = require('child_process');
var proc = child.spawn('python', ['my_script.py', '-p', 'example']);
I also set some data handling:
proc.stdin.setEncoding('utf8');
proc.stdout.setEncoding('utf8');
proc.stderr.setEncoding('utf8');
proc.stdout.on('data', function (data) {
console.log('out: ' + data);
});
proc.stderr.on('data', function (data) {
console.log('err: ' + data);
});
proc.on('close', function (code) {
console.log('subprocess exited with status ' + code);
proc.stdin.end();
});
My Python script reads lines from stdin and for each line does some operations and prints to stdout. It works fine in the shell (I write a line and I get the output immediately) but when I do this in Node:
for (var i = 0; i < 10; i++) {
proc.stdin.write('THIS IS A TEST\n');
}
I get nothing.
I got to fix it calling proc.stdin.end() but that also terminates the child process (which I want to stay in background, streaming data).
I also triggered a flush filling the buffer with lots of writes, but that's not really an option.
Is there any way to manually flush the stream?
You are not flushing the output from Python after print statement. I had similar problem and #Alfe answered my question. Take a look at this:
Stream child process output in flowing mode
Related
So far I have gotten my script to execute a windows .bat file with child_process, my issue is that it opens it in the background with no way to “connect” to it to see what happens and debug, is there a way to “listen” for a certain output to happen? For example, if the .bat outputs a “Done!” in the shell at one point, is there a way to make my node.js script detect that certain keyword and run further commands if it does?
Thanks!
Some clarification: The .bat outputs "Done!" and stays running, it doesn't stop, all I want to do is detect that "Done!" so that I can send a message to the user that the server has successfully started
My current code:
exec('D:\\servers\\game_server_1\\start.bat', {shell: true, cwd: 'D:\\servers\\game_server_1'});
Well, if you're trying to do a one and done type of NodeJS script, you can just spawn a process that launches with the given command and exits when all commands completed. This creates a one and done streaming interface that you can monitor. The stdout returns a data buffer that returns the command you ran, unless it's something like START to launch a program-- it returns null. You could just issue a KILL command after the START -- your_program.exe:
const spawn = require('child_process').spawn;
const child = spawn('cmd.exe', ['/c', 'commands.bat']);
let DONE = 0;
const done = () => {
console.log("log it");
DONE++;
};
child.stdout.on('data', function (data) {
console.log('stdout: ' + data);
//it's important to add some type of counter to
//prevent any logic from running twice, since
//this will run twice for any given command
if ( data.toString().includes("DONE") && DONE === 0 ) {
done();
}
});
child.stderr.on('data', function (data) {
console.log('stderr: ' + data);
});
child.on('exit', function (code) {
console.log('child process exited with code ' + code);
});
Keep in mind, when you run a command to launch a program and the program launches, the data buffer will be null in stdout event listener. The error event will only fire if there was an issue with launching the program.
YOUR .BAT:
ECHO starting batch script
//example launching of program
START "" https://localhost:3000
//issue a command after your program launch
ECHO DONE
EXIT
You could also issue an ECHO DONE command right after the command where you launched the program and listen for that, and try and parse out that command from stdout.
You could use a Regular expression.
const { spawn } = require('child_process');
const child = spawn(...);
child.stdout.on('data', function (data) {
console.log('stdout: ' + data);
// Now use a regular expression to detect a done event
// For example
data.toString().match(/Done!/);
});
// Error handling etc. here
I am new to node js. I have written a script that spawns 5 child process and captures log from them,
As of now the log looks like mess.
But i want to display the output in a cleaner format, like each line gets reused and data gets loged next to the process id.
something like this:
below is my code:
var state = "node stealth.js ";
var exec = require('child_process').exec;
workers = 5
p_list = [];
for (var i = 0; i < workers; i++) {
(function(i){
var child = exec(state+'user'+i);
console.log('started Worker '+i)
// Add the child process to the list for tracking
p_list.push({process:child, content:""});
// Listen for any response:
child.stdout.on('data', function (data) {
console.log(child.pid, data);
p_list[i].content += data;
process.stdout.cursorTo(i);
process.stdout.clearLine();
process.stdout.write(child.pid+data);
process.stdout.write("\n"); // end the line
});
// Listen for any errors:
child.stderr.on('data', function (data) {
console.log(child.pid, data);
p_list[i].content += data;
});
// Listen if the process closed
child.on('close', function(exit_code) {
console.log('Closed before stop: Closing code: ', exit_code);
});
})(i)
}
You are looking for VT100 Terminal Control Escape Sequences. These sequences allow to move the cursor around or color the text.
Example
console.log('Line 1');
console.log('Line 2');
console.log('\x1B[1AAgain line 2');
The \x1B[1A sequence at the start moves the cursor one line up and therefore the text after it overwrites the previous text. The result looks like this:
Line 1
Again line 2
For more sequences, check out the part of the page about cursor movement.
Finally i figured out a solution.
Since i need to gather data from different processes and display in a neat way, i started using pm2 process manager, it does a lot more than just console log from processes.
for anyone facing same problem please check out this node package
I am trying to write a GUI with Node and Electron around an existing ruby command line application. I found an example how to get output from a child process by doing something like:
var spawn = require('child_process').spawn;
var child = spawn('node', ['child.js']);
child.stdout.on('data', function (data) {
console.log('stdout: ' + data);
});
child.stderr.on('data', function (data) {
console.log('stderr: ' + data);
});
And the child.js looks like this
while(true) {
console.log('blah');
}
This is working fine for me, but if I try and do this with a ruby application switching to this
var child = spawn('ruby', ['test.rb']);
and this being the ruby code
while true
sleep 2
puts 'test'
end
I get no output. It seems to hang on the output. I would expect to see 'test' printed every 2 seconds.
Ruby doesn't know you want the output right away, and buffers it for efficiency. If you make it flush the output buffer (IO#flush), it works.
while true
sleep 2
puts 'test'
STDOUT.flush
end
Alternately, you can tell Ruby you want all output right away (IO#sync=):
STDOUT.sync = true
while true
sleep 2
puts 'test'
end
Incidentally, your original code works on JRuby, so the buffering behaviour differs... I did not know that...
I'm starting to learn and use node and I like it but I'm not really sure how certain features work. Maybe you can help me resolve one such issue:
I want to spawn local scripts and programs from my node server upon rest commands. looking at the fs library I saw the example below of how to spawn a child process and add some pipes/event handlers on it.
var spawn = require('child_process').spawn,
ps = spawn('ps', ['ax']),
grep = spawn('grep', ['ssh']);
ps.stdout.on('data', function (data) {
grep.stdin.write(data);
});
ps.stderr.on('data', function (data) {
console.log('ps stderr: ' + data);
});
ps.on('close', function (code) {
if (code !== 0) {
console.log('ps process exited with code ' + code);
}
grep.stdin.end();
});
grep.stdout.on('data', function (data) {
console.log('' + data);
});
grep.stderr.on('data', function (data) {
console.log('grep stderr: ' + data);
});
grep.on('close', function (code) {
if (code !== 0) {
console.log('grep process exited with code ' + code);
}
});
What's weird to me is that I don't understand how I can be guaranteed that the event handler code will be registered before the program starts to run. It's not like there's a 'resume' function that you run to start up the child. Isn't this a race condition? Granted the condition would be minisculy small and would almost never hit because its such a short snipping of code afterward but still, if it is I'd rather not code it this way out of good habits.
So:
1) if it's not a race condition why?
2) if it is a race condition how could I write it the right way?
Thanks for your time!
Given the slight conflict and ambiguity in the accepted answer's comments, the sample and output below tells me two things:
The child process (referring to the node object returned by spawn) emits no events even though the real underlying process is live / executing.
The pipes for the IPC are setup before the child process is executed.
Both are obvious. The conflict is w.r.t. interpretation of the OP's question:-
Actually 'yes', this is the epitome of a data race condition if one needs to consider the real child process's side effects. But 'no', there's no data race as far as IPC pipe plumbing is concerned. The data is written to a buffer and retrieved as a (bigger) blob as and when (as already well described) the context completes allowing the event loop to continue.
The first data event seen below pushes not 1 but 5 chunks written to stdout by the child process whilst we were blocking.. thus nothing is lost.
sample:
let t = () => (new Date()).toTimeString().split(' ')[0]
let p = new Promise(function (resolve, reject) {
console.log(`[${t()}|info] spawning`);
let cp = spawn('bash', ['-c', 'for x in `seq 1 1 10`; do printf "$x\n"; sleep 1; done']);
let resolved = false;
if (cp === undefined)
reject();
cp.on('error', (err) => {
console.log(`error: ${err}`);
reject(err);
});
cp.stdout.on('data', (data) => {
if (!resolved) {
console.log(`[${t()}|info] spawn succeeded`);
resolved = true;
resolve();
}
process.stdout.write(`[${t()}|data] ${data}`);
});
let ts = parseInt(Date.now() / 1000);
while (parseInt(Date.now() / 1000) - ts < 5) {
// waste some cycles in the current context
ts--; ts++;
}
console.log(`[${t()}|info] synchronous time wasted`);
});
Promise.resolve(p);
output:
[18:54:18|info] spawning
[18:54:23|info] synchronous time wasted
[18:54:23|info] spawn succeeded
[18:54:23|data] 1
2
3
4
5
[18:54:23|data] 6
[18:54:24|data] 7
[18:54:25|data] 8
[18:54:26|data] 9
[18:54:27|data] 10
It is not a race condition. Node.js is single threaded and handles events on a first come first serve basis. New events are put at the end of the event loop. Node will execute your code in a synchronous manner, part of which will involve setting up event emitters. When these event emitters emit events, they will be put to the end of the queue, and will not be handled until Node finishes executing whatever piece of code its currently working on, which happens to be the same code that registers the listener. Therefore, the listener will always be registered before the event is handled.
Consider the following C program (test.c):
#include <stdio.h>
int main() {
printf("string out 1\n");
fprintf(stderr, "string err 1\n");
getchar();
printf("string out 2\n");
fprintf(stderr, "string err 2\n");
fclose(stdout);
}
Which should print a line to stdout, a line to stderr, then wait for user input, then another line to stdout and another line to stderr. Very basic!
When compiled and run on the command line the output of the program when complete (user input is received for getchar()):
$ ./test
string out 1
string err 1
string out 2
string err 2
When trying to spawn this program as a child process using nodejs with the following code:
var TEST_EXEC = './test';
var spawn = require('child_process').spawn;
var test = spawn(TEST_EXEC);
test.stdout.on('data', function (data) {
console.log('stdout: ' + data);
});
test.stderr.on('data', function (data) {
console.log('stderr: ' + data);
});
// Simulate entering data for getchar() after 1 second
setTimeout(function() {
test.stdin.write('\n');
}, 1000);
The output appears like this:
$ nodejs test.js
stderr: string err 1
stdout: string out 1
string out 2
stderr: string err 2
Very different from the output as seen when running ./test in the terminal. This is because the ./test program isn't running in an interactive shell when spawned by nodejs. The test.c stdout stream is buffered and when run in a terminal as soon as a \n is reached the buffer is flushed but when spawned in this way with node the buffer isn't flushed. This could be resolved by either flushing stdout after every print, or changing the stdout stream to be unbuffered so it flushes everything immediately.
Assuming that test.c source isn't available or modifiable, neither of the two flushing options mentioned can be implemented.
I then started looking at emulating an interactive shell, there's pty.js (pseudo terminal) which does a good job, for example:
var spawn = require('pty.js').spawn;
var test = spawn(TEST_EXEC);
test.on('data', function (data) {
console.log('data: ' + data);
});
// Simulate entering data for getchar() after 1 second
setTimeout(function() {
test.write('\n');
}, 1000);
Which outputs:
$ nodejs test.js
data: string out 1
string err 1
data:
data: string out 2
string err 2
However both stdout and stderr are merged together (as you would see when running the program in a terminal) and I can't think of a way to separate the data from the streams.
So the question..
Is there any way using nodejs to achieve the output as seen when running ./test without modifying the test.c code? Either by terminal emulation or process spawning or any other method?
Cheers!
I tried the answer by user568109 but this does not work, which makes sense since the pipe only copies the data between streams. Hence, it only gets to process.stdout when the buffer is flushed...
The following appears to work:
var TEST_EXEC = './test';
var spawn = require('child_process').spawn;
var test = spawn(TEST_EXEC, [], { stdio: 'inherit' });
//the following is unfortunately not working
//test.stdout.on('data', function (data) {
// console.log('stdout: ' + data);
//});
Note that this effectively shares stdio's with the node process. Not sure if you can live with that.
I was just revisiting this since there is now a 'shell' option available for the spawn command in node since version 5.7.0. Unfortunately there doesn't seem to be an option to spawn an interactive shell (I also tried with shell: '/bin/sh -i' but no joy).
However I just found this which suggests using 'stdbuf' allowing you to change the buffering options of the program that you want to run. Setting them to 0 on everything produces unbuffered output for all streams and they're still kept separate.
Here's the updated javascript:
var TEST_EXEC = './test';
var spawn = require('child_process').spawn;
var test = spawn('stdbuf', ['-i0', '-o0', '-e0', TEST_EXEC]);
test.stdout.on('data', function (data) {
console.log('stdout: ' + data);
});
test.stderr.on('data', function (data) {
console.log('stderr: ' + data);
});
// Simulate entering data for getchar() after 1 second
setTimeout(function() {
test.stdin.write('\n');
}, 1000);
Looks like this isn't pre-installed on OSX and of course not available for Windows, may be similar alternatives though.
You can do this :
var TEST_EXEC = 'test';
var spawn = require('child_process').spawn;
var test = spawn(TEST_EXEC);
test.stdin.pipe(process.stdin);
test.stdout.pipe(process.stdout);
test.stderr.pipe(process.stderr);
When you use events on stdout and stderr to print the output on console.log, you will get jumbled output because of asynchronous execution of the functions. The output will be ordered for a stream independently, but output can still get interleaved among stdin,stdout and stderr.