Forgive the silly question if so, I'm relatively new to Node.
I'm spawning a child process on my node server to import a dataset to a database. The child process, executing osm2pgsl with parameters, has its own internal output that displays the currently processed data and a count of what's been processed.
I have a simple node script to spawn this process, and log information from this process as and when it arrives. The main info that I need access to isn't polled through stdout, stderr or on, which is problematic.
Node script
var util = require('util'),
spawn = require('child_process').spawn,
file = process.argv[2],
ls = spawn('osm2pgsql', ['--slim', '-d', 'gis', '-U', 'postgres', '--number-processes', '3', file]);
ls.stdout.on('data', function (data) {
process.stdout.write('Currently processing: ' + data.toString() + '\r');
});
ls.stderr.on('data', function (data) {
console.log('stderr: ' + data.toString());
});
ls.on('exit', function (code) {
console.log('child process exited with code ' + code.toString());
});
Output
Mid: pgsql, scale=100 cache=800
Setting up table: planet_osm_nodes
stderr: NOTICE: table "planet_osm_nodes" does not exist, skipping
stderr: Setting up table: planet_osm_ways
stderr: NOTICE: table "planet_osm_ways" does not exist, skipping
stderr: Setting up table: planet_osm_rels
stderr: NOTICE: table "planet_osm_rels" does not exist, skipping
stderr:
Reading in file: /OSMDATA/great-britain-latest.osm.pbf
Processing: Node(10k 10.0k/s) Way(0k 0.00k/s) Relation(0 0.00/s)
Processing: Node(20k 20.0k/s) Way(0k 0.00k/s) Relation(0 0.00/s)
Processing: Node(30k 30.0k/s) Way(0k 0.00k/s) Relation(0 0.00/s)
From the stderr: line, you can see that I'm able to access that stream, but the Processing: ... is what I need access to above all else. This is being printed from within the child process and I'm not sure how to access it directly.
Is there any way of accessing the output (above) from within my Nodejs server?
EDIT: I'm intending on piping this output to Socket.io but I need access to it first, hence the title.
Figured it out but it may be useful to people in the future.
Because Processing: ... was being recursively updated on a single line using \r in the osm2pgsql source code, it was actually coming out of the stderr in the same manner as everything else.
The output for the Processing: ... is actually the following line:
stderr:
Reading in file: /OSMDATA/great-britain-latest.osm.pbf
Processing: Node(10k 10.0k/s) Way(0k 0.00k/s) Relation(0 0.00/s)
It didn't occur to me that the output may be multiple lines long.
I'm able to access the output via ls.stderr.on('data', function(data) {} );
Related
I'm trying to do a little application with node.js that would run on mac and execute some commands.
I've successfully used spawn to run command lines such as xcodebuild, but xcrun doesn't seems to work when I try to open the iOS Simulator.
I can open on terminal by typing:
xcrun instruments -w 'iPhone 5s (9.2)' -t <template>
But if I use node and try to use spawn like this:
var args = ['instruments', '-w', `iPhone 5s (9.2)`, '-t', 'noTemp'];
var xcrun = spawn('xcrun', args);
So it got me thinking that maybe it had some limitation opening apps? I tried to run:
var args = ['/Applications/Spotify.app'];
var xcrun = spawn('open', args);
And nothing happens. I couldn't find anything related to that. My question is: is there anyway to open apps using node.js spawn? If there is, does someone know what's the problem with my code?
Here's the full code if needed:
var args = ['instruments', '-w', `${fullDevice}`, '-t', 'noTemp'];
var xcrun = spawn('xcrun', args);
xcrun.stdout.on('data', (data)=>{
console.log(data.toString('utf8'));
})
xcrun.on('close', (code) => {
socket.emit({
time: commands.getCurrentTime(),
type: 'success',
log: 'Device booted...'
});
callback();
if (code !== 0) {
console.log(`open process exited with code ${code}`);
}
});
OBS: if I run this piece of code the application doesn't terminate, the program doesn't continue and nothing happens.
EDIT: Changed:
xcrun.on('data', (data)=>{
To:
xcrun.stdout.on('data', (data)=>{
Spawned processes have two separate streams for stdout and stderr, so you will need to listen for data on those objects and not the spawned process object itself:
xcrun.stdout.on('data', function(data) {
console.log('stdout: ' + data.toString());
});
xcrun.stderr.on('data', function(data) {
console.log('stderr: ' + data.toString());
});
The problem was one line above. Not sure why, but there's a socket.emit call that is wrong and actually hold the program's execution.
When i run this pidof command by hand, it works. Then put into my server.js.
// send signal to start the install script
var spw = cp.spawn('/sbin/pidof', ['-x', 'wait4signal.py', '|', 'xargs', 'kill', '-USR1']);
spw.stderr.on('data', function(data) {
res.write('----- Install Error !!! -----\n');
res.write(data.toString());
console.log(data.toString());
});
spw.stdout.on('data', function(data) {
res.write('----- Install Data -----\n');
res.write(data.toString());
console.log(data.toString());
});
spw.on('close', function(data) {
res.end('----- Install Finished, please to to status page !!! -----\n');
console.log('88');
});
In the web i only see "----- Install Finished, please to to status page !!!". My install script seems never get this USR1 signal. Anything wrong please ?
The problem is that you have two separate commands. You are piping the output of your /sbin/pidof command to the input stream of your xargs command. If you are using spawn (rather than exec, which a string exactly as you would write on the command line), you need to spawn one process per command.
Spawn your processes like this:
const pidof = spawn('/sbin/pidof', ['-x', 'wait4signal.py']);
const xargs = spawn('xargs', ['kill', '-USR1']);
Now pipe the output of the first process to the input of the second, like so:
pidof.stdout.pipe(xargs.stdin);
Now you can listen to events on your xargs process, like so:
xargs.stdout.on('data', data => {
console.log(data.toString());
});
I am trying to get two long running node.js processes to communicate - a parent and a child - using pipes and Node's child-process module. I want the child to be able to send data back to the parent asynchronously, and I was hoping to use a pipe to do so.
Here's a simplified version of my code:
Parent:
cp = require('child_process')
es = require('event-stream')
child = cp.spawn('coffee', ['child.coffee'], {stdio: [null, null, null, 'pipe']})
so = child.stdout.pipe(es.split())
p3 = child.stdio[3].pipe(es.split())
so.on 'data', (data) ->
console.log('stdout: ' + data)
child.stderr.on 'data', (data) ->
console.log('stderr: ' + data);
p3.on 'data', (data) ->
console.log('stdio3: ' + data);
child.on 'close', (code) ->
console.log('child process exited with code ' + code)
child.stdin.write "a message from your parent", "utf8"
Child:
fs = require('fs')
p3 = fs.createWriteStream('/dev/fd/3', {encoding: 'utf8'})
process.stdin.on 'data', (data) ->
p3.write "hello #{process.pid} - #{data}\n", 'utf8'
process.stdout.write "world #{process.pid} - #{data}\n", 'utf8'
p3.end()
process.exit(0)
process.stdin.on 'end', (data) ->
console.log "end of stdin"
p3.end()
process.exit(0)
process.stdin.setEncoding('utf8')
process.stdin.resume()
The code works on OSX 10.9, but fails to run on a Ubuntu box. I have tried running it under both Ubuntu 12.04 and 14.04. I am running Node 10.2x.
/dev/fd/ under Ubuntu is symbolically linked to /proc/self/fd/ so I believe my child process is opening the right file.
The output from running the parent on Ubuntu is as follows:
$ coffee parent.coffee
stderr:
stderr: events.js:72
stderr: throw er; // Unhandled 'error' event
stderr:
stderr:
stderr:
stderr:
stderr: ^
stderr: Error: UNKNOWN, open '/dev/fd/3'
events.js:72
throw er; // Unhandled 'error' event
^
Error: read ECONNRESET
at errnoException (net.js:901:11)
at Pipe.onread (net.js:556:19)
I would expect to see (and do on a OSX box):
$ coffee parent.coffee
stdio3: hello 21101 - a message from your parent
stdout: world 21101 - a message from your parent
stdio3:
stdout:
child process exited with code 0
It is possible to communicate with the child using the command line also on Ubuntu, so the problem is likely in the parent when spawning the child process:
$ echo foo | coffee child.coffee 3>&1
hello 3077 - foo
world 3077 - foo
I have tried to investigate the kernel calls that node makes using strace, but couldn't make much sense of the output.
I figured it out myself. The error was in the child. Ubuntu linux is more strict when it comes to opening files that are already open, the line:
p3 = fs.createWriteStream('/dev/fd/3', {encoding: 'utf8'})
was throwing an error. The file descriptor 3 is already open when the child runs, so the code should look as follows:
Child:
fs = require('fs')
# parent opens the file descriptor 3 when spawning the child (and closes it when the child returns)
fd3write = (s) ->
b = new Buffer(s)
fs.writeSync(3,b,0,b.length)
process.stdin.on 'data', (data) ->
fd3write "p3 #{process.pid} - #{data}\n"
process.stdout.write "so #{process.pid} - #{data}\n", 'utf8'
process.exit(0)
process.stdin.on 'end', (data) ->
console.log "end of stdin"
process.exit(0)
process.stdin.setEncoding('utf8')
process.stdin.resume()
I hope this will be of help to someone else.
To use a pipe instead of stdin to send messages from the parent to the child this link might be of use: child-process-multiple-file-descriptors.
I cannot get any output in the following code:
var spawn = require('child_process').spawn,
script = 'ftp',
child = spawn(script);
child.stdout.on('data', function (data) {
console.log('stdout: ' + data);
});
child.stderr.on('data', function (data) {
console.log('stderr: ' + data);
});
child.on('close', function (code) {
console.log('child process exited with code ' + code);
});
It works for normal scripts such as 'ls', 'pwd' etc. But not for interactive programs such as 'ftp', 'telnet'. Any suggestions?
Edit:
Take another script for example:
#!/usr/bin/env python
name = raw_input("your name>")
print name
When spawn this script, I wish to fetch the prompt "your name>" with the data event, so that I can latter input something into stdin.
The problem is that I got nothing in the data event, and it seemed that none of these events are triggered.
ls, cat is controllable via input output and error stream.
ftp, telnet is controllable indirectly via tty.
The protocol is also base on input/output stream but it is more complicated. You can use available package to handle that protocol.
https://github.com/chjj/pty.js
var pty = require('pty.js');
var term = pty.spawn('ftp', [], options);
term.on('data', function(data) {
console.log(data);
});
term.write(ftpCmd + '\r');
The author of pty have some interesting examples, he forward pty to web via web socket, including terminal games:
https://github.com/chjj/tty.js
In interactive mode there is a command interpreter that reads user input from stdin, then accodingly prints output. So you have to write to stdin to do something. For example add following lines to your code with telnet command:
child.stdin.write('?\n');
child.stdin.write('quit\n');
Output:
stdout: Commands may be abbreviated. Commands are:
! cr mdir proxy send
$ delete mget sendport site
account debug mkdir put size
append dir mls pwd status
ascii disconnect mode quit struct
bell form modtime quote system
binary get mput recv sunique
bye glob newer reget tenex
case hash nmap rstatus trace
ccc help nlist rhelp type
cd idle ntrans rename user
cdup image open reset umask
chmod lcd passive restart verbose
clear ls private rmdir ?
close macdef prompt runique
cprotect mdelete protect safe
child process exited with code 0
I have a child process that I am using as follows in node.js. Instead of redirecting the output to the console I would like to put the output in a log file located somewhere on the machine this is running on (and should work for both windows and mac).
The code below is what I am using and I would like to output the files into a log file. What changes needed to do that here? Thanks!
My Code:
var spawn = require('child_process').spawn,
ls = spawn('ls', ['-lh', '/usr']);
ls.stdout.on('data', function (data) {
console.log('stdout: ' + data);
});
ls.stderr.on('data', function (data) {
console.log('stderr: ' + data);
});
ls.on('close', function (code) {
console.log('child process exited with code ' + code);
});
Here's an example of logging to file using streams.
var logStream = fs.createWriteStream('./logFile.log', {flags: 'a'});
var spawn = require('child_process').spawn,
ls = spawn('ls', ['-lh', '/usr']);
ls.stdout.pipe(logStream);
ls.stderr.pipe(logStream);
ls.on('close', function (code) {
console.log('child process exited with code ' + code);
});
There are two ways you can achieve this, one is using
let logConsoleStream = fs.createWriteStream('./logConsoleFile.log', {flags: 'a'});
let logErrorStream = fs.createWriteStream('./logErrorFile.log', {flags: 'a'});
and redirect all logs or errors using this
ls.stdout.pipe(logConsoleStream ); // redirect stdout/logs only
ls.stderr.pipe(logErrorStream ); // redirect error logs only
by separating log files you will have separate files for Error logs and console logs
this is exactly same as generalhenry shared above
And Second Way for Achieving this with the help of Command Line
when you execute node app from the command line
node app/src/index.js
you can specify here where you want to redirect logs and Errors from this application
there are three stream redirection commands using the command line
`>` It will redirect your only stdout or logs to the specified path
`2>` It will redirect your errors logs to the specified path
`2>&1 ` It will redirect all your stdout and stderr to a single file
example: how you will use these commands
node app/src/index.js > ./logsOnly.txt
node app/src/index.js 2> ./ErrorsOnly.txt
node app/src/index.js 2>&1 ./consoleLogsAndErrors.txt
I hope someone coming later finds this helpful
if there is I done wrong way please do let me know it will help me and others
Thanks
If you run your JS script with forever then you have the option to define a log file as parameter which will handle all your console.log messages. Not to mention the benefit of keeping your nodejs app live permanently.
Alternatively try this:
sudo forever start myapp.js 2>&1 /home/someuser/myapp/myapp.log
use forever with below options
forever start -o out.log -e err.log server.js
The best answer was in the comments and is mentioned in a previous question here: stackoverflow.com/questions/2496710/nodejs-write-to-file
It is as follows:
var fs = require('fs');
fs.writeFile("/tmp/test", "Hey there!", function(err) {
if(err) {
console.log(err);
} else {
console.log("The file was saved!");
}
});