My Scenario
In my node application I'm using child_process.spawn to query information from the current repository
I've built a small function to return a promise which resolves with the response from the command:
const spawn = require('child_process').spawn;
const gitExec = command => (
new Promise((resolve, reject) => {
const thread = spawn('git', command);
const stdOut = [];
const stdErr = [];
thread.stdout.on('data', (data) => {
stdOut.push(data.toString('utf8'));
});
thread.stderr.on('data', (data) => {
stdErr.push(data.toString('utf8'));
});
thread.on('close', () => {
if (stdErr.length) {
reject(stdErr.join(''));
return;
}
resolve(stdOut.join());
});
})
);
module.exports = gitExec;
Calling git branchworks just as expected:
gitExec(['branch'])
.then((branchInfo) => {
console.log(branchInfo);
})
(as expected) results in
* develop
feature/forever
feature/sourceconfig
feature/testing
master
From my understanding this proves the method I'm using to actually be working.
When calling git shortlog -sn the spawned process "hangs" and does not resolve in anything
gitExec(['shortlog', '-sn'])
.then((shortlogInfo) => {
console.log(shortlogInfo);
})
Calling git shortlog -sn via commandline i geht the expected result:
154 Andreas Gack
89 Some other dude
6 Whoever else
My (so far unsuccessfull) tries
using spawnSync (while changing my gitExec function to acommodate the syncronous approach) returns an object as documented - so the process seems to actually exit - but the relevant props of the object output stdout and stderr are all empty.
The status of the object is 0 which indicates the command to execute successfully
I've read about having to redefine the maxBuffer in the spawn options, but neither setting it to a (ridiculously) high value nor a very small value does make a difference in the syncronous or asynchronous approach.
Setting the shell option to true also does not make a difference in all of the above scenarios.
The issue occurrs on my Win10x64 as well as on MacOs running node v6.9.x or 7.x
Also calling the alias git log --pretty=short does not provide a result
My Actual Question
Has anyone yet managed to successfully query git shortlog -sn via child_process.spawn?
Does anyone know a module for Node which allows querying the current local git-repository?
I somehow think the two commands git branch and git shortlog internally handle their output in a different way.
I'd be happy to create an issue on their github page, but I actually don't know how to identify the actual root-cause of that problem.
Any further input is much appreciated!
git shortlog thinks it needs to read something from stdin thats why the whole process hangs. To work around that, you can pass stdin from the main process in as an option and pipe everything else as usual. Then it should run.
const spawn = require('child_process').spawn;
const gitExec = command => (
new Promise((resolve, reject) => {
const thread = spawn('git', command, { stdio: ['inherit', 'pipe', 'pipe'] });
const stdOut = [];
const stdErr = [];
thread.stdout.on('data', (data) => {
stdOut.push(data.toString('utf8'));
});
thread.stderr.on('data', (data) => {
stdErr.push(data.toString('utf8'));
});
thread.on('close', () => {
if (stdErr.length) {
reject(stdErr.join(''));
return;
}
resolve(stdOut.join());
});
})
);
module.exports = gitExec;
Perhaps some more context from the git documentation:
If no revisions are passed on the command line and either standard input is not a terminal or there is no current branch, git shortlog will output a summary of the log read from standard input, without reference to the current repository.
Which is the case when spawning a child process. So it expects something gets passed in via stdin. By setting stdin to the main process git shortlog is aware of the terminal and therefor will run.
I got it working by specifying the before and after commit hashes.
git shortlog -sn `git log --pretty=format:'%H' --reverse | head -1` `git log --pretty=format:'%H' | head -1`"
Related
I've written an Angular builder that dynamically sets up my jest command and then spawns it, as shown below. While I do get all the output, it's not showing the same way as if I were to run the child process directly from the command line.
For example, if I manually run the node command then I get VT100-like color coding, and a timer is shown as the jest tests are starting. With the below code, that doesn't happen, and the output just all comes out hodge-podge.
Is there a different way to handle running the command?
export async function runJest(options: Options, context: BuilderContext): Promise<BuilderOutput> {
...
const child = spawn('node', [
'--experimental-vm-modules',
'--no-warnings',
'./node_modules/jest/bin/jest.js',
'--no-cache',
'--config',
JSON.stringify(config)
])
child.stdout.on('data', x => process.stderr.write(x))
child.stderr.on('data', x => process.stderr.write(x))
const exitCode = await new Promise((resolve) => child.on('close', resolve))
return {success: exitCode === 0}
}
I am launching an external program (adb shell) via node's spawn method.
Then I am sending commands via childProc.stdin.write(input) and the shell returns stuff accordingly.
My data callback gets called each time there is a new data chunk available. All good!
But since the connection stays open, I don't know when the external program is finished sending data.
Is there a way to solve this? Isn't there some standard way of showing "that's the data you asked for, I am done returning"?
static async execCommand2(command, args, input) {
if(!this.adbShellProc || this.adbShellProc.killed){
this.adbShellProc = spawn(command, args)
this.adbShellProc.stdout.setEncoding('utf8')
}
const self = this
let promise = new Promise((resolve, reject) => {
log.debug('Shell.execCommand2:', command, 'args:', args)
log.debug('send input: ', input)
self.adbShellProc.stdin.write(input)
self.adbShellProc.stdin.write('\n')
self.adbShellProc.stdout.on('data', chunk => {
// resolve(chunk)
// PROBLEM: I can't resolve here, cause there might be more data coming in
})
self.adbShellProc.on('close', code => {
self.adbShellProc.stdin.end()
self.adbShellProc = null
})
})
return promise
}
I am sure your answer is in your question itself. You have added close event listener, so when process is ended, your control will be in close event(Just one modification is, close event will be of adbShellProc.stdout), so you can resolve/return data from there.
static async execCommand2(command, args, input) {
let finalOutcome = ''
if(!this.adbShellProc || this.adbShellProc.killed){
this.adbShellProc = spawn(command, args)
this.adbShellProc.stdout.setEncoding('utf8')
}
const self = this
let promise = new Promise((resolve, reject) => {
log.debug('Shell.execCommand2:', command, 'args:', args)
log.debug('send input: ', input)
self.adbShellProc.stdin.write(input)
self.adbShellProc.stdin.write('\n')
self.adbShellProc.stdout.on('data', chunk => {
// resolve(chunk)
finalOutcome += data
// PROBLEM: I can't resolve here, cause there might be more data coming in
})
self.adbShellProc.stdout.on('close', code => {
self.adbShellProc.stdin.end()
self.adbShellProc = null
resolve(finalOutcome)
})
})
return promise
}
Also, it has another event end of stdout,
ls.stdout.on('end', (d) =>{
// You can resolve from here also
})
You can get more details here Child Process
No. There's no way of knowing if the output is final unless an exit code is returned, this is the linux way.
Fortunately, adb shell gives you a way of running one off commands using the adb shell without having to start it in interactive mode.
Instead of starting the shell and then sending the commands as input, make the whole thing a single command. Example:
adb shell am start -a android.intent.action.VIEW
Then the close event will give you a clear indication that the command has finished so you can resolve and you will also get the return code, so you can check if the command has succeeded or not.
There's no real benefit in starting adb shell in interactive mode, because the real processing is not happening in adb but in the actual server, which is adbd.
https://developer.android.com/studio/command-line/adb#shellcommands
static async execCommand2(command, args, input) {
let finalOutcome = ''
//append the input to the arguments and don't use this, all your commands are one offs and do not require a state check
let adbShellProc = spawn(command, args+input)
adbShellProc.stdout.setEncoding('utf8')
let promise = new Promise((resolve, reject) => {
log.debug('Shell.execCommand2:', command, 'args:', args+input)
self.adbShellProc.stdout.on('data', chunk => {
finalOutcome += data
})
self.adbShellProc.stdout.on('close', code => {
//unix return code, 0 is good, not zero means error
if(code != 0){
reject(finalOutcome)
}
resolve(finalOutcome)
})
})
return promise
}
You can end your shell commands with ; echo some_marker and wait until some_marker appears in the stdout. (And you can also get the exit code with ; echo some_marker $?)
Example of codes that will execute command line using from Node.js. It will return full HTML of page.
const getPageHtmlResponse = (fullUrl) => {//fullUrl is come from input form in web page
return new Promise((resolve, reject) => {
try {
const exec = require('child_process').exec
exec("curl "+fullUrl, (err, stdout, stderr) => resolve(stdout))
} catch (error) {
resolve(false)
}
});
}
Is this code can be insecure? I mean the hackers can inject another command on it to manipulate the system or server?
If yes, there's good way to escape it or make it secure?
Don't use child_process.exec(). A clever crafted string from user input will launch arbitrary code from your program, which you do want to avoid.
Instead, use child_process.execFile() as follows:
const getPageHtmlResponse = (fullUrl) => {//fullUrl is come from input form in web page
return new Promise((resolve, reject) => {
try {
const execFile = require('child_process').execFile
execFile("/path/to/curl", fullUrl, (err, stdout, stderr) => resolve(stdout))
} catch (error) {
resolve(false)
}
});
}
execFile takes the pre-parsed list of commands and does not launch an intermediate shell, so there is less risk of launching a program through an untrusted URL.
See also
child_process.execFile
If the user writes a script and put it on http://blahblah.blah/b and instead of this URL it provides a tricky one: http://blahblah.blah/b | sh, now your code will create a process and execute curl http://blahblah.blah/b | sh. and the script could be anything.
one thing you should consider is that, to validate user input URL, check it to no contain extra commands, and be the only url.
I'm running Windows 10, and I have a program, let's call it program, that can be run from the command line. When run, it responds to commands the user enters. The user enters a command, presses the return key, and the program prints a response. I did not make this program and do not have the source, so I cannot modify it.
I want to run this program from within Node.js, and have my Node.js program act as the user, sending it commands and getting the responses. I spawn my program like this:
var spawn = require('child_process').spawn;
var child = spawn('program');
child.stdout.on('data', function(data) {
console.log(`stdout: ${data}`);
});
Then I attempt to send it a command, for example, help.
child.stdin.write("help\n");
And nothing happens. If I manually run the program, type help, and press the return key, I get output. I want Node.js to run the program, send it input, and receive the output exactly as a human user would. I assumed that stdin.write() would send the program a command as if the user typed it in the console. However, as the program does not respond, I assume this is not the case. How can I send the program input?
I've seen many similar questions, but unfortunately the solutions their authors report as "working" did not work for me.
Sending input data to child process in node.js
I've seen this question and answer and tried everything in it with no success. I've tried ending the command with \r\n instead of \n. I've also tried adding the line child.stdin.end() after writing. Neither of these worked.
How to pass STDIN to node.js child process
This person, in their self-answer, says that they got theirs to work almost exactly as I'm doing it, but mine does not work.
Nodejs Child Process: write to stdin from an already initialised process
This person, in their self-answer, says they got it to work by writing their input to a file and then piping that file to stdin. This sounds overly complicated to send a simple string.
This worked for me, when running from Win10 CMD or Git Bash:
console.log('Running child process...');
const spawn = require('child_process').spawn;
const child = spawn('node');
// Also worked, from Git Bash:
//const child = spawn('cat');
child.stdout.on('data', (data) => {
console.log(`stdout: "${data}"`);
});
child.stdin.write("console.log('Hello!');\n");
child.stdin.end(); // EOF
child.on('close', (code) => {
console.log(`Child process exited with code ${code}.`);
});
Result:
D:\Martin\dev\node>node test11.js
Running child process...
stdout: "Hello!
"
Child process exited with code 0.
I also tried running aws configure like this, first it didn't work because I sent only a single line. But when sending four lines for the expected four input values, it worked.
Maybe your program expects special properties for stdin, like being a real terminal, and therefore doesn't take your input?
Or did you forget to send the EOF using child.stdin.end();? (If you remove that call from my example, the child waits for input forever.)
Here is what worked for me. I have used child_process exec to create a child process. Inside this child process Promise, I am handling the i/o part of the cmd given as parameter. Its' not perfect, but its working.
Sample function call where you dont need any human input.
executeCLI("cat ~/index.html");
Sample function call where you interact with aws cli. Here
executeCLI("aws configure --profile dev")
Code for custom executeCLI function.
var { exec } = require('child_process');
async function executeCLI(cmd) {
console.log("About to execute this: ", cmd);
var child = exec(cmd);
return new Promise((resolve, reject) => {
child.stdout.on('data', (data) => {
console.log(`${data}`);
process.stdin.pipe(child.stdin);
});
child.on('close', function (err, data) {
if (err) {
console.log("Error executing cmd: ", err);
reject(err);
} else {
// console.log("data:", data)
resolve(data);
}
});
});
}
Extract the user input code from browser and save that code into a file on your system using fs module. Let that file be 'program.cpp' and save the user data input in a text file.
As we can compile our c++ code on our terminal using g++ similarly we will be using child_process to access our system terminal and run user's code.
execFile can be used for executing our program
var { execFile } = require('child_process');
execFile("g++", ['program.cpp'], (err, stdout, stderr) => {
if (err) {
console.log("compilation error: ",err);
} else{
execFile ('./a.out' ,['<', 'input.txt'], {shell: true}, (err, stdout, stderr) => {
console.log("output: ", stdout);
})
}
})
In this code we simply require the child_process and uses its execFile function.
First we compile the code present in program.cpp, which creates a default a.out as output file
Then we pass the a.out file with input that is present in input.txt
Hence you can view the generated output in your terminal and pass that back to the user.
for more details you can check: Child Processes
I'm using child_process.spawn and need to capture the shell error that occurs when the command fails. According to this question, I should be able to do:
var child_process = require('child_process');
var python = child_process.spawn(
'python', ["script.py", "someParam"]
);
python.on('error', function(error) {
console.log("Error: bad command", error);
});
When I replace 'python', ["script.py", "someParam"] with banana, like in the linked question, it works, and the error is visible. But in my case, using python with arguments, the 'error' event is never called.
How can I capture shell errors from python?
According to the Node.js docs for the ChildProcess error event, it is only fired in a few situations:
The process could not be spawned, or
The process could not be killed, or
Sending a message to the child process failed for whatever reason.
To capture the shell error output, you can additionally listen to data events on the stdout and stderr of your spawned process:
python.stdout.on('data', function(data) {
console.log(data.toString());
});
python.stderr.on('data', function(data) {
console.error(data.toString());
});
To capture the shell error code, you can attach a listener to the exit event:
python.on('exit', function(code) {
console.log("Exited with code " + code);
});
Thread is a little bit old and all but I encountered this today while working on my test casing library. I realize that the accepted answer has a solution already, but, for me, it is not clearly explained. Anyway in case someone needs it here is what you need to do.
The thing I realized is that, while executing code, if python interpreter encounters an error, or should I say, if your code has an error in it, it will write it to standard error stream and exit. So what you need to do, in your Node.js code, is to listen to the stderr stream of the spawned process. In addition, all of the data passed to print() function is written to the 'stdout' stream of the process.
So here is an example code:
const { spawn } = require('child_process');
const proc = spawn('python',['main.py','-c']);
proc.stderr.on('data',(data)=>{
//Here data is of type buffer
console.log(data.toString())
})
proc.stdout('data',(data)=>{
//Also buffer
console.log(data.toString());
})
What happens clear should already be clear if you read the first part of my answer. One other thing you could do instead of writing data to the console, is redirect it to another stream, this could be really useful if you want to write output data to a file for example. This is how you could do it:
const fs = require('fs');
const path = require('path');
const { spawn } = require('child_process');
const outputFile = path.join(__dirname,'output.txt');
const errorFile = path.join(__dirname,'output.txt');
const outputStream = fs.createWriteStream(outputFile, {
encoding: "utf8",
autoClose: true
});
const outputStream = fs.createWriteStream(errorFile, {
encoding: "utf8",
autoClose: true
});
const proc = spawn('python',['main.py','-c']);
proc.stdout.pipe(outputStream);
proc.stderr.pipe(errorStream);
What is happening here is that, using pipe function we send all data from stdout and stderr of the process to the file streams. Also you do not have to worry about files existing, it will create them for you