I try to execute long processes sequentially with node.js (docker exec commands).
I do:
const childProcess = require('child_process');
const execWithPromise = async command => {
return new Promise(async resolve => {
const process = childProcess.exec(command);
process.on('exit', err => resolve(err));
process.on('close', err => resolve(err));
});
};
const run = async () => {
await execWithPromise('/usr/local/bin/docker exec -i -t cucumber node long-running-script.js');
await execWithPromise('/usr/local/bin/docker exec -i -t cucumber node long-running-script.js');
};
run();
But the promise is resolved immediately with a result of 1. In both cases. The command runs on the commandline just fine.
Why is it returning immediately?
child_process.exec expects a callback as the second or third argument. It doesn't return a promise. You have a few choices depending on your use case and version of node. The following work with node 16.x
Use a callback and return the resolve.
const execWithPromise = command =>
new Promise((resolve, reject) => {
childProcess.exec(command, (err, stout, sterr) => {
if(err) {
reject(sterr)
} else {
resolve(stout)
}
})
})
Use spawn instead (keeping most of your code)
const execWithPromise = command =>
new Promise((resolve, reject) => {
const process = childProcess.spawn(command);
let data = '';
let error = '';
process.stdout.on('data', stdout => {
data += stdout.toString();
});
process.stderr.on('data', stderr => {
error += stderr.toString();
});
process.on('error', err => {
reject(err);
})
process.on('close', code => {
if (code !== 0) {
reject(error)
} else {
resolve(data)
}
process.stdin.end();
});
});
Use execSync
const execWithPromise = command => childProcess.execSync(command).toString();
I know this is an old question but here is a useful tool I discovered with node a while back...So, say you have a node file app.ts, in typescript that is...
app.ts
import utils from 'util'; // The thing that is useful, it has a bunch of useful functions
import { exec } from 'child_process'; // The exec import
export const execute = utils.promisify(exec);
const run = async () => {
await execute('/usr/local/bin/docker exec -i -t cucumber node long-running-script.js');
await execute('/usr/local/bin/docker exec -i -t cucumber node long-running-script.js');
};
run();
In js though it would probably something like this
app.js
const utils = require('util');
const exec = require('child_process').exec;
const execute = utils.promisify(exec);
const run = async () => {
await execute('/usr/local/bin/docker exec -i -t cucumber node long-running-script.js');
await execute('/usr/local/bin/docker exec -i -t cucumber node long-running-script.js');
};
run();
Related
I have a Node JS CLI script that automates some migrations to a third-party service. I've largely avoided asynchronous methods (e.g. prefixing with async) as 1) I don't fully understand it in this context and 2) it hasn't been necessary for the script thus far.
Where I'm having trouble, is that I'm looping through a set of files and attempting to call a method on each entry, but the method doesn't execute before the script exits.
Here's the primary method:
const migrateAll = (app, env, source) => {
const self = this;
fs.promises
.readFile(config, "utf8")
.then((contents) => {
self.config = JSON.parse(contents);
})
.then(() => {
const spaceId = self.config.applications[app].space_id;
fs.readdir(source, "utf8", (err, files) => {
if (err) throw err;
files.forEach((file) => {
console.log(chalk.yellow(`Migrating "${file}" to the "${env}" environment for "${app}"`))
migrate(file, env, app);
});
process.exit();
});
});
};
The call to migrate(file, env, app) doesn't appear to run at all. The contents of that function are:
const migrate = (space, env, migration) => {
exec(
`migrate ${migration} "${space}" ${env}`,
(error, stdout, stderr) => {
if (error) {
// A `switch` to handle errors.
}
process.exit();
}
if (stderr) {
console.log(`stderr: ${stderr}`);
process.exit();
}
success(stdout);
);
};
The rest of the script, in context, looks like this:
const parseFlags = () => {
process.argv.splice(2).forEach((arg) => {
let parts = arg.split("=");
switch (parts[0]) {
// parse flags into variables
}
});
if (all) {
migrateAll(app, env, source);
}
return { app, env, source };
};
const run = () => {
try {
intro();
checkSettings();
const { app, env, source } = parseFlags();
// continue on here if migrateAll doesn't get called
} catch (err) {
complain(err);
}
};
run();
So, with the right flags, we call migrateAll() which in turn calls the migrate function for each file we find that needs to migrate. Some things I've noticed/tried
The console.log inside of the forEach in migrateAll runs as expected
I've tried various combinations of await and async, .then(), promisify, etc, but it feels like I'm throwing things at the wall just to see what sticks to no avail.
A few things:
You're calling async functions (fs.promises.readFile, readdir and exec) from within synchronous contexts. So in your script execution you have basically this:
migrate() ---------------+
| |
execution complete |
| readFile()
exit |
parse()
|
readdir()
|
exec()
You're synchronous execution completes before you finish running the async stuff.
You seem to spawning off a bunch of child processes to run these modules, you should instead require and just run them in-process
exec is not a safe way to spawn a child process as you're passing the string directly into a shell. If I as a user can control any of those three arguments I can pop a remote shell in netcat.
Using shorthand migrate is not a safe way to call a child process as it resolves from the PATH environment variable. If I have access to the runtime environment I can make migrate point to whatever I want.
Don't call process.exit(). The exit code you pass lets the caller or operating system know whether something went wrong. On success call process.exit(0), on error use any integer that's greater than 0 and less than 256. You can assign a unique exit code to each error situation if you wish.
Try this
// migrate.js
const {spawn} = require('child_process');
module.exports = async (space,env,migration) => new Promise((resolve,reject)=>{
let stdout = '';
let stderr = '';
let args = [migration,space,env];
let cp = spawn('/absolute/path/to/migrate', args);
cp.on('error',reject);
cp.stdout.setEncoding('utf8').on('data',(d)=>stdout+=d);
cp.stderr.setEncoding('utf8').on('data',(d)=>stderr+=d);
cp.on('exit',(code,signal) => {
if(signal)
code = signal;
if(code != 0) {
console.error(stderr);
return reject(new Error(`migrate returned exit code ${code}`));
}
resolve(stdout);
});
}).then(success); // not sure what success does, but this is equivalent to what you had
// migrate-all.js
const fsp = require('fs').promises;
const migrate = require('./migrate');
module.exports = async (app,env,source) => {
let contents = await fsp.readFile(config,'utf8');
self.config = JSON.parse(contents);
const spaceId = self.config.applications[app].space_id;
let files = await fsp.readdir(source);
for(let i in files) { // avoid async calls in .forEach loops
let file = files[i];
console.log(chalk.yellow(`Migrating "${file}" to the "${env}" environment for "${app}"`))
await migrate(file, env, app);
}
}
// index.js
const migrateAll = require('./migrate-all');
const parseFlags = async () => {
process.argv.splice(2).forEach((arg) => {
let parts = arg.split("=");
switch (parts[0]) {
// parse flags into variables
}
});
if (all) {
await migrateAll(app, env, source);
}
return { app, env, source };
};
const run = async () => {
try {
intro();
checkSettings();
const { app, env, source } = await parseFlags();
// continue on here if migrateAll doesn't get called
} catch (err) {
complain(err);
}
};
run();
I am using node to execute a jar file that usually takes a CSV file as an input path.
I would like to try and circumvent writing the CSV file and pipe in the CSV as a string into the process if possible.
I have this working with execSync but I would prever to use exec wrapped with promisify
The problem is that exec does not have the input option like execSync so I can't pipe data into it. How do you get around this? Or is the best practice to wrap execSync in a Promise?
import {execSync} from 'child_process';
export const runJar = async (input: string, cwd: string) => {
const out = execSync(`java -jar model.jar`, {
cwd,
input,
})
return out.toString('utf-8');
};
Minimalistic example usage of a childs process stdio.
https://nodejs.org/dist/latest-v14.x/docs/api/child_process.html#child_process_child_process_exec_command_options_callback
const child_process = require("child_process");
const fs = require("fs");
// exec returns a child process instance
// https://nodejs.org/dist/latest-v14.x/docs/api/child_process.html#child_process_class_childprocess
const child = child_process.exec("cat");
// write to child process stdin
child.stdin.write("Hello World");
// to read/parse your csv file
//fs.createReadStream("./file.csv").pipe(child.stdin);
// listen on child process stdout
child.stdout.on("data", (chunk) => {
console.log(chunk);
child.kill();
});
To promisfy this, you can listen on the exit (status) on the child process and resolve or reject the promise based on the exit code:
child.on("close", (code) => {
if (code != 0) {
reject();
} else {
resolve();
}
});
Example given:
const readParseCSV = function (file = "./file.csv") {
return new Promise((resolve, reject) => {
const child = child_process.exec("java -jar model.jar");
fs.createReadStream(file).pipe(child.stdin);
let response = "";
// listen on child process stdout
child.stdout.on("data", (chunk) => {
response += chunk;
});
child.on("close", (code) => {
if (code != 0) {
reject();
} else {
resolve(response);
}
});
});
};
Im not sure if this works on windows the same way as on linux.
I am currently developing a program where you can create virtual machines in VirtualBox as a project. To do this I need to run 2 commands synchronously as one creates a VM and the other modifies that VM. Here is the code.
nodecmd.run(cmd);
var cmd1 = createCmd1.concat(createServ);
console.log(cmd1);
var cmd2 = cmd1.concat(' --natpf1 "ssh,tcp,,302');
console.log(cmd2);
var cmd3 = cmd2.concat(createServ);
console.log(cmd3);
var cmd4 = cmd3.concat(',,22"');
console.log(cmd4);
nodecmd.run(cmd4);
Thanks!
I have found that I can just use child processes:
const execSync = require('child_process').execSync;
code = execSync('node -v');
I think you should use promise to chain your executions steps. Promises are good for handling asynchronous chaining events.
node-cmd support promise using bluebird.
const promise = require('bluebird');
const nodecmd = require('node-cmd');
const getAsync = promise.promisify(nodecmd.get, {
multiArgs: true,
context: nodecmd
});
let cmd0 = 'node -v';
let cmd1 = 'pwd';
let cmd2 = 'mkdir -p xxx';
getAsync(cmd0)
.then(result => console.log(result))
.then(() => getAsync(cmd1))
.then(result => console.log(result))
.then(() => getAsync(cmd2))
.then(result => console.log(result))
.catch(err => {
console.log('cmd err', err)
})
Try this in action - https://jsitor.com/2ZZPZqtvb
Is there an easy way to mock the Node.js child_process spawn function?
I have code like the following, and would like to test it in a unit test, without having to rely on the actual tool calls:
var output;
var spawn = require('child_process').spawn;
var command = spawn('foo', ['get']);
command.stdout.on('data', function (data) {
output = data;
});
command.stdout.on('end', function () {
if (output) {
callback(null, true);
}
else {
callback(null, false);
}
});
Is there a (proven and maintained) library that allows me to mock the spawn call and lets me specify the output of the mocked call?
I don't want to rely on the tool or OS to keep the tests simple and isolated. I want to be able to run the tests without having to set up complex test fixtures, which could mean a lot of work (including changing system configuration).
Is there an easy way to do this?
you can use sinon.stubs sinon stubs guide
// i like the sandbox, or you can use sinon itself
let sandbox = sinon.sandbox.create();
let spawnEvent = new events.EventEmitter();
spawnEvent.stdout = new events.EventEmitter();
sandbox.stub(child_process, 'spawn').returns(spawnEvent);
// and emit your event
spawnEvent.stdout.emit('data', 'hello world');
console.log(output) // hello world
Came across this and nwinkler's answer put me on the path. Below is a Mocha, Sinon and Typescript example that wraps the spawn in a promise, resolving if the exit code is a zero, and rejecting otherwise, It gathers up STDOUT/STDERR output, and lets you pipe text in through STDIN. Testing for a failure would be just a matter of testing for the exception.
function spawnAsPromise(cmd: string, args: ReadonlyArray<string> | undefined, options: child_process.SpawnOptions | undefined, input: string | undefined) {
return new Promise((resolve, reject) => {
// You could separate STDOUT and STDERR if your heart so desires...
let output: string = '';
const child = child_process.spawn(cmd, args, options);
child.stdout.on('data', (data) => {
output += data;
});
child.stderr.on('data', (data) => {
output += data;
});
child.on('close', (code) => {
(code === 0) ? resolve(output) : reject(output);
});
child.on('error', (err) => {
reject(err.toString());
});
if(input) {
child.stdin.write(input);
child.stdin.end();
}
});
}
// ...
describe("SpawnService", () => {
it("should run successfully", async() => {
const sandbox = sinon.createSandbox();
try {
const CMD = 'foo';
const ARGS = ['--bar'];
const OPTS = { cwd: '/var/fubar' };
const STDIN_TEXT = 'I typed this!';
const STDERR_TEXT = 'Some diag stuff...';
const STDOUT_TEXT = 'Some output stuff...';
const proc = <child_process.ChildProcess> new events.EventEmitter();
proc.stdin = new stream.Writable();
proc.stdout = <stream.Readable> new events.EventEmitter();
proc.stderr = <stream.Readable> new events.EventEmitter();
// Stub out child process, returning our fake child process
sandbox.stub(child_process, 'spawn')
.returns(proc)
.calledOnceWith(CMD, ARGS, OPTS);
// Stub our expectations with any text we are inputing,
// you can remove these two lines if not piping in data
sandbox.stub(proc.stdin, "write").calledOnceWith(STDIN_TEXT);
sandbox.stub(proc.stdin, "end").calledOnce = true;
// Launch your process here
const p = spawnAsPromise(CMD, ARGS, OPTS, STDIN_TEXT);
// Simulate your program's output
proc.stderr.emit('data', STDERR_TEXT);
proc.stdout.emit('data', STDOUT_TEXT);
// Exit your program, 0 = success, !0 = failure
proc.emit('close', 0);
// The close should get rid of the process
const results = await p;
assert.equal(results, STDERR_TEXT + STDOUT_TEXT);
} finally {
sandbox.restore();
}
});
});
I've found the mock-spawn library, which pretty much does what I want. It allows to mock the spawn call and provide expected results back to the calling test.
An example:
var mockSpawn = require('mock-spawn');
var mySpawn = mockSpawn();
require('child_process').spawn = mySpawn;
mySpawn.setDefault(mySpawn.simple(1 /* exit code */, 'hello world' /* stdout */));
More advanced examples can be found on the project page.
For anyone who still has problems with this particular problem and for some reason, the recommendations in other answers don't help, I was able to get it to work with proxyrequire (https://github.com/thlorenz/proxyquire) by replacing the real child_process spawn with an event emitter that I then used in my tests to mock the emission.
var stdout = new events.EventEmitter();
var stderr = new events.EventEmitter();
var spawn = new events.EventEmitter();
spawn.stderr = stderr;
spawn.stdout = stdout;
var child_process = {
spawn: () => spawn,
stdout,
stderr
};
// proxyrequire replaces the child_process require in the file pathToModule
var moduleToTest = proxyquire("./pathToModule/", {
'child_process': child_process
});
describe('Actual test', function () {
var response;
before(function (done) {
// your regular method call
moduleToTest.methodToTest()
.then(data => {
response = data;
done();
}).catch(err => {
response = err;
done();
});
// emit your expected response
child_process.stdout.emit("data", "the success message sent");
// you could easily use the below to test an error
// child_process.stderr.emit("data", "the error sent");
});
it('test your expectation', function () {
expect(response).to.equal("the success message or whatever your moduleToTest
resolves with");
});
});
Hope this helps...
In a node.js, I'd like to find a way to obtain the output of a Unix terminal command. Is there any way to do this?
function getCommandOutput(commandString){
// now how can I implement this function?
// getCommandOutput("ls") should print the terminal output of the shell command "ls"
}
This is the method I'm using in a project I am currently working on.
var exec = require('child_process').exec;
function execute(command, callback){
exec(command, function(error, stdout, stderr){ callback(stdout); });
};
Example of retrieving a git user:
module.exports.getGitUser = function(callback){
execute("git config --global user.name", function(name){
execute("git config --global user.email", function(email){
callback({ name: name.replace("\n", ""), email: email.replace("\n", "") });
});
});
};
If you're using node later than 7.6 and you don't like the callback style, you can also use node-util's promisify function with async / await to get shell commands which read cleanly. Here's an example of the accepted answer, using this technique:
const { promisify } = require('util');
const exec = promisify(require('child_process').exec)
module.exports.getGitUser = async function getGitUser () {
// Exec output contains both stderr and stdout outputs
const nameOutput = await exec('git config --global user.name')
const emailOutput = await exec('git config --global user.email')
return {
name: nameOutput.stdout.trim(),
email: emailOutput.stdout.trim()
}
};
This also has the added benefit of returning a rejected promise on failed commands, which can be handled with try / catch inside the async code.
You're looking for child_process
var exec = require('child_process').exec;
var child;
child = exec(command,
function (error, stdout, stderr) {
console.log('stdout: ' + stdout);
console.log('stderr: ' + stderr);
if (error !== null) {
console.log('exec error: ' + error);
}
});
As pointed out by Renato, there are some synchronous exec packages out there now too, see sync-exec that might be more what yo're looking for. Keep in mind though, node.js is designed to be a single threaded high performance network server, so if that's what you're looking to use it for, stay away from sync-exec kinda stuff unless you're only using it during startup or something.
Requirements
This will require Node.js 7 or later with a support for Promises and Async/Await.
Solution
Create a wrapper function that leverage promises to control the behavior of the child_process.exec command.
Explanation
Using promises and an asynchronous function, you can mimic the behavior of a shell returning the output, without falling into a callback hell and with a pretty neat API. Using the await keyword, you can create a script that reads easily, while still be able to get the work of child_process.exec done.
Code sample
const childProcess = require("child_process");
/**
* #param {string} command A shell command to execute
* #return {Promise<string>} A promise that resolve to the output of the shell command, or an error
* #example const output = await execute("ls -alh");
*/
function execute(command) {
/**
* #param {Function} resolve A function that resolves the promise
* #param {Function} reject A function that fails the promise
* #see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
*/
return new Promise(function(resolve, reject) {
/**
* #param {Error} error An error triggered during the execution of the childProcess.exec command
* #param {string|Buffer} standardOutput The result of the shell command execution
* #param {string|Buffer} standardError The error resulting of the shell command execution
* #see https://nodejs.org/api/child_process.html#child_process_child_process_exec_command_options_callback
*/
childProcess.exec(command, function(error, standardOutput, standardError) {
if (error) {
reject();
return;
}
if (standardError) {
reject(standardError);
return;
}
resolve(standardOutput);
});
});
}
Usage
async function main() {
try {
const passwdContent = await execute("cat /etc/passwd");
console.log(passwdContent);
} catch (error) {
console.error(error.toString());
}
try {
const shadowContent = await execute("cat /etc/shadow");
console.log(shadowContent);
} catch (error) {
console.error(error.toString());
}
}
main();
Sample Output
root:x:0:0::/root:/bin/bash
[output trimmed, bottom line it succeeded]
Error: Command failed: cat /etc/shadow
cat: /etc/shadow: Permission denied
Try it online.
Repl.it.
External resources
Promises.
child_process.exec.
Node.js support table.
Thanks to Renato answer, I have created a really basic example:
const exec = require('child_process').exec
exec('git config --global user.name', (err, stdout, stderr) => console.log(stdout))
It will just print your global git username :)
You can use the util library that comes with nodejs to get a promise from the exec command and can use that output as you need. Use destructuring to store the stdout and stderr in variables.
const util = require('util');
const exec = util.promisify(require('child_process').exec);
async function lsExample() {
const {
stdout,
stderr
} = await exec('ls');
console.log('stdout:', stdout);
console.error('stderr:', stderr);
}
lsExample();
you can use ShellJS package.
ShellJS is a portable (Windows/Linux/OS X) implementation of Unix shell commands on top of the Node.js API.
see: https://www.npmjs.com/package/shelljs#execcommand--options--callback
import * as shell from "shelljs";
//usage:
//exec(command [, options] [, callback])
//example:
const version = shell.exec("node --version", {async: false}).stdout;
console.log("nodejs version", version);
Here's an async await TypeScript implementation of the accepted answer:
const execute = async (command: string): Promise<any> => {
return new Promise((resolve, reject) => {
const exec = require("child_process").exec;
exec(
command,
function (
error: Error,
stdout: string | Buffer,
stderr: string | Buffer
) {
if (error) {
reject(error);
return;
}
if (stderr) {
reject(stderr);
return;
} else {
resolve(stdout);
}
}
);
});
};