I am trying to run a command inside a async.forEach loop using ChildProcess.exec in my node job. here is the code
async.forEach( docPaths, function(docPath, callback) {
var run = [];
// some command using docPath variable here..
run.push(command);
debugger;
exec(run.join(' '), function(error, stdout, stderr){
callback();
});
}, callback);
Here is the error
"stack":"Error: spawn EMFILE\
at errnoException (child_process.js:478:11)\
at ChildProcess.spawn (child_process.js:445:11)\
at child_process.js:343:9\
at Object.execFile (child_process.js:253:15)\
at child_process.js:220:18\
a quick google shows i need to set ulimit value to increase the number of file descriptors can be open. some thing like "ulimit -n 10000".. (from link below)
https://groups.google.com/forum/#!topic/nodejs/jeec5pAqhps
where can i increase this..? or is there any other solution to circumvent the issue?
Appreciate your help.. Thanks much !!
First of all its not advisable to mess with ulimit, as it may have system wide impacts.
Instead since you are already using async, it comes with a limit paramater which you can use to limit the number of parallely executions.
async.eachLimit( docPaths, 100, function(docPath, callback) {
var run = [];
// some command using docPath variable here..
run.push(command);
debugger;
exec(run.join(' '), function(error, stdout, stderr){
callback();
});
}, callback);
Please do trial and error and replace 100 with suitable value.
Related
I'm running Windows 10, and I have a program, let's call it program, that can be run from the command line. When run, it responds to commands the user enters. The user enters a command, presses the return key, and the program prints a response. I did not make this program and do not have the source, so I cannot modify it.
I want to run this program from within Node.js, and have my Node.js program act as the user, sending it commands and getting the responses. I spawn my program like this:
var spawn = require('child_process').spawn;
var child = spawn('program');
child.stdout.on('data', function(data) {
console.log(`stdout: ${data}`);
});
Then I attempt to send it a command, for example, help.
child.stdin.write("help\n");
And nothing happens. If I manually run the program, type help, and press the return key, I get output. I want Node.js to run the program, send it input, and receive the output exactly as a human user would. I assumed that stdin.write() would send the program a command as if the user typed it in the console. However, as the program does not respond, I assume this is not the case. How can I send the program input?
I've seen many similar questions, but unfortunately the solutions their authors report as "working" did not work for me.
Sending input data to child process in node.js
I've seen this question and answer and tried everything in it with no success. I've tried ending the command with \r\n instead of \n. I've also tried adding the line child.stdin.end() after writing. Neither of these worked.
How to pass STDIN to node.js child process
This person, in their self-answer, says that they got theirs to work almost exactly as I'm doing it, but mine does not work.
Nodejs Child Process: write to stdin from an already initialised process
This person, in their self-answer, says they got it to work by writing their input to a file and then piping that file to stdin. This sounds overly complicated to send a simple string.
This worked for me, when running from Win10 CMD or Git Bash:
console.log('Running child process...');
const spawn = require('child_process').spawn;
const child = spawn('node');
// Also worked, from Git Bash:
//const child = spawn('cat');
child.stdout.on('data', (data) => {
console.log(`stdout: "${data}"`);
});
child.stdin.write("console.log('Hello!');\n");
child.stdin.end(); // EOF
child.on('close', (code) => {
console.log(`Child process exited with code ${code}.`);
});
Result:
D:\Martin\dev\node>node test11.js
Running child process...
stdout: "Hello!
"
Child process exited with code 0.
I also tried running aws configure like this, first it didn't work because I sent only a single line. But when sending four lines for the expected four input values, it worked.
Maybe your program expects special properties for stdin, like being a real terminal, and therefore doesn't take your input?
Or did you forget to send the EOF using child.stdin.end();? (If you remove that call from my example, the child waits for input forever.)
Here is what worked for me. I have used child_process exec to create a child process. Inside this child process Promise, I am handling the i/o part of the cmd given as parameter. Its' not perfect, but its working.
Sample function call where you dont need any human input.
executeCLI("cat ~/index.html");
Sample function call where you interact with aws cli. Here
executeCLI("aws configure --profile dev")
Code for custom executeCLI function.
var { exec } = require('child_process');
async function executeCLI(cmd) {
console.log("About to execute this: ", cmd);
var child = exec(cmd);
return new Promise((resolve, reject) => {
child.stdout.on('data', (data) => {
console.log(`${data}`);
process.stdin.pipe(child.stdin);
});
child.on('close', function (err, data) {
if (err) {
console.log("Error executing cmd: ", err);
reject(err);
} else {
// console.log("data:", data)
resolve(data);
}
});
});
}
Extract the user input code from browser and save that code into a file on your system using fs module. Let that file be 'program.cpp' and save the user data input in a text file.
As we can compile our c++ code on our terminal using g++ similarly we will be using child_process to access our system terminal and run user's code.
execFile can be used for executing our program
var { execFile } = require('child_process');
execFile("g++", ['program.cpp'], (err, stdout, stderr) => {
if (err) {
console.log("compilation error: ",err);
} else{
execFile ('./a.out' ,['<', 'input.txt'], {shell: true}, (err, stdout, stderr) => {
console.log("output: ", stdout);
})
}
})
In this code we simply require the child_process and uses its execFile function.
First we compile the code present in program.cpp, which creates a default a.out as output file
Then we pass the a.out file with input that is present in input.txt
Hence you can view the generated output in your terminal and pass that back to the user.
for more details you can check: Child Processes
I am having a problem with async shell executes in node.js.
In my case, node.js is installed on a Linux operating system on a raspberry pi. I want to fill an array with values that are parsed from a shell script which is called on the pi. This works fine, however, the exec() function is called asynchronously.
I need the function to be absolute synchron to avoid messing up my whole system. Is there any way to achieve this? Currently I am trying a lib called .exe, but the code still seems to behave asynchron.
Here's my code:
function execute(cmd, cb)
{
child = exec(cmd, function(error, stdout, stderr)
{
cb(stdout, stderr);
});
}
function chooseGroup()
{
var groups = [];
execute("bash /home/pi/scripts/group_show.sh", function(stdout, stderr)
{
groups_str = stdout;
groups = groups_str.split("\n");
});
return groups;
}
//Test
console.log(chooseGroup());
If what you're using is child_process.exec, it is asynchronous already.
Your chooseGroup() function will not work properly because it is asynchronous. The groups variable will always be empty.
Your chooseGroup() function can work if you change it like this:
function chooseGroup() {
execute("bash /home/pi/scripts/group_show.sh", function(stdout, stderr) {
var groups = stdout.split("\n");
// put the code here that uses groups
console.log(groups);
});
}
// you cannot use groups here because the result is obtained asynchronously
// and thus is not yet available here.
If, for some reason, you're looking for a synchronous version of .exec(), there is child_process.execSync() though it is rarely recommended in server-based code because it is blocking and thus blocks execution of other things in node.js while it is running.
Hi can anyone help me with a custom grunt task?
I'm basically using child_process to call
var cp = exec(cmd, {}, function (err, stdout, stderr) {}
this of course has the stdout etc. If I
cp.stdout.pipe(process.stdout);
it rights a bunch of stuff to the console. I am making this call in a loop so here's a skeleton
this.files.forEach(function(f) {
var src = f.src.map(function(filepath) {
var cmd = util.format("%s %s", options.xUnit, src);
var cp = exec(cmd, {}, function (err, stdout, stderr) {
....
cb();
});
if (options.stdout || grunt.option('verbose')) {
cp.stdout.pipe(process.stdout);
}
This all works fine. but since I'm doing it several times I want at the very least to be able to index how many times stderr has a value. I'm basically running collections of tests. If all pass then stderr is blank if not, I don't want to stop the rest of the tests I want them all to run, but i'd like to output at the end "you have 3 collections that have failing tests". I don't see why I can't seem to get it to work. If I declare
var output = 0;
then
if (options.stderr || grunt.option('verbose')) {
cp.stdout.pipe(process.stdout);
output ++;
}
I then try to print to console after they all run but it's always 0.
I would really like to some how get a copy of ALL the output and parse out the last line from each test session which has the number of tests that failed in it. But that just seems way out of realm of possiblity. I've tried all manner of stuff with the streams and got less than nothing.
Anyway, if anyone can help I would greatly appreciate it
I keep banging my head against the wall because of tons of different errors. This is what the code i try to use :
fs.readFile("balance.txt", function (err, data) //At the beginning of the script (checked, it works)
{
if (err) throw err;
balance=JSON.parse(data);;
});
fs.readFile("pick.txt", function (err, data)
{
if (err) throw err;
pick=JSON.parse(data);;
});
/*....
.... balance and pick are modified
....*/
if (shutdown)
{
fs.writeFile("balance2.txt", JSON.stringify(balance));
fs.writeFile("pick2.txt", JSON.stringify(pick));
process.exit(0);
}
At the end of the script, the files have not been modified the slightest. I then found out on this site that the files were being opened 2 times simultaneously, or something like that, so i tried this :
var balance, pick;
var stream = fs.createReadStream("balance.txt");
stream.on("readable", function()
{
balance = JSON.parse(stream.read());
});
var stream2 = fs.createReadStream("pick.txt");
stream2.on("readable", function()
{
pick = JSON.parse(stream2.read());
});
/****
****/
fs.unlink("pick.txt");
fs.unlink("balance.txt");
var stream = fs.createWriteStream("balance.txt", {flags: 'w'});
var stream2 = fs.createWriteStream("pick.txt", {flags: 'w'});
stream.write(JSON.stringify(balance));
stream2.write(JSON.stringify(pick));
process.exit(0);
But, this time, both files are empty... I know i should catch errors, but i just don't see where the problem is. I don't mind storing the 2 objects in the same file, if that can helps. Besides that, I never did any javascript in my life before yesterday, so, please give me a simple explanation if you know what failed here.
What I think you want to do is use readFileSync and not use readFile to read your files since you need them to be read before doing anything else in your program (http://nodejs.org/api/fs.html#fs_fs_readfilesync_filename_options).
This will make sure you have read both the files before you execute any of the rest of your code.
Make your like code do this:
try
{
balance = JSON.parse(fs.readFileSync("balance.txt"));
pick = JSON.parse(fs.readFileSync("pick.txt"));
}
catch(err)
{ throw err; }
I think you will get the functionality you are looking for by doing this.
Note, you will not be able to check for an error in the same way you can with readFile. Instead you will need to wrap each call in a try catch or use existsSync before each operation to make sure you aren't trying to read a file that doesn't exist.
How to capture no file for fs.readFileSync()?
Furthermore, you have the same problem on the writes. You are kicking off async writes and then immediately calling process.exit(0). A better way to do this would be to either write them sequentially asynchronously and then exit or to write them sequentially synchronously then exit.
Async option:
if (shutdown)
{
fs.writeFile("balance2.txt", JSON.stringify(balance), function(err){
fs.writeFile("pick2.txt", JSON.stringify(pick), function(err){
process.exit(0);
});
});
}
Sync option:
if (shutdown)
{
fs.writeFileSync("balance2.txt", JSON.stringify(balance));
fs.writeFileSync("pick2.txt", JSON.stringify(pick));
process.exit(0);
}
I am using the Node FFI module and am trying to run sync tasks on Windows. I can successfully run a task using the following code.
var ffi=require('ffi')
var nativeC = new ffi.Library("Kernel32", {
"WinExec": ["int32", ["string"]]
});
nativeC.WinExec('ls -lrt');
I presume this is the way to execute sync tasks, but this code always exits after the 1st 'ls -lrt' command; if I chain a few more commands, they won't work. So, is there a callback function over here, in the FFI module, or another way I can chain commands in node.js on Windows so they run in sync, one after the other.
I'm not sure you need WinExec to run a windows command. As Jonathan pointed out, ls isn't available.
However, if you want to chain commands you could use async.js and exec like this:
var
async = require('async');
exec = require('child_process').exec,
commands = [ 'dir /w', 'echo test'];
var executeCommand = function(command, callback){
exec(command, function (err, stdout, stderr) {
if(err) return callback(err);
console.log(stdout);
callback();
});
};
async.eachSeries(commands, executeCommand, function(err){
console.log('error: ' + err);
});