Npm child process live feed - node.js

I am writing a CLI tool for a node.js app. Some of the commands have to run npm and show the results. This is what I have so far:
import {spawn} from 'child_process';
let projectRoot = '...';
let npm = (process.platform === "win32" ? "npm.cmd" : "npm"),
childProcess = spawn(npm, ["install"], { cwd: projectRoot });
childProcess.stdout.pipe(process.stdout);
childProcess.stderr.pipe(process.stderr);
childProcess.on('close', (code) => {
// Continue the remaining operations
});
The command does run fine and outputs the results (or errors). However, it doesn't give me a live feed with the progress bar, etc. It waits until the entire operation is over and then dumps the output into the console.
I've tried different variations of the spawn configuration but I can't get it to show me the live feed.
I am on Windows 10 and use node.js 4 and npm 3.

As discussed in the comments: Run the spawn with { stdio: 'inherit' }.
However, good question is why the 'manual piping' does not do the same. I think that's because npm uses the 'fancy progress bar'. It probably uses some special way how to deal with stdout that does not play well with process.stdout. If you try some other long-running command (such as 'find ./'), your way of piping works fine.

Related

Installing NPM Modules via the Frontend

I am working on an app wherein I would like to be able to install NPM modules via the frontend. I have no idea, though, how to do so. That is, I know how to do CRUD actions via the front end, but I don't know how to either interact with the command line or run command line functions via the front end.
Are there packages that can help with this or is this built into Node.js somehow?
In short, how can I connect my front-end to my backend in such a way that I can install an NPM package?
What you want is the child_process module. It's built-in so you don't need to install any additional module.
Mostly what you're looking for is either spawn() or exec().
For example, if you want to run npm install some_module you can do:
const { exec } = require('child_process');
let command = 'npm install some_module';
let options = { cwd: '/path/to/node/project' };
exec(command, options, (error, stdout, stderr) => {
// Do anything you want with program output here:
console.log('output:', stdout, stderr);
});
You may check the documentation for child_process in Node JS:
Child Process
const { spawn } = require('child_process');
const ls = spawn('ls', ['-lh', '/usr']);
ls.stdout.on('data', (data) => {
console.log(`stdout: ${data}`);
});
ls.on('close', (code) => {
console.log(`child process exited with code ${code}`);
});
The key difference between exec() and spawn() is how they return the data. As exec() stores all the output in a buffer, it is more memory intensive than spawn(), which streams the output as it comes.
Generally, if you are not expecting large amounts of data to be returned, you can use exec() for simplicity. Good examples of use-cases are creating a folder or getting the status of a file. However, if you are expecting a large amount of output from your command, then you should use spawn()
The easiest way to install an npm package via "the front end" is to have node spawn npm as a child process based off of the package name that the client provides.
var child = require('child_process').exec(`npm i ${package_name}`);
child.on('exit',function(){
//npm finished
});
This should install the module given that package_name is the name of the npm package, in the same directory that the script is running in. In terms of getting the package name from the front end to the back end, there are several different ways to do that.
You cannot run the your backend application without NPM modules installed, one thing that I think you can do is to make a plain nodejs file without any modules, which will receive the args when invoked, and you can use that args to the required modules, because that file will run with just core modules

NodeJS child process stdout not returning completly

I'm running the following code using node and I always get only about 80% of what stdout is supposed to return when the file is past 12ko or so. If the file is 240ko it will still output 80% of it. If it's under 12 it will output completely.
When I open a cmd and run the command manually I always get the full output.
I tried exec, execFile, I tried increasing the max buffer or changing the encoding and it's not the issue. I tried to add options {shell: true, detached: true} but it vain, in fact, when I run it as detached it seems to be running completely fine as it does open an actual console but I'm not able to retrieve the stdout when the process is completed.
const spawn = require('child_process').spawn;
const process = spawn(
'C:\\Users\\jeanphilipped\\Desktop\\unrtf\\test\\unrtf.exe',
['--html' ,'C:\\Users\\jeanphilipped\\Desktop\\unrtf\\test\\tempaf45.rtf'],
);
let chunks = [];
process.stdout.on('data', function (msg) {
chunks = [...chunks, ...msg];
});
process.on('exit', function (msg) {
const buffer = Buffer.from(chunks);
console.log(buffer.toString());
});
Any clues ? It seems to be Node since when I run it manually everything works fine.
according to nodejs documentation, all of the child_process commands are asynchronous . so when you try to access your chunk variable there is no guarantee that you command has been finished , maybe your command is still on process. so it is recommended you should wrap your entire child_process command in async/await function.

How to create and manage a npm process in electron

I'm trying to make this little QoL app that can trigger my scripts inside package.json of any project I throw at it (react, polymer, etc.).
So far, it works as it should. At least until I kill the app or want to terminate the specific process (like using ctrl+c while running a npm start in a console).
I'm currently calling the command this way:
const exec = require("child_process").execFile;
...
let ps = exec("command.bat", [path, command], (error, stdout, stderr)=>{}) //command.bat contains only: cd "%1" \n npm "%2"
Previously I've used node-powershell like so:
const Shell = require('node-powershell');
...
let sh = new Shell();
sh.addCommand(`cd ${fs.realpathSync(path)}`);
sh.addCommand(`npm ${command}`);
sh.invoke();
And I've already tried using ps-tree in hopes that this will list all the processes started by my process so I can kill them but no luck and because it creates additional process or two it's getting out of hands really quickly.
const cp = require('child_process');
...
psTree(ps.pid, function (err, children) {
cp.spawn('kill', ['-9'].concat(children.map(function (p) { return p.PID })));
});
So if there is some solution I would be really grateful. I'm also open to any different solution if there is any.
Thanks in advance.

How do I debug "Error: spawn ENOENT" on node.js?

When I get the following error:
events.js:72
throw er; // Unhandled 'error' event
^
Error: spawn ENOENT
at errnoException (child_process.js:1000:11)
at Process.ChildProcess._handle.onexit (child_process.js:791:34)
What procedure can I follow to fix it?
Author note: Lots of issues with this error encouraged me to post this question for future references.
Related questions:
using spawn function with NODE_ENV=production
node.js child_process.spawn ENOENT error - only under supervisord
spawn ENOENT node.js error
https://stackoverflow.com/questions/27603713/nodejs-spawn-enoent-error-on-travis-calling-global-npm-package
Node JS - child_process spawn('npm install') in Grunt task results in ENOENT error
Running "foreman" task Fatal error: spawn ENOENT
unhandled error event in node js Error: spawn ENOENT at errnoException (child_process.js:975:11)
Node.js SpookyJS: error executing hello.js
https://stackoverflow.com/questions/26572214/run-grunt-on-a-directory-nodewebkit
Run exe file with Child Process NodeJS
Node: child_process.spawn not working on Java even though it's in the path (ENOENT)
spawn ENOENT error with NodeJS (PYTHON related)
image resizing is not working in node.js (partial.js) (non-installed dependency)
npm install error ENOENT (build dependency problem)
Cannot install node.js - oracle module on Windows 7 (build dependency problem)
Error installing gulp using nodejs on windows (strange case)
NOTE: This error is almost always caused because the command does not exist, because the working directory does not exist, or from a windows-only bug.
I found a particular easy way to get the idea of the root cause of:
Error: spawn ENOENT
The problem of this error is, there is really little information in the error message to tell you where the call site is, i.e. which executable/command is not found, especially when you have a large code base where there are a lot of spawn calls. On the other hand, if we know the exact command that cause the error then we can follow #laconbass' answer to fix the problem.
I found a very easy way to spot which command cause the problem rather than adding event listeners everywhere in your code as suggested in #laconbass' answer. The key idea is to wrap the original spawn call with a wrapper which prints the arguments send to the spawn call.
Here is the wrapper function, put it at the top of the index.js or whatever your server's starting script.
(function() {
var childProcess = require("child_process");
var oldSpawn = childProcess.spawn;
function mySpawn() {
console.log('spawn called');
console.log(arguments);
var result = oldSpawn.apply(this, arguments);
return result;
}
childProcess.spawn = mySpawn;
})();
Then the next time you run your application, before the uncaught exception's message you will see something like that:
spawn called
{ '0': 'hg',
'1': [],
'2':
{ cwd: '/* omitted */',
env: { IP: '0.0.0.0' },
args: [] } }
In this way you can easily know which command actually is executed and then you can find out why nodejs cannot find the executable to fix the problem.
Step 1: Ensure spawn is called the right way
First, review the docs for child_process.spawn( command, args, options ):
Launches a new process with the given command, with command line arguments in args. If omitted, args defaults to an empty Array.
The third argument is used to specify additional options, which defaults to:
{ cwd: undefined, env: process.env }
Use env to specify environment variables that will be visible to the new process, the default is process.env.
Ensure you are not putting any command line arguments in command and the whole spawn call is valid. Proceed to next step.
Step 2: Identify the Event Emitter that emits the error event
Search on your source code for each call to spawn, or child_process.spawn, i.e.
spawn('some-command', [ '--help' ]);
and attach there an event listener for the 'error' event, so you get noticed the exact Event Emitter that is throwing it as 'Unhandled'. After debugging, that handler can be removed.
spawn('some-command', [ '--help' ])
.on('error', function( err ){ throw err })
;
Execute and you should get the file path and line number where your 'error' listener was registered. Something like:
/file/that/registers/the/error/listener.js:29
throw err;
^
Error: spawn ENOENT
at errnoException (child_process.js:1000:11)
at Process.ChildProcess._handle.onexit (child_process.js:791:34)
If the first two lines are still
events.js:72
throw er; // Unhandled 'error' event
do this step again until they are not. You must identify the listener that emits the error before going on next step.
Step 3: Ensure the environment variable $PATH is set
There are two possible scenarios:
You rely on the default spawn behaviour, so child process environment will be the same as process.env.
You are explicity passing an env object to spawn on the options argument.
In both scenarios, you must inspect the PATH key on the environment object that the spawned child process will use.
Example for scenario 1
// inspect the PATH key on process.env
console.log( process.env.PATH );
spawn('some-command', ['--help']);
Example for scenario 2
var env = getEnvKeyValuePairsSomeHow();
// inspect the PATH key on the env object
console.log( env.PATH );
spawn('some-command', ['--help'], { env: env });
The absence of PATH (i.e., it's undefined) will cause spawn to emit the ENOENT error, as it will not be possible to locate any command unless it's an absolute path to the executable file.
When PATH is correctly set, proceed to next step. It should be a directory, or a list of directories. Last case is the usual.
Step 4: Ensure command exists on a directory of those defined in PATH
Spawn may emit the ENOENT error if the filename command (i.e, 'some-command') does not exist in at least one of the directories defined on PATH.
Locate the exact place of command. On most linux distributions, this can be done from a terminal with the which command. It will tell you the absolute path to the executable file (like above), or tell if it's not found.
Example usage of which and its output when a command is found
> which some-command
some-command is /usr/bin/some-command
Example usage of which and its output when a command is not found
> which some-command
bash: type: some-command: not found
miss-installed programs are the most common cause for a not found command. Refer to each command documentation if needed and install it.
When command is a simple script file ensure it's accessible from a directory on the PATH. If it's not, either move it to one or make a link to it.
Once you determine PATH is correctly set and command is accessible from it, you should be able to spawn your child process without spawn ENOENT being thrown.
simply adding shell: true option solved my problem:
incorrect:
const { spawn } = require('child_process');
const child = spawn('dir');
correct:
const { spawn } = require('child_process');
const child = spawn('dir', [], {shell: true});
As #DanielImfeld pointed it, ENOENT will be thrown if you specify "cwd" in the options, but the given directory does not exist.
Windows solution: Replace spawn with node-cross-spawn. For instance like this at the beginning of your app.js:
(function() {
var childProcess = require("child_process");
childProcess.spawn = require('cross-spawn');
})();
For ENOENT on Windows, https://github.com/nodejs/node-v0.x-archive/issues/2318#issuecomment-249355505 fix it.
e.g. replace spawn('npm', ['-v'], {stdio: 'inherit'}) with:
for all node.js version:
spawn(/^win/.test(process.platform) ? 'npm.cmd' : 'npm', ['-v'], {stdio: 'inherit'})
for node.js 5.x and later:
spawn('npm', ['-v'], {stdio: 'inherit', shell: true})
#laconbass's answer helped me and is probably most correct.
I came here because I was using spawn incorrectly.
As a simple example:
this is incorrect:
const s = cp.spawn('npm install -D suman', [], {
cwd: root
});
this is incorrect:
const s = cp.spawn('npm', ['install -D suman'], {
cwd: root
});
this is correct:
const s = cp.spawn('npm', ['install','-D','suman'], {
cwd: root
});
however, I recommend doing it this way:
const s = cp.spawn('bash');
s.stdin.end(`cd "${root}" && npm install -D suman`);
s.once('exit', code => {
// exit
});
this is because then the cp.on('exit', fn) event will always fire, as long as bash is installed, otherwise, the cp.on('error', fn) event might fire first, if we use it the first way, if we launch 'npm' directly.
How to research the spawn call raising the error:
Use NODE_DEBUG=child_process, Credits to #karl-richter. Simple, quick, October 2019
Use a wrapper to decorate child_process.spawn, Credits to #jiaji-zhou. Simple, quick, January 2015
Long procedure, credits to #laconbass. Complex, time-cost, December 2014
Known, usual causes
Environment issues
The command executable does not exist within the system (dependency not being installed). see prominc's answer
The command executable does not exist within a directory of those specified by PATH environment variable.
The executable binary was compiled with uncompatible libraries. see danilo-ramirez answer
Windows-only bugs/quirks
'.cmd' extension / shell: true. see li-zheng answer
Administrator permisions. see steve's answer
Wrong spawn('command', ['--argument', 'list'], { cwd, env, ...opts }) usage
Specified working directory (opts.cwd) does not exist ยท see leeroy-brun's answer
Argument list within command String spawn('command --wrong --argument list')
Env vars within command string spawn('ENV_VAR=WRONG command')
Argument list Array specified as String spawn('cmd', '--argument list')
Unset PATH env variable spawn('cmd', [], { env: { variable } } => spawn('cmd', [], { env: { ...process.env, variable } }
There are 2 posible origins for ENOENT:
Code you are writing
Code you depend on
When origin is code you depend on, usual cause is an Environment Issue (or windows quirk)
For anyone who might stumble upon this, if all the other answers do not help and you are on Windows, know that there is currently a big issue with spawn on Windows and the PATHEXT environment variable that can cause certain calls to spawn to not work depending on how the target command is installed.
In my case, I was getting this error thrown due to the necessary dependent system resources not being installed.
More specifically, I have a NodeJS app that is utilizing ImageMagick. Despite having the npm package installed, the core Linux ImageMagick was not installed. I did an apt-get to install ImageMagick and after that all worked great!
Before anyone spends to much time debugging this problem, most of the time it can be resolved by deleting node_modules and reinstalling the packages.
To Install:
If a lockfile exists you might use
yarn install --frozen-lockfile
or
npm ci
respectivly. if not then
yarn install
or
npm i
Are you changing the env option?
Then look at this answer.
I was trying to spawn a node process and TIL that you should spread the existing environment variables when you spawn else you'll loose the PATH environment variable and possibly other important ones.
This was the fix for me:
const nodeProcess = spawn('node', ['--help'], {
env: {
// by default, spawn uses `process.env` for the value of `env`
// you can _add_ to this behavior, by spreading `process.env`
...process.env,
OTHER_ENV_VARIABLE: 'test',
}
});
In case you're experiencing this issue with an application whose source you cannot modify consider invoking it with the environment variable NODE_DEBUG set to child_process, e.g. NODE_DEBUG=child_process yarn test. This will provide you with information which command lines have been invoked in which directory and usually the last detail is the reason for the failure.
I ran into the same problem, but I found a simple way to fix it.
It appears to be spawn() errors if the program has been added to the PATH by the user (e.g. normal system commands work).
To fix this, you can use the which module (npm install --save which):
// Require which and child_process
const which = require('which');
const spawn = require('child_process').spawn;
// Find npm in PATH
const npm = which.sync('npm');
// Execute
const noErrorSpawn = spawn(npm, ['install']);
Use require('child_process').exec instead of spawn for a more specific error message!
for example:
var exec = require('child_process').exec;
var commandStr = 'java -jar something.jar';
exec(commandStr, function(error, stdout, stderr) {
if(error || stderr) console.log(error || stderr);
else console.log(stdout);
});
A case that I found that is not in this list but it's worthy to be added:
On Alpine Linux, Node will error with ENOENT if the executable is not compatible.
Alpine expects binaries with libc. An executable (e.g. chrome as part of chromium) that has been compiled with glibc as a wrapper for system calls, will fail with ENOENT when called by spawn.
Ensure module to be executed is installed or full path to command if it's not a node module
I was also going through this annoying problem while running my test cases, so I tried many ways to get across it. But the way works for me is to run your test runner from the directory which contains your main file which includes your nodejs spawn function something like this:
nodeProcess = spawn('node',params, {cwd: '../../node/', detached: true });
For example, this file name is test.js, so just move to the folder which contains it. In my case, it is test folder like this:
cd root/test/
then from run your test runner in my case its mocha so it will be like this:
mocha test.js
I have wasted my more than one day to figure it out. Enjoy!!
solution in my case
var spawn = require('child_process').spawn;
const isWindows = /^win/.test(process.platform);
spawn(isWindows ? 'twitter-proxy.cmd' : 'twitter-proxy');
spawn(isWindows ? 'http-server.cmd' : 'http-server');
I ran into this problem on Windows, where calling exec and spawn with the exact same command (omitting arguments) worked fine for exec (so I knew my command was on $PATH), but spawn would give ENOENT. Turned out that I just needed to append .exe to the command I was using:
import { exec, spawn } from 'child_process';
// This works fine
exec('p4 changes -s submitted');
// This gives the ENOENT error
spawn('p4');
// But this resolves it
spawn('p4.exe');
// Even works with the arguments now
spawn('p4.exe', ['changes', '-s', 'submitted']);
I had this appear while building gulp-jekyll in Powershell on Windows 11.
nodejs 10 LTS, gulp v3.9.1, ruby 3.1, bundler 2.4.5, jekyll 4.2.2
This line of code here is the cause of the ENOENT issue I had with spawn and bundle.
return cp.spawn('bundle', [
'exec',
'jekyll',
'build',
'--source=app', '--destination=build/development', '--config=_config.yml', '--profile'
], { stdio: 'inherit' })
.on('close', done);
Two errors returned, and troubleshooting should start here. ๐Ÿค“๐Ÿง
events.js:174
throw er; // Unhandled 'error' event
^
Error: spawn bundle ENOENT
The cp.spawn is not handling an error Event, so handling that with a simple console.log() will expose the real error with debug information:
return cp.spawn('bundle', [
'exec',
'jekyll',
'build',
'--source=app', '--destination=build/development', '--config=_config.yml', '--profile'
], { stdio: 'inherit' })
.on('error', (e) => console.log(e))
.on('close', done);
This now provides a lot more information to debug with.
{ Error: spawn bundle ENOENT
errno: 'ENOENT',
code: 'ENOENT',
syscall: 'spawn bundle',
path: 'bundle',
spawnargs:
[ 'exec',
'jekyll',
'build',
'--source=app',
'--destination=build/development',
'--config=_config.yml',
'--profile' ] }
The next step to debug would be using the nodejs 10 LTS documentation for child_process.spawn(command[, args][, options]). As already described above, adding the { shell: true } to the options of spawn is a working solution. This is what solved my problem as well. ๐Ÿ˜Œ
{ stdio: 'inherit', shell: true }
This solution is merely a band-aid and could be refactored to handle all environments, but that is out of scope for how to troubleshoot & debug the spawn ENOENT error on nodejs.
I was getting this error when trying to debug a node.js program from within VS Code editor on a Debian Linux system. I noticed the same thing worked OK on Windows. The solutions previously given here weren't much help because I hadn't written any "spawn" commands. The offending code was presumably written by Microsoft and hidden under the hood of the VS Code program.
Next I noticed that node.js is called node on Windows but on Debian (and presumably on Debian-based systems such as Ubuntu) it's called nodejs. So I created an alias - from a root terminal, I ran
ln -s /usr/bin/nodejs /usr/local/bin/node
and this solved the problem. The same or a similar procedure will presumably work in other cases where your node.js is called nodejs but you're running a program which expects it to be called node, or vice-versa.
If you're on Windows Node.js does some funny business when handling quotes that may result in you issuing a command that you know works from the console, but does not when run in Node. For example the following should work:
spawn('ping', ['"8.8.8.8"'], {});
but fails. There's a fantastically undocumented option windowsVerbatimArguments for handling quotes/similar that seems to do the trick, just be sure to add the following to your opts object:
const opts = {
windowsVerbatimArguments: true
};
and your command should be back in business.
spawn('ping', ['"8.8.8.8"'], { windowsVerbatimArguments: true });
Although it may be an environment path or another issue for some people, I had just installed the Latex Workshop extension for Visual Studio Code on Windows 10 and saw this error when attempting to build/preview the PDF. Running VS Code as Administrator solved the problem for me.
In my case removing node, delete all AppData/Roaming/npm and AppData/Roaming/npm-cache and installing node once again solve the issue.
Recently I also faced similar issue.
Starting the development server...
events.js:174
throw er; // Unhandled 'error' event
^
Error: spawn null ENOENT
at Process.ChildProcess._handle.onexit (internal/child_process.js:240:19)
at onErrorNT (internal/child_process.js:415:16)
at process._tickCallback (internal/process/next_tick.js:63:19)
Emitted 'error' event at:
at Process.ChildProcess._handle.onexit (internal/child_process.js:246:12)
at onErrorNT (internal/child_process.js:415:16)
at process._tickCallback (internal/process/next_tick.js:63:19)
error Command failed with exit code 1.
It was due to having a wrong configuration in .env file for BROWSER. I had BROWSER=null, but it has to be BROWSER=none. Changing that configuration resolved my issue.
Tried all nothing worked , my system has different issue .
The working solution for me is
run command :
npm config set script-shell "C:\Program Files\git\bin\bash.exe"
I got the same error for windows 8.The issue is because of an environment variable of your system path is missing . Add "C:\Windows\System32\" value to your system PATH variable.
Local development on Emulator
Make sure to have the package installed locally. By changing the spawn command with exec i got a more detailed error and found out I didn't install the package. Simply run, to check if the package is present:
brew install imagemagick
Source
There might be a case that you have two package-lock.json files in your project directory (in/out of the main folder), simply delete the one which would have been created accidentally. Delete the build folder and run the server again.

Is it possible to run PhantomJS from node.js as a command line argument

I was recently going to test out running phantomJS from python as a commandline argument, I haven't got round to it yet but have seen examples. Because PhantomJS is run from the command line this seems to be possible. The result that PhantomJS would spit out would go straight into a variable.
Before I go down that path, making this work in node.js would actually be more useful for me and it got me thinking, can i just use to node to run PhantomJS as a program gets run from the commandline and store the data result that PhantomJS would normally spit out into a variable?
I would rather not use phantomjs-node because it seems to be using too many tricks.
The reason for all of this is to be able to run PhantomJS at the same time as another action the program takes and use the resulting data its recorded for some other stuff.
Simply put, you can run system command line stuff in python, can I do the same in node.js?
Cheers :)
Edit: I understand that node and phantom use different js environments, that's cool because I just want to run phantom as its own process and catch all that output data into a node.js variable (the data will be a array of a pair, string and floating point.) I don't want to 'drive' with phantom, I will craft the loaded javascript files todo what I want. All I want is phantom output. :)
From NPM: https://npmjs.org/package/phantomjs
var path = require('path')
var childProcess = require('child_process')
var phantomjs = require('phantomjs')
var binPath = phantomjs.path
var childArgs = [
path.join(__dirname, 'phantomjs-script.js'),
'some other argument (passed to phantomjs script)'
]
childProcess.execFile(binPath, childArgs, function(err, stdout, stderr) {
// handle results
})
I suppose you can make a simple script for Node.js to run; in that script phantomjs script will be run as a child process. You can see the working example (and links for some documentation) in this answer. I suppose this discussion might be helpful for you as well.
As an alternative to Donald Derek's answer, you can use the spawn function. It will allow you to read the child process's output as soon as it's produced rather than the output being buffered and returned to you all at once.
You can read more about it here.
An example from the documentation.
var spawn = require('child_process').spawn,
ls = spawn('ls', ['-lh', '/usr']);
ls.stdout.on('data', function (data) {
console.log('stdout: ' + data);
});
ls.stderr.on('data', function (data) {
console.log('stderr: ' + data);
});
ls.on('close', function (code) {
console.log('child process exited with code ' + code);
});

Resources