npm install does not seem to get all dependencies - node.js

My package.json looks like this (name/description/etc. omitted).
{
"dependencies": {
"express": "3.3.4",
"jade": "0.34.x",
"mongoose": "3.6.x"
},
"devDependencies": {
"vows": "0.7.x"
}
}
I used express on the repository and ran the auto-generated node app.js. This worked, but when I used curl http://localhost:port I got the error "Cannot find module character-parser." I ran npm install character-parser and then I got "Cannot find module transformers." This happened a few more times, but after I installed all of them the app started working.
I thought that npm install was supposed to install dependencies recursively. This also worries me because I obviously want the package.json to be usable when the app is deployed.

Here's a simple script to collect the dependencies in ./node_modules:
var fs = require("fs");
function main() {
fs.readdir("./node_modules", function (err, dirs) {
if (err) {
console.log(err);
return;
}
dirs.forEach(function(dir){
if (dir.indexOf(".") !== 0) {
var packageJsonFile = "./node_modules/" + dir + "/package.json";
if (fs.existsSync(packageJsonFile)) {
fs.readFile(packageJsonFile, function (err, data) {
if (err) {
console.log(err);
}
else {
var json = JSON.parse(data);
console.log('"'+json.name+'": "' + json.version + '",');
}
});
}
}
});
});
}
For one project I'm working on, the output looks like this:
"progress": "0.1.0",
"request": "2.11.4",
If you remember to remove the comma from the last entry, you can copy and paste the output.

I got this exact error while I was npm installing for https://github.com/HenrikJoreteg/humanjs-sample-app/
I'm on a Windows machine, so I suspected that it was an issue with the odd restrictions that Windows has on the number of characters in a file path.
I resolved this by shorting my base path to a three character folder name in my root (in this case going from C:\projects\humanjs-sample-app to C:\hjs). When I re-ran npm install everything worked. I'm not happy with that resolution. I shouldn't have to worry about my base path name being too long. This is one of the reasons that people often don't do node development on Windows machines.
An alternate resolution is to develop on Linux or Mac, if you aren't already. That's probably my long term strategy.

When you run npm install <name-of-package> it will install the package to your node_modules folder, but will not add it as a dependency. In order to install the package and save it as a dependency in your package.json, you must use the --save flag like so:
npm install <name-of-package> --save
The npm documentation provides further information on additional flags that can be used such as the --save-dev flag for saving development dependencies and the --save-optional flag for saving optional dependencies to your package.json.

Related

NPM - package.json#engines | How to specify Python?

I need to specify a python version on my package.json.
Can I simple do:
{ engines: { "python": "2.7.11" } } ?
Putting "engines": { "python": "2.7.11" } won't cause any issues (as far as I am aware) in your package.json, but it won't actually do anything either.
Determining an appropriate way to do this depends on particulars of your project. If it's about Python code you've written, you can check the version within the Python script itself. If it's about a build step in package.json, you can perhaps test as part of the build step.
As of NPM 7.x (same is valid for legacy NPM 6.x), the only valid entries for "engines" field in package.json, are the "node" version, and the "npm" version.
Furthermore, this is not a hard constraint unless you use "engine-strict" as well, as stated by the NPM docs:
Unless the user has set the engine-strict config flag, this field is advisory only and will only produce warnings when your package is installed as a dependency.
Your requirement, expecting a specific python version, is more related to an environment requirement than to your Node/NPM environment.
You can achieve this by implementing a "postinstall" NPM script that can result an error if the desired version is not found:
{
"scripts": {
"postinstall": "node ./check-python.js"
}
}
This script will be executed by NPM automatically after npm install. You could also use "preinstall" instead.
Consider using it on your "build" or "prebuild" scripts as well, based on your requirements. See more details on NPM pre- and post- scripts in the docs.
Then, your check-python.js script could be something like:
const { exec } = require('child_process');
const EXPECTED_PYTHON_VERSION = "2.7.11";
exec('python -c "import platform; print(platform.python_version())"',
function(err, stdout, stderr) {
const currentPythonVersion = stdout.toString();
if(currentPythonVersion !== EXPECTED_PYTHON_VERSION) {
throw new Error(`Expected Python version '${EXPECTED_PYTHON_VERSION}' but found '${currentPythonVersion}'. Please fix your Python installation.`);
}
});

Node temporary npm install done programmatically

I would like to programmatically npm install a package using a Node app, after the node app has started. Ideally, this package would not file into my node_modules folder, but rather would trash itself after runtime.
npm supports programmatic installs, however it seems to actually save the modules into node_modules. Additionally, making the entirety of npm (a big module) a requirement for this kind of sucks. However, when I looked at the source code, the npm install part uses a ton of modules and isn't something I can easily reproduce.
Is there any other module that anyone knows about that meets this requirement?
Found with NPM you can install to a path, and there's this nifty temp module that helps do that cross platform:
var temp = require('temp')
, npm = require('npm')
;
function use(module, cb) {
npm.load({}, function(){
npm.commands.install(temp.dir, [module], function(err, data){
var dir = data[0][1];
var mod = require(__dirname + '/' + dir);
cb(mod);
});
})
}
use('lodash', function(_){
// ... do things.
});
If you want to, temp has a clean function that can clean up the temp dir later.

Install node_modules to vendor

How can I install npm modules locally for each project to vendor/node_modules and make package.json file see them.
I don't want to move package.json to vendor folder
I have bower and in .bowerrc I specify the bower_components path - that is super easy.
How can I do that with npm ?
I`ve read the docs, npmrc docs, some questions here, googled, wasted more than an hour - still no luck. This is ridiculously hard for such an easy task.
I don't care about minuses, just tell me how to do that finally.
Frustrated by the fact that there seems to be no built in way to install into a node_modules folder in an arbitrary subfolder, I came up with a sneaky solution using the two following scripts:
preinstall.js
var fs = require("fs");
try
{
fs.mkdirSync("./app/node_modules/");
}
catch(e)
{
}
try
{
if(process.platform.indexOf("win32") !== -1)
{
fs.symlinkSync("./app/node_modules/","./node_modules/","junction");
}
else
{
fs.symlinkSync("./app/node_modules/","./node_modules","dir");
}
}
catch(e){}
postinstall.js
var fs = require("fs");
try
{
if(process.platform.indexOf("win32") !== -1)
{
fs.unlinkSync("./node_modules/");
}
else
{
fs.unlinkSync("./node_modules");
}
}
catch(e){}
All you need to do is use them in your package.json file by adding them to the scripts option:
"scripts": {
"preinstall": "node preinstall.js",
"postinstall": "node postinstall.js"
},
So, the big question is: what does it do?
Well, when you call npm install the preinstall.js script fires which creates a node_modules in the subfolder you want. Then it creates a symlink or (shortcut in Windows) from the node_modules that npm expects to the real node_modules.
Then npm installs all the dependencies.
Finally, once all the dependencies are installed, the postinstall.js script fires which removes the symlink!
Here's a handy gist with all that you need.
You cannot, not using built-in npm functionality.
This discussion on the npm github repository covers the issue. It's also being addressed in this answer which is part of their FAQ.
You can still do the installs "manually" by copying modules into your /vendor directory and then calling them using the require("./vendor/whatever") syntax...but that means each require needs to use your new custom location. There are a few ways you could handle this but they all mean you are doing extra work in your source to accomodate the custom location.

Is there a way to generate the bundledDependencies list automatically?

I have an app that I'm deploying to Nodejitsu. Recently, they suffered from npm issues that caused my app to go offline for several hours after I tried (and failed) to restart it, as its dependencies could not be installed. I was told that this could be averted in the future by listing all of my dependencies as bundledDependencies in my package.json, causing the dependencies to be uploaded along with the rest of the application. Which means that I need my package.json to look something like this:
"dependencies": {
"express": "2.5.8",
"mongoose": "2.5.9",
"stylus": "0.24.0"
},
"bundledDependencies": [
"express",
"mongoose",
"stylus"
]
Now, on DRY grounds, this is unappealing. But what's worse is the maintenance: Every time I add or remove a dependency, I have to make the change in two places. Is there a command I can use to sync bundledDependencies with dependencies?
How about implementing a grunt.js task to do it? This works:
module.exports = function(grunt) {
grunt.registerTask('bundle', 'A task that bundles all dependencies.', function () {
// "package" is a reserved word so it's abbreviated to "pkg"
var pkg = grunt.file.readJSON('./package.json');
// set the bundled dependencies to the keys of the dependencies property
pkg.bundledDependencies = Object.keys(pkg.dependencies);
// write back to package.json and indent with two spaces
grunt.file.write('./package.json', JSON.stringify(pkg, undefined, ' '));
});
};
Put that in the root directory of your project in a file called grunt.js. To install grunt, use npm: npm install -g grunt. Then bundle the packages by executing grunt bundle.
A commentor mentioned an npm module that could be useful: https://www.npmjs.com/package/grunt-bundled-dependencies (I haven't tried it.)
You can use a simple Node.js script to read and update the bundleDependencies property and run it via npm lifecycle hooks/scripts.
My folder structure is:
scripts/update-bundle-dependencies.js
package.json
scripts/update-bundle-dependencies.js:
#!/usr/bin/env node
const fs = require('fs');
const path = require('path');
const pkgPath = path.resolve(__dirname, "../package.json");
const pkg = require(pkgPath);
pkg.bundleDependencies = Object.keys(pkg.dependencies);
const newPkgContents = JSON.stringify(pkg, null, 2);
fs.writeFileSync(pkgPath, newPkgContents);
console.log("Updated bundleDependencies");
If you are using the latest version of npm (> 4.0.0), you can use the prepublishOnly or prepack script: https://docs.npmjs.com/misc/scripts
prepublishOnly: Run BEFORE the package is prepared and packed, ONLY on
npm publish. (See below.)
prepack: run BEFORE a tarball is packed (on npm pack, npm publish, and
when installing git dependencies)
package.json:
{
"scripts": {
"prepack": "npm run update-bundle-dependencies",
"update-bundle-dependencies": "node scripts/update-bundle-dependencies"
}
}
You can test it yourself by running npm run update-bundle-dependencies.
Hope this helps!

Install dependencies globally and locally using package.json

Using npm we can install the modules globally using -g option. How can we do this in the package.json file?
Suppose, these are my dependencies in package.json file
"dependencies": {
"mongoose": "1.4.0",
"node.io" : "0.3.3",
"jquery" : "1.5.1",
"jsdom" : "0.2.0",
"cron" : "0.1.2"
}
When i run npm install, I want only node.io to be installed globally, the rest others should be installed locally. Is there an option for this?
New Note: You probably don't want or need to do this. What you probably want to do is just put those types of command dependencies for build/test etc. in the devDependencies section of your package.json. Anytime you use something from scripts in package.json your devDependencies commands (in node_modules/.bin) act as if they are in your path.
For example:
npm i --save-dev mocha # Install test runner locally
npm i --save-dev babel # Install current babel locally
Then in package.json:
// devDependencies has mocha and babel now
"scripts": {
"test": "mocha",
"build": "babel -d lib src",
"prepublish": "babel -d lib src"
}
Then at your command prompt you can run:
npm run build # finds babel
npm test # finds mocha
npm publish # will run babel first
New NEW Note: For awhile now we have had npx, which allows you to run the devDependencies commands without needing to add them to your scripts section (if you want).
For example:
npx webpack
But if you really want to install globally, you can add a preinstall in the scripts section of the package.json:
"scripts": {
"preinstall": "npm i -g themodule"
}
So actually my npm install executes npm install again ... Which is weird but seems to work.
Note: you might have issues if you are using the most common setup for npm where global Node package installs required sudo. One option is to change your npm configuration so this isn't necessary:
npm config set prefix ~/npm, add $HOME/npm/bin to $PATH by appending export PATH=$HOME/npm/bin:$PATH to your ~/.bashrc.
Another, probably better option is to just use nvm to manage Node and you won't have that problem.
Due to the disadvantages described below, I would recommend following the accepted answer:
Use npm install --save-dev [package_name] then execute scripts with:
$ npm run lint
$ npm run build
$ npm test
My original but not recommended answer follows.
Instead of using a global install, you could add the package to your devDependencies (--save-dev) and then run the binary from anywhere inside your project:
"$(npm bin)/<executable_name>" <arguments>...
In your case:
"$(npm bin)"/node.io --help
This engineer provided an npm-exec alias as a shortcut. This engineer uses a shellscript called env.sh. But I prefer to use $(npm bin) directly, to avoid any extra file or setup.
Although it makes each call a little larger, it should just work, preventing:
potential dependency conflicts with global packages (#nalply)
the need for sudo
the need to set up an npm prefix (although I recommend using one anyway)
Disadvantages:
$(npm bin) won't work on Windows.
Tools deeper in your dev tree will not appear in the npm bin folder. (Install npm-run or npm-which to find them.)
It seems a better solution is to place common tasks (such as building and minifying) in the "scripts" section of your package.json, as Jason demonstrates above.
This is a bit old but I ran into the requirement so here is the solution I came up with.
The Problem:
Our development team maintains many .NET web application products we are migrating to AngularJS/Bootstrap. VS2010 does not lend itself easily to custom build processes and my developers are routinely working on multiple releases of our products. Our VCS is Subversion (I know, I know. I'm trying to move to Git but my pesky marketing staff is so demanding) and a single VS solution will include several separate projects. I needed my staff to have a common method for initializing their development environment without having to install the same Node packages (gulp, bower, etc.) several times on the same machine.
TL;DR:
Need "npm install" to install the global Node/Bower development environment as well as all locally required packages for a .NET product.
Global packages should be installed only if not already installed.
Local links to global packages must be created automatically.
The Solution:
We already have a common development framework shared by all developers and all products so I created a NodeJS script to install the global packages when needed and create the local links. The script resides in "....\SharedFiles" relative to the product base folder:
/*******************************************************************************
* $Id: npm-setup.js 12785 2016-01-29 16:34:49Z sthames $
* ==============================================================================
* Parameters: 'links' - Create links in local environment, optional.
*
* <p>NodeJS script to install common development environment packages in global
* environment. <c>packages</c> object contains list of packages to install.</p>
*
* <p>Including 'links' creates links in local environment to global packages.</p>
*
* <p><b>npm ls -g --json</b> command is run to provide the current list of
* global packages for comparison to required packages. Packages are installed
* only if not installed. If the package is installed but is not the required
* package version, the existing package is removed and the required package is
* installed.</p>.
*
* <p>When provided as a "preinstall" script in a "package.json" file, the "npm
* install" command calls this to verify global dependencies are installed.</p>
*******************************************************************************/
var exec = require('child_process').exec;
var fs = require('fs');
var path = require('path');
/*---------------------------------------------------------------*/
/* List of packages to install and 'from' value to pass to 'npm */
/* install'. Value must match the 'from' field in 'npm ls -json' */
/* so this script will recognize a package is already installed. */
/*---------------------------------------------------------------*/
var packages =
{
"bower" : "bower#1.7.2",
"event-stream" : "event-stream#3.3.2",
"gulp" : "gulp#3.9.0",
"gulp-angular-templatecache" : "gulp-angular-templatecache#1.8.0",
"gulp-clean" : "gulp-clean#0.3.1",
"gulp-concat" : "gulp-concat#2.6.0",
"gulp-debug" : "gulp-debug#2.1.2",
"gulp-filter" : "gulp-filter#3.0.1",
"gulp-grep-contents" : "gulp-grep-contents#0.0.1",
"gulp-if" : "gulp-if#2.0.0",
"gulp-inject" : "gulp-inject#3.0.0",
"gulp-minify-css" : "gulp-minify-css#1.2.3",
"gulp-minify-html" : "gulp-minify-html#1.0.5",
"gulp-minify-inline" : "gulp-minify-inline#0.1.1",
"gulp-ng-annotate" : "gulp-ng-annotate#1.1.0",
"gulp-processhtml" : "gulp-processhtml#1.1.0",
"gulp-rev" : "gulp-rev#6.0.1",
"gulp-rev-replace" : "gulp-rev-replace#0.4.3",
"gulp-uglify" : "gulp-uglify#1.5.1",
"gulp-useref" : "gulp-useref#3.0.4",
"gulp-util" : "gulp-util#3.0.7",
"lazypipe" : "lazypipe#1.0.1",
"q" : "q#1.4.1",
"through2" : "through2#2.0.0",
/*---------------------------------------------------------------*/
/* fork of 0.2.14 allows passing parameters to main-bower-files. */
/*---------------------------------------------------------------*/
"bower-main" : "git+https://github.com/Pyo25/bower-main.git"
}
/*******************************************************************************
* run */
/**
* Executes <c>cmd</c> in the shell and calls <c>cb</c> on success. Error aborts.
*
* Note: Error code -4082 is EBUSY error which is sometimes thrown by npm for
* reasons unknown. Possibly this is due to antivirus program scanning the file
* but it sometimes happens in cases where an antivirus program does not explain
* it. The error generally will not happen a second time so this method will call
* itself to try the command again if the EBUSY error occurs.
*
* #param cmd Command to execute.
* #param cb Method to call on success. Text returned from stdout is input.
*******************************************************************************/
var run = function(cmd, cb)
{
/*---------------------------------------------*/
/* Increase the maxBuffer to 10MB for commands */
/* with a lot of output. This is not necessary */
/* with spawn but it has other issues. */
/*---------------------------------------------*/
exec(cmd, { maxBuffer: 1000*1024 }, function(err, stdout)
{
if (!err) cb(stdout);
else if (err.code | 0 == -4082) run(cmd, cb);
else throw err;
});
};
/*******************************************************************************
* runCommand */
/**
* Logs the command and calls <c>run</c>.
*******************************************************************************/
var runCommand = function(cmd, cb)
{
console.log(cmd);
run(cmd, cb);
}
/*******************************************************************************
* Main line
*******************************************************************************/
var doLinks = (process.argv[2] || "").toLowerCase() == 'links';
var names = Object.keys(packages);
var name;
var installed;
var links;
/*------------------------------------------*/
/* Get the list of installed packages for */
/* version comparison and install packages. */
/*------------------------------------------*/
console.log('Configuring global Node environment...')
run('npm ls -g --json', function(stdout)
{
installed = JSON.parse(stdout).dependencies || {};
doWhile();
});
/*--------------------------------------------*/
/* Start of asynchronous package installation */
/* loop. Do until all packages installed. */
/*--------------------------------------------*/
var doWhile = function()
{
if (name = names.shift())
doWhile0();
}
var doWhile0 = function()
{
/*----------------------------------------------*/
/* Installed package specification comes from */
/* 'from' field of installed packages. Required */
/* specification comes from the packages list. */
/*----------------------------------------------*/
var current = (installed[name] || {}).from;
var required = packages[name];
/*---------------------------------------*/
/* Install the package if not installed. */
/*---------------------------------------*/
if (!current)
runCommand('npm install -g '+required, doWhile1);
/*------------------------------------*/
/* If the installed version does not */
/* match, uninstall and then install. */
/*------------------------------------*/
else if (current != required)
{
delete installed[name];
runCommand('npm remove -g '+name, function()
{
runCommand('npm remove '+name, doWhile0);
});
}
/*------------------------------------*/
/* Skip package if already installed. */
/*------------------------------------*/
else
doWhile1();
};
var doWhile1 = function()
{
/*-------------------------------------------------------*/
/* Create link to global package from local environment. */
/*-------------------------------------------------------*/
if (doLinks && !fs.existsSync(path.join('node_modules', name)))
runCommand('npm link '+name, doWhile);
else
doWhile();
};
Now if I want to update a global tool for our developers, I update the "packages" object and check in the new script. My developers check it out and either run it with "node npm-setup.js" or by "npm install" from any of the products under development to update the global environment. The whole thing takes 5 minutes.
In addition, to configure the environment for the a new developer, they must first only install NodeJS and GIT for Windows, reboot their computer, check out the "Shared Files" folder and any products under development, and start working.
The "package.json" for the .NET product calls this script prior to install:
{
"name" : "Books",
"description" : "Node (npm) configuration for Books Database Web Application Tools",
"version" : "2.1.1",
"private" : true,
"scripts":
{
"preinstall" : "node ../../SharedFiles/npm-setup.js links",
"postinstall" : "bower install"
},
"dependencies": {}
}
Notes
Note the script reference requires forward slashes even in a Windows
environment.
"npm ls" will give "npm ERR! extraneous:" messages for all packages
locally linked because they are not listed in the "package.json"
"dependencies".
Edit 1/29/16
The updated npm-setup.js script above has been modified as follows:
Package "version" in var packages is now the "package" value passed to npm install on the command line. This was changed to allow for installing packages from somewhere other than the registered repository.
If the package is already installed but is not the one requested, the existing package is removed and the correct one installed.
For reasons unknown, npm will periodically throw an EBUSY error (-4082) when performing an install or link. This error is trapped and the command re-executed. The error rarely happens a second time and seems to always clear up.
You could use a separate file, like npm_globals.txt, instead of package.json. This file would contain each module on a new line like this,
mongoose#1.4.0
node.io#0.3.3
jquery#1.5.1
jsdom#0.2.0
cron#0.1.2
Then in the command line run,
< npm_globals.txt xargs npm install -g
Check that they installed properly with,
npm list -g --depth=0
As for whether you should do this or not, I think it all depends on use case. For most projects, this isn't necessary; and having your project's package.json encapsulate these tools and dependencies together is much preferred.
But nowadays I find that I'm always installing create-react-app and other CLI's globally when I jump on a new machine. It's nice to have an easy way to install a global tool and its dependencies when versioning doesn't matter much.
And nowadays, I'm using npx, an npm package runner, instead of installing packages globally.
Build your own script to install global dependencies. It doesn't take much. package.json is quite expandable.
const { execSync } = require('child_process');
const fs = require('fs');
const package = JSON.parse(fs.readFileSync('package.json'));
let keys = Object.keys(package.dependencies);
let values = Object.values(package.dependencies);
for (let index = 0; index < keys.length; index++) {
const key = keys[index];
let value = values[index].replace("~", "").replace("^", "");
console.log(`Installing: ${key}#${value} globally`,);
execSync('npm i -g ' + `${key}#${value}`);
}
Using the above, you can even make it inline, below!
Look at preinstall below:
{
"name": "Project Name",
"version": "0.1.0",
"description": "Project Description",
"main": "app.js",
"scripts": {
"preinstall": "node -e \"const {execSync} = require('child_process'); JSON.parse(fs.readFileSync('package.json')).globalDependencies.forEach(globaldep => execSync('npm i -g ' + globaldep));\"",
"build": "your transpile/compile script",
"start": "node app.js",
"test": "./node_modules/.bin/mocha --reporter spec",
"patch-release": "npm version patch && npm publish && git add . && git commit -m \"auto-commit\" && git push --follow-tags"
},
"dependencies": [
},
"globalDependencies": [
"cordova#8.1.2",
"ionic",
"potato"
],
"author": "author",
"license": "MIT",
"devDependencies": {
"chai": "^4.2.0",
"mocha": "^5.2.0"
},
"bin": {
"app": "app.js"
}
}
The authors of node may not admit package.json is a project file. But it is.
All modules from package.json are installed to ./node_modules/
I couldn't find this explicitly stated but this is the package.json reference for NPM.
This probably may be door for problems in production.
if project dependencies installed outside project folder, the code may break if anybody else delete or replace your packages or change folder permissions.
Having every thing in one folder is more durable and make system predictable and maintenance tasks easier.

Resources