I am new to nodejs. And I found that there are many unknown modules in node_modules folder after I installed three modules (express, jade, gulp) in my local project.
Unknown module examples in the node_modules:
vary
statuses
send
promise
From the tutorials I see from others, after they installed gulp, there will be only one "gulp" folder in their node_modules folder, but this is not my case. Why? Thank you.
The node package manager (npm) updated recently, as part of that update, all modules are installed in the top level node_modules folder. This includes modules that your dependencies need to install. In the past these modules would be nested inside another node_modules folder in express (for example).
This is why the tutorials you read say different, likely they were written before this update.
It is mentioned in the npm changelog here
Your dependencies will now be installed maximally flat. Insofar as is
possible, all of your dependencies, and their dependencies, and THEIR
dependencies will be installed in your project's node_modules folder
with no nesting. You'll only see modules nested underneath one another
when two (or more) modules have conflicting dependencies.
#3697 This will hopefully eliminate most cases where windows users ended up with paths that were too long for Explorer and other standard
tools to deal with.
#6912 (#4761 #4037) This also means that your installs will be deduped from the start.
#5827 This deduping even extends to git deps.
#6936 (#5698) Various commands are dedupe aware now. This has some implications for the behavior of other commands:
npm uninstall removes any dependencies of the module that you
specified that aren't required by any other module. Previously, it
would only remove those that happened to be installed under it,
resulting in left over cruft if you'd ever deduped. npm ls now shows
you your dependency tree organized around what requires what, rather
than where those modules are on disk.
#6937 npm dedupe now flattens the tree in addition to deduping. And bundling of dependencies when packing or publishing changes too:
#2442 bundledDependencies no longer requires that you specify deduped sub deps. npm can now see that a dependency is required by something
bundled and automatically include it. To put that another way,
bundledDependencies should ONLY include things that you included in
dependencies, optionalDependencies or devDependencies.
#5437 When bundling a dependency that's both a devDependency and the child of a regular dependency, npm bundles the child dependency. As a
demonstration of our confidence in our own work, npm's own
dependencies are now flattened, deduped, and bundled in the npm#3
style. This means that npm#3 can't be packed or published by npm#2,
which is something to be aware of if you're hacking on npm.
Related
We work in a large team on multiple projects and everyone has a different set of node packages installed globally.
This can cause different behaviour when different members of the team build a project.
Is there a setting in a package.json file (or other file) that we can apply that forces NPM to only use local packages that have been installed in the local project node_modules.
I assume that this is more relating to development tools and modules (netlify, standard, prettier, #babel/core, rollup, eslint, etc)
Using npm i --save-dev package-a package-b package-c to ensure that you have the exact and correct versions across your development teams. I would suggest looking at package.jsons in projects like three.js, react, nextjs etc to see how they handle it. It can be quite a task to get a correct baseline of devDependencies, but once set, it generally solves a lot of package dependency problems caused by global modules.
See this: http://npm.github.io/how-npm-works-docs/npm3/how-npm3-works.html
I noticed that npm installs the same node.js packages several times when they are required by different modules.
Is there an option to switch this off and to have one repo for all the packages?
I looked but did not find anything on this...
This is the default behaviour of NPM since version 3. Quoting the changelog of that version:
Your dependencies will now be installed maximally flat. Insofar as is possible, all of your dependencies, and their dependencies, and THEIR dependencies will be installed in your project's node_modules folder with no nesting. You'll only see modules nested underneath one another when two (or more) modules have conflicting dependencies.
As you can see, if you have modules which have conflicting dependencies (i.e. require different versions of the same dependency), multiple versions of such dependencies will be installed, and this cannot be avoided.
If you suspect that your node_modules directory somehow contains duplicates, you can use the command npm dedupe (docs) to find these and reduce the duplication, leaving only one instance of each duplicate.
I have standard Lerna repository like this:
my-repo
- package.json
- packages
- api
- package.json
- web-app
- package.json
If I need same dependency in both packages (for example lodash), then people in tutorials suggest to install it to both sub modules and then bootstrap project with with lerna bootstrap --hoist flag.
Because of --hoist flag lodash dependency will be loaded only to root level node_modules but both sub-modules will contain it as dependency in their appropriate package.json
But Node’s package resolution algorithm searches up the file tree looking for node_modules folder.
So my question is why I can't just install common dependencies to root level project? Then lodash will be located under root's node_modules. And sub-modules (packages) will find it because Node will search up for node_module until the root of the file system is reached.
At least it will help me to avoid using uncommon lerna bootstrap --hoist, as well as lodash dependency will be present only once at the top level package.json (and not twice: in package.json of both submodules)
So my question is why I can't just install common dependencies to root
level project?
You can, and you're right, node's resolution algorithm will find the shared dependency fine.
The downside of doing this is that you lose flexibility, and you would need to deploy or work with the whole mono-repo which might be fine for you. A more traditional approach is to keep production dependencies in the sub packages so that you can publish the packages and use them individually without depending on the root of the monorepo, but again, this might not be important for you.
If your packages are npm packages that you intend to reuse by publishing to some npm registry, then you should give them proper package.json dependencies.
A package should always list what it requires to operate. E.g. if your "api" package requires lodash at runtime, then it MUST have lodash in its dependencies. Otherwise an app could install it without lodash; then it would fail to run.
In your lerna repo, your "root" package.json is not connected to any npm package, and is not published, so it doesn't affect your "real" npm packages: "api" and "web-app".
Finally, if you don't intend to publish your packages as npm packages, then do whatever you want. Lerna is probably overkill in that case. Even the package.json is overkill, because it's just being used for its scripts.
My node_modules has packages that are not listed in my package.json's dependencies, so I'm guessing that those packages are dependencies of my dependencies. How would I be able to check this? I want to make sure that there aren't any unnecessary packages in my node_modules directory.
If your dependency list won't take too long to reinstall, a simple option is a table-flip: remove the node_modules directory entirely and run npm install to re-create it.
If you don't want to do that, you can try tools that inspect your dependencies, like depcheck as #sagar-gopale suggests in their answer.
Related: Run npm -v to find out if you are running npm v2 or v3. Like #cartant says in their answer, with v3, your node_modules directory will be maximally flat, which means things that used to appear as subdirectories of other modules (when installed with npm v2) will now appear at the top level of node_modules itself. That may be the reason you see more modules than you expect.
If you are using NPM 3, you will likely see a large number of modules that you were not expecting to see in the node_modules directory, as NPM 3 flattens the dependency hierarchy.
Whichever version you are using, if you run the npm list command, NPM should highlight any extraneous modules that are not required.
Please checkout this package.
https://www.npmjs.com/package/depcheck
Since packages can require other packages, just because there are packages in the node_modules folder that don't exist in your packages.json file doesn't mean they aren't needed by one of your specified packages.
If you run an npm prune command on the root directory of your solution it will read the dependency tree and remove the packages that are truly no longer needed.
I just run a simple npm install morgan in a folder and for some reason it adds all the sub dependencies to the parent folder. See image attached
Yes, this is a new feature in npm 3.x, you can read about it here:
https://github.com/npm/npm/releases/tag/v3.0.0
Your dependencies will now be installed maximally flat. Insofar as is possible, all of your dependencies, and their dependencies, and THEIR dependencies will be installed in your project's node_modules folder with no nesting. You'll only see modules nested underneath one another when two (or more) modules have conflicting dependencies.
Basically, it now handles dependencies with as little nesting as possible.
Npm has changed the way they organize dependencies. So instead of 2 separate modules requiring the same dependency and installing them in their own node_modules folder. The dependency is only installed once at the same folder level the node module is installed at.