I read on github that:
grunt-mocha-test uses npm's Peer Dependencies functionality
I was unsure what "Peer Dependencies" were so I checked the npm docs and found:
npm is awesome as a package manager. In particular, it handles
sub-dependencies very well: if my package depends on request version 2
and some-other-library
Which I take to mean:
Having 'peer dependencies' mean that a dependency could need other
dependencies in order to function correctly.
npm creates a tree like structure, where the dependency is the root,
and the root dependency has children dependencies
The questions I am left with are:
Where do the children dependencies come from? Are they copies? Or are they
references to other dependencies already present in package.json?
Each of them will have a copy of the package. For example, if you have a project with those dependencies:
"dependencies": {
"node-handlebars": "*",
"less-file": "*",
"async-ls": "*",
"promise": "4.0.0"
}
and run npm install, you would have 4 copies of promise (the one you declared as a dependency and 3 others needed from each of the other dependencies)
$ find . -name promise
./node_modules/async-ls/node_modules/promise
./node_modules/promise
./node_modules/node-handlebars/node_modules/promise
./node_modules/less-file/node_modules/promise
Note that this would happen even if every one depends on a specific version of promises package (ex 4.0.0).
Despite looking a little redundant I guess this makes dependency management a lot easier, and nowadays the extra space used in general should be negligible.
Related
I have a package.json that gives a load of security warnings. Looking at the first critical item I see its open#0.0.5 which hasn't been updated for five years. Looking at npm ll it is included by npm#6.5.0 where I am using the latest that was updated about two weeks ago.
I would like to remove the insecure dependencies. In the Java world the maven package manager lets you put exclude certain transitive dependencies. Ideally, with npm or another node package manager, I should be able to exclude dependencies with vulnerabilities. Then I can retest that my app works and not see any security errors. Is there a way to quickly exclude anything that has a security vulnerability from my package.json? If there isn't a way to do this what approaches can a take to ensure that no insecure packages are used by my application?
Update: Although "npm": "^6.5.0" is specified in the package.json I was building it with an older npm which was picking up the critical issue mentioned above. I fixed all the issues with ./node_modules/.bin/npm audit fix --force
By definition, you can't exclude a package that a dependency you are using relies on. In other words, if you require package A, and package A claims it is dependent on package B, then removing package B will cause A to either stop working altogether or begin behaving erratically.
Unfortunately this does happen, and your options include:
Ignoring the security warning.
Replacing package A with something else (applies in some cases and not others).
Asking the maintainer of package A to upgrade the version of package B they rely on, possibly opening a pull request yourself.
In your case, though, I'm not sure if your investigation is complete yet - I don't see open in npm's dependency list. Might be worth scrapping your node_modules and re-running npm install, then check again to see who is using open.
This specific warning is targeting at your lockfile, and can be easily fixed by removing the yarn.lock or package-lock.json and reinstall dependencies.
Tarn package manager has feature resulution by which you can set fixed libraries to insecure thirdparties.
See
How do I override nested dependencies with `yarn`?
NPM has something similar.
So in my project, I require multiple packages, including "dep1".
"dep1" requires another dependency, "dep2".
And "dep2" requires multiple packages, including "dep3".
The problem is, "dep3" is not compatible with the licence I want to use (well actually, with the licence my boss wants to use).
Fortunatly, only one function of "dep2" uses "dep3", and "dep1" does not use this function. So if I remove "dep3" from node_modules, all will go smoothly.
The problem I have is, how to modify package.json to take this into account when doing npm install, and not install this package ?
I am aware I could branch "dep2" to supress the function, and then "dep1" to use the modifies "dep2", but this seems overkill, and I would be dependant of the owner of the packages to accept my branch. I am looking for a solution like 'well, just add the line ignore : "dep3" into package.json', but can't find any.
Thanks for your help !
The obvious way is to remove unwanted dependency from node_modules in NPM postinstall hook.
Alternatively, a stub can be provided instead of dep3. A stub should contain package.json that will identify it as a substitute with matching version:
{
"name": "dep3",
"version": "VERSION THAT MATCHES DEP2 CONSTRAINT"
}
It can be specified as local dependency in dep:
...
"dependencies": {
"dep3": "./dep3-stub",
...
Or as Git dependency:
...
"dependencies": {
"dep3": "github-user-name/dep3-stub",
...
If version constraint matches, dep3 stub will be installed and used instead of real package, otherwise dep2 may install its own copy of dep3.
I am aware I could branch "dep2" to supress the function, and then "dep1" to use the modifies "dep2", but this seems overkill, and I would be dependant of the owner of the packages to accept my branch.
This is a reasonable approach. This is no concern to the owner. dep2 fork can be used instead of dep2 NPM package in the same way as shown for dep3.
With npm, when a package requires other packages it creates a tree structure of dependencies. Sometimes a lot of these dependencies depend on the same packages from other packages.
I was wondering, would it be possible to make npm so all packages are stores in the global node_modules and any dependency is just symlinked back to the top of the global node_modules. I understand the version issue, and that can just be handled by storing the package with the version name appended, then symlinking to the proper version.
I feel this would speed up installs and reduce disk usage for duplicate files.
(Is this what npm3 is supposed to do?)
Yes, what you propose would be possible (at least on Linux the symlinks are resolved as expected).
npm (in none of its versions) however does not benefit from symlinks. To gain some of the benefits you proposed, newer versions of npm work as follows: if some package is needed multiple times, npm installs the package as high as possible in the dependency tree. This enables using the same dependency by multiple packages.
For example, no matter how many (sub-)dependencies depend on somedep v. ^1.x.x you got only one copy of somedep. This will probably be placed directly in the root node_modules, so that any sub-dependency can require it.
Older versions of npm do not do this automatically, however, you can invoke the similar effect by running 'npm dedupe'.
Note however, that this approach is weaker than proposed in the question: If 3 of your dependencies depend on somedep v. ^1.x.x and 3 other dependencies depend on somedep v. ^2.x.x, npm obviously cannot put both of these somedeps to the parent node_modules.
Also, check out ied project: https://github.com/alexanderGugel/ied . It does something similar to what you propose, but sadly, it installs only one version of each dependency, which is quite limiting.
NPM triggers a warning message: Dependency '...' exists in both dependencies and devDependencies when my package.json explicitly declares the same module in both dependencies and devDependencies. It suggests that the NPM's developers think it is a wrong way of using NPM.
This is surprising to me since I've always considered it a good practice Doing so means I can change (add / remove / update) my "app dependencies" without breaking my build scripts etc.
Do you know the rational behind this view or do you have examples where it could introduce problems?
dependencies and devDependencies are both installed into the node_modules directory at the top level. If you declare it in both sections, you might declare it at different versions (or conflicting version ranges), which would it impossible for npm to meet both requirements at the same time.
In practice, if you declare it in both sections, npm will pick the version declared in dependencies, even if it is older, since this is usually what you meant to happen.
As a corollary: if you update your app dependencies to a version that would break your build scripts, it will break your build scripts, and declaring it twice won't help.
npm allows us to specify bundledDependencies, but what are the advantages of doing so? I guess if we want to make absolutely sure we get the right version even if the module we reference gets deleted, or perhaps there is a speed benefit with bundling?
Anyone know the advantages of bundledDependencies over normal dependencies?
For the quick reader : this QA is about the package.json bundledDependencies field, not about the package.
What bundledDependencies do
"bundledDependencies" are exactly what their name implies. Dependencies that should be inside your project. So the functionality is basically the same as normal dependencies. They will also be packed when running npm pack.
When to use them
Normal dependencies are usually installed from the npm registry.
Thus bundled dependencies are useful when:
you want to re-use a third party library that doesn't come from the npm registry or that was modified
you want to re-use your own projects as modules
you want to distribute some files with your module
This way, you don't have to create (and maintain) your own npm repository, but get the same benefits that you get from npm packages.
When not to use bundled dependencies
When developing, I don't think that the main point is to prevent accidental updates though. We have better tools for that, namely code repositories (git, mercurial, svn...) or now lock files.
To pin your package versions, you can use:
Option1: Use the newer NPM version 5 that comes with node 8. It uses a package-lock.json file (see the node blog and the node 8 release)
Option2: use yarn instead of npm.
It is a package manager from facebook, faster than npm and it uses a yarn.lock file. It uses the same package.json otherwise.
This is comparable to lockfiles in other package managers like Bundler
or Cargo. It’s similar to npm’s npm-shrinkwrap.json, however it’s not
lossy and it creates reproducible results.
npm actually copied that feature from yarn, amongst other things.
Option3: this was the previously recommended approach, which I do not recommend anymore. The idea was to use npm shrinkwrap most of the time, and sometimes put the whole thing, including the node_module folder, into your code repository. Or possibly use shrinkpack. The best practices at the time were discussed on the node.js blog and on the joyent developer websites.
See also
This is a bit outside the scope of the question, but I'd like to mention the last kind of dependencies (that I know of): peer dependencies. Also see this related SO question and possibly the docs of yarn on bundledDependencies.
One of the biggest problems right now with Node is how fast it is changing. This means that production systems can be very fragile and an npm update can easily break things.
Using bundledDependencies is a way to get round this issue by ensuring, as you correctly surmise, that you will always deliver the correct dependencies no matter what else may be changing.
You can also use this to bundle up your own, private bundles and deliver them with the install.
Other advantage is that you can put your internal dependencies (application components) there and then just require them in your app as if they were independent modules instead of cluttering your lib/ and publishing them to npm.
If/when they are matured to the point they could live as separate modules, you can put them on npm easily, without modifying your code.
I'm surprised I didn't see this here already, but when carefully selected, bundledDependencies can be used to produce a distributable package from npm pack that will run on a system where npm is not configured. This is helpful if you have e.g. a system that's not networked / not on the internet: bring your package over on a thumb drive (or whatever) and unpack the tarball, then npm run or node index.js and it Just Works.
Maybe there's a better way to bundle up your application to run "offline", but if there is I haven't found it.
Operationally, I look at bundledDependencies as a module's private module store, where dependencies is more public, resolved among your module and its dependencies (and sub-dependencies). Your module may rely on an older version of, say, react, but a dependency requires latest-and-greatest. Your package/install will result in your pinned version in node_modules/$yourmodule/node_modules/react, while your dependency will get their version in node_modules/react (or node_modules/$dependency/node_modules/react if they're so inclined).
A caveat: I recently ran into a dependency that did not properly configure its dependency on react, and having react in bundledDependencies caused that dependent module to fail at runtime.