The npm-shrinkwrap ensures that installed packages also have the same version of dependencies that was used at the moment of publishing to the registry (These versions are stated in the npm-shrinkwrap which is then used on installation).
I am currently using yarn (and lerna for publishing) and working on a monorepo project with workspaces. Now I would like to have each package in the monorepo have same guarantees provided by npm-shrinkwrap.
One shortcoming of the npm-shrinkwrap is that it does not support workspaces. Hence I cannot use npm-shrinkwrap since it is only created at the root and does not influence how individual packages in the monorepo get installed.
Since I am using yarn, I was wondering if there is an alternative to npm-shrinkwrap in yarn?
Or maybe a better question is, using yarn/lerna, how do I lock version dependencies for publication, such that when my packages in the mono-repo are downloaded, they are downloaded with the exact versions of dependencies (and transitive dependencies) that was specified at the point of publication?
I found the following in the yarn docs (classic yarn 1.x)
If you are using an npm-shrinkwrap.json file right now, be aware that you may end up with a different set of dependencies. Yarn does not support npm shrinkwrap files as they don’t have enough information in them to power Yarn’s more deterministic algorithm. If you are using a shrinkwrap file it may be easier to convert everyone working on the project to use Yarn at the same time. Simply remove your existing npm-shrinkwrap.json file and check in the newly created yarn.lock file.
From here:
https://classic.yarnpkg.com/en/docs/migrating-from-npm
What does npm i --package-lock-only do exactly? The documentation is a tad shy on examples. https://docs.npmjs.com/cli/v6/configuring-npm/package-locks
I'm curious to know if I have older packages in my local node_modules folder and no package-lock.json file, will npm i --package-lock-only generate a package-lock.json according to the version in my local node_modules folder or will it generate a package-lock.json with newer package versions that is consistent with the semver ranges in the package.json that's published in the npm registry.
It will determine versions of packages to install using package.json, and then create a package-lock.json file with its resolved versions if none exists, or overwrite an existing one.
Significantly, it does not actually install anything, which is what distinguishes it from regular npm install (or the aliased npm i).
Well, #Ben Wheeler is acurate, but there's a place to give a little background on this process. In regular situation the package-lock is meant for set a complete dependency tree of every package and it's dependencies in your application, so every developer on a different machine will have the exact same tree. This is important because the dependencies packages might be updated with time and if every developer will use different versions it could break your application. So every time you do "npm i" if you do have a package.lock.json it actually install the packages from there and not from package.json.
Sometimes when developers have a dependencies errors they tend to delete the lock file and the node_modules. which is not always the best option. Most of the time it's enough to update only the lock file to reflect the package.json with the flag --package-lock-only, and then you can do again "npm i" to install your packages. The lock file should be committed to your project repo so everyone can use it to have the same packages version.
package-lock.json is automatically generated for any operations where npm modifies either the node_modules tree, or package.json. It describes the exact tree that was generated, such that subsequent installs are able to generate identical trees, regardless of intermediate dependency updates.
This file is intended to be committed into source repositories, and serves various purposes:
Describe a single representation of a dependency tree such that
teammates, deployments, and continuous integration are guaranteed to
install exactly the same dependencies.
Provide a facility for users to "time-travel" to previous states of
node_modules without having to commit the directory itself.
Facilitate greater visibility of tree changes through readable source
control diffs.
Optimize the installation process by allowing npm to skip repeated
metadata resolutions for previously-installed packages.
As of npm v7, lockfiles include enough information to gain a complete
picture of the package tree, reducing the need to read package.json
files, and allowing for significant performance improvements.
I have a monorepo where I have a /packages folder with many packages.
I can use npm i ./packages or npm i if they are already specified using using the file pointer.
Looks something like this:
"dependencies": {
"#reggi/command": "file:packages/command",
"#reggi/dep-merge": "file:packages/dep-merge",
"#reggi/dep-merge-cli": "file:packages/dep-merge-cli",
"#reggi/dep-pointer": "file:packages/dep-pointer"
}
The issue is that if I install these packages I don't get dev dependencies.
What I really want is to also install the devDependencies of these dependencies.
lerna a popular tool that has pioneered the usage of monorepos, suggests that you should add all the devDependencies for these packages in the root package. The issue with this is that it eliminates the ability for two packages to depend on different versions of a given dev dependency.
What I have done is created a script that merges all the devDependencies into dependencies at preinstall then undoes the changes. This works but can be kind of wonky at times, especially when explaining all this to shrinkwrap.
It would be nice if I could just npm i --allDevDepsFromDeps and it would install all of my dependencies dev dependencies.
Is there any other solution I am missing?
I don't see what you're trying to achieve there, aren't the devDepencies used for ... development?
If you want different version for different package just don't put them in the root but in each package.
The issue is that if I install these packages I don't get dev dependencies.
You should consider those packages as 'production'/'bundled' packages, you don't need dev dependencies in this case.
For example, when you are working on #pkg/A, it will have its own devDep but then if you work on #pkg/B that depends on #pkg/A, the #pkg/A should be the production/bundled version (without devDeps).
Maybe you should have a look at bundledDependencies or peerDependencies, that might help you.
On my current project we have dependencies listed with the exact version (e. g. babel-core: '6.5.2') in our package.json file.
But after the recent project build (which includes running npm install on the build machine) some node modules were updated and our artifact got broken.
The research has shown that although we have the exact versions mentioned in package.json file, the required module might have dependency on itself through its dependency, and both of them are using the caret dependencies. In our case it appeared to be
babel-core: '6.5.2' depending on
babel-register: '^6.16.0' depending on
babel-core: '^6.9.0' resolved as babel-core: '6.26.0' (because of the caret range rules)
And the specified babel-core version breaks our builds.
We've tried creating package-lock.json files in order to prevent these transitive dependencies to be resolved the way they were resolved.
But, as the documentation on package-lock files states:
Whenever you run npm install, npm generates or updates your package lock
And thus, all the packages depending on babel-core with caret version are resolved with incorrect version of a module.
After taking a closer look at package-lock.json file, I don't see it's worth changing it because of
the snowball effect - changing one dependency requires to change all the others, depending on it (and this is recursive process)
the need of verifying both the existence of the version you're about to change to and its hash (for the integrity field)
The question is: how do I install the exact versions of all the modules, mentioned in my package.json file, preventing NPM to resolve it again with a different version?
Try using npm shrinkwrap, it is used for transitive lock.
I just recently upgraded to npm#5. I now have a package-lock.json file with everything from package.json. I would expect that, when I run npm install that the dependency versions would be pulled from the lock file to determine what should be installed in my node_modules directory. What's strange is that it actually ends up modifying and rewriting my package-lock.json file.
For example, the lock file had typescript specified to be at version 2.1.6. Then, after the npm install command, the version was changed to 2.4.1. That seems to defeat the whole purpose of a lock file.
What am I missing? How do I get npm to actually respect my lock file?
Update 3: As other answers point out as well, the npm ci command got introduced in npm 5.7.0 as additional way to achieve fast and reproducible builds in the CI context. See the documentation and npm blog for further information.
Update 2: The issue to update and clarify the documentation is GitHub issue #18103.
Update 1: The behaviour that was described below got fixed in npm 5.4.2: the currently intended behaviour is outlined in GitHub issue #17979.
Original answer (pre-5.4.2): The behaviour of package-lock.json was changed in npm 5.1.0 as discussed in issue #16866. The behaviour that you observe is apparently intended by npm as of version 5.1.0.
That means that package.json can override package-lock.json whenever a newer version is found for a dependency in package.json. If you want to pin your dependencies effectively, you now must specify the versions without a prefix, e.g., you need to write them as 1.2.0 instead of ~1.2.0 or ^1.2.0. Then the combination of package.json and package-lock.json will yield reproducible builds. To be clear: package-lock.json alone no longer locks the root level dependencies!
Whether this design decision was good or not is arguable, there is an ongoing discussion resulting from this confusion on GitHub in issue #17979. (In my eyes it is a questionable decision; at least the name lock doesn't hold true any longer.)
One more side note: there is also a restriction for registries that don’t support immutable packages, such as when you pull packages directly from GitHub instead of npmjs.org. See this documentation of package locks for further explanation.
I've found that there will be a new version of npm 5.7.1 with the new command npm ci, that will install from package-lock.json only
The new npm ci command installs from your lock-file ONLY. If your package.json and your lock-file are out of sync then it will report an error.
It works by throwing away your node_modules and recreating it from scratch.
Beyond guaranteeing you that you'll only get what is in your lock-file it's also much faster (2x-10x!) than npm install when you don't start with a node_modules.
As you may take from the name, we expect it to be a big boon to continuous integration environments. We also expect that folks who do production deploys from git tags will see major gains.
Short Answer:
npm install honors package-lock.json only if it satisfies the requirements of package.json.
If it doesn't satisfy those requirements, packages are updated & package-lock is overwritten.
If you want the install to fail instead of overwriting package-lock when this happens, use npm ci.
Here is a scenario that might explain things (Verified with NPM 6.3.0)
You declare a dependency in package.json like:
"depA": "^1.0.0"
Then you do, npm install which will generate a package-lock.json with:
"depA": "1.0.0"
Few days later, a newer minor version of "depA" is released, say "1.1.0", then the following holds true:
npm ci # respects only package-lock.json and installs 1.0.0
npm install # also, respects the package-lock version and keeps 1.0.0 installed
# (i.e. when package-lock.json exists, it overrules package.json)
Next, you manually update your package.json to:
"depA": "^1.1.0"
Then rerun:
npm ci # will try to honor package-lock which says 1.0.0
# but that does not satisfy package.json requirement of "^1.1.0"
# so it would throw an error
npm install # installs "1.1.0" (as required by the updated package.json)
# also rewrites package-lock.json version to "1.1.0"
# (i.e. when package.json is modified, it overrules the package-lock.json)
Use the newly introduced
npm ci
npm ci promises the most benefit to large teams. Giving developers the ability to “sign off” on a package lock promotes more efficient collaboration across large teams, and the ability to install exactly what is in a lockfile has the potential to save tens if not hundreds of developer hours a month, freeing teams up to spend more time building and shipping amazing things.
Introducing npm ci for faster, more reliable builds
Use the npm ci command instead of npm install.
"ci" stands for "clean install".
It will install the project dependencies based on the package-lock.json file instead of the lenient package.json file dependencies.
It will produce identical builds to your team mates and it is also much faster.
You can read more about it in this blog post:
https://blog.npmjs.org/post/171556855892/introducing-npm-ci-for-faster-more-reliable
It appears this issue is fixed in npm v5.4.2
https://github.com/npm/npm/issues/17979
(Scroll down to the last comment in the thread)
Update
Actually fixed in 5.6.0. There was a cross platform bug in 5.4.2 that was causing the issue to still occur.
https://github.com/npm/npm/issues/18712
Update 2
See my answer here:
https://stackoverflow.com/a/53680257/1611058
npm ci is the command you should be using when installing existing projects now.
In the future, you will be able to use a --from-lock-file (or similar) flag to install only from the package-lock.json without modifying it.
This will be useful for CI, etc. environments where reproducible builds are important.
See https://github.com/npm/npm/issues/18286 for tracking of the feature.
Probably you should use something like this
npm ci
Instead of using npm install
if you don't want to change the version of your package.
According to the official documentation, both npm install and npm ci install the dependencies which are needed for the project.
The main difference is, npm install does install the packages taking packge.json as a reference. Where in the case of npm ci, it does install the packages taking package-lock.json as a reference, making sure every time the exact package is installed.
You probably have something like:
"typescript":"~2.1.6"
in your package.json which npm updates to the latest minor version, in your case being 2.4.1
Edit: Question from OP
But that doesn't explain why "npm install" would change the lock file. Isn't the lock file meant to create a reproducible build? If so,
regardless of the semver value, it should still use the same 2.1.6
version.
Answer:
This is intended to lock down your full dependency tree. Let's say typescript v2.4.1 requires widget ~v1.0.0. When you npm install it
grabs widget v1.0.0. Later on your fellow developer (or CI build)
does an npm install and gets typescript v2.4.1 but widget has been
updated to widget v1.0.1. Now your node module are out of sync. This
is what package-lock.json prevents.
Or more generally:
As an example, consider
package A:
{ "name": "A", "version": "0.1.0", "dependencies": {
"B": "<0.1.0" } }
package B:
{ "name": "B", "version": "0.0.1", "dependencies": {
"C": "<0.1.0" } }
and package C:
{ "name": "C", "version": "0.0.1" }
If these are the only versions
of A, B, and C available in the registry, then a normal npm install A
will install:
A#0.1.0 -- B#0.0.1
-- C#0.0.1
However, if B#0.0.2 is published, then a fresh npm install A will install:
A#0.1.0 -- B#0.0.2
-- C#0.0.1 assuming the new version did not modify B's dependencies. Of course, the new version of B could include a new
version of C and any number of new dependencies. If such changes are
undesirable, the author of A could specify a dependency on B#0.0.1.
However, if A's author and B's author are not the same person, there's
no way for A's author to say that he or she does not want to pull in
newly published versions of C when B hasn't changed at all.
OP Question 2: So let me see if I understand correctly. What you're
saying is that the lock file specifies the versions of the secondary
dependencies, but still relies on the fuzzy matching of package.json
to determine the top-level dependencies. Is that accurate?
Answer: No. package-lock locks the entire package tree, including the
root packages described in package.json. If typescript is locked
at 2.4.1 in your package-lock.json, it should remain that way until it is
changed. And lets say tomorrow typescript releases version 2.4.2.
If I checkout your branch and run npm install, npm will respect the
lockfile and install 2.4.1.
More on package-lock.json:
package-lock.json is automatically generated for any operations where npm modifies either the node_modules tree, or package.json. It describes the exact tree that was generated, such that subsequent installs are able to generate identical trees, regardless of intermediate dependency updates.
This file is intended to be committed into source repositories, and serves various purposes:
Describe a single representation of a dependency tree such that teammates, deployments, and continuous integration are guaranteed to install exactly the same dependencies.
Provide a facility for users to "time-travel" to previous states of node_modules without having to commit the directory itself.
To facilitate greater visibility of tree changes through readable source control diffs.
And optimize the installation process by allowing npm to skip repeated metadata resolutions for previously-installed packages.
https://docs.npmjs.com/files/package-lock.json
EDIT: the name "lock" is a tricky one, its NPM trying to catch up with Yarn. It isn't a locked file whatsoever. package.json is a user-fixed file, that once "installed" will generate node_modules folder tree and that tree will then be written in package-lock.json. So you see, its the other way around - dependency versions will be pulled from package.json as always, and package-lock.json should be called package-tree.json
(hope this made my answer clearer, after so many downvotes)
A simplistic answer: package.json have your dependencies as usual, while package-lock.json is "an exact, and more importantly reproducible node_modules tree" (taken from npm docs itself).
As for the tricky name, its NPM trying to catch up with Yarn.
There is an open issue for this on their github page: https://github.com/npm/npm/issues/18712
This issue is most severe when developers are using different operating systems.
Npm install detects any changes made to package.json file to reflect the dependency list accordingly.
Ex. If user added or removed a new dependency, the build will download or remove the dependencies in the local computer. We can compare this to .m2 repository in java where maven keeps track of pom.xml file constantly to update the dependencies.
package-lock.json is a replica of package.json used at run-time by internal processes, only difference is package-lock.json is read-only to user.