I'm working on an open source project which is currently using lerna to help manage a single repository with multiple packages. So far, so good, except that every now and then something breaks because we don't currently shrinkwrap anything. I've been trying to do some monorepo version of shrinkwrapping so that we're not at the mercy of all our dependencies (it's amazing how often everything goes wrong in the middle of a release!) but have hit a bit of a wall and wondered if I'm even on the right track.
I was originally hoping to use npm shrinkwrap which I was familiar with from previous projects. Unfortunately lerna doesn't appear to support shrinkwrap.
Plan B was to make use of yarn which, after some initial difficulties, seemed to be going ok after switching to using yarn workspaces- at least I think yarn install --frozen-lockfile was doing what I wanted.
Unfortunately, other than dependency locking, yarn doesn't seem to be helping- everything worked with npm and lerna, but lerna and yarn workspaces seems to be causing issues resolving modules (even resolving something in the same directory which is baffling).
Maybe switching to yarn is overkill anyway so I started wondering if more recent versions of npm and package-lock.json would be a better idea. Unfortunately that looks like it would need some work arounds with lerna at which point I'm beginning to wonder how much lerna is really adding. Maybe dropping lerna would help?
So, tl;dr, does anyone have a good way of locking down module dependencies in a monorepo?
I'd suggest simply using exact versioning; so in your package.json files where there are version numbers for dependencies like ^3.4.2, changing it to 3.4.2. The ^ (or ~) before the number suggest a version range. You can get that to happen with the save exact config option: --save-exact flag or by placing save-exact=true in a .npmrc file in the repo. lerna add also supports an exact option.
Hope that helps!
yarn is a production ready package manager that natively supports monorepos :)
When using yarn workspaces, there is no need to use lerna as a monorepo manager as well at the same time.
You can use other features of lerna if you want, but there is no reason to use lerna for installing monorepos (which are already uses yarn).
If there is a specific errors when installing/managing monorepo using yarn, please add them to the question.
Notes:
--frozen-lockfile doesn't do anything in yarn monorepo. yarn has a open issue on that which I think won't resolve soon.
Related
I am now in charge of an app whose dependencies were installed using yarn. I am more familiar with NPM. Is switching over as easy as deleting the yarn.lock file and installing NPM to the project? The app hasn't been updated in a year or two so I'm trying to update everything.
Or maybe a better question is, can I install both NPM and yarn globally but pick and choose which one I use for what app? Will it cause issues if they are both installed globally on machine?
Thanks in advance
You can definitely have Yarn and NPM installed on the same machine without issue. NPM generally comes packaged with Node, so most people using Yarn will also have NPM installed, wether they use it or not.
In terms of switching a project from Yarn to NPM, it's a pretty straightforward process, like you described: remove yarn.lock and remove the existing node_modules directory just to avoid any issues.
The subtle issue here is that the yarn.lock will be the current source of truth for exactly which versions of each dependency (and sub-dependencies) is installed. So by removing the yarn.lock your package.json will now become the (incomplete) source of truth which will likely result in some dependencies being upgraded when you perform your first npm install -- then your package-lock.json will become the new strict source of truth.
Given you're planning on updating everything anyway, then this likely isn't going to be an issue, but it's worth keeping in mind as you're likely to see some minor dependency changes.
The npm-shrinkwrap ensures that installed packages also have the same version of dependencies that was used at the moment of publishing to the registry (These versions are stated in the npm-shrinkwrap which is then used on installation).
I am currently using yarn (and lerna for publishing) and working on a monorepo project with workspaces. Now I would like to have each package in the monorepo have same guarantees provided by npm-shrinkwrap.
One shortcoming of the npm-shrinkwrap is that it does not support workspaces. Hence I cannot use npm-shrinkwrap since it is only created at the root and does not influence how individual packages in the monorepo get installed.
Since I am using yarn, I was wondering if there is an alternative to npm-shrinkwrap in yarn?
Or maybe a better question is, using yarn/lerna, how do I lock version dependencies for publication, such that when my packages in the mono-repo are downloaded, they are downloaded with the exact versions of dependencies (and transitive dependencies) that was specified at the point of publication?
I found the following in the yarn docs (classic yarn 1.x)
If you are using an npm-shrinkwrap.json file right now, be aware that you may end up with a different set of dependencies. Yarn does not support npm shrinkwrap files as they don’t have enough information in them to power Yarn’s more deterministic algorithm. If you are using a shrinkwrap file it may be easier to convert everyone working on the project to use Yarn at the same time. Simply remove your existing npm-shrinkwrap.json file and check in the newly created yarn.lock file.
From here:
https://classic.yarnpkg.com/en/docs/migrating-from-npm
I'm building a microservice app in a monorepo containing a bunch of microservices and a commons package. This commons package is never published to npm. (packages are managed with yarn workspaces)
Using parcel, the commons package is bundled into the production code, so I don't need to install it at run time.
Each microservice runs in its own docker container. So, when I build the docker container, Ideally, I'd want to ignore this "commons" dependency and install all the other ones. AFAIK, the only way to do this is to place the "commons" package in devDependencies.
However, it seems that even if I add it only to devDependencies and run npm i --only=production, npm still tries to resolve the package and still throws an ETARGET error.
Is there some way to completely ignore the devDependencies? My only other Idea is to write a script that removes the devDependencies field from the package.json before running npm install, but I wanted to ask here first to make sure I'm not missing anything.
There is a GitHub issue on the npm/cli repo tracking this issue here (#4967), where this behaviour is categorized as a bug.
So to answer your question, as far as the current status of the GitHub issue indicates, the intended behaviour is that devDependencies don't get attempted to be resolved in --production mode, and you shouldn't need to do anything extra to get this behaviour once the fix is made. I don't think you are missing anything.
The workaround you have thought of sounds reasonable to me.
On the GitHub issue, you can indicate "me too" with a thumbs up reaction (please don't spam the comments with "me too" comments).
To install packages only at production without devDependencies,
npm install --production
Docs about npm install is here.
when running npm install --no-optional, it takes around 3 mins every time to complete. It installs ~ 200MB of files. I would like to speed the build process, but I cannot find any ways to really speed it up.
Doesn't npm install by default cache dependencies (like any other decent tool e.g. maven, sbt or nuget) by default? If yes, shouldn't it be much faster than that? If no, then WHY and how to work around that?
I found npm-cache package, but it seems to .tar all the dependencies and when neither of them changes, npm-cache will reuse the tar file. The downside of this is that, whenever a small change in dependencies occurs, it won't be able to reuse the cache (from what I understand).
Are there any nice resources on why this is slow and how to speed it up and how caching works with npm in general? Other tools that I have used (sbt, maven, nuget) are much faster, therefore my expectations are high for npm as well.
Another option I looked into is npm install -g, but it seems not to solve any problems here, as it is meant to be used for installing some cli tools like grunt, npm-cache and etc., as it adds them to a path. So this definitely doesn't solve the problem.
npm -v: 4.0.5
node -v: 6.8.1
The problem with node was that coming from sbt background, where sbt uses a local ivy cache to cache dependencies, I expect the same behaviour from Node. So at least up to V 5.0, Node didn't have a proper dependency caching mechanism, so you basically needed to redownload all of the dependencies every time you do a node install with a clean node_modules folder.
There were some tool developed to work around that, but none of them were satisfactory.
But it seems that this might have been fixed in Node V 5.0 with some caching strategy, therefore if you have a similar issue, please take a look at the changes for the 5th version.
It's better to install pnpm package using the following command:
npm i -g pnpm
pnpm uses hard links and symlinks to save one version of a module only ever once on a disk. When using npm or Yarn for example, if you have 100 projects using the same version of lodash, you will have 100 copies of lodash on disk. With pnpm, lodash will be saved in a single place on the disk and a hard link will put it into the node_modules where it should be installed.
As an example I can mention that whenever you want to install the dependencies of package.json file, what you should do is simply that enter the pnpm i and it handles the other things by itself. Its speed is faster than the npm, because it will reuse the dependencies that you've installed them before!
I work in a team of about 20 other developers. All of our projects utilize npm packages and currently all of our developers are running npm to manage those packages. I'm very curious about Yarn and have it currently installed on my machine. However I'm nervous to actually use it to install packages in case it screws up a project for other developers.
My question is can one developer utilize Yarn on a project while other devs are using npm in the same repo? From what I've read, Yarn uses the same package.json file to get its dependencies. We ignore the node_modules in our repo, however we use npm shrinkwrap to lock dependencies. I know Yarn has a Yarn.lock file and that is where my concern lies. Has anyone attempted to run Yarn independently from their team and what issues have you run into?
Yarn doesn't read npm-shrinkwrap.json. It generates its own yarn.lock. While only you use yarn, the project in your environment might have dependencies version different then that your teammates have. However it's safe to try yarn in your own environment since it doesn't overwrite shrinkwrap file and won't impact other developers.