As a team practice, I would like to force my teammate to use yarn install/ run but not npm install/ run.
Is it possible to force a package.json 's dependency be installed only via yarn install or package.json's script be run only via yarn run?
If it cannot be done, can I at least get a warning when using npm install?
Again, this is only to align the team practice, so that reduces the possibility of error/ problem produced during dev/ops. Thanks
One of the way I can think of is CI can set a rules to detect if there is new file package-lock.json file being created, make the build fail. The developer will then realized he made a mistake since his build failed.
Alternatively you can also rely on husky pre-commit hook, which essentially runs a command to check if package-lock.json existed everytime developer trying to run git commit.
Related
Node (v14.2.0), Yarn (1.22.4), Windows 10
Context: I have several node projects hosted in a private git repo. I have several cross dependencies between projects, e.g. project C depends on projects A and B, and project D may depend on C and A (perhaps this is my problem?). I generally have my package.json files set up to use the git repos directly, and it works reasonably well for the projects with one or two dependencies.
One of my larger projects has many dependencies on my other projects. Running yarn install on this project gives me this error consistently:
EINVAL: invalid argument, mkdir [some C:\\...Yarn\\Cache\\... directory]
The install ends with that error, and node_modules not being created.
I worked around the issue by removing all (nine) git dependencies from my package.json and then adding them one-by-one and running yarn install each time. No issues, no errors, and in the end I have a fully functioning node project. Great success!
The question here then, is why can I not install (run yarn install) everything at once. I have tried the tricks I found googling - clear the yarn cache, use npm install, run npm adduser or npm login, run as administrator... every combination of those actions resulted in the same EINVAL error.
My guess would be that yarn is trying to do "too many things at once" and its resulting in filesystem errors (trying to mkdir a dir that is locked)... but why is this not documented, and more importantly, why is there not a way to tell yarn to install "one thing at a time"? If there is, and I missed it, I would love to know about it.
Cheers!
Our coorporate network is very closed down, so a normal method of:
npm install cypress#3.4.1
Doesnt work since it is being blocked by a proxy, we need to provide the parameter CYPRESS_INSTALL_BINARYin the following way with the help of cross-env (since we have mainly Microsoft environments here).
cross-env CYPRESS_INSTALL_BINARY='\\localserver\cypress\3.4.1\cypress.zip' npm install cypress#3.4.1
This is easy to do on first install, but the problem is that everyone on the team needs to run this command. And I want it to be possible to just type npm install and they will get all requirements automatically. This is extra obvious when we want to update the cypress package, since the binary url needs to change each time.
I tried to add a preinstall script to my package.json like so:
"scripts": {
...
"preinstall": "cross-env CYPRESS_INSTALL_BINARY='\\localserver\cypress\3.4.1\cypress.zip'",
...
},
But it seems like that environment variable set by cross-env is "gone" after preinstall is finished and install begins, since cypress tries and fails to download from the web. I am okay with it being temporary, but it needs to persist over the install-command. Also seen solutions with .env files but none of those have support for the install step as far as I can see.
My current solution is to run the entire cypress installation in the preinstall step, and it works but seems uneccessary to run a double install each time.
So, what I am asking for, is a way to let a developer just run the following commands on a brand new computer and be done.
git clone ...
cd ...
npm install
How can I do that?
Same situation on my side, except that I want to prevent the installation of cypress on the local machine.
Solved it by adding a .npmrc into the root of the project and adding to version control.
Contents of .npmrc:
CYPRESS_INSTALL_BINARY=0
Since the environment variable is used on install time, the solution with cross-env was not possible, since one cannot be sure that cross-env has been already installed.
Let me know whether it helped or you have already another solution.
I ended up creating a tool that takes care of supplying the correct binary, depending on environment:
https://github.com/tomasbjerre/dictator-cypress
I have it internally at my company. When we need a new version of Cypress, we release a new version of this tool internally.
I add the tool as a preinstall script:
...
"scripts": {
"preinstall": "npx dictator-cypress#0.0.28",
...
So that it runs when someone does npm install:
...
Copy linux cypress to cypress.zip
Applying: copy linux-x64/cypress.zip to .
.npmrc should have reference to cypress binary
Up to date: .npmrc should have lines
.gitignore should include the copied zip
Up to date: .gitignore should have lines
...
I am a complete new user of Yeoman. I am trying to learn the work flow of this Yeoman. So far I understand some of it but I am stuck in a very basic process.
Here is what I mean:
When I push all my directory to my repository it takes necessary files and folders. I used Yeoman .gitignor so that it ignores unwanted large files like bower_component
Other them member can clone my repository and run : bower install && npm install
to have all bower and npm components. and then run grunt build to make dist folder.
Upto now all works fine but my question is do they need to do same processes every time when they pull any new item ?
like I made some changes and pushed to my repository so what they have to do to get my new changes ? They need to do bower install && npm install again ?
Thanks in advance and sorry for really poor English
You only need to run bower install and npm install again if any dependencies changed! Otherwise, it will make no difference.
The git pull command won't touch directories that are not versioned (that is, that were ignored).
Hey, perfectly understandable english. Do continue participating!
I am trying to speed up the npm install during the build process phase. My package.json has the list of packages pretty much with locked revisions in it. I've also configured the cache directory using the command
npm config set cache /var/tmp/npm-cache --global
However, on trying to install using npm install -g --cache, I find that this step isn't reducing the time to install by just loading the packages from cache as I would expect. In fact, I doubt if it's even using the local cache to look up packages first.
Proposing two more modern approches:
1) npm ci
Use npm ci, which is available from npm version 5.7.0 (although I recommend 5.7.1 and upwards because of the broken release) - this requires package-lock.json to be present and it skips building your dependency tree off of your package.json file, respecting the already resolved dependency URLs in your lock file.
A very quick
boost for your CI/CD envs (our build time was cut down to a quarter of the original!) and/or to make sure all your developers sit on the same versions of dependencies during development (without having to hard-code strict versions in your package.json file).
Note however that npm ci removes the node_modules/ directory before installing, so it won't benefit from any caching strategies.
2) npm i --prefer-offline
Use the --prefer-offline flag with your regular npm install / npm i. With this approach, you need to make sure you've cached your node_modules/ directory between builds (in a CI/CD environment). If it fails to find packages locally with the specific version, it falls back to the network safely.
You can also add --no-audit --progress=false to reduce pre-install checks and remove the progress bar (latter is only a very slight improvement)
For pure npm solution, you may try
npm install --prefer-offline --no-audit --progress=false
Prefer offline may not be useful for the first run.
As suggested by #Daniel Serodio
You could also include your node_modules folder inside your repository but you should probably zip it first than add to repo, and while installing you can unzip it and just
npm rebuild
(which works cross platform) it is quite fast.
This would also give you the benefit of full control over all your dependencies.
Also you can set the process flag to false to increase your speed by 2x.
npm set progress=false
Read source for more info
Update:
You can also use pnpm for this
npm i -g pnpm
This basically use local cached modules (i have heard its better then YARN)
It's better to install pnpm package using the following command:
npm i -g pnpm
pnpm uses hard links and symlinks to save one version of a module only ever once on a disk. When using npm or Yarn for example, if you have 100 projects using the same version of lodash, you will have 100 copies of lodash on disk. With pnpm, lodash will be saved in a single place on the disk and a hard link will put it into the node_modules where it should be installed.
As an example I can mention that whenever you want to install the dependencies of package.json file, what you should do is simply that enter the pnpm i and it handles the other things by itself.
UPDATE: The original answer is from 2014. I wouldnt recommend checking in node_modules, as there are definitly better options around speeding up the install especially for a ci pipeline, eg. npm ci --only=production
You could also include your node_modules folder inside your repository (you are probably using git), and just npm rebuild (which works cross platform) on build/deploy processes, and is pretty fast.
This would also give you the benefit of full control over all your dependencies (I know that's what shrinkwrap usually should be used for)
Edit:
Also you can set the progress flag to false to increase your speed by at least 20%. This works only with npm#v3.x.x, and there will be hopefully fixes for that soon (see second link)
npm set progress=false
Tweet about finding
Github Issue Cause identification
As very modern solution you can start to use Docker.
Docker allows you virtualize and pre-define as image the current state of your code, including installed npm-modules and other goodies.
Once the docker image for your infrastructure/env is built locally, or retrieved from remote repository, it will be stored on the host machine, and you can spin server in seconds.
Another benefit of it is that you use same virtualized code infrastructure on any machine where you deploy your code.
Docker speeds up install/deployment processes and is widely used technology.
To start using docker is enough to (all the snippets are just mock/example for pre-setup and are not by any means most robust/elegant solution) :
1) Install docker and docker-compose using manuals and get some basic understanding of it at docker.com
2) Write Dockerfile file in root of your application
FROM node:6.9.5
RUN mkdir /usr/local/app
WORKDIR /usr/local/app
COPY package.json package.json
RUN npm install
3) create docker-compose.yml in the root of your project with such content:
version: "2"
server:
hostname: server
container_name: server
image: server
build: .
command: sh -c 'NODE_ENV=development PORT=8080 node app.js'
ports:
- "8080:8080"
volumes: #list of folders and files to use
- ${PWD}/server:/usr/local/server
- ${PWD}/app.js:/usr/local/app.js
4) To start server you will need to docker-compose up -d. To see the logs docker-compose logs -f server. If you will restart your server it will do it in seconds once it built the image already at once.
Then it will cache build layers locally so next run will take only few seconds.
I know this might be bit of a robust solution, but I am sure it is have most potential/flexibility and is widely used in industry. And while it requires some learning for anyone who did not use Docker before, in my humble oppinion, it is the best one for your problem.
Nothing helped me more than disabling antivirus (Windows Defender in my case) I got from 2:30 to 1 minute.
With npm-cache package I got to ~30 secs.
I tried to use yarn, which is very fast, but was randomly failing in my case.
We have been trying to solve this problem to speed up our deployments.
We have settled on using pac, which follows the principles in the other answers. It zips the npm modules and includues them in your repo so you don't have a million files in your commits and code reviews and you can just unzip/rebuild for the target machine.
https://www.npmjs.com/package/pac
Running git push heroku master always triggers a step that prompts:
Installing dependencies with npm
This step loads and reinstalls all of the dependencies again even it already exists. This is very time consuming and I want to skip this step sometimes when I deploy that I know the dependencies are the same.
Is there any command or options that do this?
Its been a long time since you asked this question, now the Heroku buildpack caches node_modules, so install times will be much faster.
If however you still want to block npm install, here is one solution.
As of when I write this, the default Heroku build pack does not allow skipping npm install entirely. You can see in the dependencies.sh file, this line will always run:
npm install --unsafe-perm --userconfig $build_dir/.npmrc 2>&1
However, if you create a file called .npmrc in your project folder with the following contents:
dry-run
This will cause npm install to not modify your existing node_modules directory.
Note that this change will also apply to the npm prune command that Heroku runs, but WILL NOT apply to the npm rebuild command (which is probably fine).
try to remove
node_modules
for example
from you .gitignore
Simplest ways I've found are
heroku apps:rename newTemporaryName
then
heroku apps:rename originalName
or change the NODE_ENV and get it back to previous again.
heroku config:set NODE_ENV=dev
then
heroku config:set NODE_ENV=production
There are probably other, similar hacks but these should be sufficient.