I've started using NodeJS and Grunt to do things like run LESS, Concat JS library files and Uglify them, etc. I have a total of only 5 modules installed: less, concat, watch, copy, and uglify.
My node_modules directory contains a whopping 3,500 files!
I am using this setup in 4 related projects in my SVN repository. Since Grunt doesn't support any kind of "global" installation, I have to install these modules separately for every project.
That's added 14,000 files to my repository, and and added a very, very large amount of time to me check-out process.
I can't imagine that everyone using Grunt is quietly ignoring and tolerating this kind of repository bloat, but I can't see how to reduce the module size.
Does anyone know what am I doing wrong?? How do I get the functionality of less, concat, watch, copy, and uglify, but without the ridiculous file count that I'm seeing today?
i dont know of a way to get the functionality without having the files on disk.
but the usual solution to avoid bloating your repo is to rely on npm install to get all those files. add node_modules/ and maybe bower_components/ to your .gitignore to make sure you dont commit them by accident.
npm install --save-dev grunt-contrib-copy
git add package.json; git commit -m 'Add grunt copy dep'; git push
cd ~
git checkout https://github.com/myName/myGruntProject; cd myGruntProject
npm install
grunt
Related
I have a git repository setup in git lab. Right now each time I change branch i should do:
npm install && composer install && cp .env.example .env && artisan generate key
Cause I lose .env , node_modules and composer modules. and it takes long time reinstalling them. cause I cant run it and test the branch if I dont have node_modules and other stuff installed
I wonder if Im doing something wrong or if there is a way to make it happen.
I have done lots of search but no luck.
Thanks in advance
Are you sure the files / directories you are talking about are ignored by git (they are in your .gitignore file)? If that's not the case, here is the answer to your question:
Since they are bound to the environment you are working on, they should not be touched by git by any means. That's why you should not lose them if you checkout on another branch.
Only the composer.lock, the package-lock.json and the .env.example should be versioned. Then, when you clone the repo from GitLab, you do a npm install, a composer install, you copy the .env.example etc... in order to setup your dependencies, but the dependencies directories (eg. node_modules) should not come from your repository.
Then after a while, let's imagine you want to update your Composer dependencies. You'll do a composer update. Your composer.lock file will be updated and will be committed to your repository.
Then, if somebody on another computer pulls your changes, he will only pull the newly updated composer.lock file. He will then make a composer install, which will install (or update if he already had installed them before) the dependencies from the composer.lock into his vendor folder.
I hope it helps you, feel free to ask more details in the comments :)
I have run
git init
npm install --save-dev webpack
Then, there is a package.json and a node_modules directory in the root directory.
I want to add the node_modules directory into git repo.
After I run
git add .
There is no response any more. And I found the git.exe process occupy more and more memory usage , but there is not any response after several hours.
What is wrong with it?
Is it caused by pretty much files to be added into repo by git?
How can I debug what happens in detail when that command is executed?
Everything works fine if I just git add some other files/folder which is not node_modules.
Have you git init the repo folder? If yes, then you can run git status. There you should see any new or changed folder.
If node_modules is not mentioned, I guess you have a .gitignore file in your project folder. If you want to git add the node modules folder you will need to remove node_modules/ from .gitignore.
npm install --save-dev webpack will just add one more dependency in package.json and install webpack in your node local repo. This is not a git related command.
Check also this question: Git - Ignore node_modules folder everywhere
Hopfully this will help,
I figured out myself.
This is caused by pretty much symlinks in the node_modules directory. If i evaluated the git-bash as administrator, the new installed node_module symlinks can be created as linux. And then when git add works well. Maybe, if without evaluated git-bash, git-bash can not create symlinks which cause pretty much files need to be git added, even worse maybe a circular reference between symblink and node module staff (not sure), that cause memory consumption of git increase always and no response.
Hope it can help others if met the same problem on windows.
I am building a simple node app and using Bootstrap to style my frontend. I want to deploy the app to Heroku by loading it from the GitHub repository. I don't want to put any compiled CSS/JS files in the repository, which means that they need to be compiled on Heroku after the source is pulled from GitHub.
The problem: Bootstrap's default package.json has its build dependencies in devDependencies, so the dependencies will not be installed on Heroku, which runs npm install in production mode.
Specifically, the problem comes up in my postinstall script, which consists of cd node_modules/bootstrap && npm install && ../.bin/grunt dist. My own npm install command does not end up installing anything because the overarching npm install --production is ignoring Bootstrap's devDependencies.
What is the best workaround to get Bootstrap's dependencies to be installed locally? I'm open to changing my workflow as long as (1) I don't have to put compiled files in my source repository and (2) I can still run the main npm install in production mode.
EDIT: Unless someone can think of a better solution, I think my options at this point are as follows:
Build a custom Github fork of Bootstrap that has its dependencies in dependencies instead of devDependencies
Build a custom Heroku buildpack that somehow leaves Bootstrap's dependencies intact
Abandon principle and compile Bootstrap locally, and put the compiled files in my source
Just use Heroku in development mode (obviously not a good idea)
I think I'm going to try for option 1, and go for option 3 if that fails. I'd still be interested in hearing any other ideas that people come up with.
I would look into using postinstall. The official documentation is on Heroku here.
Since you're doing some build steps at deploy time, you need some devDependencies (bootstrap sources in your case), but heroku will only install in production mode, like you said.
When connected with the heroku toolbelt client, run :
heroku config:set NPM_CONFIG_PRODUCTION=false
You'll have your devDependencies and you'll be able to build.
You only have to do that once (no need to do that each time you deploy).
PS: I don't know about your project, but maybe you need to put the NODE_ENV to production, here is the command:
heroku config:set NODE_ENV=production
You don't need yo put back the NPM_CONFIG_PRODUCTION flag back to true, this is the purpose of making the build on heroku.
But if you prefer, here is the workflow you could use (you said you were opened to change workflow):
ignore dist folder from your source repo (it should already be ignored)
make dist folder a git repo & add the git remote of heroku in repo of the dist folder
cd dist
git init
git remote add origin git#heroku.com:project-name.git
git pull origin master
make sure your build routine doesn't remove this /dist/.git folder - you can use the following glob (if you're using rimraf or something like it with grunt/gulp or whatever) : ['/dist/**/*','/dist/!.git/**/*']
This way, your workflow will be:
build (with your tool)
cd dist
git add .
git commit -m "new version bundled"
git push origin master there, you are in dist folder where origin is the remote for your heroku
You might do the same kind of things with github pages ...
I am a complete new user of Yeoman. I am trying to learn the work flow of this Yeoman. So far I understand some of it but I am stuck in a very basic process.
Here is what I mean:
When I push all my directory to my repository it takes necessary files and folders. I used Yeoman .gitignor so that it ignores unwanted large files like bower_component
Other them member can clone my repository and run : bower install && npm install
to have all bower and npm components. and then run grunt build to make dist folder.
Upto now all works fine but my question is do they need to do same processes every time when they pull any new item ?
like I made some changes and pushed to my repository so what they have to do to get my new changes ? They need to do bower install && npm install again ?
Thanks in advance and sorry for really poor English
You only need to run bower install and npm install again if any dependencies changed! Otherwise, it will make no difference.
The git pull command won't touch directories that are not versioned (that is, that were ignored).
Hey, perfectly understandable english. Do continue participating!
I am trying to speed up the npm install during the build process phase. My package.json has the list of packages pretty much with locked revisions in it. I've also configured the cache directory using the command
npm config set cache /var/tmp/npm-cache --global
However, on trying to install using npm install -g --cache, I find that this step isn't reducing the time to install by just loading the packages from cache as I would expect. In fact, I doubt if it's even using the local cache to look up packages first.
Proposing two more modern approches:
1) npm ci
Use npm ci, which is available from npm version 5.7.0 (although I recommend 5.7.1 and upwards because of the broken release) - this requires package-lock.json to be present and it skips building your dependency tree off of your package.json file, respecting the already resolved dependency URLs in your lock file.
A very quick
boost for your CI/CD envs (our build time was cut down to a quarter of the original!) and/or to make sure all your developers sit on the same versions of dependencies during development (without having to hard-code strict versions in your package.json file).
Note however that npm ci removes the node_modules/ directory before installing, so it won't benefit from any caching strategies.
2) npm i --prefer-offline
Use the --prefer-offline flag with your regular npm install / npm i. With this approach, you need to make sure you've cached your node_modules/ directory between builds (in a CI/CD environment). If it fails to find packages locally with the specific version, it falls back to the network safely.
You can also add --no-audit --progress=false to reduce pre-install checks and remove the progress bar (latter is only a very slight improvement)
For pure npm solution, you may try
npm install --prefer-offline --no-audit --progress=false
Prefer offline may not be useful for the first run.
As suggested by #Daniel Serodio
You could also include your node_modules folder inside your repository but you should probably zip it first than add to repo, and while installing you can unzip it and just
npm rebuild
(which works cross platform) it is quite fast.
This would also give you the benefit of full control over all your dependencies.
Also you can set the process flag to false to increase your speed by 2x.
npm set progress=false
Read source for more info
Update:
You can also use pnpm for this
npm i -g pnpm
This basically use local cached modules (i have heard its better then YARN)
It's better to install pnpm package using the following command:
npm i -g pnpm
pnpm uses hard links and symlinks to save one version of a module only ever once on a disk. When using npm or Yarn for example, if you have 100 projects using the same version of lodash, you will have 100 copies of lodash on disk. With pnpm, lodash will be saved in a single place on the disk and a hard link will put it into the node_modules where it should be installed.
As an example I can mention that whenever you want to install the dependencies of package.json file, what you should do is simply that enter the pnpm i and it handles the other things by itself.
UPDATE: The original answer is from 2014. I wouldnt recommend checking in node_modules, as there are definitly better options around speeding up the install especially for a ci pipeline, eg. npm ci --only=production
You could also include your node_modules folder inside your repository (you are probably using git), and just npm rebuild (which works cross platform) on build/deploy processes, and is pretty fast.
This would also give you the benefit of full control over all your dependencies (I know that's what shrinkwrap usually should be used for)
Edit:
Also you can set the progress flag to false to increase your speed by at least 20%. This works only with npm#v3.x.x, and there will be hopefully fixes for that soon (see second link)
npm set progress=false
Tweet about finding
Github Issue Cause identification
As very modern solution you can start to use Docker.
Docker allows you virtualize and pre-define as image the current state of your code, including installed npm-modules and other goodies.
Once the docker image for your infrastructure/env is built locally, or retrieved from remote repository, it will be stored on the host machine, and you can spin server in seconds.
Another benefit of it is that you use same virtualized code infrastructure on any machine where you deploy your code.
Docker speeds up install/deployment processes and is widely used technology.
To start using docker is enough to (all the snippets are just mock/example for pre-setup and are not by any means most robust/elegant solution) :
1) Install docker and docker-compose using manuals and get some basic understanding of it at docker.com
2) Write Dockerfile file in root of your application
FROM node:6.9.5
RUN mkdir /usr/local/app
WORKDIR /usr/local/app
COPY package.json package.json
RUN npm install
3) create docker-compose.yml in the root of your project with such content:
version: "2"
server:
hostname: server
container_name: server
image: server
build: .
command: sh -c 'NODE_ENV=development PORT=8080 node app.js'
ports:
- "8080:8080"
volumes: #list of folders and files to use
- ${PWD}/server:/usr/local/server
- ${PWD}/app.js:/usr/local/app.js
4) To start server you will need to docker-compose up -d. To see the logs docker-compose logs -f server. If you will restart your server it will do it in seconds once it built the image already at once.
Then it will cache build layers locally so next run will take only few seconds.
I know this might be bit of a robust solution, but I am sure it is have most potential/flexibility and is widely used in industry. And while it requires some learning for anyone who did not use Docker before, in my humble oppinion, it is the best one for your problem.
Nothing helped me more than disabling antivirus (Windows Defender in my case) I got from 2:30 to 1 minute.
With npm-cache package I got to ~30 secs.
I tried to use yarn, which is very fast, but was randomly failing in my case.
We have been trying to solve this problem to speed up our deployments.
We have settled on using pac, which follows the principles in the other answers. It zips the npm modules and includues them in your repo so you don't have a million files in your commits and code reviews and you can just unzip/rebuild for the target machine.
https://www.npmjs.com/package/pac