I've got a Node app that I'm deploying to Heroku. Their docs say it's best practice to check in your node_modules directory (i.e. don't gitignore it), for faster deploys and for dev/prod parity.
In my package.json, I've got loads of devDependencies (mostly Grunt plugins and all their deps) and a few regular production dependencies like Express. Heroku only needs the production deps. I'd rather not check in all my dev deps, because they come to about 50MB.
Is there some way to have a separate folder for you dev deps, e.g. node_modules_dev? If this was possible, then I could just add node_modules_dev to my .gitignore, and check in the regular production node_modules directory as per Heroku's advice.
Is there any way to do this? Or can you think of another way to do what I'm trying to do?
I use a CI server to build, test, and deploy my files — so I was looking for a similar solution that would prevent me from needing to deploy extra dependencies and/or re-build on Heroku.
After all my tests run, I run npm prune --production, which removes devDependencies from node_modules, and then I push the result to Heroku.
No extra files go to the server, and deployment times are much faster since Heroku avoids having to build all the binaries usually found in Gulp/Grunt plugins.
If you don't mind checking them in anyways, and your only concern is the resulting slug size (i.e.: not your git repo size, or transfer of that repo to Heroku), then simply add the relevant node_modules to .slugignore.
Docs: Ignoring files with .slugignore.
Related
I have a largish nodejs based web app, with both server and client components working together. I am currently deploying the app, but using git pull to take my latest production branch from the server repository. A git post-commit hook runs to do a npm install and a rebuild of the servers .env file, and PM2 is monitoring the various processes (3 web servers) using a change in the .env file to restart them.
node_modules is at the highest level of the project with separate server and client subdirectories. Since this is using http2 on a fast lan, I don't bother compressing the client files with web-pack or the like, although I do use rollup on lit-element and lit-html to sort out the import statements (they are not relative or absolute) that they have embedded in them.
I've just been reading that I should really have been doing an npm ci for my node dependencies, but reading the instructions for that it says it blows away the node_modules directory and starts again (whereas npm install doesn't). Since this is all running on a raspberry pi its not instantaneous.
I am not sure a temporary loss of node_modules should effect a running app too much - after all I believe the modules will all have been cached into memory, but it just might have not and there also a possibility that one of the servers falls over and pm2 restarts it, so I am wondering ....
So what is best practice here. Is it possible for instance to copy package.json, package-lock.json to a special build subdirectory, build the node_modules directory there and then move it back into place. Once built. Or is there a better way?
I am trying to Buddy.works using their CI to upload my themes in in test, staging and production mode.
I have noticed when the files are compiled using Webpack that it is also uploading node_modules to the root folder.
It is taking an age to upload everything. Do I really need those files if the build is finished within Webpack?
Normally working on my project locally all node_modules have to be installed in order to use them. Do I need to include the node_modules in the cloud also?
The standard is to not push node_modules to the SCM(Git, SVN, Mercurial) because they take a lot of space. That's why we have package.json. So on a CI/CD tool like Jenkins, Circle CI or Travis you would specify the build to run npm install --production and it would take care of creating and downloading the necessary dependencies to the node_modules folder.
This makes the build faster too as the first step would be something like,
git clone <repo_url>
If you had pushed your node_modules folder it would take longer to clone due to the extra size.
Does this answer your question?
What is the best practice for deploying a nodejs application?
1) Directly moving the node_modules folders from the development server to production server, so that our same local environment can be created in the production also. Whatever changes made to any of the node modules remotely will not affect our code.
2) Run npm install command in the production server with the help of package.json. Here the problem is, any changes in the node modules will affect our code. I have faced some issues with the loopback module (issue link).
Can anyone help me?
Running npm install in production server cannot be done in certain scenario (lack of compiling tools, restricted internet access, etc...) and also if you have to deploy the same project on multiple machines, can be a waste of cpu, memory and bandwidth.
You should run npm install --production on a machine with the same libraries and node version of the production server, compress node_modules and deploy on production server. You should also keep the package-lock.json file to pinpoint versions.
This approach allows you also to build/test your code using development packages and then pruning the node_modules before the actual deploy.
Moving node_modules folder is overkilled.
Running npm install might break the version dependencies.
The best approach is npm ci. It uses the package_lock file and installs the required dependencies without modify the versions.
npm ci meant for continuous integration projects. LINK
I am an ASP.NET Core developer but I recently started working with Node.js apps. For me this was one of the challenges you mentioned to move the node_modules folder to production. Instead of moving the whole folder to production or only running the npm install command on production server, I figured out and tried a way of bundling my Node.js app using Webpack into a single/multiple bundles, and I just got rid of the mess of managing node_modules folder. It only picks up the required node_modules packages that are being used/referred in my app and bundles up in a single file along with my app code and I deploy that single file to production without moving the entire node_modules folder.
I found this approach useful in my case but please suggest me if this is not the correct way regarding the performance of the app or if any cons of this approach.
Definitely npm install. But you shouldn't do this by your own hand when it comes to deploying your app.
Use the tool for this like PM2.
As for your concern about changes in packages, the short answer is package-lock.json.
My guess is that by asking this question you don't really understand the point of the package.json file.
The package.json file is explicitly intended for this purpose (that, and uploading to the npm registry), the transfer of a node package without having to transfer the sizeable number of dependencies along with it.
I would go as far as to say that one should never manually move the node_modules directory at all.
Definitely use the npm install command on your production server, this is the proper way of doing it. To avoid any changes to the node_modules directory as compared to your local environment, use the package lock file. That should help with minimising changes to the source code in node_modules.
I mean no bad intent by saying this
I understand, in general, the most frequent way Grunt is used on Heroku -- load the buildpack, specify your grunt heroku task(s), and include any Grunt plugins you want to use on the Heroku dyno in your package.json:dependencies.
However, I find this to be a rather poor solution, because it miscommunicates about my app. My Grunt plugins are all more like devDependencies, as I will only run anything with Grunt one time (per deploy). My app doesn't directly depend on them to run, as it's mostly minification and template compilation.
I'm also trying to keep compiled files (e.g. .css and .html files when I'm writing in Sass and Handlebars) out of source control, so pushing them over from Github is possible but definitely not what I want.
Is there a way to do all of the following?
exclude .css and .html files from Git
write stylesheets and markup using Sass and Handlebars
push a basic Express app containing these files to Heroku
have Heroku run a server-relative version of my grunt build task (naming it whatever is necessary is fine) to compile and prepare all of the views and assets before launching the web: process
have Heroku's Grunt task intelligently understand that I just want some pre-launch compilation so it will look at devDependencies and install the relevant ones in a perhaps-temporary kind of way
launch the Express server via the web: process
I'm fine with having to write my own buildpack, or whatever needs to be done to do this (what I perceive to be) the right way.
I created the buildpack heroku-buildpack-webapp-client that supports grunt, compass/sass and installs your devDependencies. Maybe you can use this one or have a look at the bin/compile script to see how it works.
while using grunt node_modules directory is created for every project I create and their is a lot of duplication going on, which I don't like....
I am using grunt-collections now but it is very inefficient, and looking for an alternative solution for this.
I don't know much about node but is it not possible to use grunt modules from the central node_modules directory within nodejs directory????
Edit:
Is their any way to install packages globally...????
yes, it is necessary. First, to deal with grunt version mismatch issues, all the global grunt command really does is delegate to a local grunt dependency installed in your project's node_modules directory.
Second, you really want a local version of your grunt plugins/modules as well. Having everything installed locally to each project allows you to have two separate projects on the same machine that depend on different versions of the same grunt plugin. This would not be possible if everything was installed globally.