Most/all examples I see online usually copy package.json into the image and then run npm install within the image. Is there a deal breaker reason for not running npm install from outside on the build server and then just copying everything including the node_modules/ folder?
My main motivation for doing this is that, we are using a private npm registry, with security, and running npm from within an image, we would need to figure out how to securely embed credentials. Also, we are using yarn, and we could just leverage the yarn cache across projects if yarn runs on the build server. I suppose there's workarounds for these, but running yarn/npm from the build server where everything is already set up seems very convenient.
thanks
Public Dockerfiles out there are trying to provide generalized solution.
Having dependencies coded in package.json makes it possible to share only one Dockerfile and not depend on anything not public available.
But at runtime Docker does not care how files got to container. So this is up to you, how you push all needed files to your container.
P.S. Consider layering. If you copy stuff under node_modules/, do it in one step, by that only one layer is used.
Related
We run multiple nodejs services using various version of nodejs. At the moment we handle this with nvm on our dev machines. I want to simplify the dev environment by dockerizing these services.
I've been looking for info, reading blogs and the process seems pretty straight forward, except when it comes to how to handle node_modules. Some recommend a a separate, external volume for node_modules. Others say just create a bind mount of the src folder into the image (including node_modules). I've also seen examples where they create a bind mount of the src folder and separate bind mount for node_modules. What is the correct way to do this?
Another thing i'm wondering is what is the proper way to handle adding/removing/updating packages? I suppose, since node_modules is mounted, i could just use npm in the src folder and the changes would be reflected in the container. But that would defeat the purpose since i would still need to use nvm locally in order for the packages to be compiled for the correct nodejs version right?
Another idea i had was to just edit the package.json and then rebuild the image, but then the changes wouldn't be reflected in the package.lock.json and that would cause problems i think.
So i guess i'm left with shelling into the container with docker exec and then running npm? This seems a bit annoying.
I would very much appreciate some advice. Thanks!
What is the best practice for deploying a nodejs application?
1) Directly moving the node_modules folders from the development server to production server, so that our same local environment can be created in the production also. Whatever changes made to any of the node modules remotely will not affect our code.
2) Run npm install command in the production server with the help of package.json. Here the problem is, any changes in the node modules will affect our code. I have faced some issues with the loopback module (issue link).
Can anyone help me?
Running npm install in production server cannot be done in certain scenario (lack of compiling tools, restricted internet access, etc...) and also if you have to deploy the same project on multiple machines, can be a waste of cpu, memory and bandwidth.
You should run npm install --production on a machine with the same libraries and node version of the production server, compress node_modules and deploy on production server. You should also keep the package-lock.json file to pinpoint versions.
This approach allows you also to build/test your code using development packages and then pruning the node_modules before the actual deploy.
Moving node_modules folder is overkilled.
Running npm install might break the version dependencies.
The best approach is npm ci. It uses the package_lock file and installs the required dependencies without modify the versions.
npm ci meant for continuous integration projects. LINK
I am an ASP.NET Core developer but I recently started working with Node.js apps. For me this was one of the challenges you mentioned to move the node_modules folder to production. Instead of moving the whole folder to production or only running the npm install command on production server, I figured out and tried a way of bundling my Node.js app using Webpack into a single/multiple bundles, and I just got rid of the mess of managing node_modules folder. It only picks up the required node_modules packages that are being used/referred in my app and bundles up in a single file along with my app code and I deploy that single file to production without moving the entire node_modules folder.
I found this approach useful in my case but please suggest me if this is not the correct way regarding the performance of the app or if any cons of this approach.
Definitely npm install. But you shouldn't do this by your own hand when it comes to deploying your app.
Use the tool for this like PM2.
As for your concern about changes in packages, the short answer is package-lock.json.
My guess is that by asking this question you don't really understand the point of the package.json file.
The package.json file is explicitly intended for this purpose (that, and uploading to the npm registry), the transfer of a node package without having to transfer the sizeable number of dependencies along with it.
I would go as far as to say that one should never manually move the node_modules directory at all.
Definitely use the npm install command on your production server, this is the proper way of doing it. To avoid any changes to the node_modules directory as compared to your local environment, use the package lock file. That should help with minimising changes to the source code in node_modules.
I mean no bad intent by saying this
I have a repo that has the backend and frontend (create-react-app) in two separate folders. For the build command, I have something like cd frontend && npm run build and for the publish directory, I have something like frontend/build, but this is not working.
disclaimer: I work for Netlify.
If you were to clone a new copy (no node modules installed in the project, for instance) of your project on a fresh laptop with nothing else except node and npm installed there, how would you build it? Imagine netlify's build process like that. So you're missing at least an "npm install" step in there :)
Anything else missing, like globally installed npm packages? Need to specify them in package.json so that Netlify's build network knows to grab them for you. Ruby gems? Better have a Gemfile in your repo!
Netlify tries to npm install (and bundle install) automatically for you, assuming there is a package.json either in the root of your repository (I'm guessing yours is in frontend/ ?) OR if you set the "base" parameter so that we start our build in the base directory. This is probably a good pattern for you, to set "base" to frontend, and then set your publish directory to build.
You can specify that base parameter in netlify.toml something like this:
[build]
base = "frontend"
Note that netlify.toml must reside in the root of your repository.
For more details on how Netlify builds, check out the following articles:
Overview of how our build network works. This article also shows how you can download our build image to test locally.
Settings that affect our build environment. Useful for telling us about what node version to use, for instance.
Some frequently experienced problems
If after some reading and experimenting, you still can't figure things out, ping the helpdesk.
The top answer is correct ^. For anyone looking to simply change the base directory (lets say there is only one npm install/start) you need to change the BASE DIRECTORY, which you will find in the build settings. Simply go to: site-settings -> build & deploy - and you will see it where I pointed in the picture attacted. Hopefully that helps someone in need of this. see here
I am building a NodeJS application using MongoDB as the database. I am thinking that it will make more sense in terms of portability across different platforms and also versioning and comparison to have the application deployed in Docker. Going through various recommendations on internet, here are my specific questions :
(a) Do I copy my application code (nodejs) within Docker? Or do I keep Source code on the host machine and have the code base available to Docker using Volumes? (Just for experimenting, I had docker file instruction pulling the code from repository within the image directly. It works, but is it a good practice, or should I pull the code outside the docker container and make it available to docker container using Volumes / copy the code)?
(b) When I install all my application dependencies, my node_module size explodes to almost 250 MB. So would you recommend that run npm install (for dependencies) as Docker step, which will increase the size of my image ? Or is there any other alternative that you can recommend?
(c) For connecting to the database, what will be the recommendation? Would you recommend, using another docker container with MongoDB image and define the dependency between the web and the db using docker? Along with that have configurable runtime property such that app in different environments (PROD, STAGE, DEV) can have the ability to connect to different database (mongodb).
Thoughts / suggestions greatly appreciated. I am sure, I may be asking questions which all of you may have run into at some point in time and have adopted different approaches, with pros and cons.
Do I copy my application code (nodejs) within Docker? Or do I keep
Source code on the host machine and have the code base available to
Docker using Volumes?
You should have the nodejs code inside the container. Keeping the source code on your machine will make your image not portable since if you switch to another machine, you need to copy the code there.
You can also pull the code directly into the container if you have git installed inside the container. But remember to remove the .git folder to have a smaller image.
When I install all my application dependencies, my node_module size
explodes to almost 250 MB. So would you recommend that run npm install
(for dependencies) as Docker step, which will increase the size of my
image ? Or is there any other alternative that you can recommend?
This is node pulling over all the internet. You have to install you dependencies. However, you should run npm cache clean --force after the install to do some clean up to have a smaller image
For connecting to the database, what will be the recommendation? Would
you recommend, using another docker container with MongoDB image and
define the dependency between the web and the db using docker? Along
with that have configurable runtime property such that app in
different environments (PROD, STAGE, DEV) can have the ability to
connect to different database (mongodb)
It is a good idea to create a new container for the database and connect your app to the database using docker networks. You can have multiple DB at the same time, but preferably keep one db container inside the network, and if you want to use another one, just remove the old one and add the new one to the network.
A
During development
Using a directory in the host is fast. You modify your code, relaunch the docker image and it will start your app quickly.
Docker image for production/deployement
It is good to pull the code from git. it's heavier to run, but easier to deploy.
B
During development
Don't run npm install inside docker, you can handle the dependencies manually.
Docker image for production/deployement
Make a single npm i in image building, because it's supposed to be static anyway.
More explanation
When you are developing, you change your code, use a new package, adapt your package.json, update packages ...
You basically need to control what happen with npm. It is easier to interact with it if you can directly execute commands lines and access the files (outside docker in local directory). You make your change, you relaunch your docker and it get started!
When you are deploying your app, you don't have the need to interact with npm modules. You want a packaged application with an appropriate version number and release date that do not move and that you can rely on.
Because npm is not 100% trustworthy, it happen that with exact same package.json some stuff you get as you npm i makes the application to crash. So I would not recommend to use npm i at every application relaunch or deployement, because imagine some package get fucked up, you gotta rush to find out a soluce. Moreover there is no need at all to reload packages that should be the exact same (they should!). It's not in deployement that you want to update the package! But in your developement environment where you can npm update safely and test everything up.
(Sorry about english!)
C
Use two docker image and connect them using a docker network. So you can deploy easily your app anywhere.
Some commands to help maybe about Docker networking! (i'm actually using it in my company)
// To create your own network with docker
sudo docker network create --subnet=172.42.0.0/24 docker-network
// Run the mondogb docker
sudo docker run -i -t --net docker-network --ip 172.42.0.2 -v ~/DIRECTORY:/database mongodb-docker
// Run the app docker
sudo docker run -i -t --net docker-network --ip 172.42.0.3 -v ~/DIRECTORY:/local-git backend-docker
I'm trying to implement a developer workflow with docker, with the ability to develop offline (as in, not having to run npm install when you switch between branches that have differing dependencies)
The most intuitive way to do that is to store dependencies in source control. This has its own issues especially when using modules that compile dependencies. I have tried nearly everything I could think of and find:
npm packing my projects dependencies, storing in source but this doesn't store my dependencies' dependencies
storing node_modules in source, copying this to the container and running npm rebuild but it doesn't actually trigger a rebuild
running npm install --no-registry so t triggers a rebuild but doesn't try to call out, but it actually calls out to the public registry anyway
other solutions I've seen like Node-PAC seem abandoned
npmbox looks the most promising but it requires that it's installed on the target globally, which would work in a container I can build but not production, unless we start deploying containers in production.
Is this a fruitless effort? Lack of network access is rare and would only really be needed when installing a new module or moving between revisions that have differing dependencies
Another option is to setup a private npm repository and to configure it to cache public repository. There are several options to implement this, I would recommend to try Nexus: https://www.sonatype.com/nexus-repository-oss