We run multiple nodejs services using various version of nodejs. At the moment we handle this with nvm on our dev machines. I want to simplify the dev environment by dockerizing these services.
I've been looking for info, reading blogs and the process seems pretty straight forward, except when it comes to how to handle node_modules. Some recommend a a separate, external volume for node_modules. Others say just create a bind mount of the src folder into the image (including node_modules). I've also seen examples where they create a bind mount of the src folder and separate bind mount for node_modules. What is the correct way to do this?
Another thing i'm wondering is what is the proper way to handle adding/removing/updating packages? I suppose, since node_modules is mounted, i could just use npm in the src folder and the changes would be reflected in the container. But that would defeat the purpose since i would still need to use nvm locally in order for the packages to be compiled for the correct nodejs version right?
Another idea i had was to just edit the package.json and then rebuild the image, but then the changes wouldn't be reflected in the package.lock.json and that would cause problems i think.
So i guess i'm left with shelling into the container with docker exec and then running npm? This seems a bit annoying.
I would very much appreciate some advice. Thanks!
Related
I am in the process of migrating a machine from an older Linux version to a newer one. We have some node projects that seem to be more difficult to transplant to the newly desired setup.
On the old machine the build script from package.json could be run without problems because node_modules is tracked by the source control.
We don't want to use that method anymore since it is not very scalable as we want to make sure we can easily upgrade all the packages for all the users at once without the hassle of putting hundreds of subfolders (from node_modules) together with various resources.
Did you encounter such issues and, if so, how did you manage it?
Can we use a global node_modules somehow to avoid multiple node_modules for each project?
Thank you!
Can we use a global node_modules somehow to avoid multiple node_modules for each project?
There are (at least) three possibilities. Here they are in the order that I would recommend using them:
If Node.js does not find a package in the application's/module's own node_modules directory, it will check in the parent directory (../node_modules) and then that directory's parent directory (../../node_modules) etc. So if all your projects have a shared parent directory, you can put the dependencies in a node_modules directory. See "Loading from node_modules folders" in the Node.js documentation.
A second option is to set the NODE_PATH environment variable to the directory where you would like your projects to search for modules not found elsewhere. See "Loading from the global folders" in the Node.js documentation.
Lastly, it's legacy behavior, but Node.js will also search in three other locations before giving up, so you can use one of those as well: $HOME/.node_modules, $HOME/.node_libraries, and $PREFIX/lib/node. See (again) "Loading from the global folders" for more information.
All that said, please note that the docs also correctly say "It is strongly encouraged to place dependencies in the local node_modules folder. These will be loaded faster, and more reliably."
If you must do this, I would use the first option above and place the node_modules folder as far down in the directory hierarchy as you can.
I have a largish nodejs based web app, with both server and client components working together. I am currently deploying the app, but using git pull to take my latest production branch from the server repository. A git post-commit hook runs to do a npm install and a rebuild of the servers .env file, and PM2 is monitoring the various processes (3 web servers) using a change in the .env file to restart them.
node_modules is at the highest level of the project with separate server and client subdirectories. Since this is using http2 on a fast lan, I don't bother compressing the client files with web-pack or the like, although I do use rollup on lit-element and lit-html to sort out the import statements (they are not relative or absolute) that they have embedded in them.
I've just been reading that I should really have been doing an npm ci for my node dependencies, but reading the instructions for that it says it blows away the node_modules directory and starts again (whereas npm install doesn't). Since this is all running on a raspberry pi its not instantaneous.
I am not sure a temporary loss of node_modules should effect a running app too much - after all I believe the modules will all have been cached into memory, but it just might have not and there also a possibility that one of the servers falls over and pm2 restarts it, so I am wondering ....
So what is best practice here. Is it possible for instance to copy package.json, package-lock.json to a special build subdirectory, build the node_modules directory there and then move it back into place. Once built. Or is there a better way?
What is the best practice for deploying a nodejs application?
1) Directly moving the node_modules folders from the development server to production server, so that our same local environment can be created in the production also. Whatever changes made to any of the node modules remotely will not affect our code.
2) Run npm install command in the production server with the help of package.json. Here the problem is, any changes in the node modules will affect our code. I have faced some issues with the loopback module (issue link).
Can anyone help me?
Running npm install in production server cannot be done in certain scenario (lack of compiling tools, restricted internet access, etc...) and also if you have to deploy the same project on multiple machines, can be a waste of cpu, memory and bandwidth.
You should run npm install --production on a machine with the same libraries and node version of the production server, compress node_modules and deploy on production server. You should also keep the package-lock.json file to pinpoint versions.
This approach allows you also to build/test your code using development packages and then pruning the node_modules before the actual deploy.
Moving node_modules folder is overkilled.
Running npm install might break the version dependencies.
The best approach is npm ci. It uses the package_lock file and installs the required dependencies without modify the versions.
npm ci meant for continuous integration projects. LINK
I am an ASP.NET Core developer but I recently started working with Node.js apps. For me this was one of the challenges you mentioned to move the node_modules folder to production. Instead of moving the whole folder to production or only running the npm install command on production server, I figured out and tried a way of bundling my Node.js app using Webpack into a single/multiple bundles, and I just got rid of the mess of managing node_modules folder. It only picks up the required node_modules packages that are being used/referred in my app and bundles up in a single file along with my app code and I deploy that single file to production without moving the entire node_modules folder.
I found this approach useful in my case but please suggest me if this is not the correct way regarding the performance of the app or if any cons of this approach.
Definitely npm install. But you shouldn't do this by your own hand when it comes to deploying your app.
Use the tool for this like PM2.
As for your concern about changes in packages, the short answer is package-lock.json.
My guess is that by asking this question you don't really understand the point of the package.json file.
The package.json file is explicitly intended for this purpose (that, and uploading to the npm registry), the transfer of a node package without having to transfer the sizeable number of dependencies along with it.
I would go as far as to say that one should never manually move the node_modules directory at all.
Definitely use the npm install command on your production server, this is the proper way of doing it. To avoid any changes to the node_modules directory as compared to your local environment, use the package lock file. That should help with minimising changes to the source code in node_modules.
I mean no bad intent by saying this
Most/all examples I see online usually copy package.json into the image and then run npm install within the image. Is there a deal breaker reason for not running npm install from outside on the build server and then just copying everything including the node_modules/ folder?
My main motivation for doing this is that, we are using a private npm registry, with security, and running npm from within an image, we would need to figure out how to securely embed credentials. Also, we are using yarn, and we could just leverage the yarn cache across projects if yarn runs on the build server. I suppose there's workarounds for these, but running yarn/npm from the build server where everything is already set up seems very convenient.
thanks
Public Dockerfiles out there are trying to provide generalized solution.
Having dependencies coded in package.json makes it possible to share only one Dockerfile and not depend on anything not public available.
But at runtime Docker does not care how files got to container. So this is up to you, how you push all needed files to your container.
P.S. Consider layering. If you copy stuff under node_modules/, do it in one step, by that only one layer is used.
When using NVM, or otherwise switching between Node.js versions, node_modules has to be rebuilt for each Node version. Would it be possible somehow use a different node_modules directory for each version, instead of having to rebuild the contents everytime you switch between versions?
something like:
node_modules_v8/
node_modules_v7/
node_modules_v6/
Maybe there is a better way to do this, not sure.
Three suggestions (in my order of pref):
Use Docker to help configure Containers and Volumes to switch up where things point to based on your version of node.
Set a symbolic link (Linux syntax, Windows syntax) to your node_modules folder that points to the correct version of modules that node expects to run.
Similar to manipulating symbolic links, use the NODE_PATH setting and point it to the correct cache of your node_modules folder (i.e. NODE_PATH=/etc/node_v8) just prior to executing your node command. If you set it globally it would restrict you running multiple instances of node on that box so, you'd need to set the NODE_PATH on a per process instance.