I am new to Docker and created following files in a large Node project folder:
Dockerfile
# syntax=docker/dockerfile:1
FROM node:16
# Update npm
RUN npm install --global npm
# WORKDIR automatically creates missing folders
WORKDIR /opt/app
# https://stackoverflow.com/a/42019654/15443125
VOLUME /opt/app
RUN useradd --create-home --shell /bin/bash app
COPY . .
RUN chown -R app /opt/app
USER app
ENV NODE_ENV=production
RUN npm install
# RUN npx webpack
CMD [ "sleep", "180" ]
docker-compose.yml
version: "3.9"
services:
app:
build:
context: .
ports:
- "3000:3000"
volumes:
- ./dist/dockerVolume/app:/opt/app
And I run this command:
docker compose up --force-recreate --build
It builds the image, starts a container and I added a sleep to make sure the container stays up for at least 3 minutes. When I open a console for that container and run cd /opt/app && ls, I can verify that there are a lot of files. project/dist/dockerVolume/app gets created by Docker, but nothing is written to it at any point.
There are no errors or warnings or other indications that something isn't set up correctly.
What am I missing?
First you should move the VOLUME declaration to the end of the Dockerfile, because:
If any build steps change the data within the volume after it has been declared, those changes will be discarded. (Documentation)
After this you will face the issue of how bind mounts and docker volumes work. Unfortunately if you use a bind mount, the contents of the host directory will always replace the files that are already in the container. Files will only appear in the host directory, if they were created during runtime by the container.
Also see:
Docker docs: bind mounts
Docker docs: volumes
To solve the issue, you could use any of these workarounds, depending on your usecase:
Use volumes in your docker-compose.yml file instead of bind mounts (Documentation)
Create the files you want to run on the host instead of in the image, and bind mount them into the container.
Use a bash script in the container that creates the neccessary files (if they are missing) when the container is starting (so the bind mount is already initialized, and the changes will persist) and after that, it starts your processes.
As far as I know, volume in Docker is some permanent data for the container, which can map local folder and container folder.
In early day, I am facing Error: Cannot find module 'winston' issue in Docker which mentioned in:
docker - Error: Cannot find module 'winston'
Someone told me in this post:
Remove volumes: - ./:/server from your docker-compose.yml. It overrides the whole directory contains node_modules in the container.
After I remove volumes: - ./:/server, the above problem is solved.
However, another problem occurs.
[solved but want explanation]nodemon --legacy-watch src/ not working in Docker
I solve the above issue by adding back volumes: - ./:/server, but I don't know what is the reason of it
Question
What is the cause and explanation for above 2 issues?
What happen between build and volumes, and what is the relationship between build and volumes in docker-compose.yml
Dockerfile
FROM node:lts-alpine
RUN npm install --global sequelize-cli nodemon
WORKDIR /server
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3030
CMD ["npm", "run", "dev"]
docker-compose.yml
version: '2.1'
services:
test-db:
image: mysql:5.7
...
test-web:
environment:
- NODE_ENV=local
- PORT=3030
build: . <------------------------ It takes Dockerfile in current directory
command: >
./wait-for-db-redis.sh test-db npm run dev
volumes:
- ./:/server <------------------------ how and when does this line works?
ports:
- "3030:3030"
depends_on:
- test-db
When you don't have any volumes:, your container runs the code that's built into the image. This is good! But, the container filesystem is completely separate from the host filesystem, and the image contains a fixed copy of your application. When you change your application, after building and testing it in a non-Docker environment, you need to rebuild the image.
If you bind-mount a volume over the application directory (.:/server) then the contents of the host directory replace the image contents; any work you do in the Dockerfile gets completely ignored. This also means /server/node_modules in the container is ./node_modules on the host. If the host and container environments don't agree (MacOS host/Linux container; Ubuntu host/Alpine container; ...) there can be compatibility issues that cause this to break.
If you also mount an anonymous volume over the node_modules directory (/server/node_modules) then only the first time you run the container the node_modules directory from the image gets copied into the volume, and then the volume content gets mounted into the container. If you update the image, the old volume contents take precedence (changes to package.json get ignored).
When the image is built only the contents of the build: block have an effect. There are no volumes: mounted, environment: variables aren't set, and the build environment isn't attached to networks:.
The upshot of this is that if you don't have volumes at all:
version: '3.8'
services:
app:
build: .
ports: ['3000:3000']
It is completely disconnected from the host environment. You need to docker-compose build the image again if your code changes. On the other hand, you can docker push the built image to a registry and run it somewhere else, without needing a separate copy of Node or the application source code.
If you have a volume mount replacing the application directory then everything in the image build is ignored. I've seen some questions that take this to its logical extent and skip the image build, just bind-mounting the host directory over an unmodified node image. There's not really benefit to using Docker here, especially for a front-end application; install Node instead of installing Docker and use ordinary development tools.
I'm a web developer who currently is working on a next.js project (it's just a framework to SSR ReactJS). I'm using Docker config on this project and I discovered an issue when I add/remove dependencies. When I add a dependency, build my project and up it with docker-compose, my new dependency isn't added to my Docker image. I have to clean my docker system with docker system prune to reset everything then I could build and up my project. After that, my dependency is added to my Docker container.
I use Dockerfile to configure my image and different docker-compose files to set different configurations depending on my environments. Here is my configuration:
Dockerfile
FROM node:10.13.0-alpine
# SET environment variables
ENV NODE_VERSION 10.13.0
ENV YARN_VERSION 1.12.3
# Install Yarn
RUN apk add --no-cache --virtual .build-deps-yarn curl \
&& curl -fSLO --compressed "https://yarnpkg.com/downloads/$YARN_VERSION/yarn-v$YARN_VERSION.tar.gz" \
&& tar -xzf yarn-v$YARN_VERSION.tar.gz -C /opt/ \
&& ln -snf /opt/yarn-v$YARN_VERSION/bin/yarn /usr/local/bin/yarn \
&& ln -snf /opt/yarn-v$YARN_VERSION/bin/yarnpkg /usr/local/bin/yarnpkg \
&& rm yarn-v$YARN_VERSION.tar.gz \
&& apk del .build-deps-yarn
# Create app directory
RUN mkdir /website
WORKDIR /website
ADD package*.json /website
# Install app dependencies
RUN yarn install
# Build source files
COPY . /website/
RUN yarn run build
docker-compose.yml (dev env)
version: "3"
services:
app:
container_name: website
build:
context: .
ports:
- "3000:3000"
- "3332:3332"
- "9229:9229"
volumes:
- /website/node_modules/
- .:/website
command: yarn run dev 0.0.0.0 3000
environment:
SERVER_URL: https://XXXXXXX.com
Here my commands to run my Docker environment:
docker-compose build --no-cache
docker-compose up
I suppose that something is wrong in my Docker's configuration but I can't catch it. Do you have an idea to help me?
Thanks!
Your volumes right now are not set up to do what you intend to do. The current set below means that you are overriding the contents of your website directory in the container with your local . directory.
volumes:
- /website/node_modules/
- .:/website
I'm sure your intention is to map your local directory into the container first, and then override node_modules with the original contents of the image's node_modules directory, i.e. /website/node_modules/.
Changing the order of your volumes like below should solve the issue.
volumes:
- .:/website
- /website/node_modules/
You are explicitly telling Docker you want this behavior. When you say:
volumes:
- /website/node_modules/
You are telling Docker you don't want to use the node_modules directory that's baked into the image. Instead, it should create an anonymous volume to hold the node_modules directory (which has some special behavior on its first use) and persist the data there, even if other characteristics like the underlying image change.
That means if you change your package.json and rebuild the image, Docker will keep using the volume version of your node_modules directory. (Similarly, the bind mount of .:/website means everything else in the last half of your Dockerfile is essentially ignored.)
I would remove the volumes: block in this setup to respect the program that's being built in the image. (I'd also suggest moving the command: to a CMD line in the Dockerfile.) Develop and test your application without using Docker, and build and deploy an image once it's essentially working, but not before.
I am aiming to configure docker so that when I modify a file on the host the change is propagated inside the container file system.
You can think of this as hot reloading for server side node code.
The nodemon file watcher should restart the server in response to file changes.
However these file changes on the host volume don't seem to be reflected inside the container when I inspect the container using docker exec pokerspace_express_1 bash and inspect a modified file the changes are not propagated inside the container from the host.
Dockerfile
FROM node:8
MAINTAINER therewillbecode
# Create app directory
WORKDIR src/app
RUN npm install nodemon -g
# Install app dependencies
COPY package.json .
# For npm#5 or later, copy package-lock.json as well
# COPY package.json package-lock.json ./
RUN npm install
CMD [ "npm", "start" ]
docker-compose.yml
version: '2'
services:
express:
build: .
depends_on:
- mongo
environment:
- MONGO_URL=mongo:27017/test
- SERVER_PORT=3000
volumes:
- ./:/src/app
ports:
- '3000:3000'
links:
- mongo
mongo:
image: mongo
ports:
- '27017:27017'
mongo-seed:
build: ./mongo-seed
links:
- mongo
.dockerignore
.git
.gitignore
README.md
docker-compose.yml
How can I ensure that host volume file changes are reflected in the container?
Try something like this in your Dockerfile:
CMD ["nodemon", "-L"]
Some people had a similar issue and were able to resolve it with passing -L (which means “legacy watch”) to nodemon.
References:
https://github.com/remy/nodemon/issues/419
http://fostertheweb.com/2016/02/nodemon-inside-docker-container/#why-isnt-nodemon-reloading
Right, so with Docker we need to re-build the image or figure out some clever solution.
You probably do not want to rebuild the image every time you make a change to your source code.
Let's figure out a clever solution. Let's generalize the Dockerfile a bit to solve your problem and also help others.
So this is the boilerplate Dockerfile:
FROM node:alpine
WORKDIR '/app'
COPY package.json .
RUN npm install
COPY . .
CMD ["npm", "run", "start"]
Remember, during the image building process we are creating a temporary container. When we make the copies we are essentially taking a snapshot of the contents /src and /public. Its a snapshot that is locked in time and by default will not be updated by making changes to the code.
So in order to get these changes to files /src and /public, we need to abandon doing a straight copy, we are going to adjust the docker run command that we use to start up our container.
We are going to make use of a feature called volume.
With Docker volume we setup a placeholder inside our Docker container, so instead of copying over our entire/src directory we can imagine we are going to put a reference to those files and give us access to the files and folders inside of the local machine.
We are setting up a mapping from a folder inside the container to a folder outside a container. The command to use is a bit painful, but once its documented here you can bookmark this answer.
docker run -p 3000:3000 -v /app/node_modules -v $(pwd):/app <image_id>
-v $(pwd):/app used to set up a volume in present working directory. This is a shortcut. So we are saying get the present working directory, get everything inside of it and map it up to our running container. It's long winded I know.
To implement this you will have to first rebuild your docker image by running:
docker build -f Dockerfile.dev .
Then run:
docker run -p 3000:3000 -v $(pwd):/app <image_id>
Then you are going to very quickly get an error message, the react-scripts not found error. You will see that message because I skipped the -v /app/node_modules.
So what's up with that?
The volume command sets up a mapping and when we do, we are saying take everything inside of our present working directory and map it up to our /appfolder, but the issue is there is no /node_modules folder which is where all our dependencies exist.
So the /node_modules folder got overwritten.
So we are essentially pointing to nothing and thats why we need that -v /app/node_modules with no colon because the colon is to map up the folder inside a container to a folder outside the container. Without the colon we are saying want it to be a placeholder, don't map it up against anything.
Now, go ahead and run: docker run -p 3000:3000 -v $(pwd):/app <image_id>
Once done, you can make all the changes you want to your project and see them "hot reload" in your browser. No need to figure out how to implement Nodemon.
So whats happening there is any changes made to your local file system is getting propagated into your container, the server inside your container sees the change and updates.
Now, I know its hard and annoying to remember such a long command, in enters Docker Compose.
We can make use of Docker Compose to dramatically simplify the command we have to run to start up the container.
So to implement that you create a Docker Compose file and inside of it you will include the port setting and the two volumes that you need.
Inside your root project, make a new file called docker-compose.yml.
Inside there you will add this:
version: '3'
services:
web:
build: .
ports:
- "3000:3000"
volumes:
- /app/node_modules
- .:/app
Then run: docker-compose up
Daniel's answer partially worked for me, but the hot reloading still doesn't work. I'm using a Windows host and had to change his docker-compose.yml to
version: '3'
services:
web:
build: .
ports:
- "3000:3000"
volumes:
- /App/node_modules
- .:/App
(I changed the volumes arguments from /app/node_modules to /App/node_modules and from .:/app to .:/App. This enables changes to be passed to the container, however the hot reloading still doesn't work. I have to use docker-compose up --build each time I want to refresh the app.)
I have an app with the following services:
web/ - holds and runs a python 3 flask web server on port 5000. Uses sqlite3.
worker/ - has an index.js file which is a worker for a queue. the web server interacts with this queue using a json API over port 9730. The worker uses redis for storage. The worker also stores data locally in the folder worker/images/
Now this question only concerns the worker.
worker/Dockerfile
FROM node:0.12
WORKDIR /worker
COPY package.json /worker/
RUN npm install
COPY . /worker/
docker-compose.yml
redis:
image: redis
worker:
build: ./worker
command: npm start
ports:
- "9730:9730"
volumes:
- worker/:/worker/
links:
- redis
When I run docker-compose build, everything works as expected and all npm modules are installed in /worker/node_modules as I'd expect.
npm WARN package.json unfold#1.0.0 No README data
> phantomjs#1.9.2-6 install /worker/node_modules/pageres/node_modules/screenshot-stream/node_modules/phantom-bridge/node_modules/phantomjs
> node install.js
<snip>
But when I do docker-compose up, I see this error:
worker_1 | Error: Cannot find module 'async'
worker_1 | at Function.Module._resolveFilename (module.js:336:15)
worker_1 | at Function.Module._load (module.js:278:25)
worker_1 | at Module.require (module.js:365:17)
worker_1 | at require (module.js:384:17)
worker_1 | at Object.<anonymous> (/worker/index.js:1:75)
worker_1 | at Module._compile (module.js:460:26)
worker_1 | at Object.Module._extensions..js (module.js:478:10)
worker_1 | at Module.load (module.js:355:32)
worker_1 | at Function.Module._load (module.js:310:12)
worker_1 | at Function.Module.runMain (module.js:501:10)
Turns out none of the modules are present in /worker/node_modules (on host or in the container).
If on the host, I npm install, then everything works just fine. But I don't want to do that. I want the container to handle dependencies.
What's going wrong here?
(Needless to say, all packages are in package.json.)
This happens because you have added your worker directory as a volume to your docker-compose.yml, as the volume is not mounted during the build.
When docker builds the image, the node_modules directory is created within the worker directory, and all the dependencies are installed there. Then on runtime the worker directory from outside docker is mounted into the docker instance (which does not have the installed node_modules), hiding the node_modules you just installed. You can verify this by removing the mounted volume from your docker-compose.yml.
A workaround is to use a data volume to store all the node_modules, as data volumes copy in the data from the built docker image before the worker directory is mounted. This can be done in the docker-compose.yml like this:
redis:
image: redis
worker:
build: ./worker
command: npm start
ports:
- "9730:9730"
volumes:
- ./worker/:/worker/
- /worker/node_modules
links:
- redis
I'm not entirely certain whether this imposes any issues for the portability of the image, but as it seems you are primarily using docker to provide a runtime environment, this should not be an issue.
If you want to read more about volumes, there is a nice user guide available here: https://docs.docker.com/userguide/dockervolumes/
EDIT: Docker has since changed it's syntax to require a leading ./ for mounting in files relative to the docker-compose.yml file.
The node_modules folder is overwritten by the volume and no more accessible in the container. I'm using the native module loading strategy to take out the folder from the volume:
/data/node_modules/ # dependencies installed here
/data/app/ # code base
Dockerfile:
COPY package.json /data/
WORKDIR /data/
RUN npm install
ENV PATH /data/node_modules/.bin:$PATH
COPY . /data/app/
WORKDIR /data/app/
The node_modules directory is not accessible from outside the container because it is included in the image.
The solution provided by #FrederikNS works, but I prefer to explicitly name my node_modules volume.
My project/docker-compose.yml file (docker-compose version 1.6+) :
version: '2'
services:
frontend:
....
build: ./worker
volumes:
- ./worker:/worker
- node_modules:/worker/node_modules
....
volumes:
node_modules:
my file structure is :
project/
│── worker/
│ └─ Dockerfile
└── docker-compose.yml
It creates a volume named project_node_modules and re-use it every time I up my application.
My docker volume ls looks like this :
DRIVER VOLUME NAME
local project_mysql
local project_node_modules
local project2_postgresql
local project2_node_modules
I recently had a similar problem. You can install node_modules elsewhere and set the NODE_PATH environment variable.
In the example below I installed node_modules into /install
worker/Dockerfile
FROM node:0.12
RUN ["mkdir", "/install"]
ADD ["./package.json", "/install"]
WORKDIR /install
RUN npm install --verbose
ENV NODE_PATH=/install/node_modules
WORKDIR /worker
COPY . /worker/
docker-compose.yml
redis:
image: redis
worker:
build: ./worker
command: npm start
ports:
- "9730:9730"
volumes:
- worker/:/worker/
links:
- redis
There's elegant solution:
Just mount not whole directory, but only app directory. This way you'll you won't have troubles with npm_modules.
Example:
frontend:
build:
context: ./ui_frontend
dockerfile: Dockerfile.dev
ports:
- 3000:3000
volumes:
- ./ui_frontend/src:/frontend/src
Dockerfile.dev:
FROM node:7.2.0
#Show colors in docker terminal
ENV COMPOSE_HTTP_TIMEOUT=50000
ENV TERM="xterm-256color"
COPY . /frontend
WORKDIR /frontend
RUN npm install update
RUN npm install --global typescript
RUN npm install --global webpack
RUN npm install --global webpack-dev-server
RUN npm install --global karma protractor
RUN npm install
CMD npm run server:dev
UPDATE: Use the solution provided by #FrederikNS.
I encountered the same problem. When the folder /worker is mounted to the container - all of it's content will be syncronized (so the node_modules folder will disappear if you don't have it locally.)
Due to incompatible npm packages based on OS, I could not just install the modules locally - then launch the container, so..
My solution to this, was to wrap the source in a src folder, then link node_modules into that folder, using this index.js file. So, the index.js file is now the starting point of my application.
When I run the container, I mounted the /app/src folder to my local src folder.
So the container folder looks something like this:
/app
/node_modules
/src
/node_modules -> ../node_modules
/app.js
/index.js
It is ugly, but it works..
Due to the way Node.js loads modules, node_modules can be anywhere in the path to your source code. For example, put your source at /worker/src and your package.json in /worker, so /worker/node_modules is where they're installed.
There is also some simple solution without mapping node_module directory into another volume. It's about to move installing npm packages into final CMD command.
Disadvantage of this approach:
run npm install each time you run container (switching from npm to yarn might also speed up this process a bit).
worker/Dockerfile
FROM node:0.12
WORKDIR /worker
COPY package.json /worker/
COPY . /worker/
CMD /bin/bash -c 'npm install; npm start'
docker-compose.yml
redis:
image: redis
worker:
build: ./worker
ports:
- "9730:9730"
volumes:
- worker/:/worker/
links:
- redis
Installing node_modules in container to different from project folder, and setting NODE_PATH to your node_modules folder helps me (u need to rebuild container).
I'm using docker-compose. My project file structure:
-/myproject
--docker-compose.yml
--nodejs/
----Dockerfile
docker-compose.yml:
version: '2'
services:
nodejs:
image: myproject/nodejs
build: ./nodejs/.
volumes:
- ./nodejs:/workdir
ports:
- "23005:3000"
command: npm run server
Dockerfile in nodejs folder:
FROM node:argon
RUN mkdir /workdir
COPY ./package.json /workdir/.
RUN mkdir /data
RUN ln -s /workdir/package.json /data/.
WORKDIR /data
RUN npm install
ENV NODE_PATH /data/node_modules/
WORKDIR /workdir
There are two seperate requirements I see for node dev environments... mount your source code INTO the container, and mount the node_modules FROM the container (for your IDE). To accomplish the first, you do the usual mount, but not everything... just the things you need
volumes:
- worker/src:/worker/src
- worker/package.json:/worker/package.json
- etc...
( the reason to not do - /worker/node_modules is because docker-compose will persist that volume between runs, meaning you can diverge from what is actually in the image (defeating the purpose of not just bind mounting from your host)).
The second one is actually harder. My solution is a bit hackish, but it works. I have a script to install the node_modules folder on my host machine, and I just have to remember to call it whenever I update package.json (or, add it to the make target that runs docker-compose build locally).
install_node_modules:
docker build -t building .
docker run -v `pwd`/node_modules:/app/node_modules building npm install
In my opinion, we should not RUN npm install in the Dockerfile. Instead, we can start a container using bash to install the dependencies before runing the formal node service
docker run -it -v ./app:/usr/src/app your_node_image_name /bin/bash
root#247543a930d6:/usr/src/app# npm install
You can also ditch your Dockerfile, because of its simplicity, just use a basic image and specify the command in your compose file:
version: '3.2'
services:
frontend:
image: node:12-alpine
volumes:
- ./frontend/:/app/
command: sh -c "cd /app/ && yarn && yarn run start"
expose: [8080]
ports:
- 8080:4200
This is particularly useful for me, because I just need the environment of the image, but operate on my files outside the container and I think this is what you want to do too.
You can just move node_modules into a / folder.
How it works
FROM node:0.12
WORKDIR /worker
COPY package.json /worker/
RUN npm install \
&& mv node_modules /node_modules
COPY . /worker/
You can try something like this in your Dockerfile:
FROM node:0.12
WORKDIR /worker
CMD bash ./start.sh
Then you should use the Volume like this:
volumes:
- worker/:/worker:rw
The startscript should be a part of your worker repository and looks like this:
#!/bin/sh
npm install
npm start
So the node_modules are a part of your worker volume and gets synchronized and the npm scripts are executed when everything is up.
If you want the node_modules folder available to the host during development, you could install the dependencies when you start the container instead of during build-time. I do this to get syntax highlighting working in my editor.
Dockerfile
# We're using a multi-stage build so that we can install dependencies during build-time only for production.
# dev-stage
FROM node:14-alpine AS dev-stage
WORKDIR /usr/src/app
COPY package.json ./
COPY . .
# `yarn install` will run every time we start the container. We're using yarn because it's much faster than npm when there's nothing new to install
CMD ["sh", "-c", "yarn install && yarn run start"]
# production-stage
FROM node:14-alpine AS production-stage
WORKDIR /usr/src/app
COPY package.json ./
RUN yarn install
COPY . .
.dockerignore
Add node_modules to .dockerignore to prevent it from being copied when the Dockerfile runs COPY . .. We use volumes to bring in node_modules.
**/node_modules
docker-compose.yml
node_app:
container_name: node_app
build:
context: ./node_app
target: dev-stage # `production-stage` for production
volumes:
# For development:
# If node_modules already exists on the host, they will be copied
# into the container here. Since `yarn install` runs after the
# container starts, this volume won't override the node_modules.
- ./node_app:/usr/src/app
# For production:
#
- ./node_app:/usr/src/app
- /usr/src/app/node_modules
I tried the most popular answers on this page but ran into an issue: the node_modules directory in my Docker instance would get cached in the the named or unnamed mount point, and later would overwrite the node_modules directory that was built as part of the Docker build process. Thus, new modules I added to package.json would not show up in the Docker instance.
Fortunately I found this excellent page which explains what was going on and gives at least 3 ways to work around it:
https://burnedikt.com/dockerized-node-development-and-mounting-node-volumes/
If you don't use docker-compose you can do it like this:
FROM node:10
WORKDIR /usr/src/app
RUN npm install -g #angular/cli
COPY package.json ./
RUN npm install
EXPOSE 5000
CMD ng serve --port 5000 --host 0.0.0.0
Then you build it: docker build -t myname . and you run it by adding two volumes, the second one without source: docker run --rm -it -p 5000:5000 -v "$PWD":/usr/src/app/ -v /usr/src/app/node_modules myname
With Yarn you can move the node_modules outside the volume by setting
# ./.yarnrc
--modules-folder /opt/myproject/node_modules
See https://www.caxy.com/blog/how-set-custom-location-nodemodules-path-yarn