I am new to Docker and created following files in a large Node project folder:
Dockerfile
# syntax=docker/dockerfile:1
FROM node:16
# Update npm
RUN npm install --global npm
# WORKDIR automatically creates missing folders
WORKDIR /opt/app
# https://stackoverflow.com/a/42019654/15443125
VOLUME /opt/app
RUN useradd --create-home --shell /bin/bash app
COPY . .
RUN chown -R app /opt/app
USER app
ENV NODE_ENV=production
RUN npm install
# RUN npx webpack
CMD [ "sleep", "180" ]
docker-compose.yml
version: "3.9"
services:
app:
build:
context: .
ports:
- "3000:3000"
volumes:
- ./dist/dockerVolume/app:/opt/app
And I run this command:
docker compose up --force-recreate --build
It builds the image, starts a container and I added a sleep to make sure the container stays up for at least 3 minutes. When I open a console for that container and run cd /opt/app && ls, I can verify that there are a lot of files. project/dist/dockerVolume/app gets created by Docker, but nothing is written to it at any point.
There are no errors or warnings or other indications that something isn't set up correctly.
What am I missing?
First you should move the VOLUME declaration to the end of the Dockerfile, because:
If any build steps change the data within the volume after it has been declared, those changes will be discarded. (Documentation)
After this you will face the issue of how bind mounts and docker volumes work. Unfortunately if you use a bind mount, the contents of the host directory will always replace the files that are already in the container. Files will only appear in the host directory, if they were created during runtime by the container.
Also see:
Docker docs: bind mounts
Docker docs: volumes
To solve the issue, you could use any of these workarounds, depending on your usecase:
Use volumes in your docker-compose.yml file instead of bind mounts (Documentation)
Create the files you want to run on the host instead of in the image, and bind mount them into the container.
Use a bash script in the container that creates the neccessary files (if they are missing) when the container is starting (so the bind mount is already initialized, and the changes will persist) and after that, it starts your processes.
Related
I am not sure what I may be doing wrong but I have the following script.sh file sitting at the root of my project:
script.sh
#!/bin/sh
npm run start
envsubst '\$PORT' < /etc/nginx/conf.d/configfile.template > /etc/nginx/conf.d/default.conf
nginx -g 'daemon off;'
Then I referenced the above script in my Dockerfile as shown below:
Dockerfile
# Build environment
FROM node:16.14.2
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install --only=production
COPY . ./
# server environment
FROM nginx:alpine
COPY nginx.conf /etc/nginx/conf.d/configfile.template
ENV HOST 0.0.0.0
ENV NODE_ENV production
EXPOSE 8080
COPY script.sh /
RUN chmod +x /script.sh
ENTRYPOINT ["/script.sh"]
After building the Docker image successfully, I attempted to run it as a container but all I keep getting back is the following error:
/script.sh: line 2: npm: not found
I expect that the script should be able to pick up the already installed npm from the environment.
What can I do differently to make this work?
You're trying to run two separate programs, so run them in two separate containers.
# Dockerfile.app
FROM node:16.14.2
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install --only=production
COPY . ./
ENV HOST 0.0.0.0
ENV NODE_ENV production
EXPOSE 8080
CMD npm run start
# Dockerfile.nginx
FROM nginx:alpine
COPY nginx.conf /etc/nginx/conf.d/configfile.template
You might use a system like Docker Compose to run the two parts together:
# docker-compose.yml
version: '3.8'
services:
app:
build:
context: .
dockerfile: Dockerfile.app
nginx:
build:
context: .
dockerfile: nginx
ports:
- 8080:80
Running docker-compose up -d will start both containers together. In your Nginx configuration, make sure to proxy_pass http://app:8080, using the Compose service name and the port number the service is listening on, to forward requests to the other container.
(The Nginx Dockerfile looks short, but it's correct. The Docker Hub nginx image already knows how to run the envsubst line from your script in its own entrypoint script and it has a correct default command already.)
There's two basic problems in the setup you show in the question, both related to trying to run two programs in the same container. The first is that you can't merge images, having a second FROM line makes Docker start over from the new base image. (So your final image contains only Nginx, not Node or your built application, hence the npm not found error.) The second you'll run into is that your script will start your application, but not start the Nginx proxy until after the application exits. There are some common workarounds to this (like using a background process) but it essentially results in one process or the other being unmonitored by Docker, so your application could potentially fail and Docker wouldn't notice it to be able to restart it.
As far as I know, volume in Docker is some permanent data for the container, which can map local folder and container folder.
In early day, I am facing Error: Cannot find module 'winston' issue in Docker which mentioned in:
docker - Error: Cannot find module 'winston'
Someone told me in this post:
Remove volumes: - ./:/server from your docker-compose.yml. It overrides the whole directory contains node_modules in the container.
After I remove volumes: - ./:/server, the above problem is solved.
However, another problem occurs.
[solved but want explanation]nodemon --legacy-watch src/ not working in Docker
I solve the above issue by adding back volumes: - ./:/server, but I don't know what is the reason of it
Question
What is the cause and explanation for above 2 issues?
What happen between build and volumes, and what is the relationship between build and volumes in docker-compose.yml
Dockerfile
FROM node:lts-alpine
RUN npm install --global sequelize-cli nodemon
WORKDIR /server
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3030
CMD ["npm", "run", "dev"]
docker-compose.yml
version: '2.1'
services:
test-db:
image: mysql:5.7
...
test-web:
environment:
- NODE_ENV=local
- PORT=3030
build: . <------------------------ It takes Dockerfile in current directory
command: >
./wait-for-db-redis.sh test-db npm run dev
volumes:
- ./:/server <------------------------ how and when does this line works?
ports:
- "3030:3030"
depends_on:
- test-db
When you don't have any volumes:, your container runs the code that's built into the image. This is good! But, the container filesystem is completely separate from the host filesystem, and the image contains a fixed copy of your application. When you change your application, after building and testing it in a non-Docker environment, you need to rebuild the image.
If you bind-mount a volume over the application directory (.:/server) then the contents of the host directory replace the image contents; any work you do in the Dockerfile gets completely ignored. This also means /server/node_modules in the container is ./node_modules on the host. If the host and container environments don't agree (MacOS host/Linux container; Ubuntu host/Alpine container; ...) there can be compatibility issues that cause this to break.
If you also mount an anonymous volume over the node_modules directory (/server/node_modules) then only the first time you run the container the node_modules directory from the image gets copied into the volume, and then the volume content gets mounted into the container. If you update the image, the old volume contents take precedence (changes to package.json get ignored).
When the image is built only the contents of the build: block have an effect. There are no volumes: mounted, environment: variables aren't set, and the build environment isn't attached to networks:.
The upshot of this is that if you don't have volumes at all:
version: '3.8'
services:
app:
build: .
ports: ['3000:3000']
It is completely disconnected from the host environment. You need to docker-compose build the image again if your code changes. On the other hand, you can docker push the built image to a registry and run it somewhere else, without needing a separate copy of Node or the application source code.
If you have a volume mount replacing the application directory then everything in the image build is ignored. I've seen some questions that take this to its logical extent and skip the image build, just bind-mounting the host directory over an unmodified node image. There's not really benefit to using Docker here, especially for a front-end application; install Node instead of installing Docker and use ordinary development tools.
I'm trying to create a Node.js based docker image. For that, I'm looking for options for Parent image. Security is one of the main considerations in the image and we wanted to harden the image by not allowing shell or bash in the container.
Google Distroless does provide this option, but Distroless-NodeJS is in the experimental stage and not recommended for production.
Possible options I could think of are (compromising Distroless feature):
Official Node Image (https://hub.docker.com/_/node/) / Alpine / CentOS based image (but all would have a shell I believe).
With that being said,
Is there any alternative for Distroless?
What are the best options for the parent image for Node.js based docker image?
Any pointers would be helpful.
One option would be to start with a Node image that meets your requirements, then delete anything that you don't want (sh, bash, etc.)
At the extreme end you could add the following to your Dockerfile:
RUN /bin/rm -R /bin/*
Although I am not certain that this wouldn't interfere with the running of node.
On the official Node image (excl Apline) you have /bin/bash, /bin/dash and /bin/sh (a symlink to /bin/dash). Just deleting these 3 flies would be sufficient to prevent shell access.
The Alpine version has a symlink /bin/sh -> /bin/busybox. You could delete this symlink, but it may not run without busybox.
I think you can build an image from scratch which only contains your node application and required dependency, nothing more even no ls or pwd etc.
FROM node as builder
WORKDIR /app
COPY . ./
RUN npm install --prod
FROM astefanutti/scratch-node
COPY --from=builder /app /app
WORKDIR /app
ENTRYPOINT ["node", "bin/www"]
scratch-node
So if someone tries to get the shell,like,
docker run --entrypoint bash -it my_node_scratch
Will get error
docker: Error response from daemon: OCI runtime create failed:
container_linux.go:348: starting container process caused "exec:
\"bash\": executable file not found in $PATH": unknown.
I am referring this from official Node.js docker image.
Create a docker file in your project.
Then build and run docker image:
docker build - t test-nodejs-app
docker run -it --rm --name running-app test-nodejs-app
If you prefer docker compose:
Version: "2"
Services:
node:
image: "node:8"
user: "node"
working_dir: /home/node/app
environment:
- NODE_ENV=production
volumes:
- ./:/home/node/app
expose:
- "8081"
command: "npm start"
Run the compose file:
docker-compose up -d
There are some similar questions but they haven't answered why docker creates the empty node_modules directory in the repo even though the dockerfile is setup to hold node_modules in the container?
I'm interested to know why directory is created on the host empty, give that yarn already installs packages inside the container within node_modules and how to avoid it.
## Dockerfile
FROM node:8.11.4-alpine
RUN apk update && apk add yarn
RUN yarn global add nodemon
WORKDIR /usr/app
COPY package.json yarn.lock /usr/app/
RUN yarn
EXPOSE 3000
## docker-compose.yml
version: "3.2"
services:
app:
build:
context: .
dockerfile: Dockerfile
command: nodemon index.js
volumes:
- .:/usr/app
- /usr/app/node_modules
ports:
- "3000:3000"
You are right, your image building process is installing the node packages into the node_modules directory in the image. So, after you build your image the image contains node_modules and you can use it to run your application.
You see node_modules on your host machine, because of your volumes setup in your Compose file. There is more to it than you see in the other answer though.
What's happening is that you are mapping .:/usr/app in your first volume definition, which means that you are mapping your current directory on the host to /usr/app in the container.
This will override the /usr/app directory in the image with the current directory on the host. And your host does not have the node_modules directory (unless you installed node_modules on the host, too) and therefore your container will not work with this mapping, beacuse you've overriden /usr/app and there is no node_modules dir in the override. Node will throw an error that it cannot find node modules.
The next volume mapping solves the situation, this is actually a common Node development setup. You create a volume /usr/app/node_modules. Note, that this volume does not have a host part there is no : in the mapping, there is only one directory here. This means that Docker will mount the /usr/app/node_modules directory from the image and add it to the previous mapping where you mapped the host dir to /usr/app.
So in the running container you'll have your source code from the host's current directory PLUS node_modules from the underlying image due to the double mapping.
As a side effect you'll see an empty node_modules directory in your host's current directory.
I am aiming to configure docker so that when I modify a file on the host the change is propagated inside the container file system.
You can think of this as hot reloading for server side node code.
The nodemon file watcher should restart the server in response to file changes.
However these file changes on the host volume don't seem to be reflected inside the container when I inspect the container using docker exec pokerspace_express_1 bash and inspect a modified file the changes are not propagated inside the container from the host.
Dockerfile
FROM node:8
MAINTAINER therewillbecode
# Create app directory
WORKDIR src/app
RUN npm install nodemon -g
# Install app dependencies
COPY package.json .
# For npm#5 or later, copy package-lock.json as well
# COPY package.json package-lock.json ./
RUN npm install
CMD [ "npm", "start" ]
docker-compose.yml
version: '2'
services:
express:
build: .
depends_on:
- mongo
environment:
- MONGO_URL=mongo:27017/test
- SERVER_PORT=3000
volumes:
- ./:/src/app
ports:
- '3000:3000'
links:
- mongo
mongo:
image: mongo
ports:
- '27017:27017'
mongo-seed:
build: ./mongo-seed
links:
- mongo
.dockerignore
.git
.gitignore
README.md
docker-compose.yml
How can I ensure that host volume file changes are reflected in the container?
Try something like this in your Dockerfile:
CMD ["nodemon", "-L"]
Some people had a similar issue and were able to resolve it with passing -L (which means “legacy watch”) to nodemon.
References:
https://github.com/remy/nodemon/issues/419
http://fostertheweb.com/2016/02/nodemon-inside-docker-container/#why-isnt-nodemon-reloading
Right, so with Docker we need to re-build the image or figure out some clever solution.
You probably do not want to rebuild the image every time you make a change to your source code.
Let's figure out a clever solution. Let's generalize the Dockerfile a bit to solve your problem and also help others.
So this is the boilerplate Dockerfile:
FROM node:alpine
WORKDIR '/app'
COPY package.json .
RUN npm install
COPY . .
CMD ["npm", "run", "start"]
Remember, during the image building process we are creating a temporary container. When we make the copies we are essentially taking a snapshot of the contents /src and /public. Its a snapshot that is locked in time and by default will not be updated by making changes to the code.
So in order to get these changes to files /src and /public, we need to abandon doing a straight copy, we are going to adjust the docker run command that we use to start up our container.
We are going to make use of a feature called volume.
With Docker volume we setup a placeholder inside our Docker container, so instead of copying over our entire/src directory we can imagine we are going to put a reference to those files and give us access to the files and folders inside of the local machine.
We are setting up a mapping from a folder inside the container to a folder outside a container. The command to use is a bit painful, but once its documented here you can bookmark this answer.
docker run -p 3000:3000 -v /app/node_modules -v $(pwd):/app <image_id>
-v $(pwd):/app used to set up a volume in present working directory. This is a shortcut. So we are saying get the present working directory, get everything inside of it and map it up to our running container. It's long winded I know.
To implement this you will have to first rebuild your docker image by running:
docker build -f Dockerfile.dev .
Then run:
docker run -p 3000:3000 -v $(pwd):/app <image_id>
Then you are going to very quickly get an error message, the react-scripts not found error. You will see that message because I skipped the -v /app/node_modules.
So what's up with that?
The volume command sets up a mapping and when we do, we are saying take everything inside of our present working directory and map it up to our /appfolder, but the issue is there is no /node_modules folder which is where all our dependencies exist.
So the /node_modules folder got overwritten.
So we are essentially pointing to nothing and thats why we need that -v /app/node_modules with no colon because the colon is to map up the folder inside a container to a folder outside the container. Without the colon we are saying want it to be a placeholder, don't map it up against anything.
Now, go ahead and run: docker run -p 3000:3000 -v $(pwd):/app <image_id>
Once done, you can make all the changes you want to your project and see them "hot reload" in your browser. No need to figure out how to implement Nodemon.
So whats happening there is any changes made to your local file system is getting propagated into your container, the server inside your container sees the change and updates.
Now, I know its hard and annoying to remember such a long command, in enters Docker Compose.
We can make use of Docker Compose to dramatically simplify the command we have to run to start up the container.
So to implement that you create a Docker Compose file and inside of it you will include the port setting and the two volumes that you need.
Inside your root project, make a new file called docker-compose.yml.
Inside there you will add this:
version: '3'
services:
web:
build: .
ports:
- "3000:3000"
volumes:
- /app/node_modules
- .:/app
Then run: docker-compose up
Daniel's answer partially worked for me, but the hot reloading still doesn't work. I'm using a Windows host and had to change his docker-compose.yml to
version: '3'
services:
web:
build: .
ports:
- "3000:3000"
volumes:
- /App/node_modules
- .:/App
(I changed the volumes arguments from /app/node_modules to /App/node_modules and from .:/app to .:/App. This enables changes to be passed to the container, however the hot reloading still doesn't work. I have to use docker-compose up --build each time I want to refresh the app.)