My webapp/Dockerfile looks like this:
FROM project_base-container
EXPOSE 9100
# ...
# Using copy instead of mount, since we need to write in sub-directories like node_modules etc.
COPY . /usr/src/app/webapp
CMD ["bash", "webapp/scripts/build_and_run.sh", "setup_deps_and_run_app"]
I want to allow the app to be able to read the source code and also write into sub-folders like node_modules, but I don't want those changes to come in my local machine. Hence, I have two choices:
Change --prefix='/tmp' in node install and mount the src as ro
Copy COPY the src and then the container can write wherever it wants.
Solution 1 wrecks havoc, because now I have to copy/link all files like package.json, index.html etc to the prefix location.
Solution 2 is what I have done above.
When I use COPY everything is fine for the first time. But now the problem is that after changes in source code, I want to update the source code in the image every time I do:
sudo docker-compose down && sudo docker-compose up --build -d
But the COPY command is cached by docker and won't be updated, even after file changes.
TL;DR: I have a src folder 'webapp' that I want to mount from host as readonly, but my app wants to write to a subfolder 'webapp/node_modules'.
To create a writable mount inside a read only mount, you need to ensure that the read-only directory has the mount points (folders), even if they are empty.
To create empty mount points and commit them, please see this answer.
For running node, you need following 4 folders writable:
$ mkdir webapp/frontend/node_modules webapp/frontend/build webapp/frontend/.config webapp/frontend/.npm
$ cat > webapp/frontend/node_modules/.gitignore
# Ignore everything in this directory
*
# Except this file
!.gitignore
$ git add -f webapp/frontend/node_modules/.gitignore
$ cat docker-compose.yml # Filtered output below
version: "2"
services:
webapp:
build: ./webapp
expose:
- "9900"
# Named volumes, defined below.
volumes:
- ./webapp:/usr/src/app/webapp:ro
- webapp_config:/usr/src/app/webapp/frontend/.config:rw
- webapp_npm:/usr/src/app/webapp/frontend/.npm:rw
- webapp_node_modules:/usr/src/app/webapp/frontend/node_modules:rw
- webapp_build:/usr/src/app/webapp/frontend/build:rw
- ./config.ini:/usr/src/app/config.ini:ro
# Named volumes. These will stay in the host, but not in the current directory.
volumes:
webapp_node_modules:
webapp_build:
webapp_config:
webapp_npm:
Related answer about writable folders in read-only mounts
See this documentation for different volume/storage options in docker
Related answer about named volumes
You can use internal docker-volumes.
version: "3"
services:
postgres:
image: postgres:9.4
volumes:
- db-data:/var/lib/db
redis:
image: redis
ports:
- "6379:6379"
volumes:
- ./data:/data
volumes:
db-data:
From documentation
Explaination:
Postgress service uses a named volume and that is located inside the directory where docker is installed. For Ubuntu, it is /var/lib/docker/aufs/. This directory may be different and can be found by using docker inspect <vulume_name>.
Redis services uses another type of volumes, the ones which are more popular and which are used to mount host directory inside container directory. The data folder is in the same directory as the one in which docker-compose.yml file is present, and this folder is mounted inside the container at /data.
Is it possible for you to store all code you want to be editable in some sub-directory (say src)? If yes, then you could do the following
Use COPY as you are doing now, which is probably needed for deployment anyway.
In addition (read-only) mount src to /usr/src/app/webapp/src during development.
Related
Supposed I have a Docker container and a folder on my host /hostFolder. Now if I want to add this folder to the Docker container as a volume, then I can do this either by using ADD in the Dockerfile or mounting it as a volume.
So far, so good.
Now /hostFolder contains a sub-folder, /hostFolder/subFolder.
I want to mount /hostFolder into the Docker container (whether as read-write or read-only does not matter, works both for me), but I do NOT want to have it included /hostFolder/subFolder. I want to exclude this, and I also want the Docker container be able to make changes to this sub-folder, without the consequence of having it changed on the host as well.
Is this possible? If so, how?
Using docker-compose I'm able to use node_modules locally, but ignore it in the docker container using the following syntax in the docker-compose.yml
volumes:
- './angularApp:/opt/app'
- /opt/app/node_modules/
So everything in ./angularApp is mapped to /opt/app and then I create another mount volume /opt/app/node_modules/ which is now empty directory - even if in my local machine ./angularApp/node_modules is not empty.
If you want to have subdirectories ignored by docker-compose but persistent, you can do the following in docker-compose.yml:
volumes:
node_modules:
services:
server:
volumes:
- .:/app
- node_modules:/app/node_modules
This will mount your current directory as a shared volume, but mount a persistent docker volume in place of your local node_modules directory. This is similar to the answer by #kernix, but this will allow node_modules to persist between docker-compose up runs, which is likely the desired behavior.
For those trying to get a nice workflow going where node_modules isn't overridden by local this might help.
Change your docker-compose to mount an anonymous persistent volume to node_modules to prevent your local overriding it. This has been outlined in this thread a few times.
services:
server:
build: .
volumes:
- .:/app
- /app/node_modules
This is the important bit we were missing. When spinning up your stack use docker-compose -V. Without this if you added a new package and rebuilt your image it would be using the node_modules from your initial docker-compose launch.
-V, --renew-anon-volumes Recreate anonymous volumes instead of retrieving
data from the previous containers.
To exclude a file, use the following
volumes:
- /hostFolder:/folder
- /dev/null:/folder/fileToBeExcluded
With the docker command line:
docker run \
--mount type=bind,src=/hostFolder,dst=/containerFolder \
--mount type=volume,dst=/containerFolder/subFolder \
...other-args...
The -v option may also be used (credit to Bogdan Mart), but --mount is clearer and recommended.
First, using the ADD instruction in a Dockerfile is very different from using a volume (either via the -v argument to docker run or the VOLUME instruction in a Dockerfile). The ADD and COPY commands just take a copy of the files at the time docker build is run. These files are not updated until a fresh image is created with the docker build command. By contrast, using a volume is essentially saying "this directory should not be stored in the container image; instead use a directory on the host"; whenever a file inside a volume is changed, both the host and container will see it immediately.
I don't believe you can achieve what you want using volumes, you'll have to rethink your directory structure if you want to do this.
However, it's quite simple to achieve using COPY (which should be preferred to ADD). You can either use a .dockerignore file to exclude the subdirectory, or you could COPY all the files then do a RUN rm bla to remove the subdirectory.
Remember that any files you add to image with COPY or ADD must be inside the build context i.e. in or below the directory you run docker build from.
for the people who also had the issue that the node_modules folder would still overwrite from your local system and the other way around
volumes:
node_modules:
services:
server:
volumes:
- .:/app
- node_modules:/app/node_modules/
This is the solution, With the trailing / after the node_modules being the fix.
Looks like the old solution doesn't work anymore(at least for me).
Creating an empty folder and mapping target folder to it helped though.
volumes:
- ./angularApp:/opt/app
- .empty:/opt/app/node_modules/
I found this link which saved me: Working with docker bind mounts and node_modules.
This working solution will create a "exclude" named volume in docker volumes manager. The volume name "exclude" is arbitrary, so you can use a custom name for the volume intead exclude.
services:
node:
command: nodemon index.js
volumes:
- ./:/usr/local/app/
# the volume above prevents our host system's node_modules to be mounted
- exclude:/usr/local/app/node_modules/
volumes:
exclude:
You can see more infos about volumes in Official docs - Use a volume with docker compose
To exclude a mounted file contained in the volume of your machine, you will have to overwrite it by allocating a volume to this same file.
In your config file:
services:
server:
build : ./Dockerfile
volumes:
- .:/app
An example in you dockerfile:
# Image Location
FROM node:13.12.0-buster
VOLUME /app/you_overwrite_file
I have Thingsboard installed in Docker in Ubuntu.
The compose file maps 2 volumes to the host system, the data and logs directories, and that works fine.
I wanted to do the same with the conf directory, to be able to modify some configuration and environment variables values in the thingsboard.yml and .conf files, and avoid having to recreate the containers each time I change something.
I've tried do it the same way it is done for the data and log directories in the docker-compose.yml:
> volumes:
> - /ThingsboardDocker/conf:/usr/share/thingsboard/conf
... but it throws this error when creating the containers:
/usr/bin/start-tb.sh: line 25: /usr/share/thingsboard/conf/thingsboard.conf: No such file or directory mytb_1 | /usr/bin/install-tb.sh: line 45: /usr/share/thingsboard/conf/thingsboard.conf: No such file or directory
For the data and logs directories works fine, but I can't figure out why it doesn't work for the configuration files.
What could be the problem?
Thank you!
The item you currently use is bind mount, what I suggest is to use volumes, this could pop up the data in container to a special place on host. A minimal example as next, you could customize it to meet your requirement:
20211008/docker-compose.yaml:
version: "3"
services:
app:
image: nginx
ports:
- 9000:80
volumes:
- export_data:/etc/nginx/conf.d
volumes:
export_data:
After the container start, e.g. has the name 20211008_app_1, then we could use next to get the path of export_data:
shubuntu1#shubuntu1:~/20211008$ docker inspect 20211008_app_1 | grep export_data
"20211008_export_data:/etc/nginx/conf.d:rw"
"Name": "20211008_export_data",
"Source": "/var/lib/docker/volumes/20211008_export_data/_data",
Here, /var/lib/docker/volumes/20211008_export_data/_data means the host directory which you could find the contents of /etc/nginx/conf.d in container:
On host:
root#shubuntu1:/var/lib/docker/volumes/20211008_export_data/_data# ls
default.conf
In container:
root#shubuntu1:~# docker exec -it 20211008_app_1 ls /etc/nginx/conf.d
default.conf
As far as I know, volume in Docker is some permanent data for the container, which can map local folder and container folder.
In early day, I am facing Error: Cannot find module 'winston' issue in Docker which mentioned in:
docker - Error: Cannot find module 'winston'
Someone told me in this post:
Remove volumes: - ./:/server from your docker-compose.yml. It overrides the whole directory contains node_modules in the container.
After I remove volumes: - ./:/server, the above problem is solved.
However, another problem occurs.
[solved but want explanation]nodemon --legacy-watch src/ not working in Docker
I solve the above issue by adding back volumes: - ./:/server, but I don't know what is the reason of it
Question
What is the cause and explanation for above 2 issues?
What happen between build and volumes, and what is the relationship between build and volumes in docker-compose.yml
Dockerfile
FROM node:lts-alpine
RUN npm install --global sequelize-cli nodemon
WORKDIR /server
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3030
CMD ["npm", "run", "dev"]
docker-compose.yml
version: '2.1'
services:
test-db:
image: mysql:5.7
...
test-web:
environment:
- NODE_ENV=local
- PORT=3030
build: . <------------------------ It takes Dockerfile in current directory
command: >
./wait-for-db-redis.sh test-db npm run dev
volumes:
- ./:/server <------------------------ how and when does this line works?
ports:
- "3030:3030"
depends_on:
- test-db
When you don't have any volumes:, your container runs the code that's built into the image. This is good! But, the container filesystem is completely separate from the host filesystem, and the image contains a fixed copy of your application. When you change your application, after building and testing it in a non-Docker environment, you need to rebuild the image.
If you bind-mount a volume over the application directory (.:/server) then the contents of the host directory replace the image contents; any work you do in the Dockerfile gets completely ignored. This also means /server/node_modules in the container is ./node_modules on the host. If the host and container environments don't agree (MacOS host/Linux container; Ubuntu host/Alpine container; ...) there can be compatibility issues that cause this to break.
If you also mount an anonymous volume over the node_modules directory (/server/node_modules) then only the first time you run the container the node_modules directory from the image gets copied into the volume, and then the volume content gets mounted into the container. If you update the image, the old volume contents take precedence (changes to package.json get ignored).
When the image is built only the contents of the build: block have an effect. There are no volumes: mounted, environment: variables aren't set, and the build environment isn't attached to networks:.
The upshot of this is that if you don't have volumes at all:
version: '3.8'
services:
app:
build: .
ports: ['3000:3000']
It is completely disconnected from the host environment. You need to docker-compose build the image again if your code changes. On the other hand, you can docker push the built image to a registry and run it somewhere else, without needing a separate copy of Node or the application source code.
If you have a volume mount replacing the application directory then everything in the image build is ignored. I've seen some questions that take this to its logical extent and skip the image build, just bind-mounting the host directory over an unmodified node image. There's not really benefit to using Docker here, especially for a front-end application; install Node instead of installing Docker and use ordinary development tools.
I am using docker container with tomcat to run application. The application is saving some data in a folder that starts with dot(.). In the yml file I have something like this:
volumes:
- /my/path/folder:/path/.* folder
It is saving the required folder on the disk, but when I'm starting again the container it doesn't persist what was saved on the disk. Is there a way to do this correctly? I prefer to not change the name of the folder.
I can confirm I have the same problem with docker-compose (1.22.0 on Fedora 29). Docker-compose doesn't seem to allow dotfiles in Volumes declaration.
This works in docker-compose.yml:
volumes:
- data:/root/folder
This does not:
volumes:
- data:/root/.folder
Where both exist in the container. I've posted on docker hub and no one there seems to know either.
I am aiming to configure docker so that when I modify a file on the host the change is propagated inside the container file system.
You can think of this as hot reloading for server side node code.
The nodemon file watcher should restart the server in response to file changes.
However these file changes on the host volume don't seem to be reflected inside the container when I inspect the container using docker exec pokerspace_express_1 bash and inspect a modified file the changes are not propagated inside the container from the host.
Dockerfile
FROM node:8
MAINTAINER therewillbecode
# Create app directory
WORKDIR src/app
RUN npm install nodemon -g
# Install app dependencies
COPY package.json .
# For npm#5 or later, copy package-lock.json as well
# COPY package.json package-lock.json ./
RUN npm install
CMD [ "npm", "start" ]
docker-compose.yml
version: '2'
services:
express:
build: .
depends_on:
- mongo
environment:
- MONGO_URL=mongo:27017/test
- SERVER_PORT=3000
volumes:
- ./:/src/app
ports:
- '3000:3000'
links:
- mongo
mongo:
image: mongo
ports:
- '27017:27017'
mongo-seed:
build: ./mongo-seed
links:
- mongo
.dockerignore
.git
.gitignore
README.md
docker-compose.yml
How can I ensure that host volume file changes are reflected in the container?
Try something like this in your Dockerfile:
CMD ["nodemon", "-L"]
Some people had a similar issue and were able to resolve it with passing -L (which means “legacy watch”) to nodemon.
References:
https://github.com/remy/nodemon/issues/419
http://fostertheweb.com/2016/02/nodemon-inside-docker-container/#why-isnt-nodemon-reloading
Right, so with Docker we need to re-build the image or figure out some clever solution.
You probably do not want to rebuild the image every time you make a change to your source code.
Let's figure out a clever solution. Let's generalize the Dockerfile a bit to solve your problem and also help others.
So this is the boilerplate Dockerfile:
FROM node:alpine
WORKDIR '/app'
COPY package.json .
RUN npm install
COPY . .
CMD ["npm", "run", "start"]
Remember, during the image building process we are creating a temporary container. When we make the copies we are essentially taking a snapshot of the contents /src and /public. Its a snapshot that is locked in time and by default will not be updated by making changes to the code.
So in order to get these changes to files /src and /public, we need to abandon doing a straight copy, we are going to adjust the docker run command that we use to start up our container.
We are going to make use of a feature called volume.
With Docker volume we setup a placeholder inside our Docker container, so instead of copying over our entire/src directory we can imagine we are going to put a reference to those files and give us access to the files and folders inside of the local machine.
We are setting up a mapping from a folder inside the container to a folder outside a container. The command to use is a bit painful, but once its documented here you can bookmark this answer.
docker run -p 3000:3000 -v /app/node_modules -v $(pwd):/app <image_id>
-v $(pwd):/app used to set up a volume in present working directory. This is a shortcut. So we are saying get the present working directory, get everything inside of it and map it up to our running container. It's long winded I know.
To implement this you will have to first rebuild your docker image by running:
docker build -f Dockerfile.dev .
Then run:
docker run -p 3000:3000 -v $(pwd):/app <image_id>
Then you are going to very quickly get an error message, the react-scripts not found error. You will see that message because I skipped the -v /app/node_modules.
So what's up with that?
The volume command sets up a mapping and when we do, we are saying take everything inside of our present working directory and map it up to our /appfolder, but the issue is there is no /node_modules folder which is where all our dependencies exist.
So the /node_modules folder got overwritten.
So we are essentially pointing to nothing and thats why we need that -v /app/node_modules with no colon because the colon is to map up the folder inside a container to a folder outside the container. Without the colon we are saying want it to be a placeholder, don't map it up against anything.
Now, go ahead and run: docker run -p 3000:3000 -v $(pwd):/app <image_id>
Once done, you can make all the changes you want to your project and see them "hot reload" in your browser. No need to figure out how to implement Nodemon.
So whats happening there is any changes made to your local file system is getting propagated into your container, the server inside your container sees the change and updates.
Now, I know its hard and annoying to remember such a long command, in enters Docker Compose.
We can make use of Docker Compose to dramatically simplify the command we have to run to start up the container.
So to implement that you create a Docker Compose file and inside of it you will include the port setting and the two volumes that you need.
Inside your root project, make a new file called docker-compose.yml.
Inside there you will add this:
version: '3'
services:
web:
build: .
ports:
- "3000:3000"
volumes:
- /app/node_modules
- .:/app
Then run: docker-compose up
Daniel's answer partially worked for me, but the hot reloading still doesn't work. I'm using a Windows host and had to change his docker-compose.yml to
version: '3'
services:
web:
build: .
ports:
- "3000:3000"
volumes:
- /App/node_modules
- .:/App
(I changed the volumes arguments from /app/node_modules to /App/node_modules and from .:/app to .:/App. This enables changes to be passed to the container, however the hot reloading still doesn't work. I have to use docker-compose up --build each time I want to refresh the app.)