I am using docker container with tomcat to run application. The application is saving some data in a folder that starts with dot(.). In the yml file I have something like this:
volumes:
- /my/path/folder:/path/.* folder
It is saving the required folder on the disk, but when I'm starting again the container it doesn't persist what was saved on the disk. Is there a way to do this correctly? I prefer to not change the name of the folder.
I can confirm I have the same problem with docker-compose (1.22.0 on Fedora 29). Docker-compose doesn't seem to allow dotfiles in Volumes declaration.
This works in docker-compose.yml:
volumes:
- data:/root/folder
This does not:
volumes:
- data:/root/.folder
Where both exist in the container. I've posted on docker hub and no one there seems to know either.
Related
Supposed I have a Docker container and a folder on my host /hostFolder. Now if I want to add this folder to the Docker container as a volume, then I can do this either by using ADD in the Dockerfile or mounting it as a volume.
So far, so good.
Now /hostFolder contains a sub-folder, /hostFolder/subFolder.
I want to mount /hostFolder into the Docker container (whether as read-write or read-only does not matter, works both for me), but I do NOT want to have it included /hostFolder/subFolder. I want to exclude this, and I also want the Docker container be able to make changes to this sub-folder, without the consequence of having it changed on the host as well.
Is this possible? If so, how?
Using docker-compose I'm able to use node_modules locally, but ignore it in the docker container using the following syntax in the docker-compose.yml
volumes:
- './angularApp:/opt/app'
- /opt/app/node_modules/
So everything in ./angularApp is mapped to /opt/app and then I create another mount volume /opt/app/node_modules/ which is now empty directory - even if in my local machine ./angularApp/node_modules is not empty.
If you want to have subdirectories ignored by docker-compose but persistent, you can do the following in docker-compose.yml:
volumes:
node_modules:
services:
server:
volumes:
- .:/app
- node_modules:/app/node_modules
This will mount your current directory as a shared volume, but mount a persistent docker volume in place of your local node_modules directory. This is similar to the answer by #kernix, but this will allow node_modules to persist between docker-compose up runs, which is likely the desired behavior.
For those trying to get a nice workflow going where node_modules isn't overridden by local this might help.
Change your docker-compose to mount an anonymous persistent volume to node_modules to prevent your local overriding it. This has been outlined in this thread a few times.
services:
server:
build: .
volumes:
- .:/app
- /app/node_modules
This is the important bit we were missing. When spinning up your stack use docker-compose -V. Without this if you added a new package and rebuilt your image it would be using the node_modules from your initial docker-compose launch.
-V, --renew-anon-volumes Recreate anonymous volumes instead of retrieving
data from the previous containers.
To exclude a file, use the following
volumes:
- /hostFolder:/folder
- /dev/null:/folder/fileToBeExcluded
With the docker command line:
docker run \
--mount type=bind,src=/hostFolder,dst=/containerFolder \
--mount type=volume,dst=/containerFolder/subFolder \
...other-args...
The -v option may also be used (credit to Bogdan Mart), but --mount is clearer and recommended.
First, using the ADD instruction in a Dockerfile is very different from using a volume (either via the -v argument to docker run or the VOLUME instruction in a Dockerfile). The ADD and COPY commands just take a copy of the files at the time docker build is run. These files are not updated until a fresh image is created with the docker build command. By contrast, using a volume is essentially saying "this directory should not be stored in the container image; instead use a directory on the host"; whenever a file inside a volume is changed, both the host and container will see it immediately.
I don't believe you can achieve what you want using volumes, you'll have to rethink your directory structure if you want to do this.
However, it's quite simple to achieve using COPY (which should be preferred to ADD). You can either use a .dockerignore file to exclude the subdirectory, or you could COPY all the files then do a RUN rm bla to remove the subdirectory.
Remember that any files you add to image with COPY or ADD must be inside the build context i.e. in or below the directory you run docker build from.
for the people who also had the issue that the node_modules folder would still overwrite from your local system and the other way around
volumes:
node_modules:
services:
server:
volumes:
- .:/app
- node_modules:/app/node_modules/
This is the solution, With the trailing / after the node_modules being the fix.
Looks like the old solution doesn't work anymore(at least for me).
Creating an empty folder and mapping target folder to it helped though.
volumes:
- ./angularApp:/opt/app
- .empty:/opt/app/node_modules/
I found this link which saved me: Working with docker bind mounts and node_modules.
This working solution will create a "exclude" named volume in docker volumes manager. The volume name "exclude" is arbitrary, so you can use a custom name for the volume intead exclude.
services:
node:
command: nodemon index.js
volumes:
- ./:/usr/local/app/
# the volume above prevents our host system's node_modules to be mounted
- exclude:/usr/local/app/node_modules/
volumes:
exclude:
You can see more infos about volumes in Official docs - Use a volume with docker compose
To exclude a mounted file contained in the volume of your machine, you will have to overwrite it by allocating a volume to this same file.
In your config file:
services:
server:
build : ./Dockerfile
volumes:
- .:/app
An example in you dockerfile:
# Image Location
FROM node:13.12.0-buster
VOLUME /app/you_overwrite_file
My simple docker-compose.yaml file:
version: '3'
services:
website:
image: php:7.4-cli
container_name: php72
volumes:
- .hi:/var/www/html
ports:
- 8000:80
in folder hi/ I have just an index.php with a hello world print in it. (Do I need to have a Dockerfile here also?)
Now I just want to run this container with docker compose up:
$ docker compose up
host path ("/Users/xy/project/TEST/hi") not allowed as volume source, you need to reference an Azure File Share defined in the 'volumes' section
What has "docker compose" up to do with Azure? - I don't want to use Azure File Share at this moment, and I never mentioned or configured anything with Azure. I logged out of azure with: $az logout but got still this strange error on my macbook.
I've encountered the same issue as you but in my case I was trying to use an init-mongo.js script for a MongoDB in an ACI. I assume you were working in an Azure context at some point, I can't speak on that logout issue but I can speak to volumes on Azure.
If you are trying to use volumes in Azure, at least in my experience, (regardless if you want to use file share or not) you'll need to reference an Azure file share and not your host path.
Learn more about Azure file share: Mount an Azure file share in Azure Container Instances
Also according to the Compose file docs:
The top-level volumes key defines a named volume and references it from each service’s volumes list. This replaces volumes_from in earlier versions of the Compose file format.
So the docker-compose file should look something like this
docker-compose.yml
version: '3'
services:
website:
image: php:7.4-cli
container_name: php72
volumes:
- hi:/var/www/html
ports:
- 8000:80
volumes:
hi:
driver: azure_file
driver_opts:
share_name: <name of share>
storage_account_name: <name of storage account>
Then just place the file/folder you wanted to use in the file share that is driving the volume beforehand. Again, I'm not sure why you are encountering that error if you've never used Azure but if you do end up using volumes with Azure this is the way to go.
Let me know if this helps!
I was testing to deploy docker compose on Azure and I faced the same problem as yours
then I tried to use docker images and that gave me the clue:
it says: image command not available in current context, try to use default
so I found the command "docker context use default"
and it worked!
so Azure somehow changed the docker context, and you need to change it back
https://docs.docker.com/engine/context/working-with-contexts/
I have a simple NodeJs app with MongoDB using Docker(docker-compose). Everything works just fine, but Mongo's mounted volume is created under ownership of user 999.
Docker is executed under the permission of a non-root user.
Here is the mounted volume permissions info:
drwxr-sr-x 4 999 www-data 4,0K Aug 5 21:56 mongo-data
Here is my docker-compose.yml file:
version: "3.3"
services:
api:
.....
mongodb:
image: mongo:latest
container_name: "mongodb"
environment:
- MONGO_DATA_DIR=/data/db
- MONGO_LOG_DIR=/dev/null
volumes:
- ./mongo-data:/data/db
ports:
- 27017:27017
command: mongod --smallfiles --logpath=/dev/null
volumes:
mongo-data:
Next time when executing: docker-compose up -d --build will throw this error:
ERROR: Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running?
If the ownership of the mounted volume is changed, everything is back to normal until next time.
I mention that I previously used this kind of configuration with MySQL and Redis, but I never encountered this issue.
Any ideas on how to fix it?
Thank you!
This also creates issues in managing those files on the host, such as backing up those files as a non-privileged user, which might be something one wants on a developer PC.
Here's the actual solution!
You can:
docker run --user some-other-user-id:some-group-id
or, in docker-compose, here's a minimal example:
version: '3.5'
services:
mongo:
image: mongo:latest
user: '1000:1000'
volumes:
- ./data:/data/db
After running this, looking at the data directory, it contains only user/group of 1000. Of course, set this to whatever you find appropriate.
I found that it was necessary to create the directory first under the user that is specified. The docker-compose runs as root, if I'm not mistaken, and the directory gets owned by root and that creates mayhem.
So,
mkdir data
docker-compose up
Enjoy!
On Ubuntu based images, 999 will be the first possible system assigned UID for unknown users, with further IDs counting down.
What this could mean is that the directory you are mounting might be a network path or might be copied from another machine, both leading to the user being unknown on your machine, leading to a system assigned UID.
Note that you can use the ADD --chown X:Y Syntax to add files under a user with a defined user ID.
Building off of the previous answer, to avoid issues like this in the future, you could consider using managed volumes rather than a specific directory in your filesystem. Keeps all those weird files out of sight, and avoids odd permissions issues like this. Here's my Docker Compose setup for Mongo:
https://github.com/alexmacarthur/local-docker-db/blob/master/mongo/docker-compose.yml#L6
My webapp/Dockerfile looks like this:
FROM project_base-container
EXPOSE 9100
# ...
# Using copy instead of mount, since we need to write in sub-directories like node_modules etc.
COPY . /usr/src/app/webapp
CMD ["bash", "webapp/scripts/build_and_run.sh", "setup_deps_and_run_app"]
I want to allow the app to be able to read the source code and also write into sub-folders like node_modules, but I don't want those changes to come in my local machine. Hence, I have two choices:
Change --prefix='/tmp' in node install and mount the src as ro
Copy COPY the src and then the container can write wherever it wants.
Solution 1 wrecks havoc, because now I have to copy/link all files like package.json, index.html etc to the prefix location.
Solution 2 is what I have done above.
When I use COPY everything is fine for the first time. But now the problem is that after changes in source code, I want to update the source code in the image every time I do:
sudo docker-compose down && sudo docker-compose up --build -d
But the COPY command is cached by docker and won't be updated, even after file changes.
TL;DR: I have a src folder 'webapp' that I want to mount from host as readonly, but my app wants to write to a subfolder 'webapp/node_modules'.
To create a writable mount inside a read only mount, you need to ensure that the read-only directory has the mount points (folders), even if they are empty.
To create empty mount points and commit them, please see this answer.
For running node, you need following 4 folders writable:
$ mkdir webapp/frontend/node_modules webapp/frontend/build webapp/frontend/.config webapp/frontend/.npm
$ cat > webapp/frontend/node_modules/.gitignore
# Ignore everything in this directory
*
# Except this file
!.gitignore
$ git add -f webapp/frontend/node_modules/.gitignore
$ cat docker-compose.yml # Filtered output below
version: "2"
services:
webapp:
build: ./webapp
expose:
- "9900"
# Named volumes, defined below.
volumes:
- ./webapp:/usr/src/app/webapp:ro
- webapp_config:/usr/src/app/webapp/frontend/.config:rw
- webapp_npm:/usr/src/app/webapp/frontend/.npm:rw
- webapp_node_modules:/usr/src/app/webapp/frontend/node_modules:rw
- webapp_build:/usr/src/app/webapp/frontend/build:rw
- ./config.ini:/usr/src/app/config.ini:ro
# Named volumes. These will stay in the host, but not in the current directory.
volumes:
webapp_node_modules:
webapp_build:
webapp_config:
webapp_npm:
Related answer about writable folders in read-only mounts
See this documentation for different volume/storage options in docker
Related answer about named volumes
You can use internal docker-volumes.
version: "3"
services:
postgres:
image: postgres:9.4
volumes:
- db-data:/var/lib/db
redis:
image: redis
ports:
- "6379:6379"
volumes:
- ./data:/data
volumes:
db-data:
From documentation
Explaination:
Postgress service uses a named volume and that is located inside the directory where docker is installed. For Ubuntu, it is /var/lib/docker/aufs/. This directory may be different and can be found by using docker inspect <vulume_name>.
Redis services uses another type of volumes, the ones which are more popular and which are used to mount host directory inside container directory. The data folder is in the same directory as the one in which docker-compose.yml file is present, and this folder is mounted inside the container at /data.
Is it possible for you to store all code you want to be editable in some sub-directory (say src)? If yes, then you could do the following
Use COPY as you are doing now, which is probably needed for deployment anyway.
In addition (read-only) mount src to /usr/src/app/webapp/src during development.
In my main project directory I have another local directory that I store my mongodb data in which is ./pokemondb. I know this directory is filled with my data because I have run mongoimport --db stats --colletion pokemon --file stats.json and can confirm the data is there in the mongo shell. I also have a docker-compose file that looks like this
pokemon-app:
build: .
ports:
- "3000:3000"
links:
- mongo
mongo:
image: mongo:3.2.4
ports:
- "27017:27017"
volumes:
- "./pokemondb:/data/db"
I run docker-compose up and no errors occur. But the problem is that the mongodb directory /data/db now doesn't contain the mounted volume I tried to pass. I can confirm that the data wasn't passed correctly by executing docker exec -ti [mongo container id] bash and checking the /data/db directory with the mongo shell, indeed nothing is there. What am I doing wrong and why is my mongodb data directory not mounting the volume correctly?
EDIT: I found an unexpected solution to my problem. One of my problems was that I had a fundamental misunderstanding of what docker volumes are. I previously thought that docker volumes were meant to copy data from your local machine into a docker container when it starts up. But in fact docker volumes are meant to save data on your local machine generated in the docker container. The solution the original problem I asked above was to create a dockerfile that copies the data into the image then import the data into the database when the container starts up. My final docker compose file looks like this.
app:
build: .
ports:
- 3000:3000
links:
- mongodb
mongodb:
image: mongo:3.2.4
ports:
- 27017:27017
stats:
build: ./stats
links:
- mongodb
If you're using Docker Toolbox for OSX then only the home directory is available to the VM. If you're outside the your home directory, you need to either add another shared volume to the Virtualbox VM, or move the project under your home directory.