In my main project directory I have another local directory that I store my mongodb data in which is ./pokemondb. I know this directory is filled with my data because I have run mongoimport --db stats --colletion pokemon --file stats.json and can confirm the data is there in the mongo shell. I also have a docker-compose file that looks like this
pokemon-app:
build: .
ports:
- "3000:3000"
links:
- mongo
mongo:
image: mongo:3.2.4
ports:
- "27017:27017"
volumes:
- "./pokemondb:/data/db"
I run docker-compose up and no errors occur. But the problem is that the mongodb directory /data/db now doesn't contain the mounted volume I tried to pass. I can confirm that the data wasn't passed correctly by executing docker exec -ti [mongo container id] bash and checking the /data/db directory with the mongo shell, indeed nothing is there. What am I doing wrong and why is my mongodb data directory not mounting the volume correctly?
EDIT: I found an unexpected solution to my problem. One of my problems was that I had a fundamental misunderstanding of what docker volumes are. I previously thought that docker volumes were meant to copy data from your local machine into a docker container when it starts up. But in fact docker volumes are meant to save data on your local machine generated in the docker container. The solution the original problem I asked above was to create a dockerfile that copies the data into the image then import the data into the database when the container starts up. My final docker compose file looks like this.
app:
build: .
ports:
- 3000:3000
links:
- mongodb
mongodb:
image: mongo:3.2.4
ports:
- 27017:27017
stats:
build: ./stats
links:
- mongodb
If you're using Docker Toolbox for OSX then only the home directory is available to the VM. If you're outside the your home directory, you need to either add another shared volume to the Virtualbox VM, or move the project under your home directory.
Related
I am trying to create a composition where two or more docker service can connect to each other in some way.
Here is my composition.
# docker-compose.yaml
version: "3.9"
services:
database:
image: "strapi-postgres:test"
restart: "always"
ports:
- "5435:5432"
project:
image: "strapi-project:test"
command: sh -c "yarn start"
restart: always
ports:
- "1337:1337"
env_file: ".env.project"
depends_on:
- "database"
links:
- "database"
Services
database
This is using a Image that is made with of Official Postgres Image.
Here is Dockerfile
FROM postgres:alpine
ENV POSTGRES_USER="root"
ENV POSTGRES_PASSWORD="password"
ENV POSTGRES_DB="strapi-postgres"
and using the default exposed port 5432 and forwarding to 5435 as defined in the Composition.
So the database service starts at some IPAddress that can be found using docker inspect.
project
This is a Image running a node application(strapi project configured to use postgres database).
Here is Dockerfile
FROM node:lts-alpine
WORKDIR /project
ADD package*.json .
ADD yarn.lock .
RUN npm install
COPY . .
RUN npm run build
EXPOSE 1337
and I am builing the Image using docker build. That gives me an Image with No Foreground Process.
Problems
When I was running the composition, the strapi-project container Exits with Error Code(0).
Solution: So I added command yarn start to run the Foreground Process.
As the project Starts it could not connect to database since it is trying to connect to 127.0.0.1:5432 (5432 since it should try to connect to the container port of database service and not 5435). This is not possible since this tries to connect to port 5432 inside the container strapi-project, which is not open for any process.
Solution: So I used the IPAddress that is found from the docker inspect and used that in a .env.project and passed this file to the project service of the Composition.
For Every docker compose up there is a incremental pattern(n'th time 172.17.0.2, n+1'th time 172.18.0.2, and so on) for the IPAddress of the Composition. So Everytime I run composition I need to edit the .env.project.
All of these are some hacky way to patch them together. I want some way to Create the Postgres database service to start first and then project to configure, connect, and to the database, start automatically.
Suggest me any edits, or other ways to configure them.
You've forgotten to put the CMD in your Dockerfile, which is why you get the "exited (0)" status when you try to run the container.
FROM node:lts-alpine
...
CMD yarn start
Compose automatically creates a Docker network and each service is accessible using its Compose container name as a host name. You never need to know the container-internal IP addresses and you pretty much never need to run docker inspect. (Other answers might suggest manually creating networks: or overriding container_name: and these are also unnecessary.)
You don't show where you set the database host name for your application, but an environment: variable is a common choice. If your database library doesn't already honor the standard PostgreSQL environment variables then you can reference them in code like process.env.PGHOST. Note that the host name will be different running inside a container vs. in your normal plain-Node development environment.
A complete Compose file might look like
version: "3.8"
services:
database:
image: "strapi-postgres:test"
restart: "always"
ports:
- "5435:5432"
project:
image: "strapi-project:test"
restart: always
ports:
- "1337:1337"
environment:
- PGHOST=database
env_file: ".env.project"
depends_on:
- "database"
Supposed I have a Docker container and a folder on my host /hostFolder. Now if I want to add this folder to the Docker container as a volume, then I can do this either by using ADD in the Dockerfile or mounting it as a volume.
So far, so good.
Now /hostFolder contains a sub-folder, /hostFolder/subFolder.
I want to mount /hostFolder into the Docker container (whether as read-write or read-only does not matter, works both for me), but I do NOT want to have it included /hostFolder/subFolder. I want to exclude this, and I also want the Docker container be able to make changes to this sub-folder, without the consequence of having it changed on the host as well.
Is this possible? If so, how?
Using docker-compose I'm able to use node_modules locally, but ignore it in the docker container using the following syntax in the docker-compose.yml
volumes:
- './angularApp:/opt/app'
- /opt/app/node_modules/
So everything in ./angularApp is mapped to /opt/app and then I create another mount volume /opt/app/node_modules/ which is now empty directory - even if in my local machine ./angularApp/node_modules is not empty.
If you want to have subdirectories ignored by docker-compose but persistent, you can do the following in docker-compose.yml:
volumes:
node_modules:
services:
server:
volumes:
- .:/app
- node_modules:/app/node_modules
This will mount your current directory as a shared volume, but mount a persistent docker volume in place of your local node_modules directory. This is similar to the answer by #kernix, but this will allow node_modules to persist between docker-compose up runs, which is likely the desired behavior.
For those trying to get a nice workflow going where node_modules isn't overridden by local this might help.
Change your docker-compose to mount an anonymous persistent volume to node_modules to prevent your local overriding it. This has been outlined in this thread a few times.
services:
server:
build: .
volumes:
- .:/app
- /app/node_modules
This is the important bit we were missing. When spinning up your stack use docker-compose -V. Without this if you added a new package and rebuilt your image it would be using the node_modules from your initial docker-compose launch.
-V, --renew-anon-volumes Recreate anonymous volumes instead of retrieving
data from the previous containers.
To exclude a file, use the following
volumes:
- /hostFolder:/folder
- /dev/null:/folder/fileToBeExcluded
With the docker command line:
docker run \
--mount type=bind,src=/hostFolder,dst=/containerFolder \
--mount type=volume,dst=/containerFolder/subFolder \
...other-args...
The -v option may also be used (credit to Bogdan Mart), but --mount is clearer and recommended.
First, using the ADD instruction in a Dockerfile is very different from using a volume (either via the -v argument to docker run or the VOLUME instruction in a Dockerfile). The ADD and COPY commands just take a copy of the files at the time docker build is run. These files are not updated until a fresh image is created with the docker build command. By contrast, using a volume is essentially saying "this directory should not be stored in the container image; instead use a directory on the host"; whenever a file inside a volume is changed, both the host and container will see it immediately.
I don't believe you can achieve what you want using volumes, you'll have to rethink your directory structure if you want to do this.
However, it's quite simple to achieve using COPY (which should be preferred to ADD). You can either use a .dockerignore file to exclude the subdirectory, or you could COPY all the files then do a RUN rm bla to remove the subdirectory.
Remember that any files you add to image with COPY or ADD must be inside the build context i.e. in or below the directory you run docker build from.
for the people who also had the issue that the node_modules folder would still overwrite from your local system and the other way around
volumes:
node_modules:
services:
server:
volumes:
- .:/app
- node_modules:/app/node_modules/
This is the solution, With the trailing / after the node_modules being the fix.
Looks like the old solution doesn't work anymore(at least for me).
Creating an empty folder and mapping target folder to it helped though.
volumes:
- ./angularApp:/opt/app
- .empty:/opt/app/node_modules/
I found this link which saved me: Working with docker bind mounts and node_modules.
This working solution will create a "exclude" named volume in docker volumes manager. The volume name "exclude" is arbitrary, so you can use a custom name for the volume intead exclude.
services:
node:
command: nodemon index.js
volumes:
- ./:/usr/local/app/
# the volume above prevents our host system's node_modules to be mounted
- exclude:/usr/local/app/node_modules/
volumes:
exclude:
You can see more infos about volumes in Official docs - Use a volume with docker compose
To exclude a mounted file contained in the volume of your machine, you will have to overwrite it by allocating a volume to this same file.
In your config file:
services:
server:
build : ./Dockerfile
volumes:
- .:/app
An example in you dockerfile:
# Image Location
FROM node:13.12.0-buster
VOLUME /app/you_overwrite_file
I have a simple NodeJs app with MongoDB using Docker(docker-compose). Everything works just fine, but Mongo's mounted volume is created under ownership of user 999.
Docker is executed under the permission of a non-root user.
Here is the mounted volume permissions info:
drwxr-sr-x 4 999 www-data 4,0K Aug 5 21:56 mongo-data
Here is my docker-compose.yml file:
version: "3.3"
services:
api:
.....
mongodb:
image: mongo:latest
container_name: "mongodb"
environment:
- MONGO_DATA_DIR=/data/db
- MONGO_LOG_DIR=/dev/null
volumes:
- ./mongo-data:/data/db
ports:
- 27017:27017
command: mongod --smallfiles --logpath=/dev/null
volumes:
mongo-data:
Next time when executing: docker-compose up -d --build will throw this error:
ERROR: Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running?
If the ownership of the mounted volume is changed, everything is back to normal until next time.
I mention that I previously used this kind of configuration with MySQL and Redis, but I never encountered this issue.
Any ideas on how to fix it?
Thank you!
This also creates issues in managing those files on the host, such as backing up those files as a non-privileged user, which might be something one wants on a developer PC.
Here's the actual solution!
You can:
docker run --user some-other-user-id:some-group-id
or, in docker-compose, here's a minimal example:
version: '3.5'
services:
mongo:
image: mongo:latest
user: '1000:1000'
volumes:
- ./data:/data/db
After running this, looking at the data directory, it contains only user/group of 1000. Of course, set this to whatever you find appropriate.
I found that it was necessary to create the directory first under the user that is specified. The docker-compose runs as root, if I'm not mistaken, and the directory gets owned by root and that creates mayhem.
So,
mkdir data
docker-compose up
Enjoy!
On Ubuntu based images, 999 will be the first possible system assigned UID for unknown users, with further IDs counting down.
What this could mean is that the directory you are mounting might be a network path or might be copied from another machine, both leading to the user being unknown on your machine, leading to a system assigned UID.
Note that you can use the ADD --chown X:Y Syntax to add files under a user with a defined user ID.
Building off of the previous answer, to avoid issues like this in the future, you could consider using managed volumes rather than a specific directory in your filesystem. Keeps all those weird files out of sight, and avoids odd permissions issues like this. Here's my Docker Compose setup for Mongo:
https://github.com/alexmacarthur/local-docker-db/blob/master/mongo/docker-compose.yml#L6
I am using Docker Compose for my local development environment for a Full Stack Javascript project.
part of my Docker Compose file look like this
version: "3.5"
services:
frontend:
build:
context: ./frontend/
dockerfile: dev.Dockerfile
env_file:
- .env
ports:
- "${FRONTEND_PORT_NUMBER}:${FRONTEND_PORT_NUMBER}"
container_name: frontend
volumes:
- ./frontend:/code
- frontend_deps:/code/node_modules
- ../my-shared-module:/code/node_modules/my-shared-module
I am trying to develop a custom Node module called my-shared-module, that's why i added - ../my-shared-module:/code/node_modules/my-shared-module to the Docker Compose file. The node module is hosted in a private Git repo, and is defined like this in package.json
"dependencies": {
"my-shared-module": "http://gitlab+deploy-token....#gitlab.com/.....git",
My problem is,
When I run update my node modules in the docker container using npm install, it download my-shared-module from my private Git repo into /code/node_modules/my-shared-module, and that overwrites the files in host ../my-shared-module, because they are synced.
So my question is, is it possible to have 1 way volume sync in Docker?
when host changes, update container
when container changes, don't update host ?
Unfortunately I don't think this is possible in Docker. Mounting a host volume is always two-way unless you consider a readonly mount to be one-way, but that prevents you from being able modify the file system with things like npm install.
Your best options here would either be to rebuild the image with the new files each time, or bake into your CMD a step to copy the mounted files into a new folder outside of the mounted volume. That way any file changes won't be persisted back to the host machine.
You can script something to do this. Mount your host node_modules to another directory inside the container, and in the entrypoint, copy the directory:
version: "3.5"
services:
frontend:
build:
context: ./frontend/
dockerfile: dev.Dockerfile
env_file:
- .env
ports:
- "${FRONTEND_PORT_NUMBER}:${FRONTEND_PORT_NUMBER}"
container_name: frontend
volumes:
- ./frontend:/code
- frontend_deps:/code/node_modules
- /code/node_modules/my-shared-module
- ../my-shared-module:/host/node_modules/my-shared-module:ro
Then add an entrypoint script to your Dockerfile with something like:
#!/bin/sh
if [ -d /host/node_modules/my-shared-module ]; then
cp -r /host/node_modules/my-shared-module/. /code/node_modules/my-shared-module/.
fi
exec "$#"
I am using docker container with tomcat to run application. The application is saving some data in a folder that starts with dot(.). In the yml file I have something like this:
volumes:
- /my/path/folder:/path/.* folder
It is saving the required folder on the disk, but when I'm starting again the container it doesn't persist what was saved on the disk. Is there a way to do this correctly? I prefer to not change the name of the folder.
I can confirm I have the same problem with docker-compose (1.22.0 on Fedora 29). Docker-compose doesn't seem to allow dotfiles in Volumes declaration.
This works in docker-compose.yml:
volumes:
- data:/root/folder
This does not:
volumes:
- data:/root/.folder
Where both exist in the container. I've posted on docker hub and no one there seems to know either.