Relatively new to Docker so trying to understand how to accomplish my task.
Locally, I:
Build the image
Push the image to some URL
SSH into Linux VM
docker pull image from URL
docker run image_name
This image, when run, downloads 2 fairly large csv.gz's. When unzipped, the two CSV's are about 15GB each.
I set up /app on the Linux VM to have 200GB available. So, in short, I need to have the Docker image download those 2 CSV's there. However no matter what I've tried within my Dockerfile, I see
'No space left on device' when it gets to the part to download the CSVs.
I've tried to set WORKDIR to /app, but that does not help.
Do I need to use a daemon.json file? Does some sort of Docker setting need to be changed on the Linux VM? Do I need to look into Docker volumes?
Relevant pieces of Dockerfile:
FROM centos/python-36-centos7
USER root
WORKDIR /usr/src/app
COPY . .
As for /usr/src/app, I've never seen anything in there. I normally use /usr/src/app since that's what I use for my Cloud Foundry deployments.
Any insight to point me in the right direction would be appreciated.
Doing the following resolved the issue:
Create (if daemon.json does not exist): /etc/docker/daemon.json
And write
{
“data-root”: “/app”
}
Looks like by default everything goes to /var, and in my case, /var only has 4GB of space. /app is where the 200GB resides.
You will have to restart docker service when creating/saving daemon.json.
Referenced this answer: the one with 88 upvotes
Related
I want to dockerize one project(ML based-deeplearning) /src file. But, the issue is about the space docker is using. During "docker build" stage, the process was stopped as my root directory volume goes to zero.
Why docker is taking so much space?
How to approach it?
Can I configure docker-engine to build docker, in other directory (like normal storage file?).
If I am doing something wrong then please correct me. Thank you for your valuable time.
I am using an image academind/node-example-1 which is a simple node image. You can check it https://hub.docker.com/r/academind/node-example-1 here. What I want while I run the image I want to get the same folder & file structure that is there in the Image. I know that can be done via volume. When I use like
docker run -d --rm --name node-test -p 5000:80 academind/node-example-1
Everything is proper But I want to get the codebase while running, so I tried like
docker run -d --rm --name node-test -p 5000:80 -v /Users/souravbanerjee/Documents/node-docker:node-example-1 academind/node-example-1
Here node-docker is my local folder where I expect the code to be. It runs but not getting the files in the local machine, I'm in doubt here where the source_path:destination_path. Please correct me to please tell me where I'm wrong, or what to do, or my entire thinking is going in the wrong direction or not.
Thanks.
If you read the official doc, you'll see that the first part of the : should be the path somewhere in the host machine (which you're doing), while the later part should match the path "inside" the container (instead you're using image name). Assuming /app being the path (I've taken that course by myself and this is the path AFAIR), it should be:
docker run -d --rm --name node-test -p 5000:80 -v /Users/souravbanerjee/Documents/node-docker:/app academind/node-example-1
I think the correct syntax is to enter the drive information in the volume mapping.
Eg. Users/xxxx/:some/drive/location
However, that would map your empty drive at xxxx over the top of 'location' folder. Thus deleting the existing files in the container.
If you are interested in seeing the contents of the files in the container, you should consider using 'Docker CP' command.
People often use volume mounts to push data (i.e. persistent database files) into a container.
Alternatively, writing log files to the volume mounted location inside the application container. Then those files are then reflected on your local drive
You can copy the files to the current host directory using the command
docker cp node-test:/app .
when the container is running
I have a very simple node js app and the project structure looks like this.
index.js
package.json
package-lock.json
Dockerfile
FROM node:12.18.2-alpine
WORKDIR /test-app
COPY package.json package-lock.json ./
RUN npm i
COPY . ./
EXPOSE 3000
ENTRYPOINT [ "node", "index.js" ]
docker-compose.yml
version: '3.2'
services:
test-app:
build: .
ports:
- "3000:3000"
volumes:
- .:/test-app
- "test_app_node_modules:/test-app/node_modules"
volumes:
test_app_node_modules:
driver: local
If you look at the volumes section in docker-compose.yml file, first I'm bind mounting my current directory on the host machine to the test-app directory on the container. This means :
whatever files or directories that were inside my current dir will get reflected on the container dir and any changes made to the container dir will also get reflected back to the host dir.
this means node_modules that were installed in the test-app dir of the container, during docker build, were overwritten as well.
and the next step in the volumes section is named volumes. This means:
It should copy everything from test-app/node_modules inside container to test_app_node_modules volume. But the test-app/node_modules is empty because step 1 overwrote it.
which means we created an empty volume and mounted it to the container.
If this is so, it should be causing missing dependency error but my app is running properly. I'm not sure where I'm getting node_modules from.
Also, I see an empty node_modules folder in the host directory. I assume the reason behind this is "test_app_node_modules:/test-app/node_modules" looks for the node_modules in the container but it doesn't exist so it creates one and as a result, it gets reflected back to the host dir.
I'm not able to grasp the idea of volume mounting. What is the flow here? How node_modules are begin stored into the volumes when there are none?
At a purely mechanical level, a couple of things happen here.
Docker sorts the volume mounts. It knows that /test-app/node_modules is a subdirectory of /test-app. So Docker will
Create the bind mount from the host directory to the container directory
Create an empty node_modules directory if required to be a mount point
Mount the Docker volume on that mount point
This is where the empty node_modules directory on the host comes from: Docker creates it to be a mount point, after it's done the bind mount, so the changes there are reflected in the host content.
When the named volume is mounted in the container, if and only if the named volume is empty, Docker will copy the content from the image into the volume; this specific step ignores that there's already a volume mounted in the container. (This also works if the volume is an anonymous volume, which you see in other Node examples; it does not work for host bind mounts, if older content is in the volume, or on Kubernetes.)
So that's why this apparently works. As the other answers to this question note, this isn't an especially effective use of Docker. I'd also recommend just deleting these volumes:, and directly running node index.js on the host if you need a live development setup.
in your docker file you have first created WORKDIR /test-app inside it you have added a package.json file and installed dependencies RUN npm i so now there is already node_module present inside docker image itself.
after that using COPY . ./ you are adding extra file like index and other all to the docker image.
if you will remove whole volume part then also it will work as your docker image contain code and it's dependencies.
version: '3.2'
services:
test-app:
build: .
ports:
- "3000:3000"
I'm not really sure why you would want to set up a docker container in this way, but the reason it's not working is due to a misunderstanding of the direction in which volumes and bind-mounts work. You say:
It should copy everything from test-app/node_modules inside container to test_app_node_modules volume. But the test-app/node_modules is empty because step 1 overwrote it.
This is back to front. When you use a volume, the volume is copied into the target. This is the whole point of volumes - they're designed to allow you to persist data even if you rebuild the container. If you use a bind mounted volume then the host directory is copied into the target in the docker container. So your test_app_node_modules directory on the host machine is copied into /test-app/node_modules in the container. Presumably test_app_node_modules contains all your node modules, hence you get no errors about missing modules.
It's only once your container is actually running that the code running in the container can update/delete data in the volume - not when you're building the container.
I'd like to create a Database image and seed it with some initial data. This seems to work fine, since I'm able to create a container with a volume managed by docker. However, when I try to mount the volume to a directory on my linux host machine, it is created empty instead of seeded.
After several hours of trying different configurations, I narrowed down the problem to it's core: the content of the folder in the container associated with the volume on the host machine is overwritten upon creation.
Below a simple Dockerfile that creates a folder containing a single file. When the container is started it prints out the content of the mounted folder.
FROM ubuntu:18.04
RUN mkdir /opt/test
RUN touch /opt/test/myFile.txt
VOLUME /opt/test
CMD ["ls", "/opt/test"]
I'm building the image with: docker build -t test .
Volume managed by docker
$ docker run -v volume-test:/opt/test --name test test
myFile.txt
Here I'm getting the expected output. With the volume mounted in the space managed by docker. So the ouput of docker volume inspect volume-test is:
{
"CreatedAt": "2020-05-13T10:09:29+02:00",
"Driver": "local",
"Labels": null,
"Mountpoint": "/var/snap/docker/common/var-lib-docker/volumes/volume-test/_data",
"Name": "volume-test",
"Options": null,
"Scope": "local"
}
Volume mounted on host machine
$ docker run -v $(pwd)/volume:/opt/test --name test test
Where nothing is returned since the folder is empty... However, I can see that the volume is created and it's owned by the user root, even though I'm executing the docker run command as another user.
drwxr-xr-x 2 root root 4096 May 13 10:11 volume
As a last test, I tried to see what happens when I create the folder for the volume beforehand and add some content to it (in my case a file called anotherFile.txt).
When I'm running now the container, I'm getting the following result:
$ docker run -v $(pwd)/volume:/opt/test --name test test
anotherFile.txt
Which let's me come to the conclusion, that the content in the folder of the container is overwritten by the content of the folder on the host machine.
I can verify as well with docker inspect -f '{{ .Mounts }}' test, that the volume is mounted at the right place:
[{bind /pathWhere/pwd/pointedTo/volume /opt/test true rprivate}]
Now my question: Is there a way to have the same behavior for volumes on the host machine as for the volumes managed by docker, where the content of the /opt/test folder in the container is copied into the folder on the host defined as volume?
Sidenote: this seems to be the case when using docker on windows and having the option Shared Folders enabled...
Furthermore, it seems as a similar question was already asked here but no answer was found. I decided to make a separate post since I think this is the most generic example to describe this issue.
Desired situation
Data from within the docker image is placed in a specified path on the host.
Your current situation
When creating the image, data is put into /opt/test
When starting the container, you mount the volume on /opt/test
Problem
Because you mount on the same path as where you have put your data, your data gets overwritten.
Solution
Create a file within image during docker-build, for example touch /opt/test/data/myFile.txt,
Use a different path to mount your volume, so that the data is not overwritten, for example /opt/test/mount
Use 'CMD' to copy the files to the volume, like so: CMD ["cp", "-n" "/opt/test/data/*", "/opt/test/mount/"]
Consulted sources
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
https://docs.docker.com/storage/volumes/
I have a Docker container (not image) that crashes when I try to start it. The Docker logs show that it is failing because and Apache2 conf file can't find a directory (/var/www/html/log/ - this is the result of me trying to get SSL setup and forgot to create this directory after I referenced it in the 000-default.conf file and restarted Apache).
How do I create this directory in the container without having to start the container itself?
You have 4.5 options that comes to my mind:
You can rebuild the image and set up the directory while doing it.
You can attach a volume while starting the image, but in this case your changes will remain in your disk and not in your container.
You can run the image overriding the entry point with --entrypoint="bash" or something. You need to do it with -ti flag so that it begins in interactive mode. Then make your changes and run docker commit -p <container> <image:tag> -p pauses container while commiting. I recommend this unless it absolutely needs to be running.
I am not sure if this one works so I give half point :P but if it does this would be the fastest option actually. You can start the container in interactive mode with docker start -i container which would attach a terminal. And if you have time until container exits or read that part of configuration, you can create the folder.
Ah finally, I have just remembered, you should be able to move files and folders from your file system to container using docker cp [container:]<source> [container:]<destination> even while container is not running.
In general, if you're using a base Docker image for Apache (for example, httpd/2.4/Dockerfile), it should already have "/var/www/html/log".
SUGGESTION 1: Please make sure you're starting with a "good" base image.
SUGGESTION 2: Add "mkdir -p /var/www/html/log" to your Dockerfile, and rebuild the image.
I'm not sure how you're using your image - what you want the image to contain besides Apache - but:
SUGGESTION 3: Google for a simple tutorial that matches your use case, and see what steps you might be "missing". For example: Dockerize your Laravel Application