I'm using docker/elk image to display my data in kibana dashboard (Version 6.6.0) and It works pretty good. I started the service like using below command.
Docker Image git repo:
https://github.com/caas/docker-elk
Command:
sudo docker-compose up --detach
Expecting that it will run background, and did as expected. After two days the server up and running the and third day the kibana alone getting stopped. and Used below command to make it up and running.
sudo docker run -d <Docer_image_name>
It's up and running when I use docker ps command. But when I tried to hit the kibana server in chrome browser it says not reachable.
So I just used to below command to restart the service.
sudo docker-compose down
After that I can see kibana server in chrome browser which is up and running but I do see all my data is lost.
I used below URL in jenkins to collect the data.
`http://hostname:9200/ecdpipe_builds/extern`al
Any idea how can I resolve this issue?
I did not see the persistent storage configuration the image you mentioned in their GitHub docker-compose file.
This is common to lost data in case of docker container if you did not provide persistent storage configuration. so docker-compose down may cause to lost you data if there is no persistent configuration docker-compose file.
Persisting log data
In order to keep log data across container restarts, this image mounts
/var/lib/elasticsearch — which is the directory that Elasticsearch
stores its data in — as a volume.
You may however want to use a dedicated data volume to persist this
log data, for instance to facilitate back-up and restore operations.
One way to do this is to mount a Docker named volume using docker's -v
option, as in:
$ sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 \
-v elk-data:/var/lib/elasticsearch --name elk sebp/elk
This command mounts the named volume elk-data to
/var/lib/elasticsearch (and automatically creates the volume if it
doesn't exist; you could also pre-create it manually using docker
volume create elk-data).
So you can set these paths in your docker-compose file accordingly. Here is the link that you can check elk-docker-persisting-log-data
Use docker volume or file location as persistant space
Related
Two examples include snap and certbot. I used to type sudo certbot and would be able to add ssl certs to my nginx servers. Now I get this every time I enter certbot. The same thing goes for snap. I'm new to docker and don't understand what is going on. Can somebody explain what is ging on?
Usage: docker compose [OPTIONS] COMMAND
Docker Compose
Options:
--ansi string Control when to print ANSI control characters ("never"|"always"|"auto") (default "auto")
--compatibility Run compose in backward compatibility mode
--env-file string Specify an alternate environment file.
-f, --file stringArray Compose configuration files
--profile stringArray Specify a profile to enable
--project-directory string Specify an alternate working directory
(default: the path of the, first specified, Compose file)
-p, --project-name string Project name
Commands:
build Build or rebuild services
convert Converts the compose file to platform's canonical format
cp Copy files/folders between a service container and the local filesystem
create Creates containers for a service.
down Stop and remove containers, networks
events Receive real time events from containers.
exec Execute a command in a running container.
images List images used by the created containers
kill Force stop service containers.
logs View output from containers
ls List running compose projects
pause Pause services
port Print the public port for a port binding.
ps List containers
pull Pull service images
push Push service images
restart Restart service containers
rm Removes stopped service containers
run Run a one-off command on a service.
start Start services
stop Stop services
top Display the running processes
unpause Unpause services
up Create and start containers
version Show the Docker Compose version information
Run 'docker compose COMMAND --help' for more information on a command.
NEVER INSTALL DOCKER WITH SNAP
I solved the problems. Not sure where everything went wrong, but I completely destroyed snapd from my system following this https://askubuntu.com/questions/1280707/how-to-uninstall-snap. Then I installed snap again and everything works.
INSTALL DOCKER WITH THE OFFICIAL GUIDE (APT)
Go here to install docker the correct way. https://docs.docker.com/engine/install/ubuntu/
If you are new to docker follow this advice and NEVER TYPE snap install docker into you terminal. Follow these words of wisdom or use the first half if you already messed up.
I have a Docker container (not image) that crashes when I try to start it. The Docker logs show that it is failing because and Apache2 conf file can't find a directory (/var/www/html/log/ - this is the result of me trying to get SSL setup and forgot to create this directory after I referenced it in the 000-default.conf file and restarted Apache).
How do I create this directory in the container without having to start the container itself?
You have 4.5 options that comes to my mind:
You can rebuild the image and set up the directory while doing it.
You can attach a volume while starting the image, but in this case your changes will remain in your disk and not in your container.
You can run the image overriding the entry point with --entrypoint="bash" or something. You need to do it with -ti flag so that it begins in interactive mode. Then make your changes and run docker commit -p <container> <image:tag> -p pauses container while commiting. I recommend this unless it absolutely needs to be running.
I am not sure if this one works so I give half point :P but if it does this would be the fastest option actually. You can start the container in interactive mode with docker start -i container which would attach a terminal. And if you have time until container exits or read that part of configuration, you can create the folder.
Ah finally, I have just remembered, you should be able to move files and folders from your file system to container using docker cp [container:]<source> [container:]<destination> even while container is not running.
In general, if you're using a base Docker image for Apache (for example, httpd/2.4/Dockerfile), it should already have "/var/www/html/log".
SUGGESTION 1: Please make sure you're starting with a "good" base image.
SUGGESTION 2: Add "mkdir -p /var/www/html/log" to your Dockerfile, and rebuild the image.
I'm not sure how you're using your image - what you want the image to contain besides Apache - but:
SUGGESTION 3: Google for a simple tutorial that matches your use case, and see what steps you might be "missing". For example: Dockerize your Laravel Application
I want to make image with persisted data to other developers can run this container and use it.
I use this https://hub.docker.com/r/centos/postgresql-96-centos7/ and do next step:
1) Run it with command
docker run -d -e POSTGRESQL_USER=svcg -e POSTGRESQL_PASSWORD=svcg1 -e POSTGRESQL_DATABASE=bcs -p 5432:5432 --expose=5432 --name=postgres --net=docker-net -v psql:/var/lib/pgsql/data centos/postgresql-96-centos7
The result is a running container "postgres" and volume "psql".
2) Restore data from the dumpName.xml
java -jar database_backup_starter.jar -u svcg -p svcg1 -url jdbc:postgresql://192.168.99.100:5432/bcs -m restore -path database_dump
The volume "psql" contain the testing data.
Is it possible to do it in one step ? For instance to create image from the volume or from the docker with volume with data? How to do it?
Since this image declares a VOLUME for the PostgreSQL data, you cannot make a derived image from this with prepopulated data in a volume, either using docker build or docker commit. It won't work (see the note "changing the volume within the Dockerfile").
The standard way to load data into a newly created database container seems to be to put a SQL-syntax database dump in the initialization directory when you start the container for the first time. In this image, it looks like you'd have to wrap the database dump in a shell script and put it in an /opt/app-root/src/postgresql-init/ directory. For the stock postgres image you'd put a *.sql file in /docker-entrypoint-initdb.d/.
I have installed the official MongoDB docker image in a VM on AWS EC2, and the database has already data on it. If I stop the VM (to save expenses overnight), will I lose all the data contained in the database? How can I make it persistent in those scenarios?
There are multiple options to achieve this but the 2 most common ways are:
Create a directory on your host to mount the data
Create a docker
volume to mount the data
1) Create a data directory on a suitable volume on your host system, e.g. /my/own/datadir. Start your mongo container like this:
$ docker run --name some-mongo -v /my/own/datadir:/data/db -d mongo:tag
The -v /my/own/datadir:/data/db part of the command mounts the /my/own/datadir directory from the underlying host system as /data/db inside the container, where MongoDB by default will write its data files.
Note that users on host systems with SELinux enabled may see issues with this. The current workaround is to assign the relevant SELinux policy type to the new data directory so that the container will be allowed to access it:
$ chcon -Rt svirt_sandbox_file_t /my/own/datadir
The source of this is the official documentation of the image.
2) Another possibility is to use a docker volume.
$ docker volume create my-volume
This will create a docker volume in the folder /var/lib/docker/volumes/my-volume. Now you can start your container with:
docker run --name some-mongo -v my-volume:/data/db -d mongo:tag
All the data will be stored in the my-volume so in the folder /var/lib/docker/my-volume. So even when you delete your container and create a new mongo container linked with this volume your data will be loaded into the new container.
You can also use the --restart=always option when you perform your initial docker run command. This mean that your container automatically will restart after a reboot of your VM. When you've persisted your data too there will be no difference between your DB before or after the reboot.
We are trying to move onto Docker for deployment purpose. Our architecture requires to have a redis, a mongodb and several nodejs and java based Docker containers.
So my question is, if suppose the redis/mongodb docker container crashes, do we loose all the data that it had?
We want isolation, but at the same time we don't want to loose data due to malfunction/crashes. Is this even possible to achieve this with Docker or is it something not relevant here?
Any help or comments will be greatly appreciated.
Thanks
The answer is: YES - If a container crashes so that it can not be restored/restarted the data is gone. But, normally containers can be restarted and continued - in that case the data is not lost.
E.g. - the following sequence from the docker docs illustrates how container startup work. Note that the data is not lost here until the container is removed.
# Start a new container
$ JOB=$(sudo docker run -d ubuntu /bin/sh -c "while true; do echo Hello world; sleep 1; done")
# Stop the container
$ sudo docker stop $JOB
# Start the container
$ sudo docker start $JOB
# Restart the container
$ sudo docker restart $JOB
# SIGKILL a container
$ sudo docker kill $JOB
# Remove a container
$ sudo docker stop $JOB # Container must be stopped to remove it
$ sudo docker rm $JOB
Whenever you execute a docker run command you start a new container with fresh data. The data is based on the image you provide and that data is consistent (unless you rebuild the image of course).
So, how should you setup docker to keep your data intact? I think that a good approach is to keep the important data mounted in a volume. Volumes are simply external folders (i.e. a folder from the host system) that holds the data and this data will not be lost even if you reinstall the entire docker daemon.
Example:
docker run -v /some/local/dir:/some/dir/in/redis-container my/redis
This mounts the host folder /some/local/dir as the folder /some/dir/in/redis-container in the running container. If e.g. redis stores its data in that folder you're all set to go and reboots/crashes can be survived.
More info about docker volumes check out the docs. Another great article is the also from the docker website, Managing Data in Containers.
EDIT: After comments I clarified the answer - the data is lost if the container can't be restarted (total crash).
If a container crashes, you won't lose any data - at least not more than with a regular application crash.
The container itself is unlikely to crash (after all, it's only an envelope for your application(s)). Your application(s) running in a container can crash, and if they do, their data will still be on the container filesystem. All you have to do in such a situation is to restart the failed container.
One case where you could lose something is if you explicitly tell Docker to remove the container when it's not running anymore (--rm option).
That being said, for IO-intensive applications such as databases, it is highly recommended to host data on Docker volumes, for performance reasons (a docker volume is a traditional filesystem, while the container default filesystem is a stack of layers and will be slower).