We've NodeJS applications running inside docker containers. Sometimes, if any process gets locked down or due to any other issue the app goes down and we've to manually login to each container n restart the application. I was wondering
if there is any sort of control panel that allow us to easily and quickly restart those and see the whole health of the system.
Please Note: we can't use --restart flag because essentially application doesn't exist with exist code. It run into problem like some process gets blocked, things are just getting bogged down vs any crashes and exist codes. That's why I don't think restart policy will help in this scenario.
I suggest you consider using the new HEALTHCHECK directive in Docker 1.12 to define a custom check for your locking condition. This feature can be combined with the new Docker swarm service feature to specify how many copies of your container you want to have running.
Related
I have a Linux based Docker container running an application which seems to have a memory leak. After around a week requests to the application start to fail and the container requires a restart to reset its state and get things working again.
The error reported by the application is:
java.lang.OutOfMemoryError: Java heap space
Is there a generic method that can be used to trigger a restart, resetting it's state, regardless of which service is being used to host it? If there's not a good generic solution, I'm about to give DigitalOcean a whirl so maybe there's a DigitalOcean specific solution that may work instead?
You can set a restart policy (with flag on-failure) as described here.
Check out the Watchtower project. This is an incredible tool that restarts Docker containers on schedule and also updates containers automatically.
I would like to use Supervisor to run multiple processes in my Docker container, as described here, in Docker docs.
It works but the doc does not say anything about what happens when one of the processes I start crashes.
Following docker behavior logic - when a process crashes - container should stop, and probably later it should be restarted by Docker according to restart policy.
But it does not happen, If one (or all) of application I start exits - container keeps working.
How can I tell Supervisor to exit (and stop the container in this way, because I run it in nodaemon=true mode) as well, when one of monitoring processes exits/crashes?
I found this article which describes that its sometimes valid to run multiple processes in one container.
He describes how to use honcho to create the behaviour you would like: stop the whole container when one of the processes fails.
I'am going to try this now, but I'm still a little bit in doubt because supervisord is used so much more in the docker world and is also described on their own site.
if you want to exit the container when your process stops, don't use supervisor (or any other process manager). just run the process in your container, directly.
but more importantly: don't run multiple critical applications in your container. the golden rule of Docker containers is not 1 process per container, but 1 concern per container. that way your container can properly shut down when that 1 concern (application) exits.
even in the example you cite, they are not running 2 critical processes. they are running 1 app process and then hosting sshd in the same container for ssh access. if sshd stops, it's probably not a big deal. if the apache server stops... well, they're using supervisor to handle that and automatically restart it.
to get what you want, separate your concerns into multiple containers and just run the app in the container directly.
Now I am developing the new content so building the server.
On my server, the base system is the Cent OS(7), I installed the Docker, pulled the cent os, and establish the "WEB SERVER container" Django with uwsgi and nginx.
However I want to up the service, (Database with postgres), what is the best way to do it?
Install postgres on my existing container (with web server)
Build up the new container only for database.
and I want to know each advantage and weak point of those.
It's idiomatic to use two separate containers. Also, this is simpler - if you have two or more processes in a container, you need a parent process to monitor them (typically people use a process manager such as supervisord). With only one process, you won't need to do this.
By monitoring, I mainly mean that you need to make sure that all processes are correctly shutdown if the container receives a SIGSTOP signal. If you don't do this properly, you will end up with zombie processes. You won't need to worry about this if you only have a signal process or use a process manager.
Further, as Greg points out, having separate containers allows you to orchestrate and schedule the containers separately, so you can do update/change/scale/restart each container without affecting the other one.
If you want to keep the data in the database after a restart, the database shouldn't be in a container but on the host. I will assume you want the db in a container as well.
Setting up a second container is a lot more work. You should find a way that the containers know about each other's address. The address changes each time you start the container, so you need to make some scripts on the host. The host must find out the ip-adresses and inform the containers.
The containers might want to update the /etc/hosts file with the address of the other container. When you want to emulate different servers and perform resilience tests this is a nice solution. You will need quite some bash knowledge before you get this running well.
In about all other situations choose for one container. Installing everything in one container is easier for setting up and for developing afterwards. Setting up Docker is just the environment where you want to do your real work. Tooling should help you with your real work, not take all your time and effort.
I am trying to find the best way to automatically start services inside a docker container once it has been restarted.
I don't mean starting the docker container on restart. I'm trying to achieve the following way:
I stop a container; and
when I start it again, the same services (processes) I was running before will start up again.
I.e. if I am running apache and ssh inside the container starting those service on container restart
That's really not the docker way (multiple processes per container). You can try to go down that path, as I did for several months, but you'll find that you'll be going against the docker team's design principles most of the time. I used the phusion/baseimage base image and it really is well designed, with a good init process and support for run-it and ssh out of the box. Tread carefully, if you go down that path however.
I'm planning to set up a jenkins-based CD workflow with Docker at the end.
My idea is to automatically build (by Jenkins) a docker image for every green build, then deploy that image either by jenkins or by 'hand' (I'm not yet sure whether I want to automatically run each green build).
Getting to the point of having a new image built is easy. My question is about the deployment itself. What's the best practice to 'reload' or 'restart' a running docker container? Suppose the image changed for the container, how do I gracefully reload it while having a service running inside? Do I need to do the traditional dance with multiple running containers and load balancing or is there a 'dockery' way?
Suppose the image changed for the container, how do I gracefully reload it while having a service running inside?
You don't want this.
Docker is a simple system for managing apps and their dependencies. It's simple and robust because ALL dependencies of an application are bundled with it. If your app runs today on your laptop, it will run tomorrow on your server. This is because we have captured 100% of the "inputs" for your application.
As soon as you introduce concepts like "upgrade" and "restart", your application can (accidentally) store state internally. That means it might behave differently tomorrow than it does today (after being restarted and upgraded 100 times).
It's better use a load balancer (or similar) to transition between your versions than to try and muck with the philosophy of Docker.
The Docker machine itself should always be immutable as you have to replace it for a new deployment. Storing state inside the Docker container will not work when you want to ship new releases often that you've built on your CI.
Docker supports Volumes which will let you write files that are permanent into some folder on the host. When you then upgrade the Docker container you use the same volume so you've got access to the same files written by the old container:
https://docs.docker.com/userguide/dockervolumes/