Given is a microservice that after some time needs to quit itself. This is no error condition but (in that case) normal behavior.
After exiting it should be automatically restarted.
So currently I have a script run_app.sh:
#!/usr/bin/env bash
while true; do ./app ; done
And in the Dockerfile (inheriting FROM ubuntu:16.04) I run it like this:
CMD ["./run_app.sh"]
It "works" but since app does not have PID 1, it does not receive SIGTERM etc. coming from Kubernetes, which is needed for a graceful shutdown during rolling updates etc.
Using while true; do exec ./app ; done in run_app.sh does not solve the problem since the loop no longer exists when app finished and no restart is performed.
How can I restart the app automatically inside the container without restarting the container / pod every time it quits, but still have the advantages of PID 1?
Well, your loop does restart your app, so this is not your problem. Your problem is that the signal sent to the docker container is not propagated into the container. Docker just isn't (AFAIK) not meant to be used like this, so it doesn't propagate signals to its app.
You have two ways of handling this:
You can teach the signal sender (Kubernetes or whatever) to instead of sending a signal to the Docker container do something more elaborate to get the information to the app inside the container. I guess this is not easy (if possible at all).
You can migrate the looping shell script outside of the container and let Kubernetes (or whatever) run this script instead. Within the loop you then can start the Docker container with your app inside. In this case you will need to catch the SIGTERM in the outside looping shell script (help trap) and either send a SIGTERM to the Docker container directly or to the app within the Docker container (using Docker exec or similar).
Related
I am very new to linux, so I can't quite understand the difference between daemon, foreground process and background process.
As I understand that:
Daemon is simply a background process that runs in the background and has init as its parent process.
Foreground process is a process that we simply invoke from the console.
Then if I run for example nginx inside a docker container with "daemon off" flag that means that nginx will be the foreground process running in the container's console?
What is the problem exactly?
Read this.
To start a container in detached mode, you use -d=true or just -d option.
In foreground mode (the default when -d is not specified), docker run can start the process in the container and attach the console to the process’s standard input, output, and standard error.
i have a docker container, which runs node.js application. This application runs a headless-chrome instance.
All work well, but if i kill chrome-instance, and check runned processes list, i will see 2(actually 3 + 2 cat process) zombie chrome process(defunct) still in system. I know this processes is a children
(of killed parent chrome process), which was not finished correct and attached to init process.
I tried to kill it directly - rejected. Also i tried to spawn chrome with detached:true flag and again kill all child processes directly, when main chrome receives "exit" signal, anyway ps -A | grep chrome shows two defunct to me. Any ideas?
UPD:
Thanks all for help. Adding --init totally solves my issue. Using another base image also works well, but i decided this approach as not neccesary. Also good description of root cause can be found here
larsks pretty much nails the reason, init (or systemd) on linux systems reaps zombie processes when their parent dies. The parent should cleanup its own zombie processes with the wait syscall. However, that automatic cleanup does not pass the namespace boundary of a container. So whatever process you run as your entrypoint, and that becomes pid 1, needs to handle these zombies for you.
With recent versions of docker, you can include an init process just by passing --init to your docker run command. If you are using a version 2.2 compose file, there's an option init: true you can define on your service for the same result.
In addition to dumb-init, there is also tini which is what docker uses under the covers as their own docker-init.
You need a process that will call wait() in order to reap any zombie processes. On a regular system this is handled by /sbin/init, but inside a container you will need to provide your own tooling. If you're developing your own application, consider just calling wait() in a loop periodically.
Alternatively, you can consider a container-specific init such as dumb-init and see if that resolves the problem.
I can't start a container on Bluemix with Debian, Centos, Alpine or other. Is there a way or is it blocked?
Image from docker hub.
Is there any getting started for the run command ?
I suppose that i need a file for parameters.
Big difference between running a container in local docker vs in the container service is that all containers in the container service are effectively running -d (i.e. daemon/disconncted mode). If you're just using the base images you listed, most of those do not have a long running process in the container, and expect that you will be running it interactive.
Result of that is that in the container service, the container starts, then exits again because it's non-interactive, and there's no other process to keep it alive. You can try adding a wait as the "cmd" for it.
i.e. for your dockerfile:
FROM alpine
build that to your registry, then run with something like cf ic run --name alpinetest -m 512 registry.ng.bluemix.net/yourregistryhere/alpine sh -c "sleep 1000000"
Then, to get an interactive shell, you can exec into the container using cf ic exec -ti alpinetest /bin/sh
Obviously, to have it do something useful, you're probably going to want to put an actual server running in there as the foreground app, and set that as the CMD or ENTRYPOINT, but that'll give you access to it running to poke at.
I know docker only listens to pid 1 and in case that pid exits (or turns into a daemon) it thinks the program exited and the container is shut down.
When apache-spark is started the ./start-master.sh script how can I kept the container running?
I do not think: while true; do sleep 1000; done is an appropriate solution.
E.g. I used command: sbin/start-master.sh to start the master. But it keeps shutting down.
How to keep it running when started with docker-compose?
As mentioned in "Use of Supervisor in docker", you could use phusion/baseimage-docker as a base image in which you can register scripts as "services".
The my_init script included in that image will take care of the exit signals management.
And the processes launched by start-master.sh would still be running.
Again, that supposes you are building your apache-spark image starting from phusion/baseimage-docker.
As commented by thaJeztah, using an existing image works too: gettyimages/spark/~/dockerfile/. Its default CMD will keep the container running.
Both options are cleaner than relying on a tail -f trick, which won't handle the kill/exit signals gracefully.
Here is another solution.
Create a file spark-env.sh with the following contents and copy it into the spark conf directory.
SPARK_NO_DAEMONIZE=true
If your CMD in the Dockerfile looks like this:
CMD ["/spark/sbin/start-master.sh"]
the container will not exit.
tail -f -n 50 /path/to/spark/logfile
This will keep the container alive and also provide useful info if you run -it interactive mode. You can run -d detached and it will stay alive.
Having an issue with Docker at the moment; I'm using it to run an image that launches an ipython notebook on startup. I'm looking to make some edits to ipython notebook itself, so I need to close it after launch.
However, hitting CTRL+C in the terminal just inputs "^C" as a string. There seems to be no real way of using CTRL+C to actually close the ipython notebook instance.
Would anyone have any clues as to what can cause this, or know of any solutions for it?
Most likely the container image you use is not handling process signals properly.
If you are authoring the image then change it as Roland Webers' answer suggests.
Otherwise try to run it with --init.
docker run -it --init ....
This fixes Ctrl+C for me.
Source: https://docs.docker.com/v17.09/engine/reference/run/#specify-an-init-process
The problem is that Ctrl-C sends a signal to the top-level process inside the container, but that process doesn't necessarily react as you would expect. The top-level process has ID 1 inside the container, which means that it doesn't get the default signal handlers that processes usually have. If the top-level process is a shell, then it can receive the signal through its own handler, but doesn't forward it to the command that is executed within the shell. Details are explained here. In both cases, the docker container acts as if it simply ignores Ctrl-C.
If you're building your own images, the solution is to run a minimal init process, such as tini or dumb-init, as the top-level process inside the container.
This post proposes CTRL-Z as a workaround for sending the process to background and then killing the process by its process id:
Cannot kill Python script with Ctrl-C
Possible problems:
The program catches ctrl-c and does nothing, very unlikely.
There are background processes that are not managed correctly. Only the main process receives the signal and sub-processes hang. Very likely what's happening.
Proposed Solution:
Check the programs documentation on how it's properly started and stopped. ctrl-c seems not to be the proper way.
Wrap the program with a docker-entrypoint.sh bash script that blocks the container process and is able to catch ctrl-c. This bash example should help: https://rimuhosting.com/knowledgebase/linux/misc/trapping-ctrl-c-in-bash
After catching ctrl-c invoke the proper shutdown method for ipython notebook.
From this post on the Docker message boards:
Open a new shell and execute
$ docker ps # get the id of the running container
$ docker stop <container> # kill it (gracefully)
This worked well for me. CTRL-Z, CTRL-\, etc. only came up as strings, but this killed the Docker container and returned the tab to terminal input.
#maybeg's answer already explains very well why this might be happening.
Regarding stopping the unresponsive container, another solution is to simply issue a docker stop <container-id> in another terminal. As opposed to CTRL-C, docker stop does not send a SIGINT but a SIGTERM signal, to which the process might react differently.
Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...]
Stop a running container by sending SIGTERM and then SIGKILL after a grace period
If that fails, use docker kill <container-id> which sends a SIGKILL immediately.