I've read How do I kill background processes / jobs when my shell script exits?, but I can't get it to work.
IDK if it's Docker shenanigans or something else.
#!/bin/bash -e
base="$(dirname "$0")"
trap 'kill $(jobs -p)' SIGINT SIGTERM EXIT
docker run --rm -p 5432:5432 -e POSTGRES_PASSWORD=password postgres:12 &
while ! nc -z localhost 5432; do
sleep 0.1
done
# uh-oh, error
false
When I run this, I am left with a running Docker container.
Why? How can stop the process when my script exits?
Docker is a client/server application, consisting of a thin client, docker, and server, dockerd. When you run a container, the client makes a few API calls to the server, one to create the container, another to start it, and since you didn't run it detached, it runs an attach API. When you kill the docker process, it detaches from the container, no longer showing you the logs, and kills that client portion. But the dockerd server is still running the container until process inside the container, running as pid 1 inside the container namespace, exits. You never killed that process since it's spawned from the dockerd daemon, not directly from the docker client.
To fix this, my suggestion is to run a docker stop, with the container name or id, as part of your trap handler. I wouldn't even bother running docker in the background, and instead pass -d to run detached.
Follow up, testing the script locally, it looks like killing the docker client does send a docker stop signal when you run the client attached like that. However, there's a race condition that can cause that stop to happen before the database is running. The command:
nc -z localhost 5432
is always going to succeed even before postgresql starts listening on the port because docker creates a port forward. E.g.:
$ nc -z localhost 5432 && echo it works
$ docker run -itd --rm -p 5432:5432 busybox tail -f /dev/null
c72427053124608fe18c31e5d6f3307d74a5cdce018503e9fff85dbc039b4fff
$ nc -z localhost 5432 && echo it works
it works
$ docker stop c72
c72
$ nc -z localhost 5432 && echo it works
However, if I run a sleep in the script, that forces it to wait long enough for the container to finish starting up, and the attach to finish, the container is stopped.
A better version of the script looks like the following, that waits for the database to completely start by checking the logs, and changing the trap to run a docker stop command:
#!/bin/bash -e
base="$(dirname "$0")"
trap 'kill $(jobs -p)' SIGINT SIGTERM EXIT
cid=$(docker run --rm -d -p 5432:5432 -e POSTGRES_PASSWORD=password postgres:12)
# leaving the kill assuming you have other background processes
trap 'docker stop $cid; kill $(jobs -p)' SIGINT SIGTERM EXIT
# waiting for the db to actually start, assuming later steps need the db to be up
while ! docker logs "$cid" 2>&1 | grep -q "database system is ready to accept connections" ; do
sleep 0.1
done
# uh-oh, error
false
It was Docker shenanigans.
I needed to use the --init option to run tini shim because
A process running as PID 1 inside a container is treated specially by Linux: it ignores any signal with the default action. As a result, the process will not terminate on SIGINT or SIGTERM unless it is coded to do so.
docker run --rm -p 5432:5432 -e POSTGRES_PASSWORD=password postgres:12 &
Related
I have bash script which runs as docker entrypoint.
This script does some things and then sleeps.
I need to be able to gracefully stop this script when docker stop command is issued. AFAIK docker sends SIGTERM signal to the process 1.
So I created bash script which reacts to SIGTERM.
It works fine without docker, I can launch it, issue kill -TERM <pid> and it gracefully stops the script. Unfortunately it does not work in docker.
Dockerfile:
FROM ubuntu:20.04
WORKDIR /root
COPY test.sh .
CMD ./test.sh
test.sh:
#!/bin/sh
trap 'echo TERM; exit' TERM
sleep 1 &
while wait $!
do
sleep 1 &
done
How should I modify script or Dockerfile to allow docker stop to work without issues?
Please note that I'm not allowed to pass -it argument to docker run.
The problem comes from the shell form of the CMD instruction: CMD ./test.sh.
In this case the command is executed with a /bin/sh -c shell. So the /bin/sh runs as PID 1 and then forks test.sh, PID 1 is /bin/sh -c ./test.sh. In consequence the SIGTERM signal is sent to the PID 1 and never reaches test.sh.
To change this you have to use the exec form: CMD ["./test.sh"]. In this case, the command will be executed without a shell. So the signal will reach your script.
FROM ubuntu:20.04
WORKDIR /root
COPY test.sh .
CMD ["./test.sh"]
Run and kill.
# run
docker run -d --name trap trap:latest
# send the signal
docker kill --signal=TERM trap
# check if the signal has been received
docker logs trap
# TERM
I suggest you to try with docker kill --signal=SIGTERM <CONTAINER_ID>.
Please let me know if it will work.
Note if your script happens to spawn child process, you may leave behind orphan process upon your docker exit. Try run with --init flag like docker run --init ..., the built-in docker-init executable tini will assumed pid 1 and ensure all child process (such as your sleep example) will be terminated accordingly upon exit.
I have created gcloud compute instance from docker image and configured it to launch shutdown script which should call docker stop in order to shut down the app in the container gracefully.
gcloud beta compute instances create-with-container mycontainername \
--container-image ypapax/trap_exit \
--metadata-from-file shutdown-script=./on_shutdown.sh
And here is my initital on_shutdown.sh:
#!/usr/bin/env bash
docker stop $(docker ps -a -q)
Although, I added more debugging lines to it and now on_shutdown.sh looks like:
#!/usr/bin/env bash
# https://cloud.google.com/compute/docs/shutdownscript
curl -X POST -d "$(date) running containers on $(hostname): $(docker ps)" http://trap_exit.requestcatcher.com/test
docker stop $(docker ps -a -q)
result=$?
curl -X POST -d "$(date) $(hostname) stop status code: $result" http://trap_exit.requestcatcher.com/test
When I reboot the google compute instance:
sudo reboot
The script on_shutdown.sh is launched (I see it checking requrest listener). But when it tries to stop docker container, there is nothing to stop yet, docker ps shows empty line.
So this line:
curl -X POST -d "$(date) running containers on $(hostname): $(docker ps)" http://trap_exit.requestcatcher.com/test
gives me
Thu Jul 12 04:29:48 UTC 2018 running containers on myinstance:
Before calling sudo reboot I checked docker ps and saw my container running:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bbaba30e65ff ypapax/trap_exit "/entrypoint.sh" 7 seconds ago Up 7 seconds myinstance
So looks like docker container is killed between calling reboot and launching on_shutdown.sh. The problem is that killing doesn't call trap cleanup EXIT in my entrypoint. It needs to be stopped in order to call the cleanup.
Here is my entry point:
#!/usr/bin/env bash
set -ex
cleanup(){
echo cleanup is called
curl -X POST -d "cleanup is called on $(hostname) $(date)" http://trap_exit.requestcatcher.com/test
}
trap cleanup EXIT
curl -X POST -d "container is started on $(hostname) $(date)" http://trap_exit.requestcatcher.com/test
sleep 10000
So I would like to run my container's cleanup on gcloud compute instance reboot or shutdown but flag --metadata-from-file shutdown-script=./on_shutdown.sh doesn't help to do it. I also tried other methods to call a script on reboot like this. But my script hadn't been launched at all.
Here is my Dockerfile if it could help.
First, there are limitations coming with this approach:
Create and run shutdown scripts that execute commands right before an instance is terminated or restarted, on a best-effort basis.
Shutdown scripts are especially useful for instances in a managed instance group with an autoscaler.
The script runs during the limited shutdown period before the instance stops
As you have seen, docker might already have stopped by the time the shutdown script run: check with docker ps -a (instead of docker ps) to see the status of all exited containers.
Try adding a supervisor (as in this example) a docker image itself, in order to see if the supervisor, or at least use the docker run --init option: the goal is to check if the containers themselves do use their supervisor scripts.
I'm trying to gracefully shutdown a service that I run in a docker container (on a raspberry pi 3 b running raspbian lite), since just issuing 'docker stop' breaks the service (a lock file does not get cleared)
My internet research has shown people working with the bash command 'trap' to execute a command when the script receives a signal to terminate.
I have tried this with the following:
docker run --rm -it --name trap resin/rpi-raspbian:jessie /bin/bash -c " trap 'echo shutting down ; exit' INT; echo waiting; read"
When I run above command, 'waiting' is being printed to the screen and when i hit CTRL+C, the message 'shutting down' appears. - Exactly as I want it!
The only problem is, that I can't seem to get that signal sent to the bash script from outside the docker container. I've tried (in a parallel ssh session):
docker kill --signal="INT" trap
docker kill --signal="SIGINT" trap
Furthermore, I tried to use the default signal that is being sent by 'docker stop', but that resulted in the console waiting ~10s (default docker stop timeout) and then killing the docker container.
docker stop trap
Neither methods got the message 'shutting down' to print onto the screen.
I'd really appreciate any help.
Edit 1:
It doesn't work with SIGTERM either (when I either of the following commands and then issue a 'docker stop' I still don't get the message 'shutting down').
docker run --rm -it --name trap resin/rpi-raspbian:jessie /bin/bash -c " trap 'echo shutting down ; exit' TERM; echo waiting; read"
docker run --rm -it --name trap resin/rpi-raspbian:jessie /bin/bash -c " trap 'echo shutting down ; exit' SIGTERM; echo waiting; read"
Workaround:
The image seems to be the problem. The entrypoint '/usr/bin/entry.sh' traps the signal. I've tried it with another image and it worked:
$ docker run -it --name trap digitallyseamless/archlinux-armv6h /bin/bash -c " trap 'echo shutting down ; exit 0' SIGTERM; echo waiting; read"
Stop the container using (from another terminal)
$ docker stop trap
Then view the container output
$ docker logs trap
waiting
shutting down
As I have no particular need to use a specific image (it really is just a means to get me to my goal), I'll just swap out images.
PS:
Huge thanks to #tkausl
As I mentioned in the comments, docker stop sends a SIGTERM. According to your edit, this doesn't work for you, however, I just tried it with exactly the same command (changed the image to ubuntu though) and it works. Are you probably detaching from the running box? In this case, you obviously don't see the message, remove the --rm flag and take a look at the docker logs.
How I did it:
In one terminal:
docker run --rm -it --name trap ubuntu /bin/bash -c " trap 'echo shutting down ; exit' SIGTERM; echo waiting; read"
In a second terminal:
$ docker stop f6
f6
What I see in the first terminal after stopping:
$ docker run --rm -it --name trap ubuntu /bin/bash -c " trap 'echo shutting down ; exit' SIGTERM; echo waiting; read"
waiting
shutting down
$
Docker stop and docker kill are sending a signal and not the trap command. You have to make a docker exec container_name trap
I am trying to run a shell script in my docker container. The problem is that the shell script spawns another process and it should continue to run unless another shutdown script is used to terminate the processes that are spawned by the startup script.
When I run the below command,
docker run image:tag /bin/sh /root/my_script.sh
and then,
docker ps -a
I see that the command has exited. But this is not what I want. My question is how to let the command run in background without exiting?
You haven't explained why you want to see your container running after your script has exited, or whether or not you expect your script to exit.
A docker container exits as soon as the container's CMD exits. If you want your container to continue running, you will need a process that will keep running. One option is simply to put a while loop at the end of your script:
while :; do
sleep 300
done
Your script will never exit so your container will keep running. If your container hosts a network service (a web server, a database server, etc), then this is typically the process the runs for the life of the container.
If instead your script is exiting unexpectedly, you will probably need to take a look at your container logs (docker logs <container>) and possibly add some debugging to your script.
If you are simply asking, "how do I run a container in the background?", then Emil's answer (pass the -d flag to docker run) will help you out.
The process that docker runs takes the place of init in the UNIX process tree. init is the topmost parent process, and once it exits the docker container stops. Any child process (now an orphan process) will be stopped as well.
$ docker pull busybox >/dev/null
$ time docker run --rm busybox sleep 3
real 0m3.852s
user 0m0.179s
sys 0m0.012s
So you can't allow the parent pid to exit, but you have two options. You can leave the parent process in place and allow it to manage its children (for example, by telling it to wait until all child processes have exited)
$ time docker run --rm busybox sh -c 'sleep 3 & wait'
real 0m3.916s
user 0m0.178s
sys 0m0.013s
…or you can replace the parent process with the child process using exec. This means that the new command is being executed in the parent process's space…
$ time docker run --rm busybox sh -c 'exec sleep 3'
real 0m3.886s
user 0m0.173s
sys 0m0.010s
This latter approach may be complex depending on the nature of the child process, but having fewer unnecessary processes running is more idiomatically Docker. (Which is not saying you should only ever have one process.)
Run you container with your script in background with below command
docker run -i -t -d image:tag /bin/sh /root/my_script.sh
Check the container id by docker ps command
Then verify your script is executing or not on container
docker exec <id> /bin/sh -l -c "ps aux"
Wrap the program with a docker-entrypoint.sh bash script that blocks the container process and is able to catch ctrl-c. This bash example should help:
https://rimuhosting.com/knowledgebase/linux/misc/trapping-ctrl-c-in-bash
The script should shutdown the process cleanly when the exit signal is sent by Docker.
You can also add a loop inside the script that repeatedly checks the running process.
I create my docker container in detached mode with the following command:
docker run [OPTIONS] --name="my_image" -d container_name /bin/bash -c "/opt/init.sh"
so I need that "/opt/init.sh" executed at container created. What I saw that the container is stopped after scripts finish executed.
How to keep container started in detached with script/services execution at container creation ?
There are 2 modes of running docker container
Detached mode - This mode you execute a command and will terminate container after the command is done
Foreground mode - This mode you run a bash shell, but will also terminate container after you exit the shell
What you need is Background mode. This is not given in parameters but there are many ways to do this.
Run an infinite command in detached mode so the command never ends and the container never stops. I usually use "tail -f /dev/null" simply because it is quite light weight and /dev/null is present in most linux images
docker run -d --name=name container tail -f /dev/null
Then you can bash in to running container like this:
docker exec -it name /bin/bash -l
If you use -l parameter, it will login as login mode which will execute .bashrc like normal bash login. Otherwise, you need to bash again inside manually
Entrypoint - You can create any sh script such as /entrypoint.sh. in entrypoint.sh you can run any never ending script as well
#!/bin/sh
#/entrypoint.sh
service mysql restart
...
tail -f /dev/null <- this is never ending
After you save this entrypoint.sh, chmod a+x on it, exit docker bash, then start it like this:
docker run --name=name container --entrypoint /entrypoint.sh
This allows each container to have their own start script and you can run them without worrying about attaching the start script each time
A Docker container will exit when its main process ends. In this case, that means when init.sh ends. If you are only trying to start a single application, you can just use exec to launch it at the end, making sure to run it in the foreground. Using exec will effectively turn the called service/application into the main process.
If you have more than one service to start, you are best off using a process manager such as supervisord or runit. You will need to start the process manager daemon in the foreground. The Docker documentation includes an example of using supervisord.