I'm trying to gracefully shutdown a service that I run in a docker container (on a raspberry pi 3 b running raspbian lite), since just issuing 'docker stop' breaks the service (a lock file does not get cleared)
My internet research has shown people working with the bash command 'trap' to execute a command when the script receives a signal to terminate.
I have tried this with the following:
docker run --rm -it --name trap resin/rpi-raspbian:jessie /bin/bash -c " trap 'echo shutting down ; exit' INT; echo waiting; read"
When I run above command, 'waiting' is being printed to the screen and when i hit CTRL+C, the message 'shutting down' appears. - Exactly as I want it!
The only problem is, that I can't seem to get that signal sent to the bash script from outside the docker container. I've tried (in a parallel ssh session):
docker kill --signal="INT" trap
docker kill --signal="SIGINT" trap
Furthermore, I tried to use the default signal that is being sent by 'docker stop', but that resulted in the console waiting ~10s (default docker stop timeout) and then killing the docker container.
docker stop trap
Neither methods got the message 'shutting down' to print onto the screen.
I'd really appreciate any help.
Edit 1:
It doesn't work with SIGTERM either (when I either of the following commands and then issue a 'docker stop' I still don't get the message 'shutting down').
docker run --rm -it --name trap resin/rpi-raspbian:jessie /bin/bash -c " trap 'echo shutting down ; exit' TERM; echo waiting; read"
docker run --rm -it --name trap resin/rpi-raspbian:jessie /bin/bash -c " trap 'echo shutting down ; exit' SIGTERM; echo waiting; read"
Workaround:
The image seems to be the problem. The entrypoint '/usr/bin/entry.sh' traps the signal. I've tried it with another image and it worked:
$ docker run -it --name trap digitallyseamless/archlinux-armv6h /bin/bash -c " trap 'echo shutting down ; exit 0' SIGTERM; echo waiting; read"
Stop the container using (from another terminal)
$ docker stop trap
Then view the container output
$ docker logs trap
waiting
shutting down
As I have no particular need to use a specific image (it really is just a means to get me to my goal), I'll just swap out images.
PS:
Huge thanks to #tkausl
As I mentioned in the comments, docker stop sends a SIGTERM. According to your edit, this doesn't work for you, however, I just tried it with exactly the same command (changed the image to ubuntu though) and it works. Are you probably detaching from the running box? In this case, you obviously don't see the message, remove the --rm flag and take a look at the docker logs.
How I did it:
In one terminal:
docker run --rm -it --name trap ubuntu /bin/bash -c " trap 'echo shutting down ; exit' SIGTERM; echo waiting; read"
In a second terminal:
$ docker stop f6
f6
What I see in the first terminal after stopping:
$ docker run --rm -it --name trap ubuntu /bin/bash -c " trap 'echo shutting down ; exit' SIGTERM; echo waiting; read"
waiting
shutting down
$
Docker stop and docker kill are sending a signal and not the trap command. You have to make a docker exec container_name trap
Related
I'd like to block CTRL-C but it doesn't work as expected.
I was following the answer described [here] (https://stackoverflow.com/a/37148777/12512199) but without success.
I must be missing something but can't figure out what. It's as if CTRL-C is intercepted but still propagated:
First I ran the following script and hit CTRL-C; the message is displayed but the script exited.
echo "
#!/bin/bash
trap 'echo "Ctrl + C happened"' SIGINT
sleep infinity
" > test.sh
chmod +x test.sh
./test.sh
Then I checked if it would behave differently in a container as pid 1:
echo "
#!/bin/bash
trap 'echo "Ctrl + C happened"' SIGINT
sleep infinity
" > test.sh
chmod +x test.sh
docker rm -f conttest
docker container create --name conttest -it --entrypoint="bash" ubuntu:20.04 -x -c /test.sh
docker cp test.sh conttest:/test.sh
docker container start --attach -i conttest
But no, it's the same behavior.
I ran those tests on Unbuntu 20.04.
Read https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#trap but still haven't found any clue ...
Any idea?
Control+C or any other key combination mapped to intr in output of
stty -a sends SIGINT to all processes in the foreground. Shell
receives it but so does sleep infinity which dies and shell exits
because it has nothing to do. If you want your script it to run
continuously and do something on SIGINT you have to use an endless
loop:
#!/bin/bash
trap 'echo "Ctrl + C happened"' SIGINT
while true
do
sleep infinity
done
If you only want to ignore SIGINT:
#!/bin/bash
trap '' SIGINT
sleep infinity
I have bash script which runs as docker entrypoint.
This script does some things and then sleeps.
I need to be able to gracefully stop this script when docker stop command is issued. AFAIK docker sends SIGTERM signal to the process 1.
So I created bash script which reacts to SIGTERM.
It works fine without docker, I can launch it, issue kill -TERM <pid> and it gracefully stops the script. Unfortunately it does not work in docker.
Dockerfile:
FROM ubuntu:20.04
WORKDIR /root
COPY test.sh .
CMD ./test.sh
test.sh:
#!/bin/sh
trap 'echo TERM; exit' TERM
sleep 1 &
while wait $!
do
sleep 1 &
done
How should I modify script or Dockerfile to allow docker stop to work without issues?
Please note that I'm not allowed to pass -it argument to docker run.
The problem comes from the shell form of the CMD instruction: CMD ./test.sh.
In this case the command is executed with a /bin/sh -c shell. So the /bin/sh runs as PID 1 and then forks test.sh, PID 1 is /bin/sh -c ./test.sh. In consequence the SIGTERM signal is sent to the PID 1 and never reaches test.sh.
To change this you have to use the exec form: CMD ["./test.sh"]. In this case, the command will be executed without a shell. So the signal will reach your script.
FROM ubuntu:20.04
WORKDIR /root
COPY test.sh .
CMD ["./test.sh"]
Run and kill.
# run
docker run -d --name trap trap:latest
# send the signal
docker kill --signal=TERM trap
# check if the signal has been received
docker logs trap
# TERM
I suggest you to try with docker kill --signal=SIGTERM <CONTAINER_ID>.
Please let me know if it will work.
Note if your script happens to spawn child process, you may leave behind orphan process upon your docker exit. Try run with --init flag like docker run --init ..., the built-in docker-init executable tini will assumed pid 1 and ensure all child process (such as your sleep example) will be terminated accordingly upon exit.
I've read How do I kill background processes / jobs when my shell script exits?, but I can't get it to work.
IDK if it's Docker shenanigans or something else.
#!/bin/bash -e
base="$(dirname "$0")"
trap 'kill $(jobs -p)' SIGINT SIGTERM EXIT
docker run --rm -p 5432:5432 -e POSTGRES_PASSWORD=password postgres:12 &
while ! nc -z localhost 5432; do
sleep 0.1
done
# uh-oh, error
false
When I run this, I am left with a running Docker container.
Why? How can stop the process when my script exits?
Docker is a client/server application, consisting of a thin client, docker, and server, dockerd. When you run a container, the client makes a few API calls to the server, one to create the container, another to start it, and since you didn't run it detached, it runs an attach API. When you kill the docker process, it detaches from the container, no longer showing you the logs, and kills that client portion. But the dockerd server is still running the container until process inside the container, running as pid 1 inside the container namespace, exits. You never killed that process since it's spawned from the dockerd daemon, not directly from the docker client.
To fix this, my suggestion is to run a docker stop, with the container name or id, as part of your trap handler. I wouldn't even bother running docker in the background, and instead pass -d to run detached.
Follow up, testing the script locally, it looks like killing the docker client does send a docker stop signal when you run the client attached like that. However, there's a race condition that can cause that stop to happen before the database is running. The command:
nc -z localhost 5432
is always going to succeed even before postgresql starts listening on the port because docker creates a port forward. E.g.:
$ nc -z localhost 5432 && echo it works
$ docker run -itd --rm -p 5432:5432 busybox tail -f /dev/null
c72427053124608fe18c31e5d6f3307d74a5cdce018503e9fff85dbc039b4fff
$ nc -z localhost 5432 && echo it works
it works
$ docker stop c72
c72
$ nc -z localhost 5432 && echo it works
However, if I run a sleep in the script, that forces it to wait long enough for the container to finish starting up, and the attach to finish, the container is stopped.
A better version of the script looks like the following, that waits for the database to completely start by checking the logs, and changing the trap to run a docker stop command:
#!/bin/bash -e
base="$(dirname "$0")"
trap 'kill $(jobs -p)' SIGINT SIGTERM EXIT
cid=$(docker run --rm -d -p 5432:5432 -e POSTGRES_PASSWORD=password postgres:12)
# leaving the kill assuming you have other background processes
trap 'docker stop $cid; kill $(jobs -p)' SIGINT SIGTERM EXIT
# waiting for the db to actually start, assuming later steps need the db to be up
while ! docker logs "$cid" 2>&1 | grep -q "database system is ready to accept connections" ; do
sleep 0.1
done
# uh-oh, error
false
It was Docker shenanigans.
I needed to use the --init option to run tini shim because
A process running as PID 1 inside a container is treated specially by Linux: it ignores any signal with the default action. As a result, the process will not terminate on SIGINT or SIGTERM unless it is coded to do so.
docker run --rm -p 5432:5432 -e POSTGRES_PASSWORD=password postgres:12 &
I have created gcloud compute instance from docker image and configured it to launch shutdown script which should call docker stop in order to shut down the app in the container gracefully.
gcloud beta compute instances create-with-container mycontainername \
--container-image ypapax/trap_exit \
--metadata-from-file shutdown-script=./on_shutdown.sh
And here is my initital on_shutdown.sh:
#!/usr/bin/env bash
docker stop $(docker ps -a -q)
Although, I added more debugging lines to it and now on_shutdown.sh looks like:
#!/usr/bin/env bash
# https://cloud.google.com/compute/docs/shutdownscript
curl -X POST -d "$(date) running containers on $(hostname): $(docker ps)" http://trap_exit.requestcatcher.com/test
docker stop $(docker ps -a -q)
result=$?
curl -X POST -d "$(date) $(hostname) stop status code: $result" http://trap_exit.requestcatcher.com/test
When I reboot the google compute instance:
sudo reboot
The script on_shutdown.sh is launched (I see it checking requrest listener). But when it tries to stop docker container, there is nothing to stop yet, docker ps shows empty line.
So this line:
curl -X POST -d "$(date) running containers on $(hostname): $(docker ps)" http://trap_exit.requestcatcher.com/test
gives me
Thu Jul 12 04:29:48 UTC 2018 running containers on myinstance:
Before calling sudo reboot I checked docker ps and saw my container running:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bbaba30e65ff ypapax/trap_exit "/entrypoint.sh" 7 seconds ago Up 7 seconds myinstance
So looks like docker container is killed between calling reboot and launching on_shutdown.sh. The problem is that killing doesn't call trap cleanup EXIT in my entrypoint. It needs to be stopped in order to call the cleanup.
Here is my entry point:
#!/usr/bin/env bash
set -ex
cleanup(){
echo cleanup is called
curl -X POST -d "cleanup is called on $(hostname) $(date)" http://trap_exit.requestcatcher.com/test
}
trap cleanup EXIT
curl -X POST -d "container is started on $(hostname) $(date)" http://trap_exit.requestcatcher.com/test
sleep 10000
So I would like to run my container's cleanup on gcloud compute instance reboot or shutdown but flag --metadata-from-file shutdown-script=./on_shutdown.sh doesn't help to do it. I also tried other methods to call a script on reboot like this. But my script hadn't been launched at all.
Here is my Dockerfile if it could help.
First, there are limitations coming with this approach:
Create and run shutdown scripts that execute commands right before an instance is terminated or restarted, on a best-effort basis.
Shutdown scripts are especially useful for instances in a managed instance group with an autoscaler.
The script runs during the limited shutdown period before the instance stops
As you have seen, docker might already have stopped by the time the shutdown script run: check with docker ps -a (instead of docker ps) to see the status of all exited containers.
Try adding a supervisor (as in this example) a docker image itself, in order to see if the supervisor, or at least use the docker run --init option: the goal is to check if the containers themselves do use their supervisor scripts.
I want to run an ubuntu container and enter bash:
[root#localhost backup]# docker run ubuntu bash
[root#localhost backup]#
The ubuntu container exits directly. How can I enter the bash?
Use -i and -t options.
Example:
$ docker run -i -t ubuntu /bin/bash
root#9055316d5ae4:/# echo "Hello Ubuntu"
Hello Ubuntu
root#9055316d5ae4:/# exit
See: Docker run Reference
$ docker run --help | egrep "(-i,|-t,)"
-i, --interactive=false Keep STDIN open even if not attached
-t, --tty=false Allocate a pseudo-TTY
Update: The reason this works and keeps the container running (running /bin/bash) is because the -i and -t options (specifically -i) keep STDIN open and so /bin/bash does not immediately terminate thus terminate the container. -- The reason you also need/want -t is because you presumably want to have an interactive terminal-like session so t creates a new pseudo-tty for you. -- Furthermore if you looked at the output of docker ps -a without using the -i/-t options you'd see that your container terminated normally with an exit code of 0.