I have docker container and it has supervisord inside.
I wish to kill that process
root 1 0.0 0.1 59768 13360 ? Ss+ 20:29 0:01 /usr/bin/python /usr/bin/supervisord
I login
sudo docker exec -ti blahblah bash
root# kill -KILL 1
it does not kills process 1 but I can kill any another process
If you kill the process the whole container would stop. So you might as well run.
docker stop containerName
or if you want to force it you can change stop to "kill" or "rm -f"(if you also want to remove the container)
Related
I have bash script which runs as docker entrypoint.
This script does some things and then sleeps.
I need to be able to gracefully stop this script when docker stop command is issued. AFAIK docker sends SIGTERM signal to the process 1.
So I created bash script which reacts to SIGTERM.
It works fine without docker, I can launch it, issue kill -TERM <pid> and it gracefully stops the script. Unfortunately it does not work in docker.
Dockerfile:
FROM ubuntu:20.04
WORKDIR /root
COPY test.sh .
CMD ./test.sh
test.sh:
#!/bin/sh
trap 'echo TERM; exit' TERM
sleep 1 &
while wait $!
do
sleep 1 &
done
How should I modify script or Dockerfile to allow docker stop to work without issues?
Please note that I'm not allowed to pass -it argument to docker run.
The problem comes from the shell form of the CMD instruction: CMD ./test.sh.
In this case the command is executed with a /bin/sh -c shell. So the /bin/sh runs as PID 1 and then forks test.sh, PID 1 is /bin/sh -c ./test.sh. In consequence the SIGTERM signal is sent to the PID 1 and never reaches test.sh.
To change this you have to use the exec form: CMD ["./test.sh"]. In this case, the command will be executed without a shell. So the signal will reach your script.
FROM ubuntu:20.04
WORKDIR /root
COPY test.sh .
CMD ["./test.sh"]
Run and kill.
# run
docker run -d --name trap trap:latest
# send the signal
docker kill --signal=TERM trap
# check if the signal has been received
docker logs trap
# TERM
I suggest you to try with docker kill --signal=SIGTERM <CONTAINER_ID>.
Please let me know if it will work.
Note if your script happens to spawn child process, you may leave behind orphan process upon your docker exit. Try run with --init flag like docker run --init ..., the built-in docker-init executable tini will assumed pid 1 and ensure all child process (such as your sleep example) will be terminated accordingly upon exit.
Sending SIGHUP with
kill -HUP <pid>
to a busybox sh process on my native system works as expected and the shell hangs up. However, if I use docker kill to send the signal to a container with
docker kill -s HUP <container>
it doesn't do anything. The Alpine container is still running:
$ CONTAINER=$(docker run -dt alpine:latest)
$ docker ps -a --filter "id=$CONTAINER" --format "{{.Status}}"
Up 1 second
$ docker kill -s HUP $CONTAINER
4fea4f2dabe0f8a717b0e1272528af1a97050bcec51babbe0ed801e75fb15f1b
$ docker ps -a --filter "id=$CONTAINER" --format "{{.Status}}"
Up 7 seconds
By the way, with an Ubuntu container (which runs bash) it does work as expected:
$ CONTAINER=$(docker run -dt debian:latest)
$ docker ps -a --filter "id=$CONTAINER" --format "{{.Status}}"
Up 1 second
$ docker kill -s HUP $CONTAINER
9a4aff456716397527cd87492066230e5088fbbb2a1bb6fc80f04f01b3368986
$ docker ps -a --filter "id=$CONTAINER" --format "{{.Status}}"
Exited (129) 1 second ago
Sending SIGKILL does work, but I'd rather find out why SIGHUP does not.
Update: I'll add another example. Here you can see that busybox sh generally does hang up on SIGHUP successfully:
$ busybox sh -c 'while true; do sleep 10; done' &
[1] 28276
$ PID=$!
$ ps -e | grep busybox
28276 pts/5 00:00:00 busybox
$ kill -HUP $PID
$
[1]+ Hangup busybox sh -c 'while true; do sleep 10; done'
$ ps -e | grep busybox
$
However, running the same infinite sleep loop inside the docker container doesn't quit. As you can see, the container is still running after SIGHUP and only exits after SIGKILL:
$ CONTAINER=$(docker run -dt alpine:latest busybox sh -c 'while true; do sleep 10; done')
$ docker ps -a --filter "id=$CONTAINER" --format "{{.Status}}"
Up 14 seconds
$ docker kill -s HUP $CONTAINER
31574ba7c0eb0505b776c459b55ffc8137042e1ce0562a3cf9aac80bfe8f65a0
$ docker ps -a --filter "id=$CONTAINER" --format "{{.Status}}"
Up 28 seconds
$ docker kill -s KILL $CONTAINER
31574ba7c0eb0505b776c459b55ffc8137042e1ce0562a3cf9aac80bfe8f65a0
$ docker ps -a --filter "id=$CONTAINER" --format "{{.Status}}"
Exited (137) 2 seconds ago
$
(I don't have Docker env at hand for a try. Just guessing.)
For your case, docker run must be running busybox/sh or bash as PID 1.
According to Docker doc:
Note: A process running as PID 1 inside a container is treated specially by Linux: it ignores any signal with the default action. So, the process will not terminate on SIGINT or SIGTERM unless it is coded to do so.
For the differece between busybox/sh and bash regarding SIGHUP ---
On my system (Debian 9.6, x86_64), the signal masks for busybox/sh and bash are as follows:
busybox/sh:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 82817 0.0 0.0 6952 1904 pts/2 S+ 10:23 0:00 busybox sh
PENDING (0000000000000000):
BLOCKED (0000000000000000):
IGNORED (0000000000284004):
3 QUIT
15 TERM
20 TSTP
22 TTOU
CAUGHT (0000000008000002):
2 INT
28 WINCH
bash:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 4871 0.0 0.1 21752 6176 pts/16 Ss 2019 0:00 /usr/local/bin/bash
PENDING (0000000000000000):
BLOCKED (0000000000000000):
IGNORED (0000000000380004):
3 QUIT
20 TSTP
21 TTIN
22 TTOU
CAUGHT (000000004b817efb):
1 HUP
2 INT
4 ILL
5 TRAP
6 ABRT
7 BUS
8 FPE
10 USR1
11 SEGV
12 USR2
13 PIPE
14 ALRM
15 TERM
17 CHLD
24 XCPU
25 XFSZ
26 VTALRM
28 WINCH
31 SYS
As we can see busybox/sh does not handle SIGHUP so the signal is ignored. Bash catches SIGHUP so docker kill can deliver the signal to Bash and then Bash will be terminated because, according to its manual, "the shell exits by default upon receipt of a SIGHUP".
UPDATE 2020-03-07 #1:
Did a quick test and my previous analysis is basically correct. You can verify like this:
[STEP 104] # docker run -dt debian busybox sh -c \
'trap exit HUP; while true; do sleep 1; done'
331380090c59018dae4dbc17dd5af9d355260057fdbd2f2ce9fc6548a39df1db
[STEP 105] # docker ps
CONTAINER ID IMAGE COMMAND CREATED
331380090c59 debian "busybox sh -c 'trap…" 11 seconds ago
[STEP 106] # docker kill -s HUP 331380090c59
331380090c59
[STEP 107] # docker ps
CONTAINER ID IMAGE COMMAND CREATED
[STEP 108] #
As I showed earlier, by default busybox/sh does not catch SIGHUP so the signal will be ignored. But after busybox/sh explicitly trap SIGHUP, the signal will be delivered to it.
I also tried SIGKILL and yes it'll always terminate the running container. This is reasonable since SIGKILL cannot be caught by any process so the signal will always be delivered to the container and kill it.
UPDATE 2020-03-07 #2:
You can also verify it this way (much simpler):
[STEP 110] # docker run -ti alpine
/ # ps
PID USER TIME COMMAND
1 root 0:00 /bin/sh
7 root 0:00 ps
/ # kill -HUP 1 <-- this does not kill it because linux ignored the signal
/ #
/ # trap 'echo received SIGHUP' HUP
/ # kill -HUP 1
received SIGHUP <-- this indicates it can receive SIGHUP now
/ #
/ # trap exit HUP
/ # kill -HUP 1 <-- this terminates it because the action changed to `exit`
[STEP 111] #
Like the other answer already points out, the docs for docker run contain the following note:
Note: A process running as PID 1 inside a container is treated specially by Linux: it ignores any signal with the default action. So, the process will not terminate on SIGINT or SIGTERM unless it is coded to do so.
This is the reason why SIGHUP doesn't work on busybox sh inside the container. However, if I run busybox sh on my native system, it won't have PID 1 and therefore SIGHUP works.
There are various solutions:
Use --init to specify an init process which should be used as PID 1.
You can use the --init flag to indicate that an init process should be used as the PID 1 in the container. Specifying an init process ensures the usual responsibilities of an init system, such as reaping zombie processes, are performed inside the created container.
The default init process used is the first docker-init executable found in the system path of the Docker daemon process. This docker-init binary, included in the default installation, is backed by tini.
Trap SIGHUP and call exit yourself.
docker run -dt alpine busybox sh -c 'trap exit HUP ; while true ; do sleep 60 & wait $! ; done'
Use another shell like bash which exits on SIGHUP by default, doesn't matter if PID 1 or not.
Lets say I have a container running and I do
docker exec -ti container-id /bin/bash
Then I detach from this container and want to attach again
If I do this
docker attach container-id
I wont go back to that bash that I created. Instead I will go to the main process.
How can I attach to that bash again ?
You can't. While the docker exec documentation suggests it supports the same "detach" key sequence as docker run, the exec'd process doesn't have any Docker-level identity (beyond its host and container pids) and there's no way to re-attach to that shell.
(In the Docker API, "exec instance" is an actual object so this isn't technically impossible; the CLI just has no support for it.)
The workflow you're describing sounds more like what you'd run with screen or tmux in a virtual machine.
I have one container and I have started it using, and checked the pid of /bin/bash
[root#ip-10-0-1-153 centos]# docker exec -ti 78c2e4a46b58 /bin/bash
root#78c2e4a46b58:/# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 03:24 ? 00:00:00 bash
root 10 0 0 03:24 ? 00:00:00 /bin/bash
root 20 10 0 03:24 ? 00:00:00 ps -ef
Now I detach from container using CTR+p and CTR+q sequence and container is detached.
Now I reattach using the container id and I see the same pid of /bin/bash
root#78c2e4a46b58:/# [root#ip-10-0-1-153 centos]# docker attach 78c2e4a46b58
root#78c2e4a46b58:/# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 03:24 ? 00:00:00 bash
root 10 0 0 03:24 ? 00:00:00 /bin/bash
root 21 1 0 03:25 ? 00:00:00 ps -ef
root#78c2e4a46b58:/#
I hope you're using CTR+p CTR+q sequence to detach from the container.
I have a v6.10.0 Node server on my macOS that is automatically started from the CMD in the Dockerfile. Normally in my local development un-containerized environment I will use CTRL+C to kill the server. Not being able to (or not knowing how to) do this in the container, I resort to ps aux | grep node to try to manually kill the processes. So, I get something like this:
myapp [master] :> kubectl exec -it web-3127363242-xb50k bash
root#web-3127363242-xb50k:/usr/src/app# ps aux | grep node
root 15 0.4 0.9 883000 35804 ? Sl 05:49 0:00 node /usr/src/app/node_modules/.bin/concurrent --kill-others npm run start-prod npm run start-prod-api
root 43 0.1 0.6 743636 25240 ? Sl 05:49 0:00 node /usr/src/app/node_modules/.bin/better-npm-run start-prod
root 44 0.1 0.6 743636 25140 ? Sl 05:49 0:00 node /usr/src/app/node_modules/.bin/better-npm-run start-prod-api
root 55 0.0 0.0 4356 740 ? S 05:49 0:00 sh -c node ./bin/server.js
root 56 0.0 0.0 4356 820 ? S 05:49 0:00 sh -c node ./bin/api.js
root 57 18.6 4.9 1018088 189416 ? Sl 05:49 0:08 node ./bin/server.js
root 58 13.9 5.2 1343296 197576 ? Sl 05:49 0:06 node ./bin/api.js
root 77 0.0 0.0 11128 1024 ? S+ 05:50 0:00 grep node
When I try to kill one of them by
kill -9 15
I am taken out of my container's shell and back to my computer's shell. When I enter the container again, I see that the process is still there with the same process id. This example uses a Kubernetes pod but I believe I have the same result with entering a Docker container using the docker exec command.
Every docker container has an ENTRYPOINT that will either be set in the dockerfile, using ENTRYPOINTor CMD declarations, or specified in the run command docker run myimage:tag "entrypoint_command". When the ENTRYPOINT process is killed, I think the container gets killed as well. The docker exec, as I understand it, is kind of like "attaching" command to a container. But if the ENTRYPOINT is down there is no container to attach to.
Kubernetes will restart a container after failure as far as I understand it. Which might be the reason you see the process is back up. I haven't really worked with Kubernetes but I'd try and play around with the way that the replications are scaled to terminate your process.
Containers isolate your desired app as pid 1 inside the namespace. The desired app being your entrypoint or cmd if you don't have an entrypoint defined. If killing a process results in pid 1 exiting, the container will immediately stop (similar to killing pid 1 on a linux host) along with killing all of the other pids. If this container has a restart policy, it will be restarted and the processes will get the same pids as last time it ran (all else being equal which it often is inside of a container).
To keep the container from stopping, you'll need to adjust your entrypoint to remain up even with the child process being killed. That side, having the container exit is typically a preferred behavior to handle unexpected errors by getting back to a clean state.
I don't understand the difference in
$> docker top lamp-test
PID USER COMMAND
31263 root {supervisord} /usr/bin/python /usr/bin/supervisord -n
31696 root {mysqld_safe} /bin/sh /usr/bin/mysqld_safe
31697 root apache2 -D FOREGROUND
...
and
$> docker exec lamp-test ps
PID TTY TIME CMD
1 ? 00:00:00 supervisord
433 ? 00:00:00 mysqld_safe
434 ? 00:00:00 apache2
831 ? 00:00:00 ps
So, the question is, why are the PID different ? I would say that the output from ps is namespaced, but if that is true, what is top showing!
docker exec lamp-test ps show pids inside docker container.
docker top lamp-test show host system pids.
You can see a container processes, but You cannot kill them. This "flawed" isolation actually has some great benefits, like the ability to monitor the processes running inside all your containers from a single monitor process running on the host machine.
I don't think you should worry about this. You can't kill the PID in Host environment, but can do it in container.
docker exec <CONTAINER NAME> ps remember the PID
docker exec <CONTAINER NAME> kill <PID>