I have a v6.10.0 Node server on my macOS that is automatically started from the CMD in the Dockerfile. Normally in my local development un-containerized environment I will use CTRL+C to kill the server. Not being able to (or not knowing how to) do this in the container, I resort to ps aux | grep node to try to manually kill the processes. So, I get something like this:
myapp [master] :> kubectl exec -it web-3127363242-xb50k bash
root#web-3127363242-xb50k:/usr/src/app# ps aux | grep node
root 15 0.4 0.9 883000 35804 ? Sl 05:49 0:00 node /usr/src/app/node_modules/.bin/concurrent --kill-others npm run start-prod npm run start-prod-api
root 43 0.1 0.6 743636 25240 ? Sl 05:49 0:00 node /usr/src/app/node_modules/.bin/better-npm-run start-prod
root 44 0.1 0.6 743636 25140 ? Sl 05:49 0:00 node /usr/src/app/node_modules/.bin/better-npm-run start-prod-api
root 55 0.0 0.0 4356 740 ? S 05:49 0:00 sh -c node ./bin/server.js
root 56 0.0 0.0 4356 820 ? S 05:49 0:00 sh -c node ./bin/api.js
root 57 18.6 4.9 1018088 189416 ? Sl 05:49 0:08 node ./bin/server.js
root 58 13.9 5.2 1343296 197576 ? Sl 05:49 0:06 node ./bin/api.js
root 77 0.0 0.0 11128 1024 ? S+ 05:50 0:00 grep node
When I try to kill one of them by
kill -9 15
I am taken out of my container's shell and back to my computer's shell. When I enter the container again, I see that the process is still there with the same process id. This example uses a Kubernetes pod but I believe I have the same result with entering a Docker container using the docker exec command.
Every docker container has an ENTRYPOINT that will either be set in the dockerfile, using ENTRYPOINTor CMD declarations, or specified in the run command docker run myimage:tag "entrypoint_command". When the ENTRYPOINT process is killed, I think the container gets killed as well. The docker exec, as I understand it, is kind of like "attaching" command to a container. But if the ENTRYPOINT is down there is no container to attach to.
Kubernetes will restart a container after failure as far as I understand it. Which might be the reason you see the process is back up. I haven't really worked with Kubernetes but I'd try and play around with the way that the replications are scaled to terminate your process.
Containers isolate your desired app as pid 1 inside the namespace. The desired app being your entrypoint or cmd if you don't have an entrypoint defined. If killing a process results in pid 1 exiting, the container will immediately stop (similar to killing pid 1 on a linux host) along with killing all of the other pids. If this container has a restart policy, it will be restarted and the processes will get the same pids as last time it ran (all else being equal which it often is inside of a container).
To keep the container from stopping, you'll need to adjust your entrypoint to remain up even with the child process being killed. That side, having the container exit is typically a preferred behavior to handle unexpected errors by getting back to a clean state.
Related
I encountered something weird when deploying a new service with supervisord. These are the relevant parts:
# supervisord.conf
[program:express]
command=yarn re-express-start
# package.json
{
"scripts": {
"re-express-start": "node lib/js/client/Express.bs.js",
}
}
When I run supervisorctl start, the node server is started as expected. But after I run supervisorctl stop, the server keeps on running even though supervisor thinks it's been killed.
If I change the supervisord.conf file to execute node lib/js/client/Express.bs.js directly (without going through yarn), then this works as expected. But I want to go through the package.json-defined script.
I looked into how the process tree looks like and but I don't quite understand why. Below are the processes before and after stopping the supervisord-managed service.
$ ps aux | grep node
user 12785 1.4 3.5 846404 72912 ? Sl 16:30 0:00 node /usr/bin/yarn re-express-start
user 12796 0.0 0.0 4516 708 ? S 16:30 0:00 /bin/sh -c node lib/js/client/Express.bs.js
user 12797 5.2 2.7 697648 56384 ? Sl 16:30 0:00 /usr/bin/node lib/js/client/Express.bs.js
root 12830 0.0 0.0 14216 1004 pts/1 S+ 16:30 0:00 grep --color=auto node
$ pstree -c -l -p -s 12785
systemd(1)───supervisord(7153)───node(12785)─┬─sh(12796)───node(12797)─┬─{node}(12798)
│ └─{node}(12807)
├─{node}(12786)
└─{node}(12795)
$ supervisorctl stop express
$ ps aux | grep node
user 12797 0.7 2.7 697648 56384 ? Sl 16:30 0:00 /usr/bin/node lib/js/client/Express.bs.js
root 12975 0.0 0.0 14216 980 pts/1 S+ 16:32 0:00 grep --color=auto node
$ pstree -c -l -p -s 12797
systemd(1)───node(12797)─┬─{node}(12798)
└─{node}(12807)
$ kill 12797
$ ps aux | grep node
root 13426 0.0 0.0 14216 976 pts/1 S+ 16:37 0:00 grep --color=auto node
From the above, the "actual" workload process doing the server stuff has PID 12797. It is spawned by the supervisor process and nested under a few more.
Stopping the supervisor process stops the processes with PIDs 12785 and 12796, but not the 12797 which is actually reattached to the init process.
Any ideas on what is happening here? Is this due to something ignoring some SIGxxx signals? I assume it's the yarn invocation somehow eating those,
but I don't know how and how to reconfigure.
I ran into this issue as well when I was running a node Express app. The problem seemed to be that I was having supervisor call npm start which refers to the package.json start script. That script simply calls node app.js. The solution seemed to be to directly call that command from the supervisor config file like so:
[program:node]
...
command=node app.js
...
stopasgroup=true
stopsignal=QUIT
In addition, I added stopasgroup and changed the stopsignal to QUIT. The stopsignal seemed to be required in order to properly kill the process.
I can now freely call supervisorctl restart node:node_00 without having any ERROR (spawn error) errors.
Lets say I have a container running and I do
docker exec -ti container-id /bin/bash
Then I detach from this container and want to attach again
If I do this
docker attach container-id
I wont go back to that bash that I created. Instead I will go to the main process.
How can I attach to that bash again ?
You can't. While the docker exec documentation suggests it supports the same "detach" key sequence as docker run, the exec'd process doesn't have any Docker-level identity (beyond its host and container pids) and there's no way to re-attach to that shell.
(In the Docker API, "exec instance" is an actual object so this isn't technically impossible; the CLI just has no support for it.)
The workflow you're describing sounds more like what you'd run with screen or tmux in a virtual machine.
I have one container and I have started it using, and checked the pid of /bin/bash
[root#ip-10-0-1-153 centos]# docker exec -ti 78c2e4a46b58 /bin/bash
root#78c2e4a46b58:/# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 03:24 ? 00:00:00 bash
root 10 0 0 03:24 ? 00:00:00 /bin/bash
root 20 10 0 03:24 ? 00:00:00 ps -ef
Now I detach from container using CTR+p and CTR+q sequence and container is detached.
Now I reattach using the container id and I see the same pid of /bin/bash
root#78c2e4a46b58:/# [root#ip-10-0-1-153 centos]# docker attach 78c2e4a46b58
root#78c2e4a46b58:/# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 03:24 ? 00:00:00 bash
root 10 0 0 03:24 ? 00:00:00 /bin/bash
root 21 1 0 03:25 ? 00:00:00 ps -ef
root#78c2e4a46b58:/#
I hope you're using CTR+p CTR+q sequence to detach from the container.
I don't understand the difference in
$> docker top lamp-test
PID USER COMMAND
31263 root {supervisord} /usr/bin/python /usr/bin/supervisord -n
31696 root {mysqld_safe} /bin/sh /usr/bin/mysqld_safe
31697 root apache2 -D FOREGROUND
...
and
$> docker exec lamp-test ps
PID TTY TIME CMD
1 ? 00:00:00 supervisord
433 ? 00:00:00 mysqld_safe
434 ? 00:00:00 apache2
831 ? 00:00:00 ps
So, the question is, why are the PID different ? I would say that the output from ps is namespaced, but if that is true, what is top showing!
docker exec lamp-test ps show pids inside docker container.
docker top lamp-test show host system pids.
You can see a container processes, but You cannot kill them. This "flawed" isolation actually has some great benefits, like the ability to monitor the processes running inside all your containers from a single monitor process running on the host machine.
I don't think you should worry about this. You can't kill the PID in Host environment, but can do it in container.
docker exec <CONTAINER NAME> ps remember the PID
docker exec <CONTAINER NAME> kill <PID>
I have docker container and it has supervisord inside.
I wish to kill that process
root 1 0.0 0.1 59768 13360 ? Ss+ 20:29 0:01 /usr/bin/python /usr/bin/supervisord
I login
sudo docker exec -ti blahblah bash
root# kill -KILL 1
it does not kills process 1 but I can kill any another process
If you kill the process the whole container would stop. So you might as well run.
docker stop containerName
or if you want to force it you can change stop to "kill" or "rm -f"(if you also want to remove the container)
I have started an app with
forever start app.js
After that I typed,
forever list
and it shows that
The "sys" module is now called "util". It should have a similar interface.
info: No forever processes running
But I checked my processes with
ps aux | grep node
and it shows that
root 1184 0.1 1.5 642916 9672 ? Ss 05:37 0:00 node
/usr/local/bin/forever start app.js
root 1185 0.1 2.1 641408 13200 ? Sl 05:37 0:00 node
/var/www/app.js
ubuntu 1217 0.0 0.1 7928 1060 pts/0 S+ 05:41 0:00 grep --color=auto node
I cannot control over the process, since I cannot list the process in "forever list"
How can I let "Forever" knowing its running processes and let having control over its running processes.
forever list should be invoked with same user as that of processes.
Generally it is root user (in case of ubuntu upstart unless specified) so you can switch to root user using sudo su and then try forever list.
PS. Moved to pm2 recently which has lot more features than forever.
i had the same problem today.
in my case: i'm using NVM and forgot that it doesn't set/modify the global node path, so i had to set it manually
export NODE_PATH="/root/.nvm/v0.6.0/bin/node"
If you exec the forever start app.js within init.d you should later type sudo HOME=/home/pi/devel/web-app -u root forever list to have the correct list.
A fix would be great for this.
Encountered this one as well.
I believe an this issue was logged here.
What I could recommend for now is to find the process that's using your node port e.g. 3000. Like so:
sudo lsof -t -i:3000
That command will show the process id.
Then kill the process by performing:
kill PID
sudo su
forever list
This will output the correct list (processes started by root user).