I have a K8s cluster created in the context of the Linux Foundation's CKAD course (LFD259). So it is a "bare metal" cluster created with kubeadm.
So I have a metrics-server deployment running on the worker node:
student#master:~$ k get deployments.apps metrics-server -o yaml | grep -A10 args
- args:
- --secure-port=4443
- --cert-dir=/tmp
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --metric-resolution=15s
- --kubelet-insecure-tls
image: k8s.gcr.io/metrics-server/metrics-server:v0.6.1
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
student#master:~$ k get pod metrics-server-6894588c69-fpvtt -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
metrics-server-6894588c69-fpvtt 1/1 Running 0 4d15h 192.168.171.98 worker <none> <none>
student#master:~$
It is my understanding that the pod's process runs inside a container running on the worker node. However, I am completely puzzled by the fact that the linux ps command "sees" it:
student#worker:~$ ps aux | grep kubelet-preferred-address-types
ubuntu 1343092 0.3 0.6 752468 49612 ? Ssl Oct28 20:25 /metrics-server --secure-port=4443 --cert-dir=/tmp --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --metric-resolution=15s --kubelet-insecure-tls
student 3310743 0.0 0.0 8184 2532 pts/0 S+ 17:39 0:00 grep --color=auto kubelet-preferred-address-types
student#worker:~$
What am I missing?
A container is just a process running on your host with some isolation features enabled. The isolation only works in one way: a container can't see resources on your host, but your host has access to all the resources running in a container.
Because a container is just a process, it shows up in ps (as do any processes that are spawned inside the container).
See e.g.:
"What is a Linux Container?"
Related
I am running an Elasticsearch container as Podman pod using podman play kube and a yaml definition of a pod. Pod is created, cluster of three nodes is created and everything works as expected. But: Podman pod dies after a few days of staying idle.
Podman podman ps command says:
ERRO[0000] Error refreshing container af05fafe31f6bfb00c2599255c47e35813ecf5af9bbe6760ae8a4abffd343627: error acquiring lock 1 for container af05fafe31f6bfb00c2599255c47e35813ecf5af9bbe6760ae8a4abffd343627: file exists
ERRO[0000] Error refreshing container b4620633d99f156bb59eb327a918220d67145f8198d1c42b90d81e6cc29cbd6b: error acquiring lock 2 for container b4620633d99f156bb59eb327a918220d67145f8198d1c42b90d81e6cc29cbd6b: file exists
ERRO[0000] Error refreshing pod 389b0c34313d9b23ecea3faa0e494e28413bd15566d66297efa9b5065e025262: error retrieving lock 0 for pod 389b0c34313d9b23ecea3faa0e494e28413bd15566d66297efa9b5065e025262: file exists
POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS
389b0c34313d elasticsearch-pod Created 1 week ago af05fafe31f6 2
What's weird is that the process is still listening if we try to find the process id listening on port 9200 or 9300:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::9200 :::* LISTEN 1328607/containers-
tcp6 0 0 :::9300 :::* LISTEN 1328607/containers-
The process ID that is hanging (and making the process still listening is):
user+ 1339220 0.0 0.1 45452 8284 ? S Jan11 2:19 /bin/slirp4netns --disable-host-loopback --mtu 65520 --enable-sandbox --enable-seccomp -c -e 3 -r 4 --netns-type=path /tmp/run-1002/netns/cni-e4bb2146-d04e-c3f1-9207-380a234efa1f tap0
The only actions I do to the pod is regular: podman pod stop, podman pod rm and podman play kube that is starting pod.
What can be causing such strange behaviour of Podman? What may be causing the lock not to be released properly?
System information:
NAME="Red Hat Enterprise Linux"
VERSION="8.3 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.3"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.3"
Red Hat Enterprise Linux release 8.3 (Ootpa)
Red Hat Enterprise Linux release 8.3 (Ootpa)
Podman version:
podman --version
podman version 2.2.1
The workaround that worked for me is to add this configuration file from the Podman repository [1] under /usr/lib/tmpfiles.d/ and /etc/tmpfiles.d/, in this way we are preventing the removal of Podman temporary files from /tmp directory [2]. As stated in [3], additionally CNI leaves Network information in /var/lib/cni/networks when the system crashes or containers do not shut down properly. This behaviour has been fixed in the latest Podman release [4] and it happens when using rootless Podman.
Workaround
First, check the runRoot default directory set for your Podman rootless user:
podman info | grep runRoot
Create the temporary configuration file:
sudo vim /usr/lib/tmpfiles.d/podman.conf
Add the following content, replacing /tmp/podman-run-* by your default runRoot directory. E.g. If your output is /tmp/run-6695/containers then use: x /tmp/run-*
# /tmp/podman-run-* directory can contain content for Podman containers that have run
# for many days. This following line prevents systemd from removing this content.
x /tmp/podman-run-*
x /tmp/containers-user-*
D! /run/podman 0700 root root
D! /var/lib/cni/networks
Copy the temporary file from /usr/lib/tmpfiles.d to /etc/tmpfiles.d/
sudo cp -p /usr/lib/tmpfiles.d/podman.conf /etc/tmpfiles.d/
After you have done all the steps according to your configuration, the error should disappear.
References
https://github.com/containers/podman/blob/master/contrib/tmpfile/podman.conf
https://bugzilla.redhat.com/show_bug.cgi?id=1888988#c9
https://github.com/containers/podman/commit/2e0a9c453b03d2a372a3ab03b9720237e93a067c
https://github.com/containers/podman/pull/8241
Problem
Docker containers started via Jenkins pipeline command
docker.image(imageToStart).inside('--init')
can not be stopped due to zombie processes left by container.
Questions
How is it possible to get zombie processes from a Docker container, when it was started with '--init' option?
Has someone else hit the same issue?
Used environment
Docker 18.03.1-ce
Jenkins 2.60.2
Docker Pipeline plugin 1.12
Details
When a container is started from Jenkins pipeline with a command like:
docker.image('alpine').inside('--init') {
sh ('ps -efa -o pid,ppid,user,comm')
}
There are several processes in this container with parent PID 0:
[Pipeline] withDockerContainer
loco does not seem to be running inside a container
$ docker run -t -d -u 1001:1002 \
--init \
-w /lhome/ci<br>admin/jenkins/workspace/bli-groovy-test \
-v /lhome/ciadmin/jenkins/workspace/bli-groovy-test:/lhome/ciadmin/jenkins/workspace/bli-groovy-test:rw,z \
-v /lhome/ciadmin/jenkins/workspace/bli-groovy-test-tmp:/lhome/ciadmin/jenkins/workspace/bli-groovy-test-tmp:rw,z \
-e ******** \
--entrypoint cat alpine
[Pipeline] {
[Pipeline] sh
[bli-groovy-test] Running shell script
+ ps -efa -o pid,ppid,user,comm
PID PPID USER COMMAND
1 0 1001 init
7 1 1001 cat
8 0 1001 sh
14 8 1001 script.sh
15 14 1001 ps
[Pipeline] }
PID 1 / PPID 0 is the 'init' command used to start the container
PID 8 / PPID 0 is the 'sh' command from the closure to execute 'ps' command
The 'sh' process does not reap its child processes. When the process itself exits its descendants are assigned to a PPID from outside the container and not to PPID 1 from the 'init' process of the container.
The new parent PID is the PID of the 'docker-containerd-shim' process of the container.
With the small example I could not reproduce the zombie processes, but here is the situation from a more complex Jenkins job:
Docker command from Jenkins job
$ docker run -t -d -u 1001:1002 \
--init \
-w /lhome/testadmin/jenkins-coreloops/workspace/test-job/database \
-v /lhome/testadmin/jenkins-coreloops/workspace/test-job/database:/lhome/testadmin/jenkins-coreloops/workspace/test-job/database:rw,z \
-v /lhome/testadmin/jenkins-coreloops/workspace/test-job/database-tmp:/lhome/testadmin/jenkins-coreloops/workspace/test-job/database-tmp:rw,z \
-e ******** \
--entrypoint cat richmond.lhs-systems.com:5000/ait/mpde
[Pipeline] {
[Pipeline] sh
10:03:09 [database] Running shell script
10:03:09 + ./db-upgrade.sh
inside the container shell scripts are started
shell scripts call perl scripts
perl scripts start SQL*Plus (1 instance per database login)
perl scripts send SQL commands to SQL*Plus instances via STDIN
When the closure ends and Jenkins tries to stop the container, the following processes are left:
[testadmin#testhost] ~ # ps -efa | grep -vw grep | grep -w 47077
root 1725 47077 0 10:03 ? 00:00:00 [ps] <defunct>
root 1732 47077 0 10:03 ? 00:00:00 [docker-runc] <defunct>
root 2887 47077 0 10:04 ? 00:00:00 [sqlplus] <defunct>
root 2915 47077 0 10:04 ? 00:00:00 [sqlplus] <defunct>
root 47077 17349 0 10:03 ? 00:00:00 docker-containerd-shim
-namespace moby
-workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/1863503ca54f75168db8ce20c78b821c0e5280f07d59875e8f651db4f0b67d9f
-address /var/run/docker/containerd/docker-containerd.sock
-containerd-binary /usr/bin/docker-containerd
-runtime-root /var/run/docker/runtime-runc
root 47098 47077 0 10:03 pts/0 00:00:00 /dev/init -- cat
root 47506 47077 0 10:03 ? 00:00:00 [sh] <defunct>
[testadmin#testhost] ~ #
and command 'docker stop' is aborted after timeout of 180 seconds.
To cleanup the remaining processes of the container this docker-containerd-shim process has to be killed with SIGKILL.
Note
We observed this issue on our recently installed CentOS server:
- CentOS Linux release 7.5.1804 (Core)
- environment related parts from 'docker info':
Server Version: 18.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-862.6.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 48
Total Memory: 377.6GiB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
The behavior on other of our Docker hosts is similar with respect to the multiple processes with parent PID 0, but we didn't observe that containers were hanging on shutdown or that there was a similar number of zombie processes.
For comparisation the similar 'docker info' extract from one of these other hosts:
Server Version: 17.05.0-ce
Storage Driver: devicemapper
Pool Name: dock-thinpool
Pool Blocksize: 524.3kB
Base Device Size: 16.11GB
Backing Filesystem: xfs
Data file:
Metadata file:
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-693.11.6.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 48
Total Memory: 377.6GiB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Edit 2018-Aug-01:
As workaround I added the init process as subreaper also to problematic 'docker exec' calls and problematic 'sh' calls in the Jenkins docker().inside() closure.
This eliminated the zombie processes in our environment.
Every time I boot up my Lubuntu 16.04 laptop I can see I have a running docker container:
$ ps -ef | grep docker
root 1724 1 3 21:17 ? 00:01:30 /usr/bin/dockerd -H fd://
root 1774 1724 0 21:17 ? 00:00:04 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
root 4750 1774 0 21:17 ? 00:00:00 docker-containerd-shim 72541a4648b890132985daf2357d1130b8b5208cf12ede607b93ab2987629719 /var/run/docker/libcontainerd/72541a4648b890132985daf2357d1130b8b5208cf12ede607b93ab2987629719 docker-runc
stephane 10755 1793 0 22:07 pts/0 00:00:00 grep docker
It serves a Jenkins application on the port 80 and requesting localhost/ in the browser redirects to http://localhost/login?from=%2F and shows a Jenkins warning page:
Unlock Jenkins
To ensure Jenkins is securely set up by the administrator, a password has been written to the log (not sure where to find it?) and this file on the server:
A wget request shows:
$ wget localhost/
--2017-05-23 22:09:55-- http://localhost/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2017-05-23 22:09:55 ERROR 403: Forbidden.
How can I know which service is firing up this docker process ?
I looked in the /etc/init.d/ directory:
$ l /etc/init.d/
alsa-utils* checkroot-bootclean.sh* halt* mattermostd* nginxd* rc* single* uuidd*
anacron* checkroot.sh* hostname.sh* mountall-bootclean.sh* ntp* rc.local* skeleton whoopsie*
apachedsd* console-setup* httpd* mountall.sh* ondemand* rcS* ssh* x11-common*
apparmor* cron* hwclock.sh* mountdevsubfs.sh* openvpn* README tomcatd*
apport* cups* irqbalance* mountkernfs.sh* php-fpm* reboot* udev*
avahi-daemon* cups-browsed* keyboardd* mountnfs-bootclean.sh* plymouth* redis* ufw*
bluetooth* dbus* killprocs* mountnfs.sh* plymouth-log* resolvconf* umountfs*
bootmisc.sh* docker* kmod* mysqld* postfix* rsync* umountnfs.sh*
cgroupfs-mount* dropboxd* lightdm* networking* pppd-dns* rsyslog* umountroot*
checkfs.sh* grub-common* mariadbd* network-manager* procps* sendsigs* urandom*
The /etc/init.d/docker is mine and removing it from the directory, a reboot still comes up with a running docker process.
I removed the /etc/init.d/docker file, rebooted, and there is a docker process:
$ ps -ef | grep docker
root 1560 1 5 22:15 ? 00:00:06 /usr/bin/dockerd -H fd://
root 1645 1560 0 22:15 ? 00:00:00 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
root 4644 1645 0 22:15 ? 00:00:00 docker-containerd-shim 069db46cca05d43c35f05ff50aaa836507cbf69e4e3d9443b6b859d0edb5b076 /var/run/docker/libcontainerd/069db46cca05d43c35f05ff50aaa836507cbf69e4e3d9443b6b859d0edb5b076 docker-runc
stephane 5520 1741 0 22:17 pts/0 00:00:00 grep docker
So I looked up for anything docker in all these files, but found nothing named docker:
$ cd /etc/init.d/
[stephane#stephane-ThinkPad-X301 init.d]
$ grep.sh docker
[stephane#stephane-ThinkPad-X301 init.d]
This docker process is there every time I start my laptop, even when off line.
What starts this docker process ?
Lubuntu 16.04 comes with systemd by default. At some point you must have started up a jenkins instance in docker - it's hard to tell exactly what started the process initially. However, systemd would be what is currently causing it to start. In order to stop it from running, run the following commands:
systemctl status docker <- Find out of systemctl thinks docker is running.
It'll likely show something like this:
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2017-05-21 22:59:46 EDT; 1 day 17h ago
Docs: http://docs.docker.com
Main PID: 1314 (dockerd-current)
Tasks: 14 (limit: 8192)
CGroup: /system.slice/docker.service
└─1314 /usr/bin/dockerd-current --add-runtime oci=/usr/libexec/docker/docker-runc-current --default-runtime=oci --containerd /run/containerd.sock --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --selinux-enabled --log-driver=journald
To stop it, run systemctl stop docker and then systemctl disable docker. As a last resort if this doesn't work, you can run systemctl mask docker.
Docker is being started by systemd in your environment. You can disable the entire engine by running:
sudo systemctl disable docker
sudo systemctl stop docker
You can also stop only the container that is running (the shim and Jenkins application):
sudo docker ps # lists the running containers along with their container id
sudo docker update --restart=no $container_id
sudo docker stop $container_id
If you know that you do not need this container and want to permanently delete it, you can run this instead of the above two last commands:
sudo docker rm -f $container_id
The -f switch also stops the container if it's currently running.
Edit: from your comment, your container is running under swarm mode which is redeploying it. To stop that first find the stack or service that is running it.
sudo docker stack ls
sudo docker service ls
If you see a stack listed, you can remove that with:
sudo docker stack rm $stack_name
If there are no stacks listed, or they don't apply to this container, you can delete the service with:
sudo docker service rm $service_name
I have 5 machines running Ubuntu 16.04.1 LTS. I want to set them up as a Kubernetes Cluster. Iḿ trying to follow this getting started guide where they're using kubeadm.
It all worked fine until step 3/4 Installing a pod network. I've looked at there addon page to look for a pod network and chose the flannel overlay network. Iǘe copied the yaml file to the machine and executed:
root#up01:/home/up# kubectl apply -f flannel.yml
Which resulted in:
configmap "kube-flannel-cfg" created
daemonset "kube-flannel-ds" created
So i thought that it went ok, but when I display all the pod stuff:
root#up01:/etc/kubernetes/manifests# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system dummy-2088944543-d5f50 1/1 Running 0 50m
kube-system etcd-up01 1/1 Running 0 48m
kube-system kube-apiserver-up01 1/1 Running 0 50m
kube-system kube-controller-manager-up01 1/1 Running 0 49m
kube-system kube-discovery-1769846148-jvx53 1/1 Running 0 50m
kube-system kube-dns-2924299975-prlgf 0/4 ContainerCreating 0 49m
kube-system kube-flannel-ds-jb1df 2/2 Running 0 32m
kube-system kube-proxy-rtcht 1/1 Running 0 49m
kube-system kube-scheduler-up01 1/1 Running 0 49m
The problem is that the kube-dns keeps in the ContainerCreating state. I don't know what to do.
It is very likely that you missed this critical piece of information from the guide:
If you want to use flannel as the pod network, specify
--pod-network-cidr 10.244.0.0/16 if you’re using the daemonset manifest below.
If you omit this kube-dns will never leave the ContainerCreating STATUS.
Your kubeadm init command should be:
# kubeadm init --pod-network-cidr 10.244.0.0/16
and not
# kubeadm init
Did you try restarting NetworkManager ...? it worked for me.. Plus, it also worked when I also disabled IPv6.
Tried installing kubernetes v1.2.0 on azure environment but after installation cannot access kube apis at port 8080.
Following services are running :
root 1473 0.2 0.5 536192 42812 ? Ssl 09:22 0:00 /home/weave/weaver --port 6783 --name 22:95:7a:6e:30:ed --nickname kube-00 --datapath datapath --ipalloc-range 10.32.0.0/12 --dns-effective-listen-address 172.17.42.1 --dns-listen-address 172.17.42.1:53 --http-addr 127.0.0.1:6784
root 1904 0.1 0.2 30320 20112 ? Ssl 09:22 0:00 /opt/kubernetes/server/bin/kube-proxy --master=http://kube-00:8080 --logtostderr=true
root 1907 0.0 0.0 14016 2968 ? Ss 09:22 0:00 /bin/bash -c until /opt/kubernetes/server/bin/kubectl create -f /etc/kubernetes/addons/; do sleep 2; done
root 1914 0.2 0.3 35888 22212 ? Ssl 09:22 0:00 /opt/kubernetes/server/bin/kube-scheduler --logtostderr=true --master=127.0.0.1:8080
root 3129 2.2 0.3 42488 25192 ? Ssl 09:27 0:00 /opt/kubernetes/server/bin/kube-controller-manager --master=127.0.0.1:8080 --logtostderr=true
curl -v http://localhost:8080 returns error
Rebuilt URL to: http://localhost:8080/
Trying 127.0.0.1...
connect to 127.0.0.1 port 8080 failed: Connection refused
Failed to connect to localhost port 8080: Connection refused
Closing connection 0 curl: (7) Failed to connect to localhost port 8080: Connection refused
Same works fine with v1.1.2.
I'm using following guidelines https://github.com/kubernetes/kubernetes/tree/master/docs/getting-started-guides/coreos/azure and updated line https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/coreos/azure/cloud_config_templates/kubernetes-cluster-main-nodes-template.yml#L187 to user version v1.2.0.
The services you show running do not include the apiserver. For a quick breakdown I can explain what each service does that you show running.
Weave: This is a software overlay network and assigns IP addresses to your pods.
kube-proxy: This runs on your worker nodes allow pods to run and route traffic between exposed services.
kubectl create: Kubectl is actually the management cli tool but in this case using -f /etc/kubernetes/addons/; sleep 2 is watching the /etc/kubernetes/addons/ folder and automatically creating any objects (pods, replication controllers, services, etc.) that are put in that folder.
kube-scheduler: Responsible for scheduling pods onto nodes. Uses policies and rules.
kube-controller-manager: Manages the state of the cluster by always making sure the current state and desired state are the same. This includes starting/stopping pods and creating objects (services, replication-controllers, etc) that do not yet exist or killing them if they shouldn't exist.
All of these services interact with the kube-apiserver which should be a separate service that coordinates all of the information these other services use. You'll need the apiserver running in order for all of the other components to do their jobs.
I won't go into the details of getting it running in your environment but from it looks like in the comments on your original thread you found some missing documentation to get it running.