kubernetes securitycontext runAsNonRoot Not working - security

Question

I am testing with securityContext but I cant start a pod when I set runAsNonRoot to true.
I use vagrant to deploy a master and two minions and ssh to the host machine as the user abdelghani :
id $USER
uid=1001(abdelghani) gid=1001(abdelghani) groups=1001(abdelghani),27(sudo)
Cluster information:
Kubernetes version: 4.4.0-185-generic
Cloud being used: (put bare-metal if not on a public cloud)
Installation method: manual
Host OS: ubuntu16.04.6
CNI and version:
CRI and version:
apiVersion: v1
kind: Pod
metadata:
name: buggypod
spec:
containers:
- name: container
image: nginx
securityContext:
runAsNonRoot: true
I do :
kubectl apply -f pod.yml
it says pod mybugypod created but when I check with :
kubectl get pods
the pod’s status is CreateContainerConfigError
what is it I am doing wrong?

Answer 1

I try to run the pod based on your requirement. And the reason it failed is the Nginx require to modify some configuration in /etc/ owned by root and when you runAsNonRoot it fails as it cannot edit the Nginx default config.
This is the error you actually get when you run it.
10-listen-on-ipv6-by-default.sh: error: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2020/08/13 17:28:55 [warn] 1#1: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:2
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:2
2020/08/13 17:28:55 [emerg] 1#1: mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)
nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)
The spec I ran.
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: buggypod
name: buggypod
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
containers:
- image: nginx
name: buggypod
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {}
My suggestion is you create a custom Nginx image with a Dockerfile that also creates user and provides permissions to the folders /var/cache/nginx, /etc/nginx/conf.d, /var/log/nginx for the newly created user. Such that you achieve running the container as Non-Root.

Answer 2

Nginx service will expect a read and write permission to its configuration path (/etc/nginx) by default non root user would have that access to the path that is the reason it is failing.
You just set runAsNonRoot but you can't expect or guarantee that container will start the service as user 1001. Please try setting runAsUser explicitly to 1001 like below, this should resolve your issue.
apiVersion: v1
kind: Pod
metadata:
name: buggypod
spec:
containers:
- name: container
image: nginx
securityContext:
runAsUser: 1001