nodeSelector constraint ignored on AKS on mixed node pools (windows/linux)? - azure

Question

Tried installing a basic Nginx ingress using Helm by running the following command:
helm install nginx-ingress --namespace ingress-basic ingress-nginx/ingress-nginx \
--set controller.service.loadBalancerIP='52.232.109.226' \
--set controller.nodeSelector."beta\.kubernetes\.io/os"='linux' \
--set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"='linux' \
--set controller.replicaCount=1 \
--set rbac.create=true
Shortly after installing I noticed the pod was scheduled onto a Windows node instead of a Linux node:
wesley#Azure:~$ kubectl get pods -n ingress-basic -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ingress-ingress-nginx-admission-create-jcp6x 0/1 ContainerCreating 0 18s <none> akswin000002 <none> <none>
wesley#Azure:~$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
aks-agentpool-59412422-vmss000000 Ready agent 5h32m v1.17.11 10.240.0.4 <none> Ubuntu 16.04.7 LTS 4.15.0-1096-azure docker://19.3.12
aks-linuxpool-59412422-vmss000000 Ready agent 5h32m v1.17.11 10.240.0.128 <none> Ubuntu 16.04.7 LTS 4.15.0-1096-azure docker://19.3.12
akswin000000 Ready agent 5h28m v1.17.11 10.240.0.35 <none> Windows Server 2019 Datacenter 10.0.17763.1397 docker://19.3.11
akswin000001 Ready agent 5h28m v1.17.11 10.240.0.66 <none> Windows Server 2019 Datacenter 10.0.17763.1397 docker://19.3.11
akswin000002 Ready agent 5h28m v1.17.11 10.240.0.97 <none> Windows Server 2019 Datacenter 10.0.17763.1397 docker://19.3.11
Running a describe on the nginx pod revealed that the field Node-selectors remains to be set to <none>.
Name: nginx-ingress-ingress-nginx-admission-create-jcp6x
Namespace: ingress-basic
Priority: 0
PriorityClassName: <none>
Node: akswin000002/10.240.0.97
Start Time: Fri, 16 Oct 2020 20:09:36 +0000
Labels: app.kubernetes.io/component=admission-webhook
app.kubernetes.io/instance=nginx-ingress
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/version=0.40.2
controller-uid=d03091cd-8138-4923-a369-afeca669099c
helm.sh/chart=ingress-nginx-3.7.1
job-name=nginx-ingress-ingress-nginx-admission-create
Annotations: <none>
**Status: Pending**
IP:
Controlled By: Job/nginx-ingress-ingress-nginx-admission-create
Containers:
create:
Container ID:
Image: docker.io/jettech/kube-webhook-certgen:v1.3.0
Image ID:
Port: <none>
Host Port: <none>
Args:
create
--host=nginx-ingress-ingress-nginx-controller-admission,nginx-ingress-ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
--namespace=$(POD_NAMESPACE)
--secret-name=nginx-ingress-ingress-nginx-admission
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
POD_NAMESPACE: ingress-basic (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from nginx-ingress-ingress-nginx-admission-token-8x6ct (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
nginx-ingress-ingress-nginx-admission-token-8x6ct:
Type: Secret (a volume populated by a Secret)
SecretName: nginx-ingress-ingress-nginx-admission-token-8x6ct
Optional: false
QoS Class: BestEffort
**Node-Selectors: <none>**
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SandboxChanged 5m (x5401 over 2h) kubelet, akswin000002 Pod sandbox changed, it will be killed and re-created.
Warning FailedCreatePodSandBox 31s (x5543 over 2h) kubelet, akswin000002 (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "nginx-ingress-ingress-nginx-admission-create-jcp6x": Error response from daemon: container a734c23d20338d7fed800752c19f5e94688fd38fe82c9e90fc14533bae90c6bc encountered an error during hcsshim::System::CreateProcess: failure in a Windows system call: The user name or password is incorrect. (0x52e) extra info: {"CommandLine":"cmd /S /C pauseloop.exe","User":"2000","WorkingDirectory":"C:\\","Environment":{"PATH":"c:\\Windows\\System32;c:\\Windows"},"CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]}
I expected the pod to be scheduled onto a Linux node instead. Does anyone have a clue why this is happening? I saw no taints or anything and this is just a newly spin up cluster. The only workaround for now seems to be to scale the windows node back to 0. Install nginx ingress and then scale up the windows nodes again.
Kubernetes version: 1.17.11

Answer 1

Works with below. Added admissionWebhooks.patch.nodeSelector.
https://learn.microsoft.com/en-us/azure/aks/ingress-basic#create-an-ingress-controller
helm install nginx-ingress ingress-nginx/ingress-nginx \
--namespace ingress-basic \
--set controller.replicaCount=2 \
--set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set controller.admissionWebhooks.patch.nodeSelector."beta\.kubernetes\.io/os"=linux