Kubernetes - Wait for available does not work as expected - azure

Question

I'm using a Gitlab CI/CD pipeline to deploy a few containers to a Kubernetes environment. The script (excerpt) basically just deploys a few resources like this:
.deploy-base:
# For deploying, we need an image that can interact with k8s
image:
name: registry.my-org.de/public-projects/kubectl-gettext:1-21-9
entrypoint: ['']
variables:
# Define k8s namespace and domain used for deployment:
NS: $KUBE_NAMESPACE
before_script:
- echo $NS
- cp $KUBECONFIG ~/.kube/config
- export CI_ENVIRONMENT_DOMAIN=$(echo "$CI_ENVIRONMENT_URL" | sed -e 's/[^/]*\/\/\([^#]*#\)\?\([^:/]*\).*/\2/')
script:
- kubectl config get-contexts
- kubectl config use-context org-it-infrastructure/org-fastapi-backend:azure-aks-agent
# Make Docker credentials available for deployment:
- kubectl create secret -n $NS docker-registry gitlab-registry-secret --docker-server=$CI_REGISTRY --docker-username=$CI_DEPLOY_USER --docker-password=$CI_DEPLOY_PASSWORD --docker-email=$GITLAB_USER_EMAIL -o yaml --dry-run | kubectl replace --force -n $NS -f -
- kubectl -n $NS patch serviceaccount default -p '{"imagePullSecrets":[{"name":"gitlab-registry-secret"}]}'
# Create config map for container env variables
- envsubst < dev/config-map.yml | kubectl -n $NS replace --force -f -
# Start and expose deployment, set up ingress:
- envsubst < dev/backend-deploy.yml | kubectl -n $NS replace --force -f -
# Set up ingress with env var expansion from template:
- envsubst < dev/ingress.yml | kubectl -n $NS replace --force -f -
# Wait for pod
- kubectl -n $NS wait --for=condition=available deployment/backend --timeout=180s
The last command should wait for the deployment to become available and return as soon as it does. Since the latest Gitlab 15 update and the switch from certificate based authentication vs agent based authentication towards K8s it doesn't work anymore and yields the following error message:
error: timed out waiting for the condition on deployments/backend
It also takes way longer than the specified 180s, its more like 15-20 minutes.
The application is available and works as expected, also the deloyment looks good:
$kubectl -n org-fastapi-backend-development describe deployment backend
Name: backend
Namespace: org-fastapi-backend-development
CreationTimestamp: Thu, 02 Jun 2022 14:15:18 +0200
Labels: app=app
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=app
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=app
Containers:
app:
Image: registry.my-org.de/org-it-infrastructure/org-fastapi-backend:development
Port: 80/TCP
Host Port: 0/TCP
Environment Variables from:
backend-config ConfigMap Optional: false
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: backend-6bb4f4bcd5 (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 83s deployment-controller Scaled up replica set backend-6bb4f4bcd5 to 1
As you can see, the Condition Available has the status set to true, still the wait command does not return positively.
Both kubectl as well as the Kubernetes environment (its Azure AKS) is running version 1.21.9