I have been following this guide to deploy application on Azure using AKS
Every thing was fine until I deployed, one node is in not ready state with ImagePullBackOff status
kubectl describe pod output
Performing below command I get success command, so I am sure authentication is happening
az acr login --name siddacr
and this command lists out the image which was uploaded
az acr repository list --name <acrName> --output table
I figured out.
The error was in the name of the image in deployment.yml file
imagebackpulloff might be caused because of the following reasons:
The image or tag doesn’t exist
You’ve made a typo in the image name or tag
The image registry requires authentication
You’ve exceeded a rate or download limit on the registry
Related
I am deploying my services Azure AKS. I am running into an issue where I get a ImagePullBackOff error. Here is some context.
I have 2 nodepools one with --enable-node-public-ip option and another one without the node public-ip enabled option. I am trying to deploy a Daemonset resource. The container image is hosted on GitHub package registry. The issue is, the nodes that don't have a public IP enabled are successfully able to pull the images whereas, the nodes that have ppublic-ip enabled has an error.
Here is the error:
Failed to pull image "docker.pkg.github.com/xyz": rpc error: code = NotFound desc = failed to pull and unpack image "docker.pkg.github.com/xyz"
I would appreciate help on this.
ImagePullBackOff and ErrImagePull indicate that the image used by a container cannot be loaded from the image registry. Make sure you don't have a typo in image definition.
Try docker pull docker.pkg.github.com/OWNER/REPOSITORY/IMAGE_NAME:TAG_NAME beforehand. Afterwards, you should give docker.pkg.github.com/OWNER/REPOSITORY/IMAGE_NAME:TAG_NAME in Daemonset definition.
It is not an authentication issue.
I have followed this tutorial microsoft_website to pull images from an azure container. My yaml successfully creates a pod job, which can pull the image, BUT only when it runs on the agentpool node in my cluster.
For example, adding nodeName: aks-agentpool-33515997-vmss000000 to the yamlworks fine, but specifying a different node name, e.g. nodeName: aks-cpu1-33515997-vmss000000, the pod fails. The error message I get with describe pods is Failed to pull image and then kubelet Error: ErrImagePull.
What I'm missing?
Create secret:
kubectl create secret docker-registry <secret-name> \
--docker-server=<container-registry-name>.azurecr.io \
--docker-username=<service-principal-ID> \
--docker-password=<service-principal-password>
As #user1571823 told solution to the problem is deleting the old image from the acr and creating/pushing a new one.
The problem was related to some sort of corruption in the image saved in the azure container registry (acr). The reason why one agent pool could pulled the image was actually because the image already existed in the VM.
Henceforth as #andov said it is good option to open an incident case to Azure support for AKS from your subscription, where AKS is deployed. The support team has full access to the AKS service backend and they can tell exactly what was causing your problem.
Four things to check:
Is it a subscription issue? Are the nodes in different subscriptions?
Is it a rights issue? Does the service principle of the node have rights to pull the image.
Is it a network issue? Are the nodes on different subnets?
Is there something with the image size or configuration, that means that it cannot run on the other cluster.
Edit
New-AzAksNodePool has a parameter -DefaultProfile
It can be AzContext, AzureRmContext, AzureCredential
If this is different between your nodes it would explain the error
I'm using Minikube for development and I need to build a k8s app that pull all images from ACR, all images stored already on ACR.
To pull images from azure what I need to is to create secret with user&pass of the azure account and pass this secret to every image that I want to pull using imagePullSecrets (documentation here)
There is a way to add this registry as a global setting for namespace, or the project?
I don't understand why every image needs to get the secret implicitly in the spec.
Edit:
Thanks for the comments I'll check them later, for now I resolve this problem at minikube level. there is a way to set a private registry in minikube (doc here)
In my version this bug exists, and this answer resolve the problem.
As I know, if you do not use the K8s in Azure, I mean the Azure Kubernetes Service, then there are two ways I know the pull the images from ACR. One is the way you know that using the secrets. And another is to use the service account, but you also need to configure it in each deployment or the pods the same way as the secrets.
If you use the Azure Kubernetes Service, then you just need to assign the AcrPull role to the service principal of the AKS, and then you need to set nothing for each image.
You can add imagePullSecrets to a service account (e.g. to the default serviceaccout).
It will automatically add imagePullSecrets to the pod spec that has assigned this specific (e.g. default) serviceaccount, so you don't have to do it explicitly.
You can do it running:
kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "myregistrykey"}]}'
You can verify it with:
$ kubectl run nginx --image=nginx --restart=Never
$ kubectl get pod nginx -o=jsonpath='{.spec.imagePullSecrets[0].name}{"\n"}'
myregistrykey
Also checkout the k8s docs add-image-pull-secret-to-service-account.
In my case, I had a local Minikube installed in order to test locally my charts and my code. I tried most of the solutions suggested here and in other Stack Overflow posts and the following are the options I found out :
Move the image from the local Docker registry to Minikube's registry and set the pullPolicy to Never or IfNotPresent in your chart.
docker build . -t my-docker-image:v1
minikube image load my-docker-image:v1
$ minikube image list
rscoreacr.azurecr.io/decibel:0.0.1
k8s.gcr.io/pause:3.5
k8s.gcr.io/kube-scheduler:v1.22.3
k8s.gcr.io/kube-proxy:v1.22.3
...
##Now edit your chart and change the `pullPolicy`.
helm install my_name chart/ ## should work.
I think that the main disadvantage of this option is that you need to change your chart and remember to change the values to their previous value.
Create a secret that holds the credentials to the acr.
First login to the acr via :
az acr login --name my-registry.azurecr.io --expose-token
The output of the command should show you a user and an access token.
Now you should create a Kubernetes secret (make sure that you are on the right Kubernetes context - Minikube) :
kubectl create secret docker-registry my-azure-secret --docker-server=my-registry.azurecr.io --docker-username=<my-user> --docker-password=<access-token>
Now, if your chart uses the default service account (When you create a pod, if you do not specify a service account, it is automatically assigned the default service account in the same namespace) you should edit the service account via the following command :
kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "my-azure-secret"}]}'
I didn't like this option because if I have a different secret provider for every helm chart I need to overwrite the yaml with the imagePullSecrets.
Another alternative you have is using Minikube's registry creds
Personally, the solution I went for is the first solution with a tweak, instead of adding the pullPolicy in the yaml itself, I overwrite it when I install the chart :
$ helm install --set image.pullPolicy=IfNotPresent <name> charts/
We recently had an issue with our Azure Kubernetes Cluster not reporting back any data through the Azure Portal. To fix this, I updated the Kubernetes version to the latest version as was recommended on GitHub. After the upgrade was complete, we were able to view logs and monitoring data through the portal, but one of the containers stored in our Azure Container Registry is not able to be pulled by the Kubernetes Cluster.
The error I see in the Kuberenetes Management page is:
Failed to pull image "myacr.azurecr.io/container:190305.191": [rpc error: code = Unknown desc = Error response from daemon: Get https://myacr.azurecr.io/v2/mycontainer/manifests/190305.191: unauthorized: authentication required, rpc error: code = Unknown desc = Error response from daemon: Get https://myacr.azurecr.io/v2/mycontainer/manifests/190305.191: unauthorized: authentication required]
My original setup used the first script provided in this document and it worked correctly without issue. Once I started receiving the error, I ran it again just to make sure.
Once I saw that failed, I then deleted the account from the permissions on both the ACR and the AKS. Again, it failed to pull the image.
After that, I tried using the second method of creating an Kubernetes secret and received the same error.
At this point, I'm unsure what else to check. I've verified that I can run docker pull on my machine and pull the image, but there seems to be a breakdown between the AKS and the ACR that I can not sort out.
It's been a while since I originally posted this, but I did stumble across a currently stable solution to the problem.
The service principal, for whatever reason, is not able to maintain a connection to the ACR. So if your cluster ever goes down, you lose the ability to pull from the ACR. I had this happen multiple times over the last year and as I moved more of my Kubernetes deployment to Azure, it became a bigger and bigger issue.
I stumbled across this Microsoft Doc and noticed the mention of the --attach-acr command.
This is what the full command looks like:
az aks create -n myAKSCluster -g myResourceGroup --generate-ssh-keys --attach-acr $MYACR
Since setting it up with that flag, I have had 0 issues with it.
knock on wood
I'm using Azure for my Continuous Deployment, My secret name is "cisecret" using
kubectl create secret docker-registry cisecret --docker-username=XXXXX --docker-password=XXXXXXX --docker-email=SomeOne#outlook.com --docker-server=XXXXXXXXXX.azurecr.io
In my Visual Studio Online Release Task
kubectl run
Under Secrets section
Type of secret: dockerRegistry
Container Registry type: Azure Container Registry
Secret name: cisecret
My Release is successfully, but when proxy into kubernetes
Failed to pull image xxxxxxx unauthorized: authentication required.
Could this be due to your container name possibly? I had an issue where I wasn't properly prepending the ACR domain in front of the image name in my Kubernetes YAML which meant I wasn't pointed at the container registry / image and therefore my secret (which was working) appeared to be broken.
Can you post your YAML? Maybe there is something simple amiss since it seems you are on the right track from the secrets perspective.
I need to grant AKS access to ACR.
Please refer to the link here
How to pass image pull secret while using 'kubectl run' command?
This should help, you need to override the kubectl command with "imagepullsecrets":"cisecret".
Add the following in yaml file.
imagePullSecrets:
- name: acr-auth