I am deploying containers from Azure Container Registry using Azure Release and the Azure App Service Deploy task. I have a Subscription Level Service connection defined. Container Registry Settings are set as:
Registry or Namespace: <registry>.azurecr.io
Image: rrez/vnext/dev/booking
Tag: latest
Startup command: "dotnet", "Booking.API.dll"
The task completes successfully, but when reviewing in Container Log on the App Service I see:
2019_02_25_RD00155D9B2488_docker.log:
2019-02-25 22:27:07.101 INFO - Issuing docker pull: imagename =<registry>.azurecr.io/rrez/vnext/dev/booking:latest
2019-02-25 22:27:07.260 INFO - Issuing docker pull: imagename =<registry>.azurecr.io/rrez/vnext/dev/booking:latest
2019-02-25 22:27:07.410 INFO - Issuing docker pull <registry>.azurecr.io/rrez/vnext/dev/booking:latest
2019-02-25 22:27:07.487 ERROR - docker pull returned STDERR>> Error response from daemon: Get https://<registry>.azurecr.io/v2/rrez/vnext/dev/booking/manifests/latest: unauthorized: authentication required
This seems to be an ACR access issue, but I would expect the Subscription Level Service Connection to be authority enough.
Since this is Azure Release there is no YAML configuration available to add additional authentication details.
I pasted the YAML from the Release Task into the successfully completing Build and received the same error.
The YAML is:
- task: AzureRmWebAppDeployment#4
displayName: 'Azure App Service Deploy: P-RREZ-BOOKING-PREPROD'
inputs:
ConnectionType: AzureRM
azureSubscription: 'RightRez.Services SubscriptionSC'
appType: webAppContainer
WebAppName: 'P-RREZ-BOOKING-PREPROD'
deployToSlotOrASE: true
ResourceGroupName: '<ResourceGroup>-RG'
DockerNamespace: <registry>.azurecr.io
DockerRepository: rrez/vnext/dev/booking
DockerImageTag: latest
StartupCommand: '"dotnet", "Booking.API.dll"'
Not sure where the problem lies.
Have you set the Access level in the ACR? If not please try setting up one from here. I had similar issue and I tried setting up the role assignment and it worked.
Related
I am trying to automate my deployment process by creating a pipeline in Azure DevOps that does the following
Build my project, create a docker image, then pushed the image to a private Azure Registry service.
Deploy the image on a slot called staging in Azure Web Service.
Here is the .yaml file that I am using
trigger:
- master
resources:
- repo: self
variables:
dockerRegistryServiceConnection: 'MyPrivateRegistry'
imageRepository: 'MyPrivateRepositoryName'
containerRegistry: 'MyPrivateRepositoryName.azurecr.io'
dockerfilePath: '$(Build.SourcesDirectory)/Dockerfile'
tag: '$(Build.BuildId)'
azureSubscription: 'MyPrivateSubscribtionName(5c4b9a4b-private-subscribtion-id-91503531e1a0)'
appName: 'private_appname'
resourceGroupName: 'PrivateResourceGroup'
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Push and Build
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: Docker#2
displayName: Build and push an image
inputs:
command: buildAndPush
repository: $(imageRepository)
dockerfile: $(dockerfilePath)
containerRegistry: '$(dockerRegistryServiceConnection)'
tags: |
$(tag)
- job: DeployToStaging
displayName: Deploy To staging
dependsOn: Build
condition: succeeded()
pool:
vmImage: $(vmImageName)
steps:
- task: AzureWebAppContainer#1
inputs:
azureSubscription: $(azureSubscription)
appName: $(appName)
deployToSlotOrASE: true
resourceGroupName: $(resourceGroupName)
slotName: 'staging'
containers: '$(containerRegistry)/$(imageRepository):$(tag)'
The projects is built successfully and pushed to the private registry as expected. I can verify that the new image in pushed with a new tagId. However, my container fails to start with the following error
Image pull failed since Inspect image returned null: MyPrivateRepositoryName.azurecr.io/MyPrivateRepositoryName:151
Here is the suggestion I see
Please check the repository name, image name, and container definitions defined by DOCKER_REGISTRY_SERVER_USERNAME, DOCKER_REGISTRY_SERVER_URL, and DOCKER_REGISTRY_SERVER_PASSWORD.
When I go to the staging slot configuration I see the following and the values are all correct. I copied these values from the "Access keys" section in the Container Registry service after enabling the Admin user
What am I missing here? How can I get the slot to correctly pull the docker image from the contain registry?
Updated
Looking more at the logs gives me this error
2022-04-01T20:36:21.409Z ERROR - Pull image threw Exception: Input string was not in a correct format.
2022-04-01T20:36:21.411Z INFO - Pulling image from Docker hub: privateregistry.azurecr.io/privateimage:152
2022-04-01T20:36:21.594Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://privateregistry.azurecr.io/v2/privateimage/manifests/152: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information."}
It sounds like something does not have permission to pull the docker image from the repository. Question is what object need to have this permission? Would I add permission to the private repository of would I add it to the Web Service?
Few things you can check if Enable Admin Access does not work.
Is your app service in the same subscription as your ACR? If not, try moving it to the same subscription.
In Azure Portal -> App Service -> Deployment Center, see if you have conflicting settings. If it's empty, try set up the container registry information here instead of passing in app settings as env variables.
If still having errors, I recommend you creating another credential to login to ACR instead of using the admin one. (For this, it won't involve system identity or managed identity)
Register a new app in Azure that will read from ACR
In ACR -> Access Control -> give this app ACR PULL permission
Replace your app service app setting with the following
DOCKER_REGISTRY_SERVER_USERNAME = Client Id of the created app
DOCKER_REGISTRY_SERVER_PASSWORD = Client Secret of the created app
Pushing and pulling of image to Azure Container Registry task in Azure DevOps pipeline fails. When tried to pull or push from local system, there's no problem but when tried to do it using the Azure Devops pipeline it fails. Docker login was successful but it fails when I want to pull the image from the ACR with the following result:
**Error response from daemon: Head "***/a2/abcd/manifest/latest": unauthorized: Invalid clientid or client secret.
##[error]Bash exited with code '1'.
##[debug]Processed: ##vso[task.issue type=error;]Bash exited with code '1'.
I checked all the service connections in Az Devops, and they all look correctly configured. Checked the associated service principals as well if they have AcrPull and AcrPush permissions, all of them are in place. Just couldn't understand what's going wrong.
My Yaml looks like this:
trigger: none
schedules:
- cron: "0 0 0 * *"
displayName: **** *
branches:
include:
- abcd
always: true
pool:
vmImage: 'ubuntu-latest'
variables:
- name: acrname
value: *****.azurecr.io
stages:
- stage: abcd
displayName: "pull images from acr"
jobs:
- job: abcdef
displayName: "abcdef"
pool:
vmImage: ubuntu-latest
steps:
- task: Docker#2
displayName: Login to ACR
inputs:
command: login
containerRegistry: '*****.azurecr.io'
- bash: |
docker pull $(acrname)/abc-def:latest
docker pull $(acrname)/igh-jkl:latest
name: pull
displayName: 'pull acr images'
Can anyone help?
I had the same issue with the ACR service connection was expired. So I had to create a new service connection by using these steps.
Docker ID and Docker Password can be obtained from ACR --> Settings --> Access keys
Update your pipeline with this new service connection and you are good to go. Please rate if this solution helps you.
In my case, when I ran into this issue, the simple and clean resolution was to use the docker login. In your situation, it looks like this would be a good solution :
docker login $(acrname)
prior to your calls to get your images
docker pull $(acrname)/abc-def:latest
In my case, the docker login password expired. So I have to do the following:
Go azure and generate the new password for the docker app authentication.
Copy the newly generated password.
Go to your virtual machine where docker is running.
Try the below command
docker login blah.azurecr.io --username your-user-name-here --password yourhaspasswordhere~5Crf9b
Now you are good to go.
I have a weird issue, possible I am missing something. The web app is created as a container with default settings(Quickstart) and I am using Azure Container Registry to push my docker image. I have pipeline which logins to the ACR, build and pushes image, and then deploys the image to the web app. All these tasks are successful. But when I got to the webapp in azure portal, the image source is set to docker hub and not the Azure Container registry.The full inage name and tag are of the ACR. Any suggestion what I am missing?
Update: I have added the pipeline.yml file for reference, if it helps. I am logging in to registry while pushing docker image, I confirm that the image is push in the ACR.
- task: Docker#2
displayName: Login to QA Azure Container Registry
inputs:
command: login
containerRegistry: $(azureSubscriptionContainer) #Docker Registry type Service connection
- task: Docker#2
displayName: Build and Push
inputs:
command: buildAndPush
repository: $(repository) #custom name of repository in ACR
tags: $(Build.BuildId)
- task: AzureRMWebAppDeployment#4
displayName: Azure App Service Deploy
inputs:
appType: webAppContainer
ConnectedServiceName: $(azureSubscription) #ARM service connection
WebAppName: $(webApp)
DockerNamespace: $(dockerNameSpace) #ACR namespace myacr.azurecr.io
DockerRepository: $(repository) #custom name of repository in ACR
DockerImageTag: $(Build.BuildId)
TLDR: You need to either select the repository in the UI; or add the credentials via ARM or AzureCLI.
https://learn.microsoft.com/en-us/azure/devops/pipelines/targets/webapp-on-container-linux?view=azure-devops&tabs=dotnet-core%2Cyaml#configure-registry-credentials-in-web-app
You can use your deployment task like you are doing right now, not all developers need access to the WebApp. However, you need to set up the credentials to the registry at least once. The UI will basically show its "best guess" for how to intepret the underlying configuration.
https://learn.microsoft.com/en-us/azure/app-service/containers/tutorial-custom-docker-image#configure-registry-credentials-in-web-app
You could also do that with every pipeline run, tough I would not recommend that.
Sidenote: You can also have your webapp automatically pull a certain tag whenever it is updated in the container registry if you select "Continuous deployment".
Create App Service with CLI:
az webapp create --resource-group myResourceGroup --plan myAppServicePlan --name <app-name> --deployment-container-image-name <azure-container-registry-name>.azurecr.io/mydockerimage:v1.0.0
This is for public images, but according the documentation if you want to use private image you must also configure registry credentials to yours Web App and if this is not Docker registry you must provide -docker-registry-server-url:
az webapp config container set --name <app-name> --resource-group myResourceGroup --docker-custom-image-name <azure-container-registry-name>.azurecr.io/mydockerimage:v1.0.0 --docker-registry-server-url https://<azure-container-registry-name>.azurecr.io --docker-registry-server-user <registry-username> --docker-registry-server-password <password>
I have a very weird (and I suppose easy to fix) problem :) I am trying to have a working CI/CD pipeline in Azure. For this purpose, I have a repository in Azure devops and build and release pipeline created. I am publishing docker images to Azure Container Registry and during release, I am pulling this image (or at least - I am trying because it doesn't work) and I am trying to publish it on Webapp for containers. The "app" in my case it is SingalR hub on .NET Core 3.1 (but I don't suppose it makes a difference in the problem I am having)
If somebody wants to know in details how did i configure it - here is the tutorial i did use:
https://wikiazure.com/devops/azure-devops-automate-your-release-pipeline-to-provision-a-docker-container-to-azure-web-app-for-containers/
There were some doubts/differences in the tutorial (for example - why initially in the tutorial web app is being configured on Docker hub, when in fact it is using ACR. And why to connect to ACR the tutorial uses Azure Resource Manager connection (And not dedicated Docker container --> ACR connetion) And why later on in build pipeline there is some weird id set for dockerRegistryServiceConnection (i am giving in this place name of my ACR docker service connection)
But the whole build pipeline is working. It is publishing image to ACR. Everything is fine till this step.
The problem starts when I want to publish Azure WebApp with this image. The problem is with ... TAGS :) They are mismatching. I have automatic CI/CD - so when i push some change to the repo i see that release pipeline is working. It is creating the image in the ACR. Then i see, that release pipeline is running. Everything is "correct" - meaning no error are seen and the release is green.
But when i go to App service and Container settings i see from logs:
2020-04-21 18:02:28.321 INFO - Pulling image: myAcrName.azurecr.io/mobile/signalr:c7aead0c46b66afc4131935efc7e6a51280dfb1a
2020-04-21 18:02:28.761 ERROR - DockerApiException: Docker API responded with status code=NotFound, response={"message":"manifest for myAcrName.azurecr.io/mobile/signalr:c7aead0c46b66afc4131935efc7e6a51280dfb1a not found: manifest unknown: manifest unknown"}
2020-04-21 18:02:28.761 ERROR - Pulling docker image myAcrName.azurecr.io/mobile/signalr:c7aead0c46b66afc4131935efc7e6a51280dfb1a failed:
2020-04-21 18:02:28.762 INFO - Pulling image from Docker hub: myAcrName.azurecr.io/mobile/signalr:c7aead0c46b66afc4131935efc7e6a51280dfb1a
2020-04-21 18:02:28.867 ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://myAcrName.azurecr.io/v2/mobile/signalr/manifests/c7aead0c46b66afc4131935efc7e6a51280dfb1a: unauthorized: authentication required"}
2020-04-21 18:02:28.870 ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
Very sophisticated error but the root cause is, that he is trying to get the image with non-existing tag, which is GIT COMMIT tag. And it suppose to get image by $(Build.BuildId) (this was my first attempt) or by $(Build.BuilNumber) (this was my second attempt)
Here is how this pipeline step (Deploy Azure App Service) looks like:
- task: AzureRmWebAppDeployment#4
displayName: 'Deploy Azure App Service'
inputs:
azureSubscription: mySubcsriptionARM
appType: webAppContainer
WebAppName: myProductsignalr
DockerNamespace: myAcrName.azurecr.io
DockerRepository: mobile/signalr
DockerImageTag: '$(Build.BuildNumber)'
When i go to Release pipeline logs as a "Deploy Azure App Service" log i see that
2020-04-21T18:41:01.6012767Z ##[section]Starting: Deploy Azure App Service
2020-04-21T18:41:01.6367124Z ==============================================================================
2020-04-21T18:41:01.6367787Z Task : Azure App Service deploy
2020-04-21T18:41:01.6368381Z Description : Deploy to Azure App Service a web, mobile, or API app using Docker, Java, .NET, .NET Core, Node.js, PHP, Python, or Ruby
2020-04-21T18:41:01.6368765Z Version : 4.163.5
2020-04-21T18:41:01.6369158Z Author : Microsoft Corporation
2020-04-21T18:41:01.6369603Z Help : https://aka.ms/azureappservicetroubleshooting
2020-04-21T18:41:01.6369976Z ==============================================================================
2020-04-21T18:41:03.8970184Z Got service connection details for Azure App Service:'myProductsignalr'
2020-04-21T18:41:04.5534864Z Trying to update App Service Configuration settings. Data: {"appCommandLine":null,"linuxFxVersion":"DOCKER|myAcrName.azurecr.io/mobile/signalr:1f283100"}
2020-04-21T18:41:05.5465725Z Updated App Service Configuration settings.
2020-04-21T18:41:05.5495890Z Trying to update App Service Application settings. Data: {"DOCKER_CUSTOM_IMAGE_NAME":"myAcrName.azurecr.io/mobile/signalr:1f283100"}
2020-04-21T18:41:06.2703349Z Updated App Service Application settings and Kudu Application settings.
2020-04-21T18:41:32.4715682Z Updated App Service Application settings and Kudu Application settings.
2020-04-21T18:41:33.4179962Z Successfully updated deployment History at https://myProductsignalr.scm.azurewebsites.net/api/deployments/111587494492765
2020-04-21T18:41:33.5945654Z App Service Application URL: http://myProductsignalr.azurewebsites.net
2020-04-21T18:41:33.6180118Z ##[section]Finishing: Deploy Azure App Service
What amazes me, that it is showing, that everything was ok - when it was far from "ok" :)
When i go to container settings after:
a) new code is published
b) build pipeline fires
c) release pipeline fires
i see it like this:
The tag is empty. If i would pick some tag manually:
And would choose: "SAVE" everything works correctly (SingalR is up and running correctly)
Clearly, I am missing something :/ Help me to see what;)
The root cause for me is that this fragment:
DockerImageTag: '$(Build.BuildNumber)'
should insert build number (as stated) and the info from container settings should be:
Pulling image: myAcrName.azurecr.io/mobile/signalr:20200421.09 (for BuildNumber 20200421.09) and it is inserting GIT COMMIT there as a tag and ends up with: Pulling image: myAcrName.azurecr.io/mobile/signalr:c7aead0c46b66afc4131935efc7e6a51280dfb1a Why o why?:)
[UPDATE 22.04 10:56]
I am posting build pipeline that i am using currently. I don't suppose it is important as it is working correctly, and the problem is more with deployment of correctly created docker image (on ACR), than with creating this image by the build pipeline. Nevertheless, here is the pipeline:
# Docker
# Build a Docker image
# https://learn.microsoft.com/azure/devops/pipelines/languages/docker
trigger:
- master
resources:
- repo: self
variables:
dockerRegistryServiceConnection: 'MyProductDockerACR'
imageRepository: 'mobile/signalr'
containerRegistry: 'myAcrName.azurecr.io'
dockerfilePath: '**/Dockerfile'
tag: '$(Build.BuildNumber)'
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build and push stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: Docker#2
displayName: Build and push image to container registry
inputs:
containerRegistry: $(dockerRegistryServiceConnection)
repository: $(imageRepository)
command: 'buildAndPush'
Dockerfile: $(dockerfilePath)
tags: |
$(tag)
I saw the release you are using is configured by UI. It's work logic much different with the one which configured by YAML.
In fact, here what you received just be the different performance produced while the running reason of the release are different.
I guess this release has the artifact source which targeting to Repos, right? You can confirm by checking its icon.
While the release source is coming from Repos, then the Build.BuildNumber would be the short part of the commit id(8 characters). And the Build.BuildId is the complete commit id.
If you want the release keep using the Build.Buildnumber value which the corresponding build(created/pushed image) was using, you must make sure the release source is targeting to this build. Also, this build need has artifacts generated. According to the YAML you shared, obviously, you haven't done that.
Only the release triggered by build along with artifact, then the Build.BuildNumber can be like 20200422.1 which the build was using.
So, please go your release definition, and re-configure its source to make sure it is coming from build artifact instead of repository.
Yes. You are right. You have mismatch in tags.
In Docker#2 task you can define tags:
steps:
- task: Docker#2
displayName: Login to ACR
inputs:
command: login
containerRegistry: devopsmanual-acr
- task: Docker#2
displayName: Build and Push
inputs:
repository: $(imageName)
command: buildAndPush
Dockerfile: build-docker-image/SampleAppForDocker/DOCKERFILE
tags: |
$(Build.BuildNumber)
- task: Docker#2
displayName: Logout of ACR
inputs:
command: logout
containerRegistry: devopsmanual-acr
Your definition should be pretty much like this one. Whre devopsmanual-acr is connection to your ACR.
.
I recently mad a blog post about creating docker images on Azure DevOps so maybe this will be also helpful for you.
If this won't be enough to solve your issue, please edit your question and show how you create and push your images.
I try to build a CI/CD Pipeline with Azure Devops.
My goal is to
Build a docker Image an upload this to a private docker Respository in Dockerhub within the CI Pipeline
Deploy this image to an Azure Kubernetes Cluster within the CD Pipeline
The CI Pipeline works well:
The image is pushed successfully to dockerhub
The pipeline docker push task:
steps:
- task: Docker#1
displayName: 'Push an image'
inputs:
containerregistrytype: 'Container Registry'
dockerRegistryEndpoint: DockerHubConnection
command: 'Push an image'
imageName: 'jastechgmbh/microservice-demo:$(Build.BuildId)'
After that I trigger my release pipeline manually an it shows success as well
The apply pipeline task:
steps:
- task: Kubernetes#0
displayName: 'kubectl apply'
inputs:
kubernetesServiceConnection: MicroserviceTestClusterConnection
command: apply
useConfigurationFile: true
configuration: '$(System.DefaultWorkingDirectory)/_MicroservicePlayground-MavenCI/drop/deployment.azure.yaml'
containerRegistryType: 'Container Registry'
dockerRegistryConnection: DockerHubConnection
But when I check the deployment on my kubernetes dashboard an error message pops up:
Failed to pull image "jastechgmbh/microservice-demo:38": rpc error: code = Unknown desc = Error response from daemon: pull access denied for jastechgmbh/microservice-demo, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
I use the same dockerhub service connection in the CI & CD Pipeline.
I would be very happy about your help.
I believe this error indicates your kubernetes cluster doesnt have access to docker registry. You'd need to create docker secret for that. like so:
kubectl create secret generic regcred \
--from-file=.dockerconfigjson=<path/to/.docker/config.json> \
--type=kubernetes.io/dockerconfigjson
or from command line:
kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>
https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
Configure ACR integration for existing AKS clusters
az aks update -n myAKSClusterName -g myAKSResourceGroupName --attach-acr acr-name
https://learn.microsoft.com/en-us/azure/aks/cluster-container-registry-integration
Solved the issue for me
The answer above is correct, just need to add that you have to put imagePullsecrets on your deployment. Read the link provided on the other answer, it explain it in detail:
https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/