I have a Linux App Service deployed within Azure, using an image from a private Azure Container Registry.
The service was up and running fine, and despite there being no changes to the image (no new pushes), the App Service was redeployed. I understand this to be an expected part of how Azure handles containers, swapping instances in and out with as it sees fit.
However, this image is no longer able to deploy successfully, here are the logs from Kudu (names changed):
2018-04-09 07:28:11.817 INFO - Starting container for site
2018-04-09 07:28:11.818 INFO - docker run -d -p 36165:8080 --name my-service_0 -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITES_PORT=8080 -e WEBSITE_SITE_NAME=my-service -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_INSTANCE_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX -e HTTP_LOGGING_ENABLED=1 myacr.azurecr.io/images/my-image:latest
2018-04-09 07:28:14.200 INFO - Issuing docker pull myacr.azurecr.io/images/my-image:latest
2018-04-09 07:28:14.327 ERROR - docker pull returned STDERR>> Error response from daemon: Get https://myacr.azurecr.io/v2/images/my-image/manifests/latest: unauthorized: authentication required
Here are the results of pulling the image locally:
docker pull myacr.azurecr.io/images/my-image:latest
latest: Pulling from images/my-image
53478ce18e19: Already exists
d1c225ed7c34: Already exists
c6724ba0c09a: Already exists
377ae380bb14: Pulling fs layer
ab6e212b3598: Pulling fs layer
377ae380bb14: Downloading
ab6e212b3598: Verifying Checksum
ab6e212b3598: Download complete
unknown blob
I'm fairly sure that the image was successfully deployed 3 days ago and that there has been no pushes to the tag since then.
Is there anything I can do to recover this image?
Is there any way I could have known this deployment had started failing?
It seems your Web App for Containers instance is unable to authenticate with Azure Container Registry.
The Azure Container Registry to Linux Web App connection is a bit problematic at times. Using the "private registry" tab on the container settings page instead of ACR usually solves the problem for me.
Just provide your ACR repo credentials as below instead of selecting your repo on the ACR tab.
Related
This question already has answers here:
How to authenticate with Azure ACR from Azure container app service
(4 answers)
How to configure an Azure app service to pull images from an ACR with terraform?
(2 answers)
Azure Container Registry authorization for Azure Web App
(1 answer)
Closed 1 year ago.
I'm deploying an App Service in Azure that has a related container image. I created both the App Service and the image with the Owner user (the only one I have). Still, when I start the App Service I get an error that the "image is not authorized" (see below). How to fix this problem?
The image is created and stored in the registry with no issues.
2021-07-26T15:03:31.157Z ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2021-07-26T15:03:37.000Z INFO - Stopping site sofiana because it failed during startup.
2021-07-26T15:06:32.588Z INFO - Starting container for site
2021-07-26T15:06:32.590Z INFO - docker run -d -p 8081:8081 --name sofiana_1_e0d93acf_msiProxy -e PORT=8081 -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=sofiana.azurewebsites.net -e WEBSITE_INSTANCE_ID=4c49e5ee1c9d26e3d207f81f51be02bb2c4d67839b9f7b25c24fceef7fd9b44e appsvc/msitokenservice:2007200210
2021-07-26T15:06:32.592Z INFO - Logging is not enabled for this container.
Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2021-07-26T15:06:43.616Z INFO - Pulling image from Docker hub: sofacr.azurecr.io/nginx-visualbank:latest
2021-07-26T15:06:43.697Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://sofacr.azurecr.io/v2/nginx-visualbank/manifests/latest: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information."}
2021-07-26T15:06:43.700Z ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2021-07-26T15:06:49.634Z INFO - Stopping site sofiana because it failed during startup.
You need to auth against the repository
az acr login --name sofacr --subscription AZURE-SUB
I have an Azure release pipeline that uses an Azure Web App for Containers task to deploy a docker image on an Azure App Service.
The image is specified in the form of some_image:$(Build.BuildId). The pipeline works as intended and successfully updates the App Service with the latest built of the image.
I want from an other release pipeline to execute a docker run command using that image. I've noticed that version 1 of the Docker task allows me to execute such a docker run command on a docker image (no idea why run is missing from version 2), but how can I specify the docker image? How can I get which image is the currently deployed on that App Service?
You can either use PowerShell or Shell script in the YAML pipeline. Since you already know the container registry and the image name, just use the below command to get the latest version
az acr repository show-tags -n MyRegistry --repository MyRepository --top 1 --orderby time_desc --detail
https://learn.microsoft.com/en-us/cli/azure/acr/repository?view=azure-cli-latest#az_acr_repository_show_tags
Might be too late now, but what you want to do is to get the value of LinuxFXVersion (if you're running docker on Linux) property from Azure Resource Explorer.
Using a combination of Azure PowerShell and CLI, you can have these commands to retrieve the current image running on your web app:
$webAppProperties = (az webapp config show --subscription "<subscription-id>" --resource-group "<resource-group-name>" -n "<webapp-name>") | ConvertFrom-Json
$webAppProperties.linuxFXVersion
Assuming you have the right permissions to your subscription from Azure Pipelines, you should be able to use this information for the next steps.
We have a SpringBoot app which requires a keystore file located at "/secrets/app.keystore.jks" to run.
We want to run the app in a container on a Azure App Service Linux instance. And For security reasons we don't want to include the "/secrets/app.keystore.jks" file in the container itself. Instead, we have managed to upload the file to the "/home/site/wwwroot/secrets/" folder on the app service.
And we use the following command to start up the container on the app service
docker run -d myacr.azurecr.io/myAPp:latest -p 80:80 --name myApp
-e WEBSITES_ENABLE_APP_SERVICE_STORAGE=TRUE -v /home/site/wwwroot/secrets:/secrets
In the app service's log, we have the error:
java.lang.IllegalStateException: java.io.IOException: Could not open
/secrets/app.keystore.jks as a file, class path resource, or URL.
It looks to me the volume was not set up and the app cannot access to the file "/secrets/app.keystore.jks"
Does anyone know how to setup a volume so the app in the container can access a file on the host?
There are two ways to achieve your purpose. One is set the environment variable WEBSITES_ENABLE_APP_SERVICE_STORAGE as true and you can mount the persistent volume to your container like belowin the docker-compose file:
volumes:
- ${WEBAPP_STORAGE_HOME}/site/wwwroot/secrets:/secrets
Get more details here.
Another way is that you can mount the Azure Storage to your container and upload the files to the storage. Follow the steps here.
This is what happens to run the container. I wonder if there is a way to start the web app for containers with a custom DNS.
I have 5 microservices in my ILB-ASE
they need to be able to call each other using my custom DNS server in the VNet. When I check the resolv.conf i see 127.0.0.11. I need that to be set to my own custom dns server.
how can we inject my custom DNS value here?
Should we use the appsettings if so what are the values in the web app for containers?
So I can use the --dns option
The mystery part that Azure runs it. Some values are coming up from the appsettings.
2018-08-23 14:12:56.100 INFO - docker run -d -p 13940:5001 --name xxx
-e DOCKER_CUSTOM_IMAGE_NAME=xxx.azurecr.io/xxx:558 -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITES_PORT=5001 -e
WEBSITE_SITE_NAME=xxx -e WEBSITE_AUTH_ENABLED=False -e
WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_INSTANCE_ID=xxx -e
HTTP_LOGGING_ENABLED=1 xxx.azurecr.io/xxx:558
=====DOCKER LOG=========
2018_08_23_RD0003FF2D0408_default_docker.log:
2018-08-23T14:12:49.755843301Z [40m[1m[33mwarn[39m[22m[49m:
Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
2018-08-23T14:12:49.755897801Z No XML encryptor configured. Key
{xxx-xxx-xxx-xxx-xxx} may be persisted to storage in unencrypted form.
2018-08-23T14:12:54.761216323Z [40m[1m[33mwarn[39m[22m[49m:
Microsoft.AspNetCore.Server.Kestrel[0]
2018-08-23T14:12:54.761251623Z Overriding address(es) 'http://+:80'.
Binding to endpoints defined in UseKestrel() instead.
2018-08-23T14:12:54.908189021Z Hosting environment: Production
2018-08-23T14:12:54.908386123Z Content root path: /app
2018-08-23T14:12:54.908961927Z Now listening on: http://0.0.0.0:5001
2018-08-23T14:12:54.909256229Z Application started. Press Ctrl+C to
shut down.
2018_08_23_RD0003FF2D0408_docker.log:
2018-08-23 14:12:44.125 INFO - Recycling container because of
AppFrameworkVersionChange and appFrameworkVersion = xxx.xxx.io/xxx:558
2018-08-23 14:12:45.900 INFO - Starting container for site
2018-08-23 14:12:45.900 INFO - docker run -d -p 30464:5001 --name xxx
-e DOCKER_CUSTOM_IMAGE_NAME=xxx.azurecr.io/xxx:549 -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITES_PORT=5001 -e
WEBSITE_SITE_NAME=xxx -e WEBSITE_AUTH_ENABLED=False -e
WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_INSTANCE_ID=xxx -e
HTTP_LOGGING_ENABLED=1 xxx.xxx.io/xxx:558
2018-08-23 14:12:55.972 INFO - Container xxx for site xxx initialized
successfully.
2018-08-23 14:12:55.976 INFO - Recycling container because of
AppSettingsChange and isMainSite = True
2018-08-23 14:12:56.099 INFO - Starting container for site
2018-08-23 14:12:56.100 INFO - docker run -d -p 13940:5001 --name xxx
-e DOCKER_CUSTOM_IMAGE_NAME=xxx.azurecr.io/xxx:558 -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITES_PORT=5001 -e
WEBSITE_SITE_NAME=xxx -e WEBSITE_AUTH_ENABLED=False -e
WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_INSTANCE_ID=xxx -e
HTTP_LOGGING_ENABLED=1 xxx.azurecr.io/xxx:558
2018-08-23 14:13:05.385 INFO - Container xxx for site xxx initialized
successfully.
we responded to your question on Github and Reddit. Re-posting our response here for visibility.
"Currently, there is a workaround for this: you should modify the default resolv.conf to the custom DNS IP and then add your custom resolv.conf on docker build by adding a COPY command in your entrypoint script and pointing a custom resolv.conf to /etc.
However, we are investigating a better solution for this, so that manually updating the resolv.conf wouldn’t be necessary, so stay tuned."
You shouldn't use DNS to communicate with microservices, instead, you should make use of service registry.
Check this Microsoft paper talking about this:
Each microservice has a unique name (URL) that is used to resolve its
location. Your microservice needs to be addressable wherever it is
running. If you have to think about which computer is running a
particular microservice, things can go bad quickly. In the same way
that DNS resolves a URL to a particular computer, your microservice
needs to have a unique name so that its current location is
discoverable. Microservices need addressable names that make them
independent from the infrastructure that they are running on. This
implies that there is an interaction between how your service is
deployed and how it is discovered, because there needs to be a service
registry. In the same vein, when a computer fails, the registry
service must be able to indicate where the service is now running.
As you can see, the best solution will depend on your deployment model. Check this note about containers:
In some microservice deployment environments (called clusters, to be
covered in a later section), service discovery is built-in. For
example, within an Azure Container Service environment, Kubernetes and
DC/OS with Marathon can handle service instance registration and
deregistration. They also run a proxy on each cluster host that plays
the role of server-side discovery router. Another example is Azure
Service Fabric, which also provides a service registry through its
out-of-the-box Naming Service.
Hope it helps!
I used following quick start doc to spin up my first Azure container.
https://learn.microsoft.com/en-us/azure/container-instances/container-instances-quickstart#feedback
It worked fine. but how do I connect to container if I want to debug something?
You cannot connect to the container itself directly to debug, IE you can't SSH or RDP to it. Take a look at this graphic which highlights how a container differs from virtual machines:
You can however pull logs from your container from the container engine. In your case you would want to use the following command in the Azure CLI: az container logs.
https://aka.ms/container_logs
When you invoke CLI through the Portal, you should already be connected through your subscription.To debug or troubleshoot you can look at the container logs. Check out this documentation for the exact commands
https://learn.microsoft.com/en-us/cli/azure/container?view=azure-cli-latest#az-container-logs
When I am building containers to run on ACI, I build them first in a local docker instance where they can be connected to and interactively debugged. When you're happy with how they run locally push them into ACI, and debug from the output logs if needed.
I get to the bash shell in my Azure containers by either the azure-cli package, as the OP noted in a comment:
az container exec --exec-command "/bin/bash"
Or by navigating to a container instance in the Azure portal, then under Settings/Containers there is a "Connect" tab: