There is an intermittent issue while downloading docker images in a Linux VM. As per docker service journalctl logs I can see below error-
level=error msg="Handler for POST /images/create returned error: Get
"https://<containerregistry>": dial tcp: lookup
https://<containerregistry>: device or resource busy".
Initialy I thought it could be intermittent connectivity issue with container registry (Azure Container Registry) but that is not the case since even with retries at different time periods I got the same error.
I have checked the docker daemon.json config and don't see any issue with that. I also verified DNS servers in systemd-resolve and can see the expected IPs as required. Interestingly, another VM with same configs works fine.
I am using .Net Client for Docker Remote API (https://github.com/dotnet/Docker.DotNet) to trigger docker image updates. The client is hosted in a Linux VM which is triggered by via socket as an RPC call. It reaches the client and executes the API ImageOperation.CreateImageAsync.
Can someone help with this? Please let me know if I can add more details.
• Regarding the error that you are encountering, I would suggest you to please check the azure container registry health first by executing the below command: -
az acr check-health --name registryname --vnet virtualnetworkname
Then, check the internet upload and download speed through ‘AzureSpeed’ tool to the container registry which hosts the images layers. Also, it may be that the image size may not be supported for the registry service tier due to which it is not available.
• Ensure that both your ‘docker client’ and ‘docker daemon’ are configured for proxy behavior behind HTTPS proxy. If you have changed or updated any proxy settings, kindly ensure to restart the daemon service.
• Check the ‘Registry resource logs in the ContainerRegistryLoginEvents table’ for further diagnosis of an attempted connection. Also, check whether the registry is accessible over the internet and allow access to the public registry endpoints from all networks. If the container registry is configured for selected virtual network with a service endpoint, then disabling public network access also disables access over the service endpoint.
• If a private endpoint is configured for the container registry, confirm that the DNS resolves the registry's public FQDN such as myregistry.azurecr.io to the registry's private IP address. Use a network utility such as ‘dig’ or ‘nslookup’ for DNS lookup. Ensure that DNS records are configured for the registry FQDN and for each of the data endpoint FQDNs as per the below documentation link: -
https://learn.microsoft.com/en-us/azure/container-registry/container-registry-private-link#dns-configuration-options
• If a service endpoint to the registry is configured, confirm that a network rule is added to the registry that allows access from that network subnet. Also, ensure that the resource provider for Azure Container Registry is registered.
For more details regarding the further steps to take to perform regarding the error that you have encountered, kindly refer to the documentation link below: -
https://learn.microsoft.com/en-us/azure/container-registry/container-registry-troubleshoot-access#configure-service-access
Related
I am trying to run Sentiment analysis from Azure Cognitive Services using an official docker image. Azure Cognitive Service for Language resource is up and running in Azure, but the container can't reach the Billing endpoint.
The container is started as described in the documentation (with the correct billing and key):
docker run --rm -it -p 5010:5000 --name sentiment_cmd mcr.microsoft.com/azure-cognitive-services/textanalytics/sentiment Eula=accept Billing=https://RESOURCENAME.cognitiveservices.azure.com/ ApiKey=KEY
but in result I always get
Failed to reach billing endpoint: 'Resource temporarily unavailable'. Trying 9 more times.
fail: Microsoft.CloudAI.Containers.Http.CloudClient[0]
Failed to reach billing endpoint: 'Resource temporarily unavailable'. Trying 9 more times.
I tried changing the default IP subnet choosen by docker engine, but it didn't fix the issue. Any ideas?
As you are telling the Azure Cognitive Services for Language service is up and running. The problem is with the subnet. Need to check with whether billing endpoint is running with the subnet of the applications is running. Check whether the billing is in private or public subnet and also application is. Even though the service is running and if still it is not responding. Check with the network connections. Check with the network traffic also for the same issue. Reset or recheck the key before connecting.
We have private AKS cluster create under a virtual network (MC resource group). We also have an APIM crated under the same virtual network. A few services are deployed in K8 and exposed via internal loadbalancer of K8. when we do kubectl get svc get a cluster IP and an external IP.
I've used the external IP to connect to APIM. but its failing with the following error
Unable to download specified file. Please ensure the URL is valid and the file is publicly accessible.
The error of unable to download ... might be from the URL of the IP address, because when the field is filled with a azure blob storage url, no errors should come.
I think you can try to add api instance by uploading json file from localhost.
In the Network Security Groups, you should also add a new Inbound Security Rule.
Change the label in the Source service tag by selecting Service Tag from the dropdown menu that appears. There is an option for ApiManagement in this dropdown.
You must select an IP address from the drop-down list and type the API Management Service's IP address.
For more information to resolve this error, refer here and also this SO Thread.
I have a private network (10.0.0.0/16) which has multiple subnets. I have an Application Gateway configured in the 10.0.0.0/24 subnet and I would like to expose Container Instances from the 10.0.2.0/24 subnet to the internet via HTTP.
The gateway has only one HTTP rule simply mapping 80:80 from the internet to the backend pool. In the "Backend Health" section the status of my container (added via IP address) is unknown and the following error is displayed:
Unable to retrieve health status data. Check presence of NSG/UDR blocking access to ports 65503-65534 from Internet to Application Gateway.
Initially I had no NSG set but after setting one (and even allowing any:any) is not solving the status check problem.
When opening the url in the browser I get an 502 Bad Gateway error.
Note: I have added a container isntance which is not part of the VNet to the Application Gateway and despite also showing unknonw status in the Backend Pool the requests actually got redirected to the container, so I am almost certain I am misconfiguring someting in the VNet.
Note2: I have pushed a test container and the test container can reach the container I am trying to expose in the same subnet, so something goes wrong between the container and the application gateway.
My questions is what NSG rules I have to set to
make the container visible to the health check
make the redirect work between the Application Gateway and the Container
there is really nothing to misconfigure, you can create a vm in one of the subnets (create a third one) and test if it can access the container instance? if it can 0 this might be a platform limitation, tbh.
But looking at the error: you should detach all NSG\User Defined routes from the Vnet (for simplicity) and do several other checks: https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-backend-health-troubleshooting#backend-health-status-unknown
I have a Linux Virtual Machine (Debian 9) deployed in Azure with Service Endpoints for Sql enabled and properly added -if I navigate the portal towards the VNet and enter the Service Endpoints tab, I can clearly see the Sql Service Endpoint listed. Just FYI, the reason for the Service Endpoint is that the VM has a dynamic IP, so I can't just whitelist it in the DB resource's configuration.
On the other hand, I have an 'Azure Database for MySQL server' deployed in the same resource group, same location and whatnot, but I can't seem to connect to it.
The steps I take when I try to connect are as follow:
I connect to the VM through SSH.
In my VM I have mysql-server installed
I write mysql --host <fully qualified server name> --user <server admin login name>#<server name> -p
I get the following error: "ERROR 9002 (28000): Server is not ready for incoming connections."
I've been reading the documentation and searching in forums for a reason why this might be happening, but I simply cannot seem to make it work. I have tried changing the status of the "Allow access to Azure services" option in the Connection security tab of the DB resource, but it doesn't seem to matter.
Could anyone have any idea of how I might go about solving this??
You said you enabled the SQL endpoint on the virtual network, but did you add a VNET rule to the instance (Attach an existing VNET)? You can find this in Azure Database for MySQL server -> Connection Security -> VNET Rules -> Attach existing VNET.
If you can't see your VNET listed then there is a mismatch between the regions of your SQL server and your VNET: They must be deployed to the same one. Additionally check that you have a General Purpose or Memory Optimized server, this feature is not available in Basic tier.
If all of this is in place, try enabling Diagnostics on the SQL Server, try logging in again a few times, then view the log file and post anything strange.
I was following the process mention on the azure site to create a docker machine in azure.
The docker was able to create the necessary components in Azure but comes back with the message of
No connection could be made because the target machine actively refused it.
Building the docker image or simply trying to query it with docker images returns this message.
I suspect that the IP Address assigned is not made public although it is configured in Azure. .
This is probably not docker related but to the Azure configuration. How does one ensure the IPAddress here is accessible and connectable externally?