Failed to pull image myapidemodocker.azurecr.io/apidemo:v4.0: rpc error: code = Unknown desc = unknown blob - azure

Question

Any idea why I keep getting this annoying and unhelpful error code/description?
Failed to pull image myapidemodocker.azurecr.io/apidemo:v4.0: rpc error: code = Unknown desc = unknown blob
I thought of incorrect secret and followed this documentation from Microsoft with no success! [https://learn.microsoft.com/en-us/azure/container-registry/container-registry-auth-aks][1].
Context:
I am using Visual Studio with Docker for Windows to create Windows
Container image.
Image is pushed to Azure Container Register (ACR) and Deployed as
Azure Container Instance. Unfortunately, I can't use ACI as
production application because it is not connected to a private vNET.
Can't use public IP for security reason but that's what is done just
for poc!
Next step, Created Kubernetes cluster in Azure and trying to deploy
the same image (Windows container) into Kubernetes POD but it is not
working.
Let me share my yml definition and event logs
Here is my yml definition:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: apidemo
spec:
template:
metadata:
labels:
app: apidemo
spec:
containers:
- name: apidemo
image: myapidemodocker.azurecr.io/apidemo:v4.0
imagePullSecrets:
- name: myapidemosecret
nodeSelector:
beta.kubernetes.io/os: windows
Event logs:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m default-scheduler Successfully assigned apidemo-57b5fc58fb-zxk86 to aks-agentp
ool-18170390-1
Normal SuccessfulMountVolume 4m kubelet, aks-agentpool-18170390-1 MountVolume.SetUp succeeded for volume "default-token-gsjhl"
Normal SandboxChanged 2m kubelet, aks-agentpool-18170390-1 Pod sandbox changed, it will be killed and re-created.
Normal Pulling 2m (x2 over 4m) kubelet, aks-agentpool-18170390-1 pulling image "apidemodocker.azurecr.io/apidemo:v4.0"
Warning Failed 20s (x2 over 2m) kubelet, aks-agentpool-18170390-1 Failed to pull image "apidemodocker.azurecr.io/apidemo:v4
.0": [rpc error: code = Unknown desc = unknown blob, rpc error: code = Unknown desc = unknown blob]
Warning Failed 20s (x2 over 2m) kubelet, aks-agentpool-18170390-1 Error: ErrImagePull
Normal BackOff 10s kubelet, aks-agentpool-18170390-1 Back-off pulling image "apidemodocker.azurecr.io/apidemo:
v4.0"
Warning Failed 10s kubelet, aks-agentpool-18170390-1 Error: ImagePullBackOff
(5) I don't understand why Kubernetes is still using /var/run/secrets/kubernetes.io/serviceaccount from default-token-gsjhl as secrete while I specified my own!
Thanks for taking time to provide feedback.

Answer 1

I was able to resolve the issue. It had nothing to do with error message! The actual problem was, I was trying to use Windows Container image and Kubernetes in Azure only support Linux Container images.
This are the actions I had to do:
Configured Ubuntu (Linux Container on Windows 10)
Configured Docker to use Linux (Switch to Linux Container).
Converted ASP.NET MVC project to ASP.NET Core using Visual Studio 2017. This was a big change to support multiple platforms including Linux.
Updated the dockerfile and docker-compose project.
Created new docker image (Linux Container).
Pushed the image to Azure Container Registry.
Created a new deployment in Kubernetes with same credential. It worked!
Created a new Service to expose the app in Kubernetes. This step created an endpoint that client can use.
My Kubernetes cluster is vNET joined and all IP's are private. So, I exposed the Kubernetes endpoint (service) via Azure API Gateway. Just for the sake of demo, I allowed anonymous access to API (API Key and jwt token are must for production app).
Here is the application flow: Client App -> Azure API Gateway -> Kubernetes Endpoint(private IP) -> Kubernetes PODs -> My Linux Container
There are lots of complexities and technology specifications are changing rapidly. So, it took me lots of reading to get it right! I am sure you can do it. Try my API from Azure Kubernetes Service here-
https://gdtapigateway.azure-api.net/containerdemo/aks/api/address/GetTop10Cities?StateProvince=Texas&CountryRegion=United%20States
https://gdtapigateway.azure-api.net/containerdemo/aks/api/address/GetAddressById?addressID=581
Here are some the configurations that I used for your information-
Dockerfile:
FROM microsoft/aspnetcore:2.0
ARG source
WORKDIR /app
ENV ASPNETCORE_URLS=http://+:80
EXPOSE 80
COPY ${source:-obj/Docker/publish} .
ENTRYPOINT ["dotnet", "gdt.api.demo.dotnetcore.dll"]
Docker-compose:
version: '3'
services:
gdt-api-demo:
image: gdt.api.demo.dotnetcore
build:
context: .\gdt.api.demo.dotnetcore
dockerfile: Dockerfile
Kubernetes Deployment Definition:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: gdtapidemo
spec:
template:
metadata:
labels:
app: gdtapidemo
spec:
containers:
- name: gdtapidemo
image: gdtapidemodocker.azurecr.io/gdtapidemo-ubuntu:v1.0
imagePullSecrets:
- name: gdtapidemosecret
Kubernetes Service Definition:
kind: Service
apiVersion: v1
metadata:
name: gdtapidemo-service
spec:
selector:
app: gdtapidemo-app
ports:
- protocol: TCP
port: 80
targetPort: 9200
Service as Deployed in Kubernetes