Expected behavior
Skaffold should pull the image from insecure Harbor registry running on HTTP. I have tried everything from these docs:
https://skaffold.dev/docs/environment/image-registries/#insecure-image-registries
but without success.
Actual behavior
Jib is pushing image to the insecure Harbor registry without a problem, but error is thrown when trying to pull the image and deploy microservice to Kubernetes:
192.168.2.24:30002/trm/redis-spring:latest#sha256:0f8d21819d845bd55aa699afa8b21e141d41f10d9d9fb1a2c6dbb2d468d89e81 can't be pulled.
Specified image can be pulled using docker:
docker pull 192.168.2.24:30002/trm/redis-spring:latest#sha256:0f8d21819d845bd55aa699afa8b21e141d41f10d9d9fb1a2c6dbb2d468d89e81
Information
Skaffold version: v1.35.1
Operating system: Windows 10 Home
Installed via: skaffold.dev
Contents of skaffold.yaml:
apiVersion: skaffold/v2beta25
kind: Config
metadata:
name: redis
build:
insecureRegistries:
- 192.168.2.24:30002/trm
- 192.168.2.24:30002/trm/redis-spring
- 192.168.2.24:30002/trm/redis-spring:latest#sha256:0f8d21819d845bd55aa699afa8b21e141d41f10d9d9fb1a2c6dbb2d468d89e81
artifacts:
- image: redis-spring
jib:
args:
- -Pjib
- -DsendCredentialsOverHttp=true
tagPolicy:
gitCommit: {}
deploy:
kubectl:
manifests:
- redis-spring-boot.yaml
time="2022-02-02T11:12:40+01:00" level=debug msg="marking resource failed due to error code STATUSCHECK_IMAGE_PULL_ERR" subtask=-1 task=Deploy
- mdm-dev:deployment/redis-spring-boot: container redis-spring is waiting to start: 192.168.2.24:30002/trm/redis-spring:latest#sha256:0f8d21819d845bd55aa699afa8b21e141d41f10d9d9fb1a2c6dbb2d468d89e81 can't be pulled
- mdm-dev:pod/redis-spring-boot-68ccfdc688-tj7pp: container redis-spring is waiting to start: 192.168.2.24:30002/trm/redis-spring:latest#sha256:0f8d21819d845bd55aa699afa8b21e141d41f10d9d9fb1a2c6dbb2d468d89e81 can't be pulled
- mdm-dev:deployment/redis-spring-boot failed. Error: container redis-spring is waiting to start: 192.168.2.24:30002/trm/redis-spring:latest#sha256:0f8d21819d845bd55aa699afa8b21e141d41f10d9d9fb1a2c6dbb2d468d89e81 can't be pulled.
time="2022-02-02T11:12:40+01:00" level=debug msg="setting skaffold deploy status to STATUSCHECK_IMAGE_PULL_ERR." subtask=-1 task=Deploy```
You need to configure a registry pull secret for your cluster, and then either annotate your pod-specs or your service account to use this registry pull secret.
Related
I have Azure Container Registry
xxxxxxxx.azurecr.io
Inside of it I have 1 image
learning-docker01/api
Next, I create App Service. I select Docker Container as publish option and following options:
Options: - Docker Compose (Preview)
Image Source - Azure Container Registry
Registry - xxxxxxxx
Next I select my docker-compose.yaml file. The configuration looks like this:
version: "3.4"
services:
docker001.api:
image: xxxxxxxx.azurecr.io/learning-docker01/api
build:
context: ./Docker001.API
environment:
- ASPNETCORE_ENVIRONMENT=Development
- ASPNETCORE_URLS=http://+:80
ports:
- "8000:80"
mongo:
image: mongo
environment:
- MONGO_INITDB_ROOT_USERNAME=root
- MONGO_INITDB_ROOT_PASSWORD=root
- MONGO_INITDB_DATABASE=docker
ports:
- 27017:27017
Then I hit Review & Create and everything deploys smoothly.
The problem - after deployment, if I view Log Stream (Preview) I see the following Errors:
DockerApiException: Docker API responded with status code=InternalServerError,
response={"message":"Get https://registry-1.docker.io/v2/library/mongo/manifests/latest:
unauthorized: incorrect username or password"}
----
Pulling docker image mongo failed:
---
Container for learningdocker001_mongo_0_986c7f1a site learningdocker001 is unhealthy, Stopping site.
I'm not sure what the problem is. Everything runs fine locally. On Azure docker001.api service runs but mongo service doesn't.
You need to set up authentication to allow the App Service to pull images from the ACR.
Recommended way is to use the Managed Identity. Detailed setup description here: https://learn.microsoft.com/en-us/azure/app-service/tutorial-custom-container?pivots=container-linux#configure-app-service-to-deploy-the-image-from-the-registry
Just study the core of K8S on local machine (Linux Mint 20.2).
Created one node cluster locally with:
k3d cluster create mycluster
And now I want to run spring boot application in a container.
I build local image:
library:0.1.0
And here is snippet from Deployment.yml:
spec:
terminationGracePeriodSeconds: 40
containers:
- name: 'library'
image: library:0.1.0
imagePullPolicy: IfNotPresent
Despite the fact that image is already built:
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
library 0.1.0 254c13416f46 About an hour ago 462MB
Starting the container fails:
pod/library-867dfb64db-vndtj Pulling image "library:0.1.0"
pod/library-867dfb64db-vndtj Failed to pull image "library:0.1.0": rpc error: code = Unknown desc = failed to pull and unpack image "library:0.1.0": failed to resolve reference "library:0.1.0": failed to do request: Head "https://...com/v2/library/manifests/0.1.0": x509: certificate signed by unknown authority
pod/library-867dfb64db-vndtj Error: ErrImagePull
pod/library-867dfb64db-vndtj Error: ImagePullBackOff
pod/library-867dfb64db-vndtj Back-off pulling image "library:0.1.0"
How to resolve local images visibility for k3d cluster?
Solution:
Update the Deployment.yml:
spec:
terminationGracePeriodSeconds: 40
containers:
- name: 'library-xp'
image: xpinjection/library:0.1.0
imagePullPolicy: Never
And import the image to cluster:
k3d image import xpinjection/library:0.1.0 -c mycluster
If you don't want to use a docker registry, you have to import the locally built image into the k3d cluster:
k3d image import [IMAGE | ARCHIVE [IMAGE | ARCHIVE...]] [flags]
But don't forget to configure in your deployment:
imagePullPolicy: Never
I have Harbor local docker registry and all needed images are there and connected GitLab to the Harbor and all the images are received from the Harbor but after November 2, Docker put a limit on the number of pulls and it seems dind service pulls from Docker hub.
Is it possible to use dind service to pull from Harbor?
Pipeline output:
Running with gitlab-runner 12.10.1 (ce065b93)
on docker_runner_7 WykGNjC6
Preparing the "docker" executor
30:20
Using Docker executor with image **harbor**.XXX.XXXX.net/library/docker_maven_jvm14 ...
Starting service docker:**dind** ...
**Pulling docker image docker:dind** ...
**ERROR**: Preparation failed: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (docker.go:198:2s)
Will be retried in 3s ...
Using Docker executor with image harbor.XXX.XXX.net/library/docker_maven_jvm14 ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
ERROR: Preparation failed: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (docker.go:198:4s)
Will be retried in 3s ...
Using Docker executor with image harbor.XXX.XXX.net/library/docker_maven_jvm14 ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
ERROR: Preparation failed: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (docker.go:198:3s)
Will be retried in 3s ...
ERROR: Job failed (system failure): Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (docker.go:198:3s)
Another way:
If you don't want to add daemon.json, you can do this:
Pull docker-dind from docker hub
docker pull docker:stable-dind
Login to harbor
docker login harbor.XXX.com
Tag image to harbor
docker tag docker:stable-dind harbor.XXX.com/library/docker:stable-dind
Push to harbor
docker push harbor.XXX.com/library/docker:stable-dind
Go to the .gitlab-ci.yml
Instead of
services:
- docker:dind
write:
services:
- name: harbor.XXX.com/library/docker:stable-dind
alias: docker
My .gitlab-ci.yml :
stages:
- build_and_push
Build:
image: ${DOCKER_REGISTRY}/library/docker:ci_tools
stage: build_and_push
tags:
- dind
services:
- name: ${DOCKER_REGISTRY}/library/docker:stable-dind
alias: docker
script:
- docker login -u $DOCKER_REGISTRY_USERNAME -p $DOCKER_REGISTRY_PASSWORD $DOCKER_REGISTRY
- make build test release REGISTRY=${DOCKER_REGISTRY}/library/ TELEGRAF_DOWNLOAD_URL="https://storage.XXX.com/ops/packages/telegraf-1.15.3_linux_amd64.tar.gz" TELEGRAF_SHA256="85a1ee372fb06921d09a345641bba5f3488d2db59a3fafa06f3f8c876523801d"
I can't found the solution for Gitlab but you can tell the docker to ignore the docker hub registry and go to the local registry.
Add daemon.json in /etc/docker/daemon.json
, if doesn't exist you can simply add in the path.
daemon.json
{
"registry-mirrors": ["https://harbor.XXX.com"]
}
sudo systemctl restart docker
I too faced the same issue while deploying some micro-services to kube cluster, here is a blog that I wrote that provides a workaround to optimize the deployment workflow: https://mailazy.com/blog/optimize-docker-pull-gitlab-pipelines/
Any idea why I keep getting this annoying and unhelpful error code/description?
Failed to pull image myapidemodocker.azurecr.io/apidemo:v4.0: rpc error: code = Unknown desc = unknown blob
I thought of incorrect secret and followed this documentation from Microsoft with no success! [https://learn.microsoft.com/en-us/azure/container-registry/container-registry-auth-aks][1].
Context:
I am using Visual Studio with Docker for Windows to create Windows
Container image.
Image is pushed to Azure Container Register (ACR) and Deployed as
Azure Container Instance. Unfortunately, I can't use ACI as
production application because it is not connected to a private vNET.
Can't use public IP for security reason but that's what is done just
for poc!
Next step, Created Kubernetes cluster in Azure and trying to deploy
the same image (Windows container) into Kubernetes POD but it is not
working.
Let me share my yml definition and event logs
Here is my yml definition:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: apidemo
spec:
template:
metadata:
labels:
app: apidemo
spec:
containers:
- name: apidemo
image: myapidemodocker.azurecr.io/apidemo:v4.0
imagePullSecrets:
- name: myapidemosecret
nodeSelector:
beta.kubernetes.io/os: windows
Event logs:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m default-scheduler Successfully assigned apidemo-57b5fc58fb-zxk86 to aks-agentp
ool-18170390-1
Normal SuccessfulMountVolume 4m kubelet, aks-agentpool-18170390-1 MountVolume.SetUp succeeded for volume "default-token-gsjhl"
Normal SandboxChanged 2m kubelet, aks-agentpool-18170390-1 Pod sandbox changed, it will be killed and re-created.
Normal Pulling 2m (x2 over 4m) kubelet, aks-agentpool-18170390-1 pulling image "apidemodocker.azurecr.io/apidemo:v4.0"
Warning Failed 20s (x2 over 2m) kubelet, aks-agentpool-18170390-1 Failed to pull image "apidemodocker.azurecr.io/apidemo:v4
.0": [rpc error: code = Unknown desc = unknown blob, rpc error: code = Unknown desc = unknown blob]
Warning Failed 20s (x2 over 2m) kubelet, aks-agentpool-18170390-1 Error: ErrImagePull
Normal BackOff 10s kubelet, aks-agentpool-18170390-1 Back-off pulling image "apidemodocker.azurecr.io/apidemo:
v4.0"
Warning Failed 10s kubelet, aks-agentpool-18170390-1 Error: ImagePullBackOff
(5) I don't understand why Kubernetes is still using /var/run/secrets/kubernetes.io/serviceaccount from default-token-gsjhl as secrete while I specified my own!
Thanks for taking time to provide feedback.
I was able to resolve the issue. It had nothing to do with error message! The actual problem was, I was trying to use Windows Container image and Kubernetes in Azure only support Linux Container images.
This are the actions I had to do:
Configured Ubuntu (Linux Container on Windows 10)
Configured Docker to use Linux (Switch to Linux Container).
Converted ASP.NET MVC project to ASP.NET Core using Visual Studio 2017. This was a big change to support multiple platforms including Linux.
Updated the dockerfile and docker-compose project.
Created new docker image (Linux Container).
Pushed the image to Azure Container Registry.
Created a new deployment in Kubernetes with same credential. It worked!
Created a new Service to expose the app in Kubernetes. This step created an endpoint that client can use.
My Kubernetes cluster is vNET joined and all IP's are private. So, I exposed the Kubernetes endpoint (service) via Azure API Gateway. Just for the sake of demo, I allowed anonymous access to API (API Key and jwt token are must for production app).
Here is the application flow: Client App -> Azure API Gateway -> Kubernetes Endpoint(private IP) -> Kubernetes PODs -> My Linux Container
There are lots of complexities and technology specifications are changing rapidly. So, it took me lots of reading to get it right! I am sure you can do it. Try my API from Azure Kubernetes Service here-
https://gdtapigateway.azure-api.net/containerdemo/aks/api/address/GetTop10Cities?StateProvince=Texas&CountryRegion=United%20States
https://gdtapigateway.azure-api.net/containerdemo/aks/api/address/GetAddressById?addressID=581
Here are some the configurations that I used for your information-
Dockerfile:
FROM microsoft/aspnetcore:2.0
ARG source
WORKDIR /app
ENV ASPNETCORE_URLS=http://+:80
EXPOSE 80
COPY ${source:-obj/Docker/publish} .
ENTRYPOINT ["dotnet", "gdt.api.demo.dotnetcore.dll"]
Docker-compose:
version: '3'
services:
gdt-api-demo:
image: gdt.api.demo.dotnetcore
build:
context: .\gdt.api.demo.dotnetcore
dockerfile: Dockerfile
Kubernetes Deployment Definition:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: gdtapidemo
spec:
template:
metadata:
labels:
app: gdtapidemo
spec:
containers:
- name: gdtapidemo
image: gdtapidemodocker.azurecr.io/gdtapidemo-ubuntu:v1.0
imagePullSecrets:
- name: gdtapidemosecret
Kubernetes Service Definition:
kind: Service
apiVersion: v1
metadata:
name: gdtapidemo-service
spec:
selector:
app: gdtapidemo-app
ports:
- protocol: TCP
port: 80
targetPort: 9200
Service as Deployed in Kubernetes
I'm using Ansible to provision my server with anything required to make a my website work. The goal is to install a base system and provide it with docker containers running apps (at the moment it's just one app).
The problem I'm facing is that my docker image isn't hosted at dockerhub or something else. Instead it's being built by an Ansible task. However, when I'm trying to run the built image, Ansible tries to pull it (which isn't possible) and then dies.
This is what the playbook section looks like:
- name: check or build image
docker_image:
path=/srv/svenv.nl-docker
name='svenv/svenv.nl'
state=build
- name: start svenv/svenv.nl container
docker:
name: svenv.nl
volumes:
- /srv/svenv.nl-docker/data/var/lib/mysql/:/var/lib/mysql/
- /srv/svenv.nl-docker/data/svenv.nl/svenv/media:/svenv.nl/svenv/media
ports:
- 80:80
- 3306:3306
image: svenv/svenv.nl
When I run this, a failure indicates that the svenv/svenv.nl get's pulled from the repository, it isn't there so it crashes:
failed: [vps02.svenv.nl] => {"changes": ["{\"status\":\"Pulling repository svenv/svenv.nl\"}\r\n", "{\"errorDetail\":{\"message\":\"Error: image svenv/svenv.nl:latest not found\"},\"error\":\"Error: image svenv/svenv.nl:latest not found\"}\r\n"], "failed": true, "status": ""}
msg: Unrecognized status from pull.
FATAL: all hosts have already failed -- aborting
My question is:
How can I
Build a local docker
Then start it as a container without pulling it
You are hitting this error:
https://github.com/ansible/ansible-modules-core/issues/1707
Ansible is attempting to create a container, but the create is failing with:
docker.errors.InvalidVersion: mem_limit has been moved to host_config in API version 1.19
Unfortunately, there is catch-all except: that is hiding this error. The result is that rather than failing with the above message, ansible assumes that the image is simply missing locally and attempts to pull it.
You can work around this by setting docker_api_version to something earlier than 1.19:
- name: start svenv/svenv.nl container
docker:
name: svenv.nl
ports:
- 80:80
- 3306:3306
image: svenv/svenv.nl
docker_api_version: 1.18