Just study the core of K8S on local machine (Linux Mint 20.2).
Created one node cluster locally with:
k3d cluster create mycluster
And now I want to run spring boot application in a container.
I build local image:
library:0.1.0
And here is snippet from Deployment.yml:
spec:
terminationGracePeriodSeconds: 40
containers:
- name: 'library'
image: library:0.1.0
imagePullPolicy: IfNotPresent
Despite the fact that image is already built:
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
library 0.1.0 254c13416f46 About an hour ago 462MB
Starting the container fails:
pod/library-867dfb64db-vndtj Pulling image "library:0.1.0"
pod/library-867dfb64db-vndtj Failed to pull image "library:0.1.0": rpc error: code = Unknown desc = failed to pull and unpack image "library:0.1.0": failed to resolve reference "library:0.1.0": failed to do request: Head "https://...com/v2/library/manifests/0.1.0": x509: certificate signed by unknown authority
pod/library-867dfb64db-vndtj Error: ErrImagePull
pod/library-867dfb64db-vndtj Error: ImagePullBackOff
pod/library-867dfb64db-vndtj Back-off pulling image "library:0.1.0"
How to resolve local images visibility for k3d cluster?
Solution:
Update the Deployment.yml:
spec:
terminationGracePeriodSeconds: 40
containers:
- name: 'library-xp'
image: xpinjection/library:0.1.0
imagePullPolicy: Never
And import the image to cluster:
k3d image import xpinjection/library:0.1.0 -c mycluster
If you don't want to use a docker registry, you have to import the locally built image into the k3d cluster:
k3d image import [IMAGE | ARCHIVE [IMAGE | ARCHIVE...]] [flags]
But don't forget to configure in your deployment:
imagePullPolicy: Never
Related
Expected behavior
Skaffold should pull the image from insecure Harbor registry running on HTTP. I have tried everything from these docs:
https://skaffold.dev/docs/environment/image-registries/#insecure-image-registries
but without success.
Actual behavior
Jib is pushing image to the insecure Harbor registry without a problem, but error is thrown when trying to pull the image and deploy microservice to Kubernetes:
192.168.2.24:30002/trm/redis-spring:latest#sha256:0f8d21819d845bd55aa699afa8b21e141d41f10d9d9fb1a2c6dbb2d468d89e81 can't be pulled.
Specified image can be pulled using docker:
docker pull 192.168.2.24:30002/trm/redis-spring:latest#sha256:0f8d21819d845bd55aa699afa8b21e141d41f10d9d9fb1a2c6dbb2d468d89e81
Information
Skaffold version: v1.35.1
Operating system: Windows 10 Home
Installed via: skaffold.dev
Contents of skaffold.yaml:
apiVersion: skaffold/v2beta25
kind: Config
metadata:
name: redis
build:
insecureRegistries:
- 192.168.2.24:30002/trm
- 192.168.2.24:30002/trm/redis-spring
- 192.168.2.24:30002/trm/redis-spring:latest#sha256:0f8d21819d845bd55aa699afa8b21e141d41f10d9d9fb1a2c6dbb2d468d89e81
artifacts:
- image: redis-spring
jib:
args:
- -Pjib
- -DsendCredentialsOverHttp=true
tagPolicy:
gitCommit: {}
deploy:
kubectl:
manifests:
- redis-spring-boot.yaml
time="2022-02-02T11:12:40+01:00" level=debug msg="marking resource failed due to error code STATUSCHECK_IMAGE_PULL_ERR" subtask=-1 task=Deploy
- mdm-dev:deployment/redis-spring-boot: container redis-spring is waiting to start: 192.168.2.24:30002/trm/redis-spring:latest#sha256:0f8d21819d845bd55aa699afa8b21e141d41f10d9d9fb1a2c6dbb2d468d89e81 can't be pulled
- mdm-dev:pod/redis-spring-boot-68ccfdc688-tj7pp: container redis-spring is waiting to start: 192.168.2.24:30002/trm/redis-spring:latest#sha256:0f8d21819d845bd55aa699afa8b21e141d41f10d9d9fb1a2c6dbb2d468d89e81 can't be pulled
- mdm-dev:deployment/redis-spring-boot failed. Error: container redis-spring is waiting to start: 192.168.2.24:30002/trm/redis-spring:latest#sha256:0f8d21819d845bd55aa699afa8b21e141d41f10d9d9fb1a2c6dbb2d468d89e81 can't be pulled.
time="2022-02-02T11:12:40+01:00" level=debug msg="setting skaffold deploy status to STATUSCHECK_IMAGE_PULL_ERR." subtask=-1 task=Deploy```
You need to configure a registry pull secret for your cluster, and then either annotate your pod-specs or your service account to use this registry pull secret.
I'm trying to deploy a docker compose file to Azure Container Instances (ACI) using public Dockerhub images with these two tutorials, docker, youtube. However, it keeps saying I can't pull a public dockerhub image
containerinstance.ContainerGroupsClient#CreateOrUpdate:
Failure sending request: StatusCode=400 -- Original Error: Code="MultipleErrorsOccurred"
Message="Multiple error occurred:
'BadRequest':'InaccessibleImage':'The image 'selenium/standalone-firefox:latest' in container group 'test_ui_automation' is not accessible. Please check the image and registry credential.
The process itself should be when an ACI instance gets setup so there's no register details existing and the image is public on docker hub.
I've logged into Azure and Docker Hub. Based on the youtube tutorial, it should just be a process of creating an ACI instance, then running docker compose -f azure-testproject-docker.yaml -d.
azure-testproject-docker.yaml
version: '3.1'
services:
testproject-agent:
image: testproject/agent:latest
container_name: testproject-agent
depends_on:
- chrome
- firefox
volumes:
- mydata:/var/testproject/agent
environment:
TP_AGENT_ALIAS: "MY DOCKER AGENT"
TP_API_KEY: "MY KEY"
TP_JOB_PARAMS: '"jobParameters" : { "browsers": [ "chrome", "firefox" ] }'
CHROME: "chrome:4444"
FIREFOX: "firefox:4444"
chrome:
image: selenium/standalone-chrome:latest
volumes:
- mydata:/dev/shm
firefox:
image: selenium/standalone-firefox:latest
volumes:
- mydata:/dev/shm
volumes:
mydata:
driver: azure_file
driver_opts:
share_name: myfileshare
storage_account_name: mystorageaccount
The docker-compose file is quite basic, just used TestProject's default docker compose file. The images download on local docker, but the azure part doesn't work.
The whole point was to deploy with only a Docker Compose file, no Dockerfile as it's just using public dockerhub images.
Has anyone experienced a similar issue?
Solution
Remove container_name as not supported in ACI
Turns out the issue was simply due to container_name not being supported.
As per comment by andriy-bilous
First you should look at supported ACI docker-compose features, as at least container_name is not supported. docs.docker.com/cloud/aci-compose-features – Andriy Bilous 17 hours ago
Resolved the issue just by removing the container_name line.
I have Harbor local docker registry and all needed images are there and connected GitLab to the Harbor and all the images are received from the Harbor but after November 2, Docker put a limit on the number of pulls and it seems dind service pulls from Docker hub.
Is it possible to use dind service to pull from Harbor?
Pipeline output:
Running with gitlab-runner 12.10.1 (ce065b93)
on docker_runner_7 WykGNjC6
Preparing the "docker" executor
30:20
Using Docker executor with image **harbor**.XXX.XXXX.net/library/docker_maven_jvm14 ...
Starting service docker:**dind** ...
**Pulling docker image docker:dind** ...
**ERROR**: Preparation failed: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (docker.go:198:2s)
Will be retried in 3s ...
Using Docker executor with image harbor.XXX.XXX.net/library/docker_maven_jvm14 ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
ERROR: Preparation failed: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (docker.go:198:4s)
Will be retried in 3s ...
Using Docker executor with image harbor.XXX.XXX.net/library/docker_maven_jvm14 ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
ERROR: Preparation failed: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (docker.go:198:3s)
Will be retried in 3s ...
ERROR: Job failed (system failure): Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit (docker.go:198:3s)
Another way:
If you don't want to add daemon.json, you can do this:
Pull docker-dind from docker hub
docker pull docker:stable-dind
Login to harbor
docker login harbor.XXX.com
Tag image to harbor
docker tag docker:stable-dind harbor.XXX.com/library/docker:stable-dind
Push to harbor
docker push harbor.XXX.com/library/docker:stable-dind
Go to the .gitlab-ci.yml
Instead of
services:
- docker:dind
write:
services:
- name: harbor.XXX.com/library/docker:stable-dind
alias: docker
My .gitlab-ci.yml :
stages:
- build_and_push
Build:
image: ${DOCKER_REGISTRY}/library/docker:ci_tools
stage: build_and_push
tags:
- dind
services:
- name: ${DOCKER_REGISTRY}/library/docker:stable-dind
alias: docker
script:
- docker login -u $DOCKER_REGISTRY_USERNAME -p $DOCKER_REGISTRY_PASSWORD $DOCKER_REGISTRY
- make build test release REGISTRY=${DOCKER_REGISTRY}/library/ TELEGRAF_DOWNLOAD_URL="https://storage.XXX.com/ops/packages/telegraf-1.15.3_linux_amd64.tar.gz" TELEGRAF_SHA256="85a1ee372fb06921d09a345641bba5f3488d2db59a3fafa06f3f8c876523801d"
I can't found the solution for Gitlab but you can tell the docker to ignore the docker hub registry and go to the local registry.
Add daemon.json in /etc/docker/daemon.json
, if doesn't exist you can simply add in the path.
daemon.json
{
"registry-mirrors": ["https://harbor.XXX.com"]
}
sudo systemctl restart docker
I too faced the same issue while deploying some micro-services to kube cluster, here is a blog that I wrote that provides a workaround to optimize the deployment workflow: https://mailazy.com/blog/optimize-docker-pull-gitlab-pipelines/
Any idea why I keep getting this annoying and unhelpful error code/description?
Failed to pull image myapidemodocker.azurecr.io/apidemo:v4.0: rpc error: code = Unknown desc = unknown blob
I thought of incorrect secret and followed this documentation from Microsoft with no success! [https://learn.microsoft.com/en-us/azure/container-registry/container-registry-auth-aks][1].
Context:
I am using Visual Studio with Docker for Windows to create Windows
Container image.
Image is pushed to Azure Container Register (ACR) and Deployed as
Azure Container Instance. Unfortunately, I can't use ACI as
production application because it is not connected to a private vNET.
Can't use public IP for security reason but that's what is done just
for poc!
Next step, Created Kubernetes cluster in Azure and trying to deploy
the same image (Windows container) into Kubernetes POD but it is not
working.
Let me share my yml definition and event logs
Here is my yml definition:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: apidemo
spec:
template:
metadata:
labels:
app: apidemo
spec:
containers:
- name: apidemo
image: myapidemodocker.azurecr.io/apidemo:v4.0
imagePullSecrets:
- name: myapidemosecret
nodeSelector:
beta.kubernetes.io/os: windows
Event logs:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m default-scheduler Successfully assigned apidemo-57b5fc58fb-zxk86 to aks-agentp
ool-18170390-1
Normal SuccessfulMountVolume 4m kubelet, aks-agentpool-18170390-1 MountVolume.SetUp succeeded for volume "default-token-gsjhl"
Normal SandboxChanged 2m kubelet, aks-agentpool-18170390-1 Pod sandbox changed, it will be killed and re-created.
Normal Pulling 2m (x2 over 4m) kubelet, aks-agentpool-18170390-1 pulling image "apidemodocker.azurecr.io/apidemo:v4.0"
Warning Failed 20s (x2 over 2m) kubelet, aks-agentpool-18170390-1 Failed to pull image "apidemodocker.azurecr.io/apidemo:v4
.0": [rpc error: code = Unknown desc = unknown blob, rpc error: code = Unknown desc = unknown blob]
Warning Failed 20s (x2 over 2m) kubelet, aks-agentpool-18170390-1 Error: ErrImagePull
Normal BackOff 10s kubelet, aks-agentpool-18170390-1 Back-off pulling image "apidemodocker.azurecr.io/apidemo:
v4.0"
Warning Failed 10s kubelet, aks-agentpool-18170390-1 Error: ImagePullBackOff
(5) I don't understand why Kubernetes is still using /var/run/secrets/kubernetes.io/serviceaccount from default-token-gsjhl as secrete while I specified my own!
Thanks for taking time to provide feedback.
I was able to resolve the issue. It had nothing to do with error message! The actual problem was, I was trying to use Windows Container image and Kubernetes in Azure only support Linux Container images.
This are the actions I had to do:
Configured Ubuntu (Linux Container on Windows 10)
Configured Docker to use Linux (Switch to Linux Container).
Converted ASP.NET MVC project to ASP.NET Core using Visual Studio 2017. This was a big change to support multiple platforms including Linux.
Updated the dockerfile and docker-compose project.
Created new docker image (Linux Container).
Pushed the image to Azure Container Registry.
Created a new deployment in Kubernetes with same credential. It worked!
Created a new Service to expose the app in Kubernetes. This step created an endpoint that client can use.
My Kubernetes cluster is vNET joined and all IP's are private. So, I exposed the Kubernetes endpoint (service) via Azure API Gateway. Just for the sake of demo, I allowed anonymous access to API (API Key and jwt token are must for production app).
Here is the application flow: Client App -> Azure API Gateway -> Kubernetes Endpoint(private IP) -> Kubernetes PODs -> My Linux Container
There are lots of complexities and technology specifications are changing rapidly. So, it took me lots of reading to get it right! I am sure you can do it. Try my API from Azure Kubernetes Service here-
https://gdtapigateway.azure-api.net/containerdemo/aks/api/address/GetTop10Cities?StateProvince=Texas&CountryRegion=United%20States
https://gdtapigateway.azure-api.net/containerdemo/aks/api/address/GetAddressById?addressID=581
Here are some the configurations that I used for your information-
Dockerfile:
FROM microsoft/aspnetcore:2.0
ARG source
WORKDIR /app
ENV ASPNETCORE_URLS=http://+:80
EXPOSE 80
COPY ${source:-obj/Docker/publish} .
ENTRYPOINT ["dotnet", "gdt.api.demo.dotnetcore.dll"]
Docker-compose:
version: '3'
services:
gdt-api-demo:
image: gdt.api.demo.dotnetcore
build:
context: .\gdt.api.demo.dotnetcore
dockerfile: Dockerfile
Kubernetes Deployment Definition:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: gdtapidemo
spec:
template:
metadata:
labels:
app: gdtapidemo
spec:
containers:
- name: gdtapidemo
image: gdtapidemodocker.azurecr.io/gdtapidemo-ubuntu:v1.0
imagePullSecrets:
- name: gdtapidemosecret
Kubernetes Service Definition:
kind: Service
apiVersion: v1
metadata:
name: gdtapidemo-service
spec:
selector:
app: gdtapidemo-app
ports:
- protocol: TCP
port: 80
targetPort: 9200
Service as Deployed in Kubernetes
I'm using Ansible to provision my server with anything required to make a my website work. The goal is to install a base system and provide it with docker containers running apps (at the moment it's just one app).
The problem I'm facing is that my docker image isn't hosted at dockerhub or something else. Instead it's being built by an Ansible task. However, when I'm trying to run the built image, Ansible tries to pull it (which isn't possible) and then dies.
This is what the playbook section looks like:
- name: check or build image
docker_image:
path=/srv/svenv.nl-docker
name='svenv/svenv.nl'
state=build
- name: start svenv/svenv.nl container
docker:
name: svenv.nl
volumes:
- /srv/svenv.nl-docker/data/var/lib/mysql/:/var/lib/mysql/
- /srv/svenv.nl-docker/data/svenv.nl/svenv/media:/svenv.nl/svenv/media
ports:
- 80:80
- 3306:3306
image: svenv/svenv.nl
When I run this, a failure indicates that the svenv/svenv.nl get's pulled from the repository, it isn't there so it crashes:
failed: [vps02.svenv.nl] => {"changes": ["{\"status\":\"Pulling repository svenv/svenv.nl\"}\r\n", "{\"errorDetail\":{\"message\":\"Error: image svenv/svenv.nl:latest not found\"},\"error\":\"Error: image svenv/svenv.nl:latest not found\"}\r\n"], "failed": true, "status": ""}
msg: Unrecognized status from pull.
FATAL: all hosts have already failed -- aborting
My question is:
How can I
Build a local docker
Then start it as a container without pulling it
You are hitting this error:
https://github.com/ansible/ansible-modules-core/issues/1707
Ansible is attempting to create a container, but the create is failing with:
docker.errors.InvalidVersion: mem_limit has been moved to host_config in API version 1.19
Unfortunately, there is catch-all except: that is hiding this error. The result is that rather than failing with the above message, ansible assumes that the image is simply missing locally and attempts to pull it.
You can work around this by setting docker_api_version to something earlier than 1.19:
- name: start svenv/svenv.nl container
docker:
name: svenv.nl
ports:
- 80:80
- 3306:3306
image: svenv/svenv.nl
docker_api_version: 1.18