Docker container always shows ssl connection error - python-3.x

I'm getting started with docker compose and have been working through the simple demo flask application. The thing is, I'm running this from inside of an organization that intercepts all communication in such a way that SSL errors are thrown right and left. They provide us with three root certificates we need to install, and I've generally got these working on my own machine, but I'm lost when it comes to getting these to work inside docker-compose deployments.
When I run docker-compose up, I get the following:
$ sudo docker-compose up
Creating network "project_default" with the default driver
Building web
Step 1/5 : FROM python:3.4-alpine
3.4-alpine: Pulling from library/python
81033e7c1d6a: Pull complete
9b61101706a6: Pull complete
415e2a07c89b: Pull complete
f22df7a3f000: Pull complete
8c16bf19c1f9: Pull complete
Digest: sha256:fe436cb066394d81cf49448a04dec7c765082445a500bc44f1ae5e8a455793bd
Status: Downloaded newer image for python:3.4-alpine
---> 5c72717ec319
Step 2/5 : ADD . /code
---> a5790c0e3e94
Removing intermediate container 052c614e41d0
Step 3/5 : WORKDIR /code
---> a2ea9acb3005
Removing intermediate container 77f2375ca0a6
Step 4/5 : RUN pip install -r requirements.txt
---> Running in 5f4fe856776d
Collecting flask (from -r requirements.txt (line 1))
Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb0061f1d30>: Failed to establish a new connection: [Errno -3] Try again',)': /simple/flask/
Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb0061f19b0>: Failed to establish a new connection: [Errno -3] Try again',)': /simple/flask/
Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb0061f1828>: Failed to establish a new connection: [Errno -3] Try again',)': /simple/flask/
Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb0061f1588>: Failed to establish a new connection: [Errno -3] Try again',)': /simple/flask/
Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb0061f1390>: Failed to establish a new connection: [Errno -3] Try again',)': /simple/flask/
Could not find a version that satisfies the requirement flask (from -r requirements.txt (line 1)) (from versions: )
No matching distribution found for flask (from -r requirements.txt (line 1))
Pip fails to install anything.
The docker-compose.yml file looks like this:
version: '3'
services:
web:
build: .
ports:
- "5000:5000"
redis:
image: "redis:alpine"
And the main Dockerfile looks like this:
FROM python:3.4-alpine
ADD . /code
WORKDIR /code
RUN pip install -r requirements.txt
CMD ["python", "app.py"]
Is there any way to be able to make this work in this particular case? Is there a general solution to this sort of problem that would allow me to pass to any container deployed the SSL certificates and have them be used?

In my case, I must add in my Dockerfile these sentences:
COPY company.crt /usr/local/share/ca-certificates/company.crt
RUN update-ca-certificates
...
RUN pip install --cert /etc/ssl/certs/company.pem -r requirements.txt
You need the certificate of your company in .crt format. When docker execute update-ca-certificates, linux will create a .pem file with the same name in the path: /etc/ssl/certs/. It will convert the network between pip in SSL.

This isn't really a docker-specific question: you are asking, in effect, "how do I install certificate authorities under Linux"? The answer is going to be the same regardless of whether you are running your ssl client inside or outside of a container.
Your Python image is based on alpine, and alpine uses the "ca-certificates" package to manage CA certificiates. To install your local CA certificates, you would need to (a) copy them into the /usr/share/ca-certificates directory and (b) run update-ca-certificates.
For example, adding something like this to your Dockerfile (before your pip install):
COPY company-ca.crt /usr/share/ca-certificates
RUN update-ca-certificates

In my case, Host machine's MTU is 1450, and Docker's MTU is 1500.
Which causes docker set MSS to 1460, and then TLS "server hello" packet got bigger than 1450 bytes, so the Host machine discard it.
To see if it's your case too, run ifconfig on both you Docker container and your host machine. If Host's MTU is less than 1500, it's easy to run into this discarding packet situation. Especially in HTTPS, for that "server hello" needs to send the certificate, which is a big packet

I was trying to read data from an API in my Go code and I was facing similar ssl error:
x509: certificate signed by unknown authority
My container was based on debian:stretch which is really really small ~100MB.
This happens when ca-certificates are not installed.
I installed ca-certificates (which also installs openssl) as shown below:
FROM debian:stretch
RUN apt-get update && apt-get install -y ca-certificates --no-install-recommends && rm -rf /var/lib/apt/lists/*
# ...
This will also work for any other base images which might not have ca-certificates installed.

Related

Unable to push image to OpenShift internal registry with i/o timeout

Pushing image docker-registry.default.svc:5000/th/th:source ...
Warning: Push failed, retrying in 5s ...
Warning: Push failed, retrying in 5s ...
Warning: Push failed, retrying in 5s ...
Warning: Push failed, retrying in 5s ...
Warning: Push failed, retrying in 5s ...
Warning: Push failed, retrying in 5s ...
Warning: Push failed, retrying in 5s ...
Registry server Address:
Registry server User Name: serviceaccount
Registry server Email: serviceaccount#example.org
Registry server Password: <<non-empty>>
error: build error: Failed to push image: After retrying 6 times, Push image still failed due to error: Get https://docker-registry.default.svc:5000/v1/_ping: dial TCP<ip>:5000: i/o timeout
Manually pushing an image from the CLI to the internal registry is working fine.
I have deployed the OpenShift instance 3.11 on a couple of azure VMs, while deploying I took care of adding external IP to the same.
All other images are also present in the docker registry and the curl command to the docker registry returns with exit code 0
What seemed curious was while deploying my app I tried pinging the registry from the build pods terminal. This resulted in the connection being hung up and no response.
Any ideas on how to fix this?
The sdn was causing this networking issue.
Does Azure support Calico networking?
Calico in VXLAN mode is supported on Azure. However, IPIP packets are
blocked by the Azure network fabric.
The above quote from calico reference was the reason this issue was caused. This could be resolved by changing to VXLAN mode in calico config. More details on how to switch can be found here.
For my solution I just switched to the default openshift sdn 'ovs-subnet' from calico in the inventory file.

Cannot access cdn.redhat.com from my Linux VM. please find the attached screenshot and provide me the solution

[root#localhost ~]# yum install ksh
Updating Subscription Management repositories.
Last metadata expiration check: 22:24:52 ago on Tue 25 May 2021 01:12:00 AM EDT.
Dependencies resolved.
Package Architecture Version Repository Size
Installing:
ksh x86_64 20120801-254.el8 rhel-8-for-x86_64-appstream-rpms 926 k
Transaction Summary
Install 1 Package
Total download size: 926 k
Installed size: 3.2 M
Is this ok [y/N]: y
Downloading Packages:
[MIRROR] ksh-20120801-254.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for https://cdn.redhat.com/content/dist/rhel8/8/x86_64/appstream/os/Packages/k/ksh-20120801-254.el8.x86_64.rpm [Failed to connect to cdn.redhat.com port 443: No route to host]
[MIRROR] ksh-20120801-254.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for https://cdn.redhat.com/content/dist/rhel8/8/x86_64/appstream/os/Packages/k/ksh-20120801-254.el8.x86_64.rpm [Failed to connect to cdn.redhat.com port 443: No route to host]
[MIRROR] ksh-20120801-254.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for https://cdn.redhat.com/content/dist/rhel8/8/x86_64/appstream/os/Packages/k/ksh-20120801-254.el8.x86_64.rpm [Failed to connect to cdn.redhat.com port 443: No route to host]
[MIRROR] ksh-20120801-254.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for https://cdn.redhat.com/content/dist/rhel8/8/x86_64/appstream/os/Packages/k/ksh-20120801-254.el8.x86_64.rpm [Failed to connect to cdn.redhat.com port 443: No route to host]
[FAILED] ksh-20120801-254.el8.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: Error downloading packages:
Cannot download Packages/k/ksh-20120801-254.el8.x86_64.rpm: All mirrors were tried

Problem executing "minikube start" command

malik#malik:~$ minikube start
😄 minikube v1.12.0 on Ubuntu 18.04
✨ Using the docker driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🎉 minikube 1.12.1 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.12.1
💡 To disable this notice, run: 'minikube config set WantUpdateNotification false'
🚜 Pulling base image ...
💾 Downloading Kubernetes v1.18.3 preload ...
E0727 07:25:35.757871 14015 cache.go:63] save image to file "k8s.gcr.io/kube-apiserver:v1.18.3" -> "/home/malik/.minikube/cache/images/k8s.gcr.io/kube-apiserver_v1.18.3" failed: write: Get https://k8s.gcr.io/v2/kube-apiserver/blobs/sha256:83b4483280e5187b2801b449338d5755e5874ab80c44bf1ce615d258142e7c8b: dial tcp: lookup k8s.gcr.io: no such host
E0727 07:25:35.757643 14015 cache.go:63] save image to file "k8s.gcr.io/coredns:1.6.7" -> "/home/malik/.minikube/cache/images/k8s.gcr.io/coredns_1.6.7" failed: write: Get https://k8s.gcr.io/v2/coredns/blobs/sha256:c6568d217a0023041ef9f729e8836b19f863bcdb612bb3a329ebc165539f5a80: dial tcp: lookup k8s.gcr.io: no such host
E0727 07:25:35.757512 14015 cache.go:63] save image to file "k8s.gcr.io/kube-scheduler:v1.18.3" -> "/home/malik/.minikube/cache/images/k8s.gcr.io/kube-scheduler_v1.18.3" failed: write: Get https://k8s.gcr.io/v2/kube-scheduler/blobs/sha256:83b4483280e5187b2801b449338d5755e5874ab80c44bf1ce615d258142e7c8b: dial tcp: lookup k8s.gcr.io: no such host
E0727 07:26:22.529729 14015 cache.go:63] save image to file "kubernetesui/dashboard:v2.0.1" -> "/home/malik/.minikube/cache/images/kubernetesui/dashboard_v2.0.1" failed: nil image for kubernetesui/dashboard:v2.0.1: Get https://index.docker.io/v2/: dial tcp: lookup index.docker.io: no such host
E0727 07:26:22.544151 14015 cache.go:63] save image to file "kubernetesui/metrics-scraper:v1.0.4" -> "/home/malik/.minikube/cache/images/kubernetesui/metrics-scraper_v1.0.4" failed: nil image for kubernetesui/metrics-scraper:v1.0.4: Get https://index.docker.io/v2/: dial tcp: lookup index.docker.io: no such host
E0727 07:26:22.579102 14015 cache.go:63] save image to file "k8s.gcr.io/etcd:3.4.3-0" -> "/home/malik/.minikube/cache/images/k8s.gcr.io/etcd_3.4.3-0" failed: write: error calculating manifest: Get https://storage.googleapis.com/eu.artifacts.k8s-artifacts-prod.appspot.com/containers/images/sha256:303ce5db0e90dab1c5728ec70d21091201a23cdf8aeca70ab54943bbaaf0833f: dial tcp: lookup storage.googleapis.com: no such host
E0727 07:26:22.579102 14015 cache.go:63] save image to file "k8s.gcr.io/kube-controller-manager:v1.18.3" -> "/home/malik/.minikube/cache/images/k8s.gcr.io/kube-controller-manager_v1.18.3" failed: write: error calculating manifest: Get https://storage.googleapis.com/eu.artifacts.k8s-artifacts-prod.appspot.com/containers/images/sha256:da26705ccb4b5eb623a7cc42e566d21b0e23c1f59a0b4d6acac3fb810538c0d5: dial tcp: lookup storage.googleapis.com: no such host
E0727 07:26:22.579194 14015 cache.go:63] save image to file "k8s.gcr.io/kube-proxy:v1.18.3" -> "/home/malik/.minikube/cache/images/k8s.gcr.io/kube-proxy_v1.18.3" failed: write: error calculating manifest: Get https://storage.googleapis.com/eu.artifacts.k8s-artifacts-prod.appspot.com/containers/images/sha256:3439b7546f29bec22edd737bc0a5770ead18b5ee5ce0aea5af9047a554715f9f: dial tcp: lookup storage.googleapis.com: no such host
E0727 07:26:22.579229 14015 cache.go:63] save image to file "gcr.io/k8s-minikube/storage-provisioner:v1.8.1" -> "/home/malik/.minikube/cache/images/gcr.io/k8s-minikube/storage-provisioner_v1.8.1" failed: write: error calculating manifest: Get https://storage.googleapis.com/artifacts.k8s-minikube.appspot.com/containers/images/sha256:4689081edb103a9e8174bf23a255bfbe0b2d9ed82edc907abab6989d1c60f02c: dial tcp: lookup storage.googleapis.com: no such host
E0727 07:26:22.619544 14015 cache.go:172] Error downloading kic artifacts: failed to download kic base image or any fallback image
❗ Executing "docker container inspect minikube --format={{.State.Status}}" took an unusually long time: 5.369799649s
💡 Restarting the docker service may improve performance.
🤷 docker "minikube" container is missing, will recreate.
🔥 Creating docker container (CPUs=2, Memory=2200MB) ...
🤦 StartHost failed, but will try again: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --security-opt apparmor=unconfined --cpus=2 --memory=2200mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.10#sha256:f58e0c4662bac8a9b5dda7984b185bad8502ade5d9fa364bf2755d636ab51438: exit status 125
stdout:
stderr:
Unable to find image 'gcr.io/k8s-minikube/kicbase:v0.0.10#sha256:f58e0c4662bac8a9b5dda7984b185bad8502ade5d9fa364bf2755d636ab51438' locally
docker: Error response from daemon: Get https://gcr.io/v2/k8s-minikube/kicbase/manifests/sha256:f58e0c4662bac8a9b5dda7984b185bad8502ade5d9fa364bf2755d636ab51438: Get https://gcr.io/v2/token?scope=repository%3Ak8s-minikube%2Fkicbase%3Apull&service=gcr.io: net/http: request canceled (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.
🤷 docker "minikube" container is missing, will recreate.
🔥 Creating docker container (CPUs=2, Memory=2200MB) ...
😿 Failed to start docker container. "minikube start" may fix it: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --security-opt apparmor=unconfined --cpus=2 --memory=2200mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.10#sha256:f58e0c4662bac8a9b5dda7984b185bad8502ade5d9fa364bf2755d636ab51438: exit status 125
stdout:
stderr:
Unable to find image 'gcr.io/k8s-minikube/kicbase:v0.0.10#sha256:f58e0c4662bac8a9b5dda7984b185bad8502ade5d9fa364bf2755d636ab51438' locally
docker: Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io: no such host.
See 'docker run --help'.
❌ [INVALID_PROXY_HOSTNAME] error provisioning host Failed to start host: recreate: creating host: create: creating: create kic node: create container: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --volume minikube:/var --security-opt apparmor=unconfined --cpus=2 --memory=2200mb -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 gcr.io/k8s-minikube/kicbase:v0.0.10#sha256:f58e0c4662bac8a9b5dda7984b185bad8502ade5d9fa364bf2755d636ab51438: exit status 125
stdout:
stderr:
Unable to find image 'gcr.io/k8s-minikube/kicbase:v0.0.10#sha256:f58e0c4662bac8a9b5dda7984b185bad8502ade5d9fa364bf2755d636ab51438' locally
docker: Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io: no such host.
See 'docker run --help'.
💡 Suggestion: Verify that your HTTP_PROXY and HTTPS_PROXY environment variables are set correctly.
📘 Documentation: https://minikube.sigs.k8s.io/docs/handbook/vpn_and_proxy/
When I run the "minikube start" command the Docker fails to load the images that are needed as it is being used as the virtual machine manager by Minikube and Kubectl is providing the interface for Minikube to be used on the terminal. I executed commands and operations using the Virtualbox as well but no use. The version of Docker, Kubectl & Minikube is up-to-date.
I have tried installing Minikube and Kubectl several times using different packages and methods but useless
Please help me so that I can be able to start a cluster and make my PC a Worker Node and get going on the road of Cloud Computing Development
You might have a minikube VM that has an old version or/and minikube cannot connect to. You can try deleting the VM and/or wipe out ~/.minikube
$ minikube delete
$ rm -rf ~/.minikube
If that doesn't work then you have a problem with VirtualBox. Uninstall/Re-install
3 years ago, I spent a lot of time struggling with minikube even though it has been the official way to run kubernetes locally.
If you get stuck, i would suggest to have a KinD cluster up in few seconds by just running this script:
kind_version="v0.8.1"
kind_bin_path=/usr/local/bin/kind
if [ ! -f ${kind_bin_path} ]; then
curl -Lo ./kind "https://kind.sigs.k8s.io/dl/${kind_version}/kind-$(uname)-amd64"
chmod +x ./kind
sudo mv ./kind ${kind_bin_path}
fi
cat <<EOF | kind create cluster --config -
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
EOF
export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"
# Now check
kubectl get nodes
# Congrats!
To customize more the cluster, check other YAML config here.
Good luck for whatever works for you.

Docker - Cannot build multi-platform images with docker buildx

I'm trying to build a multi-platform (amd64, arm64 and armv7) image using docker buildx. Since I'm using an amd64 machine running Ubuntu 18.04, I followed the instructions on the Docker website and installed qemu via:
sudo apt install qemu-user
However, a weird error appears when I execute the previous command. More specifically, there seems to be an issue with the binfmt-support service. Here's the full log:
Reading package lists... Done
Building dependency tree
Reading state information... Done
Starting pkgProblemResolver with broken count: 0
Starting 2 pkgProblemResolver with broken count: 0
Done
The following additional packages will be installed:
binfmt-support qemu-user-binfmt
The following NEW packages will be installed:
binfmt-support qemu-user qemu-user-binfmt
0 upgraded, 3 newly installed, 0 to remove and 1 not upgraded.
Need to get 0 B/7.409 kB of archives.
After this operation, 63,4 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Selecting previously unselected package binfmt-support.
(Reading database ... 245278 files and directories currently installed.)
Preparing to unpack .../binfmt-support_2.1.8-2_amd64.deb ...
Unpacking binfmt-support (2.1.8-2) ...
Selecting previously unselected package qemu-user.
Preparing to unpack .../qemu-user_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...
Unpacking qemu-user (1:2.11+dfsg-1ubuntu7.21) ...
Selecting previously unselected package qemu-user-binfmt.
Preparing to unpack .../qemu-user-binfmt_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...
Unpacking qemu-user-binfmt (1:2.11+dfsg-1ubuntu7.21) ...
Setting up binfmt-support (2.1.8-2) ...
Job for binfmt-support.service failed because the control process exited with error code.
See "systemctl status binfmt-support.service" and "journalctl -xe" for details.
invoke-rc.d: initscript binfmt-support, action "start" failed.
● binfmt-support.service - Enable support for additional executable binary formats
Loaded: loaded (/lib/systemd/system/binfmt-support.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2020-02-05 17:20:29 CET; 4ms ago
Docs: man:update-binfmts(8)
Process: 7766 ExecStart=/usr/sbin/update-binfmts --enable (code=exited, status=2)
Main PID: 7766 (code=exited, status=2)
feb 05 17:20:29 XPS-15-9570 systemd[1]: Starting Enable support for additional executable binary formats...
feb 05 17:20:29 XPS-15-9570 update-binfmts[7766]: update-binfmts: warning: unable to close /proc/sys/fs/binfmt_misc/register: No such file or directory
feb 05 17:20:29 XPS-15-9570 update-binfmts[7766]: update-binfmts: exiting due to previous errors
feb 05 17:20:29 XPS-15-9570 systemd[1]: binfmt-support.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
feb 05 17:20:29 XPS-15-9570 systemd[1]: binfmt-support.service: Failed with result 'exit-code'.
feb 05 17:20:29 XPS-15-9570 systemd[1]: Failed to start Enable support for additional executable binary formats.
Setting up qemu-user (1:2.11+dfsg-1ubuntu7.21) ...
Setting up qemu-user-binfmt (1:2.11+dfsg-1ubuntu7.21) ...
update-binfmts: warning: current package is qemu-user-binfmt, but binary format already installed by qemu-user-static
update-binfmts: exiting due to previous errors
dpkg: error processing package qemu-user-binfmt (--configure):
installed qemu-user-binfmt package post-installation script subprocess returned error exit status 2
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for neon-settings (0.0+p18.04+git20191212.1343) ...
Processing triggers for systemd (237-3ubuntu10.33) ...
Errors were encountered while processing:
qemu-user-binfmt
E: Sub-process /usr/bin/dpkg returned an error code (1)
Despite that, I tried to go on with the usual procedure, namely:
docker buildx create --name mybuilder
docker buildx use mybuilder
docker buildx inspect --bootstrap
Where the output of the last command is:
[+] Building 5.0s (1/1) FINISHED
=> [internal] booting buildkit 5.0s
=> => pulling image moby/buildkit:buildx-stable-1 4.3s
=> => creating container buildx_buildkit_mybuilder0 0.7s
Name: mybuilder
Driver: docker-container
Nodes:
Name: mybuilder0
Endpoint: unix:///var/run/docker.sock
Status: running
Platforms: linux/amd64, linux/386
As you can see, "linux/amd64" and "linux/386" are listed as the only available platforms, however I would need to build the image for "linux/arm64" and "linux/arm/v7" platforms as well.
I've been looking for a solution to this problem for hours, though I didn't find anything that worked
------------------------------------ EDIT ------------------------------------
Looks like I was able to solve part of the issue by running:
sudo apt purge --auto-remove qemu-user qemu-user-binfmt binfmt-support
And then reinstalling them. In fact, running again this command:
sudo apt install qemu-user
gives no error at all:
Reading package lists... Done
Building dependency tree
Reading state information... Done
Starting pkgProblemResolver with broken count: 0
Starting 2 pkgProblemResolver with broken count: 0
Done
The following additional packages will be installed:
binfmt-support qemu-user-binfmt
The following NEW packages will be installed:
binfmt-support qemu-user qemu-user-binfmt
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/7.409 kB of archives.
After this operation, 63,4 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Selecting previously unselected package binfmt-support.
(Reading database ... 245437 files and directories currently installed.)
Preparing to unpack .../binfmt-support_2.1.8-2_amd64.deb ...
Unpacking binfmt-support (2.1.8-2) ...
Selecting previously unselected package qemu-user.
Preparing to unpack .../qemu-user_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...
Unpacking qemu-user (1:2.11+dfsg-1ubuntu7.21) ...
Selecting previously unselected package qemu-user-binfmt.
Preparing to unpack .../qemu-user-binfmt_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...
Unpacking qemu-user-binfmt (1:2.11+dfsg-1ubuntu7.21) ...
Setting up binfmt-support (2.1.8-2) ...
Created symlink /etc/systemd/system/multi-user.target.wants/binfmt-support.service → /lib/systemd/system/binfmt-support.service.
Setting up qemu-user (1:2.11+dfsg-1ubuntu7.21) ...
Setting up qemu-user-binfmt (1:2.11+dfsg-1ubuntu7.21) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for neon-settings (0.0+p18.04+git20191212.1343) ...
Processing triggers for systemd (237-3ubuntu10.38) ...
Similarly, the output of systemctl status binfmt-support.service is as expected:
● binfmt-support.service - Enable support for additional executable binary formats
Loaded: loaded (/lib/systemd/system/binfmt-support.service; enabled; vendor preset: enabled)
Active: active (exited) since Mon 2020-02-10 11:42:23 CET; 1min 11s ago
Docs: man:update-binfmts(8)
Main PID: 7161 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 4915)
CGroup: /system.slice/binfmt-support.service
feb 10 11:42:23 XPS-15-9570 systemd[1]: Starting Enable support for additional executable binary formats...
feb 10 11:42:23 XPS-15-9570 systemd[1]: Started Enable support for additional executable binary formats.
However, part of the issue is still there, as the output after running these three commands:
docker buildx create --name mybuilder
docker buildx use mybuilder
docker buildx inspect --bootstrap
is the same as before, namely:
[+] Building 2.6s (1/1) FINISHED
=> [internal] booting buildkit 2.6s
=> => pulling image moby/buildkit:buildx-stable-1 2.0s
=> => creating container buildx_buildkit_mybuilder0 0.6s
Name: mybuilder
Driver: docker-container
Nodes:
Name: mybuilder0
Endpoint: unix:///var/run/docker.sock
Status: running
Platforms: linux/amd64, linux/386
Why is that? Why is it showing me linux/amd64 and linux/386 as the only available platforms?
EDIT #2 (concerning #LinPy's comment)
The output of docker context ls is:
NAME DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR
default * Current DOCKER_HOST based configuration unix:///var/run/docker.sock swarm
I've also tried to restart docker after qemu's installation, but to no success. Also, specifying the target platforms in the docker buildx command:
docker buildx build -t <mytag> --platform linux/amd64,linux/arm64,linux/arm/v7 --load .
results in this error:
[+] Building 0.6s (5/20)
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 32B 0.0s
=> [linux/arm/v7 internal] load metadata for docker.io/alegeno92/opencv_python3:3.4.2 0.6s
=> CANCELED [linux/arm64 internal] load metadata for docker.io/alegeno92/opencv_python3:3.4.2 0.6s
=> CANCELED [linux/amd64 internal] load metadata for docker.io/alegeno92/opencv_python3:3.4.2 0.6s
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to load LLB: runtime execution on platform linux/arm/v7 not supported
By the way, my version of the kernel is 4.15.0-76-generic
Run the multiarch container first
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker buildx rm builder
docker buildx create --name builder --driver docker-container --use
docker buildx inspect --bootstrap
And you should have your alternate architectures.
Tagging on this answer in response to the first error. The commands have been updated per https://docs.docker.com/buildx/working-with-buildx/.
QEMU is a cross-platform emulator responsible for sourcing the binaries for different architectures (through the binfmt_misc handler).
Will save some people some time to start with this command first:
docker run --privileged --rm tonistiigi/binfmt --install all
There are multiple binfmt packages, and there's a configuration that I think was missed when this question was asked.
For the various packages, I would opt for qemu-user-static over qemu-user-binfmt to avoid any dynamic linking issues. The two packages are doing the same thing, so you'll need to pick one or the other.
The next part should be fixed in current releases, but I think you were stumbling on this before. That's the fix binary or F flag you'll see when catting the files in /proc/sys/fs/binfmt_misc, e.g. see the F flag here:
$ cat /proc/sys/fs/binfmt_misc/qemu-arm
enabled
interpreter /usr/libexec/qemu-binfmt/arm-binfmt-P
flags: POCF
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff
Details on what the F flag means can be found on this kernel.org post but the short of it is container namespaces include a different filesystem namespace, and trying to access the interpreter from that namespace will fail (unless you do something like bind mount /usr/libexec/qemu-binfmt into your container). Newer versions of the qemu packages automatically set this flag, so if your flags section doesn't have the F defined, see these bug reports for the version you'll need to upgrade to:
Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=868030
Ubuntu: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1815100
The easy button is to use the binaries from the multiarch image. This is good in CI if you have a dedicated VM (less ideal if you are modifying the host used by other builds). However if you reboot, it breaks until you run the container again. And it requires you to remember to update it for any upstream patches. So I wouldn't recommend it for a long running build host.
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
For github CI, add following plugin solve this for me
- name: Set up QEMU
id: qemu
uses: docker/setup-qemu-action#v1
with:
image: tonistiigi/binfmt:latest
platforms: all

install TOR on a centOS 7 server

I have tried downloading TOR by following this article but I am getting 503 errors. So is there no other way to download TOR? Please can someone help me as I have to do a research project on it.
I am using a centOS server: CentOS Linux release 7.3.1611 (Core)
but am getting the follwing errors,
[sam#xx etc]$ sudo yum install tor
[sudo] password for sam:
Loaded plugins: fastestmirror, langpacks
base | 3.6 kB 00:00
extras | 3.4 kB 00:00
http://deb.torproject.org/torproject.org/rpm/el/7/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
Trying other mirror.
http://deb.torproject.org/torproject.org/rpm/el/7/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
Trying other mirror.
http://deb.torproject.org/torproject.org/rpm/el/7/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
Trying other mirror.
http://deb.torproject.org/torproject.org/rpm/el/7/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
Trying other mirror.
http://deb.torproject.org/torproject.org/rpm/el/7/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
Trying other mirror.
http://deb.torproject.org/torproject.org/rpm/el/7/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
Trying other mirror.
http://deb.torproject.org/torproject.org/rpm/el/7/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
Trying other mirror.
http://deb.torproject.org/torproject.org/rpm/el/7/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
Trying other mirror.
http://deb.torproject.org/torproject.org/rpm/el/7/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
Trying other mirror.
http://deb.torproject.org/torproject.org/rpm/el/7/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 503 - Service Unavailable
Trying other mirror.
Tor no longer recommends using their repo for CentoOS and to instead use epel repos. You'll just end up with a very old version out of their repos.
Instead do:
yum install epel-release
yum install tor
This will get you a current version of Tor managed by a repo. This is perfectly fine.
For CentOS, I have been self-compiling for a while. I have a hacky shell script I'm not yet willing to post here ;) but really it's just compiling Tor and OpenSSL. I'd still recommend using the epel-release since its more tested.
To try building Tor statically linked to OpenSSL yourself, grab a recent copy of OpenSSL (e.g. 1.1.1x), then grab the version of Tor you want to build (e.g. 0.4.1.6).
First you will need to install some prerequisites:
yum install -y \
gcc gcc-c++ \
zlib-devel \
perl-Module-Load-Conditional perl-Test-Harness \
libevent-devel \
libzstd-devel xz-devel \
libscrypt-devel libseccomp-devel
From OpenSSL source dir:
./config --prefix=/opt/openssl --openssldir=/opt/openssl \
-fPIC zlib-dynamic no-shared enable-ec_nistp_64_gcc_128
make
make test
make install
OpenSSL 1.1.1 note: Remove the no-shared option when building OpenSSL, otherwise Tor configuration will fail with an error that it can't find a linkable OpenSSL even though it is being built statically. Tor will still link a static OpenSSL but it seems to require the shared libraries to work. This appears to be fixed in 1.1.1c and later.
This installs OpenSSL to /opt/openssl so it doesn't interfere or replace the system's OpenSSL.
Then, build Tor:
./configure --prefix=/opt/tor-VERSION --sysconfdir=/etc --localstatedir=/var \
--enable-static-openssl --with-openssl-dir=/opt/openssl \
--with-tor-user=tor --with-tor-group=tor \
--enable-lzma --enable-zstd
make
make test
make install
unlink /usr/bin/tor && ln -s /opt/tor-VERSION/bin/tor /usr/bin/tor
The systemd service file I use is:
[Unit]
Description=Anonymizing overlay network for TCP
After=syslog.target network.target nss-lookup.target
[Service]
Type=forking
PidFile=/var/run/tor/tor.pid
NotifyAccess=all
ExecStartPre=/usr/bin/tor -f /etc/tor/torrc --verify-config
ExecStart=/usr/bin/tor -f /etc/tor/torrc --RunAsDaemon 1
ExecReload=/bin/kill -HUP ${MAINPID}
KillSignal=SIGINT
TimeoutStartSec=120
TimeoutStopSec=60
Restart=on-failure
LimitNOFILE=65536
# Hardening
PrivateTmp=yes
PrivateDevices=yes
ProtectHome=yes
ProtectSystem=full
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/tor
ReadWriteDirectories=-/var/log/tor
NoNewPrivileges=yes
CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_NET_BIND_SERVICE CAP_DAC_OVERRIDE CAP_CHOWN CAP_FOWNER
[Install]
WantedBy=multi-user.target

Resources