ERROR: The certificate of 'dl.eff.org' is not trusted - node.js

I'm studying about docker, node and graphql and would like to make my project available in HTTPS.
I found this tutorial (https://techsparx.com/nodejs/docker/express-https.html) that explains how to do this, but I ran into an error and couldn't find a solution that works.
Building certbot
[+] Building 2.8s (9/11)
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 840B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/debian:jessie 1.9s
=> [auth] library/debian:pull token for registry-1.docker.io 0.0s
=> [1/7] FROM docker.io/library/debian:jessie#sha256:32ad5050caffb2c7e969dac873bce2c370015c2256ff984b70c1c08b3a2816a0 0.0s
=> CACHED [2/7] RUN apt-get update && apt-get install -y cron bash wget 0.0s
=> CACHED [3/7] RUN mkdir -p /webroots/bgl8700.com.br/.well-known /scripts 0.0s
=> CACHED [4/7] WORKDIR /scripts 0.0s
=> ERROR [5/7] RUN wget https://dl.eff.org/certbot-auto 0.7s
------
> [5/7] RUN wget https://dl.eff.org/certbot-auto:
#11 0.347 converted 'https://dl.eff.org/certbot-auto' (ANSI_X3.4-1968) -> 'https://dl.eff.org/certbot-auto' (UTF-8)
#11 0.347 --2021-10-24 20:20:12-- https://dl.eff.org/certbot-auto
#11 0.361 Resolving dl.eff.org (dl.eff.org)... 151.101.176.201, 2a04:4e42:2a::201
#11 0.569 Connecting to dl.eff.org (dl.eff.org)|151.101.176.201|:443... connected.
#11 0.640 ERROR: The certificate of 'dl.eff.org' is not trusted.
#11 0.640 ERROR: The certificate of 'dl.eff.org' has expired.
------
executor failed running [/bin/sh -c wget https://dl.eff.org/certbot-auto]: exit code: 5
ERROR: Service 'certbot' failed to build : Build failed
This is the Dockerfile:
FROM debian:jessie
# Install cron, certbot, bash, plus any other dependencies
RUN apt-get update \
&& apt-get install -y cron bash wget
RUN mkdir -p /webroots/bgl8700.com.br/.well-known /scripts
WORKDIR /scripts
RUN wget https://dl.eff.org/certbot-auto
RUN chmod a+x ./certbot-auto
# Run certbot-auto so that it installs itself
RUN /scripts/certbot-auto -n certificates
# /webroots/DOMAIN.TLD/.well-known/... files go here
VOLUME /webroots
VOLUME /etc/letsencrypt
# This installs a Crontab entry which
# runs "certbot renew" on several days of each week at 03:22 AM
#
# RUN echo "22 03 * * 2,4,6,7 root /scripts/certbot-auto renew" >/etc/cron.d/certbot
CMD [ "cron", "-f" ]
This is the register:
#!/bin/sh
/scripts/certbot-auto certonly --webroot -w /webroots/$1 -d $1
Folder Structure:
certbot/
- Dockerfile
- register

You need to take a closer look at the SSL certificate that is used in your setup. After all you have the messages
#11 0.640 ERROR: The certificate of 'dl.eff.org' is not trusted.
#11 0.640 ERROR: The certificate of 'dl.eff.org' has expired.
So check where in your process the certificate is created - and if it is not, then you need to add that part.

Related

How to set up the same Docker cache behavior during build on Ubuntu as it is on Windows?

We have such Dockerfile:
FROM alpine:latest as step1
RUN echo "STEP1"
FROM alpine:latest as step2
RUN echo "STEP2"
FROM step1 as step3
RUN echo "STEP3"
On Windows (or on WSL2) we have such results for building multi-stage images.
Building all steps on Windows:
$ docker build .
[+] Building 0.2s (7/7) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 32B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.0s
=> [step1 1/2] FROM docker.io/library/alpine:latest 0.0s
=> CACHED [step1 2/2] RUN echo "STEP1" 0.0s
=> CACHED [step3 1/1] RUN echo "STEP3" 0.0s
=> exporting to image 0.1s
=> => exporting layers 0.0s
=> => writing image sha256:38ba33425558f5373099ad59aa187456c38b3f5eab14a60fdbb3cb7b1c484392 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Building Step 2 on Windows:
$ docker build --target step2 .
[+] Building 0.5s (6/6) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 32B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.0s
=> CACHED [step2 1/2] FROM docker.io/library/alpine:latest 0.0s
=> [step2 2/2] RUN echo "STEP2" 0.3s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:855f35039a5a902e1d33111fc781f748955f8b3cad7fb56d6eb420fc5d2fa55b 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Docker version on Windows:
$ docker version
Client:
Cloud integration: 1.0.17
Version: 20.10.7
API version: 1.41
Go version: go1.16.4
Git commit: f0df350
Built: Wed Jun 2 12:00:56 2021
OS/Arch: windows/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.7
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: b0f5bc3
Built: Wed Jun 2 11:54:58 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.6
GitCommit: d71fcd7d8303cbf684402823e425e9dd2e99285d
runc:
Version: 1.0.0-rc95
GitCommit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
docker-init:
Version: 0.19.0
GitCommit: de40ad0
While on Ubuntu we have slightly different results.
Building all steps on Ubuntu:
$ docker build .
Sending build context to Docker daemon 2.048kB
Step 1/6 : FROM alpine:latest as step1
---> d4ff818577bc
Step 2/6 : RUN echo "STEP1"
---> Using cache
---> ca7b2711269c
Step 3/6 : FROM alpine:latest as step2
---> d4ff818577bc
Step 4/6 : RUN echo "STEP2"
---> Using cache
---> 50668d3ea1c2
Step 5/6 : FROM step1 as step3
---> ca7b2711269c
Step 6/6 : RUN echo "STEP3"
---> Using cache
---> f529239cd29a
Successfully built f529239cd29a
Building Step 2 on Ubuntu:
$ docker build --target step2 .
Sending build context to Docker daemon 2.048kB
Step 1/4 : FROM alpine:latest as step1
---> d4ff818577bc
Step 2/4 : RUN echo "STEP1"
---> Using cache
---> ca7b2711269c
Step 3/4 : FROM alpine:latest as step2
---> d4ff818577bc
Step 4/4 : RUN echo "STEP2"
---> Using cache
---> 50668d3ea1c2
Successfully built 50668d3ea1c2
Docker version on Ubuntu:
$ docker version
Client: Docker Engine - Community
Version: 20.10.7
API version: 1.41
Go version: go1.13.15
Git commit: f0df350
Built: Wed Jun 2 11:56:38 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.7
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: b0f5bc3
Built: Wed Jun 2 11:54:50 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.6
GitCommit: d71fcd7d8303cbf684402823e425e9dd2e99285d
runc:
Version: 1.0.0-rc95
GitCommit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
docker-init:
Version: 0.19.0
GitCommit: de40ad0
As you can see Docker Desktop (Windows) skips unneeded stages while Docker on Ubuntu builds them. The Docker Engine versions are the same.
For complex multi-stage images, the caching behavior the Docker on Windows can save a lot of build time comparing to the one on Ubuntu.
How to make Docker on Ubuntu behave the same way with cache as on Windows?
I found the answer by myself. On Ubuntu we need to enable BuildKit by DOCKER_BUILDKIT=1. See https://docs.docker.com/engine/reference/builder/#buildkit
In this case, the behavior on both platforms is the same.

Why Docker COPY doesn't change file permissions? (--chmod)

Given this Dockerfile:
FROM docker.io/alpine
RUN mkdir test
# RUN umask 0022
COPY README /test/README
COPY --chmod=777 README /test/README-777
COPY --chmod=755 README /test/README-755
COPY FORALL /test/FORALL
COPY --chmod=777 FORALL /test/FORALL-777
COPY --chmod=755 FORALL /test/FORALL-755
RUN ls -la /test
I'd expect to have the read, write, execute permissions be set accordingly by Docker during the build process (docker build ./).
But the last command returns
total 8
drwxr-xr-x 1 root root 4096 Jun 9 19:20 .
drwxr-xr-x 1 root root 4096 Jun 9 19:20 ..
-rwxrwxrwx 1 root root 0 Jun 9 19:19 FORALL
-rwxrwxrwx 1 root root 0 Jun 9 19:19 FORALL-755
-rwxrwxrwx 1 root root 0 Jun 9 19:19 FORALL-777
-rw-rw-r-- 1 root root 0 Jun 9 19:19 README
-rw-rw-r-- 1 root root 0 Jun 9 19:19 README-755
-rw-rw-r-- 1 root root 0 Jun 9 19:19 README-777
No file permission was changed, and no error was raised.
Why doesn't it work?
How to fix this?
I figured out:
the flag --chmod is a new feature from Docker Buildkit, so it is necessary to run the build enabling it via:
DOCKER_BUILDKIT=1 docker build ./
However, it is really not clear why Docker swallows the --chmod option without any error or warn about the non-existing option 😕.
This is fixed in 20.10.6 (pull request, tracking issue):
$ cat df.chmod
FROM busybox as base
RUN touch /test
FROM busybox as release
COPY --from=base --chmod=777 /test /test-777
COPY --from=base --chmod=555 /test /test-555
CMD ls -l /test*
$ DOCKER_BUILDKIT=0 docker build -t test-chmod-classic -f df.chmod .
Sending build context to Docker daemon 22.02kB
Step 1/6 : FROM busybox as base
---> a9d583973f65
Step 2/6 : RUN touch /test
---> Running in ed48f45a5dca
Removing intermediate container ed48f45a5dca
---> 5606d2d23861
Step 3/6 : FROM busybox as release
---> a9d583973f65
Step 4/6 : COPY --from=base --chmod=777 /test /test-777
the --chmod option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ to learn how to build images with BuildKit enabled
And if the build is run with buildkit, the expected result occurs:
$ DOCKER_BUILDKIT=1 docker build -t test-chmod-buildkit -f df.chmod .
[+] Building 1.0s (8/8) FINISHED
=> [internal] load build definition from df.chmod 0.0s
=> => transferring dockerfile: 214B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 49B 0.0s
=> [internal] load metadata for docker.io/library/busybox:latest 0.0s
=> CACHED [base 1/2] FROM docker.io/library/busybox 0.0s
=> [base 2/2] RUN touch /test 0.6s
=> [release 2/3] COPY --from=base --chmod=777 /test /test-777 0.1s
=> [release 3/3] COPY --from=base --chmod=555 /test /test-555 0.1s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:a4df92175046e36a72a769f9c7b297bc04a825708c5f6ca5873428b55c340036 0.0s
=> => naming to docker.io/library/test-chmod-buildkit 0.0s
$ docker run --rm test-chmod-buildkit
-r-xr-xr-x 1 root root 0 Jun 10 13:00 /test-555
-rwxrwxrwx 1 root root 0 Jun 10 13:00 /test-777

Docker - Cannot build multi-platform images with docker buildx

I'm trying to build a multi-platform (amd64, arm64 and armv7) image using docker buildx. Since I'm using an amd64 machine running Ubuntu 18.04, I followed the instructions on the Docker website and installed qemu via:
sudo apt install qemu-user
However, a weird error appears when I execute the previous command. More specifically, there seems to be an issue with the binfmt-support service. Here's the full log:
Reading package lists... Done
Building dependency tree
Reading state information... Done
Starting pkgProblemResolver with broken count: 0
Starting 2 pkgProblemResolver with broken count: 0
Done
The following additional packages will be installed:
binfmt-support qemu-user-binfmt
The following NEW packages will be installed:
binfmt-support qemu-user qemu-user-binfmt
0 upgraded, 3 newly installed, 0 to remove and 1 not upgraded.
Need to get 0 B/7.409 kB of archives.
After this operation, 63,4 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Selecting previously unselected package binfmt-support.
(Reading database ... 245278 files and directories currently installed.)
Preparing to unpack .../binfmt-support_2.1.8-2_amd64.deb ...
Unpacking binfmt-support (2.1.8-2) ...
Selecting previously unselected package qemu-user.
Preparing to unpack .../qemu-user_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...
Unpacking qemu-user (1:2.11+dfsg-1ubuntu7.21) ...
Selecting previously unselected package qemu-user-binfmt.
Preparing to unpack .../qemu-user-binfmt_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...
Unpacking qemu-user-binfmt (1:2.11+dfsg-1ubuntu7.21) ...
Setting up binfmt-support (2.1.8-2) ...
Job for binfmt-support.service failed because the control process exited with error code.
See "systemctl status binfmt-support.service" and "journalctl -xe" for details.
invoke-rc.d: initscript binfmt-support, action "start" failed.
● binfmt-support.service - Enable support for additional executable binary formats
Loaded: loaded (/lib/systemd/system/binfmt-support.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2020-02-05 17:20:29 CET; 4ms ago
Docs: man:update-binfmts(8)
Process: 7766 ExecStart=/usr/sbin/update-binfmts --enable (code=exited, status=2)
Main PID: 7766 (code=exited, status=2)
feb 05 17:20:29 XPS-15-9570 systemd[1]: Starting Enable support for additional executable binary formats...
feb 05 17:20:29 XPS-15-9570 update-binfmts[7766]: update-binfmts: warning: unable to close /proc/sys/fs/binfmt_misc/register: No such file or directory
feb 05 17:20:29 XPS-15-9570 update-binfmts[7766]: update-binfmts: exiting due to previous errors
feb 05 17:20:29 XPS-15-9570 systemd[1]: binfmt-support.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
feb 05 17:20:29 XPS-15-9570 systemd[1]: binfmt-support.service: Failed with result 'exit-code'.
feb 05 17:20:29 XPS-15-9570 systemd[1]: Failed to start Enable support for additional executable binary formats.
Setting up qemu-user (1:2.11+dfsg-1ubuntu7.21) ...
Setting up qemu-user-binfmt (1:2.11+dfsg-1ubuntu7.21) ...
update-binfmts: warning: current package is qemu-user-binfmt, but binary format already installed by qemu-user-static
update-binfmts: exiting due to previous errors
dpkg: error processing package qemu-user-binfmt (--configure):
installed qemu-user-binfmt package post-installation script subprocess returned error exit status 2
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for neon-settings (0.0+p18.04+git20191212.1343) ...
Processing triggers for systemd (237-3ubuntu10.33) ...
Errors were encountered while processing:
qemu-user-binfmt
E: Sub-process /usr/bin/dpkg returned an error code (1)
Despite that, I tried to go on with the usual procedure, namely:
docker buildx create --name mybuilder
docker buildx use mybuilder
docker buildx inspect --bootstrap
Where the output of the last command is:
[+] Building 5.0s (1/1) FINISHED
=> [internal] booting buildkit 5.0s
=> => pulling image moby/buildkit:buildx-stable-1 4.3s
=> => creating container buildx_buildkit_mybuilder0 0.7s
Name: mybuilder
Driver: docker-container
Nodes:
Name: mybuilder0
Endpoint: unix:///var/run/docker.sock
Status: running
Platforms: linux/amd64, linux/386
As you can see, "linux/amd64" and "linux/386" are listed as the only available platforms, however I would need to build the image for "linux/arm64" and "linux/arm/v7" platforms as well.
I've been looking for a solution to this problem for hours, though I didn't find anything that worked
------------------------------------ EDIT ------------------------------------
Looks like I was able to solve part of the issue by running:
sudo apt purge --auto-remove qemu-user qemu-user-binfmt binfmt-support
And then reinstalling them. In fact, running again this command:
sudo apt install qemu-user
gives no error at all:
Reading package lists... Done
Building dependency tree
Reading state information... Done
Starting pkgProblemResolver with broken count: 0
Starting 2 pkgProblemResolver with broken count: 0
Done
The following additional packages will be installed:
binfmt-support qemu-user-binfmt
The following NEW packages will be installed:
binfmt-support qemu-user qemu-user-binfmt
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/7.409 kB of archives.
After this operation, 63,4 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Selecting previously unselected package binfmt-support.
(Reading database ... 245437 files and directories currently installed.)
Preparing to unpack .../binfmt-support_2.1.8-2_amd64.deb ...
Unpacking binfmt-support (2.1.8-2) ...
Selecting previously unselected package qemu-user.
Preparing to unpack .../qemu-user_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...
Unpacking qemu-user (1:2.11+dfsg-1ubuntu7.21) ...
Selecting previously unselected package qemu-user-binfmt.
Preparing to unpack .../qemu-user-binfmt_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...
Unpacking qemu-user-binfmt (1:2.11+dfsg-1ubuntu7.21) ...
Setting up binfmt-support (2.1.8-2) ...
Created symlink /etc/systemd/system/multi-user.target.wants/binfmt-support.service → /lib/systemd/system/binfmt-support.service.
Setting up qemu-user (1:2.11+dfsg-1ubuntu7.21) ...
Setting up qemu-user-binfmt (1:2.11+dfsg-1ubuntu7.21) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for neon-settings (0.0+p18.04+git20191212.1343) ...
Processing triggers for systemd (237-3ubuntu10.38) ...
Similarly, the output of systemctl status binfmt-support.service is as expected:
● binfmt-support.service - Enable support for additional executable binary formats
Loaded: loaded (/lib/systemd/system/binfmt-support.service; enabled; vendor preset: enabled)
Active: active (exited) since Mon 2020-02-10 11:42:23 CET; 1min 11s ago
Docs: man:update-binfmts(8)
Main PID: 7161 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 4915)
CGroup: /system.slice/binfmt-support.service
feb 10 11:42:23 XPS-15-9570 systemd[1]: Starting Enable support for additional executable binary formats...
feb 10 11:42:23 XPS-15-9570 systemd[1]: Started Enable support for additional executable binary formats.
However, part of the issue is still there, as the output after running these three commands:
docker buildx create --name mybuilder
docker buildx use mybuilder
docker buildx inspect --bootstrap
is the same as before, namely:
[+] Building 2.6s (1/1) FINISHED
=> [internal] booting buildkit 2.6s
=> => pulling image moby/buildkit:buildx-stable-1 2.0s
=> => creating container buildx_buildkit_mybuilder0 0.6s
Name: mybuilder
Driver: docker-container
Nodes:
Name: mybuilder0
Endpoint: unix:///var/run/docker.sock
Status: running
Platforms: linux/amd64, linux/386
Why is that? Why is it showing me linux/amd64 and linux/386 as the only available platforms?
EDIT #2 (concerning #LinPy's comment)
The output of docker context ls is:
NAME DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR
default * Current DOCKER_HOST based configuration unix:///var/run/docker.sock swarm
I've also tried to restart docker after qemu's installation, but to no success. Also, specifying the target platforms in the docker buildx command:
docker buildx build -t <mytag> --platform linux/amd64,linux/arm64,linux/arm/v7 --load .
results in this error:
[+] Building 0.6s (5/20)
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 32B 0.0s
=> [linux/arm/v7 internal] load metadata for docker.io/alegeno92/opencv_python3:3.4.2 0.6s
=> CANCELED [linux/arm64 internal] load metadata for docker.io/alegeno92/opencv_python3:3.4.2 0.6s
=> CANCELED [linux/amd64 internal] load metadata for docker.io/alegeno92/opencv_python3:3.4.2 0.6s
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to load LLB: runtime execution on platform linux/arm/v7 not supported
By the way, my version of the kernel is 4.15.0-76-generic
Run the multiarch container first
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker buildx rm builder
docker buildx create --name builder --driver docker-container --use
docker buildx inspect --bootstrap
And you should have your alternate architectures.
Tagging on this answer in response to the first error. The commands have been updated per https://docs.docker.com/buildx/working-with-buildx/.
QEMU is a cross-platform emulator responsible for sourcing the binaries for different architectures (through the binfmt_misc handler).
Will save some people some time to start with this command first:
docker run --privileged --rm tonistiigi/binfmt --install all
There are multiple binfmt packages, and there's a configuration that I think was missed when this question was asked.
For the various packages, I would opt for qemu-user-static over qemu-user-binfmt to avoid any dynamic linking issues. The two packages are doing the same thing, so you'll need to pick one or the other.
The next part should be fixed in current releases, but I think you were stumbling on this before. That's the fix binary or F flag you'll see when catting the files in /proc/sys/fs/binfmt_misc, e.g. see the F flag here:
$ cat /proc/sys/fs/binfmt_misc/qemu-arm
enabled
interpreter /usr/libexec/qemu-binfmt/arm-binfmt-P
flags: POCF
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff
Details on what the F flag means can be found on this kernel.org post but the short of it is container namespaces include a different filesystem namespace, and trying to access the interpreter from that namespace will fail (unless you do something like bind mount /usr/libexec/qemu-binfmt into your container). Newer versions of the qemu packages automatically set this flag, so if your flags section doesn't have the F defined, see these bug reports for the version you'll need to upgrade to:
Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=868030
Ubuntu: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1815100
The easy button is to use the binaries from the multiarch image. This is good in CI if you have a dedicated VM (less ideal if you are modifying the host used by other builds). However if you reboot, it breaks until you run the container again. And it requires you to remember to update it for any upstream patches. So I wouldn't recommend it for a long running build host.
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
For github CI, add following plugin solve this for me
- name: Set up QEMU
id: qemu
uses: docker/setup-qemu-action#v1
with:
image: tonistiigi/binfmt:latest
platforms: all

Docker container always shows ssl connection error

I'm getting started with docker compose and have been working through the simple demo flask application. The thing is, I'm running this from inside of an organization that intercepts all communication in such a way that SSL errors are thrown right and left. They provide us with three root certificates we need to install, and I've generally got these working on my own machine, but I'm lost when it comes to getting these to work inside docker-compose deployments.
When I run docker-compose up, I get the following:
$ sudo docker-compose up
Creating network "project_default" with the default driver
Building web
Step 1/5 : FROM python:3.4-alpine
3.4-alpine: Pulling from library/python
81033e7c1d6a: Pull complete
9b61101706a6: Pull complete
415e2a07c89b: Pull complete
f22df7a3f000: Pull complete
8c16bf19c1f9: Pull complete
Digest: sha256:fe436cb066394d81cf49448a04dec7c765082445a500bc44f1ae5e8a455793bd
Status: Downloaded newer image for python:3.4-alpine
---> 5c72717ec319
Step 2/5 : ADD . /code
---> a5790c0e3e94
Removing intermediate container 052c614e41d0
Step 3/5 : WORKDIR /code
---> a2ea9acb3005
Removing intermediate container 77f2375ca0a6
Step 4/5 : RUN pip install -r requirements.txt
---> Running in 5f4fe856776d
Collecting flask (from -r requirements.txt (line 1))
Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb0061f1d30>: Failed to establish a new connection: [Errno -3] Try again',)': /simple/flask/
Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb0061f19b0>: Failed to establish a new connection: [Errno -3] Try again',)': /simple/flask/
Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb0061f1828>: Failed to establish a new connection: [Errno -3] Try again',)': /simple/flask/
Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb0061f1588>: Failed to establish a new connection: [Errno -3] Try again',)': /simple/flask/
Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb0061f1390>: Failed to establish a new connection: [Errno -3] Try again',)': /simple/flask/
Could not find a version that satisfies the requirement flask (from -r requirements.txt (line 1)) (from versions: )
No matching distribution found for flask (from -r requirements.txt (line 1))
Pip fails to install anything.
The docker-compose.yml file looks like this:
version: '3'
services:
web:
build: .
ports:
- "5000:5000"
redis:
image: "redis:alpine"
And the main Dockerfile looks like this:
FROM python:3.4-alpine
ADD . /code
WORKDIR /code
RUN pip install -r requirements.txt
CMD ["python", "app.py"]
Is there any way to be able to make this work in this particular case? Is there a general solution to this sort of problem that would allow me to pass to any container deployed the SSL certificates and have them be used?
In my case, I must add in my Dockerfile these sentences:
COPY company.crt /usr/local/share/ca-certificates/company.crt
RUN update-ca-certificates
...
RUN pip install --cert /etc/ssl/certs/company.pem -r requirements.txt
You need the certificate of your company in .crt format. When docker execute update-ca-certificates, linux will create a .pem file with the same name in the path: /etc/ssl/certs/. It will convert the network between pip in SSL.
This isn't really a docker-specific question: you are asking, in effect, "how do I install certificate authorities under Linux"? The answer is going to be the same regardless of whether you are running your ssl client inside or outside of a container.
Your Python image is based on alpine, and alpine uses the "ca-certificates" package to manage CA certificiates. To install your local CA certificates, you would need to (a) copy them into the /usr/share/ca-certificates directory and (b) run update-ca-certificates.
For example, adding something like this to your Dockerfile (before your pip install):
COPY company-ca.crt /usr/share/ca-certificates
RUN update-ca-certificates
In my case, Host machine's MTU is 1450, and Docker's MTU is 1500.
Which causes docker set MSS to 1460, and then TLS "server hello" packet got bigger than 1450 bytes, so the Host machine discard it.
To see if it's your case too, run ifconfig on both you Docker container and your host machine. If Host's MTU is less than 1500, it's easy to run into this discarding packet situation. Especially in HTTPS, for that "server hello" needs to send the certificate, which is a big packet
I was trying to read data from an API in my Go code and I was facing similar ssl error:
x509: certificate signed by unknown authority
My container was based on debian:stretch which is really really small ~100MB.
This happens when ca-certificates are not installed.
I installed ca-certificates (which also installs openssl) as shown below:
FROM debian:stretch
RUN apt-get update && apt-get install -y ca-certificates --no-install-recommends && rm -rf /var/lib/apt/lists/*
# ...
This will also work for any other base images which might not have ca-certificates installed.

Gitlab CI 9.5 service is not running

I am searching a solution since 2 weeks on the web and I really need some help.
I am facing 3 problems:
Linux Gitlab-runner is not running
I have been trying to install gilab-runner with all the ways (GitLab's official repository, manualy, docker).
Everytime, when I am launching the command "gitlab-runner status" the answer is always "The server is not running." I have tried a million times to uninstall the service and re-install it but I do not want to work. I have register runners of all kind and with/without the sudo user. Without any success. This is my setup server:
Config
Ubuntu 16.04.1
Docker container gitlab 9.4.3
Port:
webservice :8088
https : 4433
ssh : 2222
gitlab-runner 9.5.0
How to reproduce
Register a shell runner http://192.168.1.10:8088/
Launch the command "sudo service gitlab-runner status"
Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since ven. 2017-08-25 15:17:45 CEST; 45s ago
Process: 13201 ExecStart=/usr/bin/gitlab-ci-multi-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user gitlab-runner (code=exited, status=1/FAILURE)
Main PID: 13201 (code=exited, status=1/FAILURE)
systemd1: gitlab-runner.service: Unit entered failed state.
systemd1: gitlab-runner.service: Failed with result 'exit-code'.
Windows gitlab-runner Error 500
Because of my problem to install gitlab-runner in Linux, I have tried to install it on another computer on Windows 10.
It worked and finally the commande gitlab-runner status answered me "Service is running" (but this is just a temporary solution, I really need to make it work on linux).
Anyway, I have added a CI script to a test program and launch the job but it was turning in loop over and over.
When I launch the command "gitlab-runner --debug run":
...
passfile: true
extension: cmd
job=183 project=19 runner=679ccd01
Using Shell executor... job=183 project=19 runner=679ccd01
Waiting for signals... job=183 project=19 runner=679ccd01
WARNING: Job failed: exit status 128 job=183 project=19 runner=679ccd01
WARNING: Submitting job to coordinator... failed job=183 runner=679ccd01 status=500 Internal Server Error
WARNING: Submitting job to coordinator... failed job=183 runner=679ccd01 status=500 Internal Server Error
...
Gitlab.com and run command
So I have decided to add my project on gitlab.com, to test it.
git#gitlab.com:sandbox_test/test_ci.git
Once again the job was turning in infinite loop until I launch on my Windows computer the command "gitlab-runner run".
Dialing: tcp gitlab.com:443 ...
Feeding runners to channel builds=0
Checking for jobs... received job=30315630 repo_url=https://gitlab.com/sandbox_test/test_ci.git runner=d98c0af1
Failed to requeue the runner: builds=1 runner=d98c0af1
Running with gitlab-ci-multi-runner 9.5.0 (413da38)
on Windows_shell_gitlab_com (d98c0af1) job=30315630 project=3992201 runner=d98c0af1
Shell configuration: environment: []
dockercommand: []
command: cmd
arguments:
- /C
passfile: true
extension: cmd
job=30315630 project=3992201 runner=d98c0af1
Using Shell executor... job=30315630 project=3992201 runner=d98c0af1
Waiting for signals... job=30315630 project=3992201 runner=d98c0af1
Job succeeded job=30315630 project=3992201 runner=d98c0af1
Why is it necessary to launch the run command to make work my job on gitlab.com?
I expect when I run a new job it will figure out by itself without to launch manually the gitlab-runner on the CI computer...
Script .gitlab-ci.yml
Validate on CI Lint
stages:
- build
- test
- deploy
build:
stage: build
script:
- echo "building"
test:
stage: test
script:
- echo "test"
I really need answers very fast, thanks for your help.
Best Regards,Clement
UPDATE 1
I have resoved a part of my problems :
Linux Gitlab-runner is not running
Launch the command "gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user gitlab-runner"
First Error : chdir /home/gitlab-runner: no such file or directory
Solution: sudo mkdir /home/gitlab-runner
Second Error : open /etc/gitlab-runner/config.toml: permission denied
Solution : sudo chmod 755 /etc/gitlab-runner/config.toml
I have resoved a part of my problems :
Linux Gitlab-runner is not running
Launch the command "gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user gitlab-runner"
First Error : chdir /home/gitlab-runner: no such file or directory
Solution: sudo mkdir /home/gitlab-runner
Second Error : open /etc/gitlab-runner/config.toml: permission denied
Solution : sudo chmod 755 /etc/gitlab-runner/config.toml

Resources