CIS benchmark issue for Kubernetes cluster - security

I'm running the CIS kube-bench tool on the master node and trying to resolve this error
[FAIL] 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated).
I understand that I need to update the API server manifest YAML file with this flag pointing to the right CA file --kubelet-certificate-authority however, I'm not sure which one is the right CA Certififace for Kubelet.
These are my files in the PKI directory:-
apiserver-etcd-client.crt
apiserver-etcd-client.key
apiserver-kubelet-client.crt
apiserver-kubelet-client.key
apiserver.crt
apiserver.key
ca.crt
ca.key
etcd
front-proxy-ca.crt
front-proxy-ca.key
front-proxy-client.crt
front-proxy-client.key
sa.key
sa.pub

3 very similar discussions on the same topic. I wont provide you all steps cause it well written in documentation and related questions on stack. Only high-level overview
How Do I Properly Set --kubelet-certificate-authority apiserver parameter?
Kubernetes kubelet-certificate-authority on premise with kubespray causes certificate validation error for master node
Kubernetes kubelet-certificate-authority on premise with kubespray causes certificate validation error for master node
Your actions:
Follow the Kubernetes documentation and setup the TLS connection between the apiserver and kubelets.
These connections terminate at the kubelet's HTTPS endpoint. By
default, the apiserver does not verify the kubelet's serving
certificate, which makes the connection subject to man-in-the-middle
attacks and unsafe to run over untrusted and/or public networks.
Enable Kubelet authentication and Kubelet authorization
Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --kubelet-certificate-authority parameter to the path to the cert file for the certificate authority.
From #Matt answer
Use /etc/kubernetes/ssl/ca.crt to sign new certificate for kubelet with valid IP SANs.
Set --kubelet-certificate-authority=/etc/kubernetes/ssl/ca.crt (valid CA).
In /var/lib/kubelet/config.yaml (kubelet config file) set tlsCertFile and tlsPrivateKeyFile to point to newly created kubelet crt and key files.
And from clarifications:
Yes you have to generate certificates for kubelets and sign sign them
the provided certificate authority located here on the master
/etc/kubernetes/ssl/ca.crt

By default in Kubernetes there are 3 different Parent CA (kubernetes-ca, etcd-ca, kubernetes-front-proxy-ca). You are looking for kubernetes-ca because kubelet using kubernetes-ca, and you can check the documentation. kubernetes-ca default path is /etc/kubernetes/pki/ca.crt But also you verify it via kubelet configmap with below commands
kubectl get configmap -n kube-system $(kubectl get configmaps -n kube-system | grep kubelet | awk '{print $1}') -o yaml | grep -i clientca

Related

Fetch secrets and certificates from AzureKeyVault inside Docker container

I have a .net framework console application. Inside this application, I'm fetching secrets and certificates from keyvault using tenantId, client Id and Client Secret.
Application is fetching secrets and certificates properly.
Now I have containerized the application using Docker. After running the image I'm unable to fetch secrets and certificates. I'm getting below error:
" Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry. (No such host is known.) (No such host is known.) (No such
host is known.) (No such host is known.)"
To resolve the error, please try the following workarounds:
Check whether your container was setup behind an nginx reverse proxy.
If yes, then try removing the upstream section from the nginx reverse proxy and set proxy_pass to use docker-compose service's hostname.
After any change make sure to restart WSL and Docker.
Check if DNS is resolving the host names successfully or not, otherwise try adding the below in your docker-compose.yml file.
dns:
- 8.8.8.8
Try removing auto generated values by WSL in /etc/resolv.conf and add DNS like below if above doesn't work.
# [network]
# generateResolvConf = false
nameserver 8.8.8.8
Try restarting the WSL by running below command as an Admin:
Restart-NetAdapter -Name "vEthernet (WSL)"
Try installing a Docker Desktop update as a workaround.
For more in detail, please refer below links:
Getting "Name or service not known (login.microsoftonline.com:443)" regularly, but occasionally it succeeds? · Discussion #3102 · dotnet/dotnet-docker · GitHub
ssl - How to fetch Certificate from Azure Key vault to be used in docker image - Stack Overflow

Gitlab : Peer's certificate issuer has been marked as not trusted by the user

I have a on-prem gitlab where I am trying to run some builds/pipeline but getting the below error -
fatal: unable to access 'https://gitlab-ci-token:[MASKED]#gitlab.systems/testing/test-project-poc.git/': Peer's certificate issuer has been marked as not trusted by the user.
I have already looked into this - Gitlab:Peer's Certificate issuer is not recognized and followed the steps of obtaining the .pem file by merging the server certificate, intermediate certificate and root certificate but I am still getting the below error and really struggling to find the root cause.
/etc/gitlab/gitlab.rb config
##! enable/disable 2-way SSL client authentication
#nginx['ssl_verify_client'] = "off"
##! if ssl_verify_client on, verification depth in the client certificates chain
#nginx['ssl_verify_depth'] = "1"
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.systems.pem"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.systems.key"
Is there any other configuration which i need to update/modify? Any guidance is really appreciated.
I am guessing you are using a self signed certificate. If that is the case you have two options to rectify this issue:
Recommended option: Here again I assume that you have already solved the issue between the gitlab-runner and gitlab itseld, hence you registered the runner successfully. So you have already the certificate file in a /etc/gitlab-runner/certs. So on the server hosting the gitlab-runner, run the below command:
git config --system http.sslCAInfo /etc/gitlab-runner/certs/CERITIFICATE_NAME.crt
This is unsafe: Here you just disable the git https certificate verification:
git config --system http.sslverify false

How to replace default certificates on a cloud2edge instance?

I deployed a cloud2edge instance and now i want to replace the default certificates with other ones generated with the create_certs.sh script. According to the Hono documentation i can specify the configuration (including the certificates path) in the values.yaml, but i am not sure how to do it with the cloud2edge package.
Where should i take a look in order to achieve my goal?
Is there any possibility to set the certificates path without re-installing the package?
This is what i did in order to replace the keys/certificate for the mqtt adapter:
Create a secret containing the keys and the certificate
kubectl create secret generic mqtt-key-cert --from-file=certs/mqtt-adapter-cert.pem --from-file=mqtt-adapter-key.pem -n $NS
Mount the secret into the adapter's container filesystem
helm upgrade -n $NS --set hono.adapters.mqtt.extraSecretMounts.tls.secretName="mqtt-key-cert" --set hono.adapters.mqtt.extraSecretMounts.tls.mountPath="/etc/tls" --reuse-values $RELEASE eclipse-iot/cloud2edge
Set the corresponding environment variables into the mqtt adapter deployment
kubectl edit deployments c2e-adapter-mqtt-vertx -n $NS
YAML:

Azure Linux web app: change OpenSSL default security level?

In my Azure Linux web app, I'm trying to perform an API call to an external provider, with a certificate. That call fails, while it's working fine when deploying the same code on a Windows app service plan. The equivalent cURL command line is:
curl --cert-type p12 --cert /var/ssl/private/THUMBPRINT.p12 -X POST https://www.example.com
The call fails with the following error:
curl: (58) could not load PKCS12 client certificate, OpenSSL error error:140AB18E:SSL routines:SSL_CTX_use_certificate:ca md too weak
The issue is caused by OpenSSL 1.1.1d, which by defaults requires a security level of 2, and my certificate is signed with SHA1 with RSA encryption:
openssl pkcs12 -in THUMBPRINT.p12 -nodes | openssl x509 -noout -text | grep 'Signature Algorithm'
Signature Algorithm: sha1WithRSAEncryption
Signature Algorithm: sha1WithRSAEncryption
On a normal Linux VM, I could edit /etc/ssl/openssl/cnf to change
CipherString = DEFAULT#SECLEVEL=2
to security level 1, but on an Azure Linux web app, the changes I make to that file are not persisted..
So my question is: how do I change the OpenSSL security level on an Azure web app? Or is there a better way to allow the use of my weak certificate?
Note: I'm not the issuer of the certificate, so I can't regenerate it myself. I'll check with the issuer if they can regenerate it, but in the meantime I'd like to proceed if possible :)
A call with Microsoft support led me to a solution. It's possible to run a script whenever the web app container starts, which means it's possible to edit the openssl.cnf file before the dotnet app in launched.
To do this, navigate to the Configuration blade of your Linux web app, then General settings, then Startup command:
The Startup command is a command that's ran when the container starts. You can do what you want, but it HAS to launch your app, because it's no longer done automatically.
You can SSH to your Linux web app, and edit that custom_startup.sh file:
#!/usr/sh
# allow weak certificates (certificate signed with SHA1)
# by downgrading OpenSSL security level from 2 to 1
sed -i 's/SECLEVEL=2/SECLEVEL=1/g' /etc/ssl/openssl.cnf
# run the dotnet website
cd /home/site/wwwroot
dotnet APPLICATION_DLL_NAME.dll
The relevant doc can be found here: https://learn.microsoft.com/en-us/azure/app-service/containers/app-service-linux-faq#built-in-images
Note however that the Startup command is not working for Azure Functions (at the time of writing May 19th, 2020). I've opened an issue on Github.
To work around this, I ended up creating custom Docker images:
Dockerfile for a webapp:
FROM mcr.microsoft.com/appsvc/dotnetcore:3.1-latest_20200502.1
# allow weak certificates (certificate signed with SHA1)
# by downgrading OpenSSL security level from 2 to 1
RUN sed -i 's/SECLEVEL=2/SECLEVEL=1/g' /etc/ssl/openssl.cnf
Dockerfile for an Azure function:
FROM mcr.microsoft.com/azure-functions/dotnet:3.0.13614-appservice
# allow weak certificates (certificate signed with SHA1)
# by downgrading OpenSSL security level from 2 to 1
RUN sed -i 's/SECLEVEL=2/SECLEVEL=1/g' /etc/ssl/openssl.cnf

kubectl version returns error

I am trying to install a Kubernetes cluster on CentOS 7.3 servers. After some progress I got stuck on getting installing CNI plugin. To install plugin I need to pass a parameter which extracted from "kubectl version" command output. However command gets error when getting the required information, Server version:
[root#bigdev1 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Error from server (NotFound): the server could not find the requested resource
Actually I started using default documentation (https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/) with version kubeadm 1.7.3 (and Docker 17) but got stuck on a check:
[root#bigdev1 ~]# kubeadm init --pod-network-cidr=10.244.0.0/16
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.7.4
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks
[preflight] WARNING: docker version is greater than the most recently validated version. Docker version: 17.03.1-ce. Max validated version: 1.12
[preflight] Starting the kubelet service
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [bigdev1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.109.20]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready
(waits here forever)
Then I decreased Docker version to 1.12.6 and kubernetes version to 1.6.0
After modifying kubeadm config. Also stopped passing cidr parameter to kubeadm init.
I will be glad if you can give any suggestions to get cleared of this issue or give the result of below command:
kubectl version | base64 | tr -d '\n'
Thanks in advance.
not sure which document your following. I would recommend using the kubeadm to configure the cluster.
https://kubernetes.io/docs/setup/independent/install-kubeadm/
This should give you the result of the command:
kubectl version 2>&1| base64 | tr -d '\n'

Resources