I have recently started learning and implementing istio in AWS EKS cluster. For configuring TLS for ingress gateway, I followed this guide which simply asks you to add AWS ACM ARN id to istio-ingressgateway as an annotation. So, I had to neither use certs to create secret nor use envoyproxy's SDS.
This setup terminates TLS at gateway, but I also want to enable mTLS within mesh for securing service-service communication. By following their documentation, I created this policy to enforce mTLS within a namespace:
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: xyz-mtls-policy
namespace: xyz-dev
spec:
mtls:
mode: STRICT
But even after applying this, I see one service being able to call another service using http.
So my question is: how do I use the ACM certs to implement mTLS in my namespace?
If you're calling from inside the mesh I would say it's working fine, take a look here and here.
Mutual TLS in Istio
Istio offers mutual TLS as a solution for service-to-service authentication.
Istio uses the sidecar pattern, meaning that each application container has a sidecar Envoy proxy container running beside it in the same pod.
When a service receives or sends network traffic, the traffic always
goes through the Envoy proxies first.
When mTLS is enabled between two services, the client side and server side Envoy proxies verify each other’s identities before sending requests.
If the verification is successful, then the client-side proxy encrypts the traffic, and sends it to the server-side proxy.
The server-side proxy decrypts the traffic and forwards it locally to the actual destination service.
I am on istio 1.6.8, think it enables mTLS by default.
Yes, it's enabled by default since istio 1.5 version. There are related docs about this.
Automatic mutual TLS is now enabled by default. Traffic between sidecars is automatically configured as mutual TLS. You can disable this explicitly if you worry about the encryption overhead by adding the option -- set values.global.mtls.auto=false during install. For more details, refer to automatic mutual TLS.
Is there any clear process to prove that it is indeed using mTLS?
I would say there are 3 ways
Test with pods
You can change it from strict to permissive and call it from outside the mesh, it should work. Then change it to strict and call it again, it shouldn't work. In both ways you should be able to call it from a pod inside the mesh.
Kiali
If you want to see it visual way kiali should have something like a padlock when mtls is enabled, there is github issue about that.
Prometheus
It was already mentioned in the banzaicloud, and you mentioned that in the comments, you can check the Connection Security Policy metric label. Istio sets this label to mutual_tls if the request has actually been encrypted.
Let me know if have any more questions.
Related
Background
I have a system running on Azure iot edge. The system is composed of multiple modules that expose REST interfaces. To make everything look tidy from the client's perspective (a browser on another machine in the same network) we use an Azure Function and its reverse proxy capabilities.
So, basically, the client makes a request to an endpoint of the function, if the route matches one in the "proxies" config, it is routed to the correct module using the docker network provided by the iot edge product.
Problem
Now, what I would like to accomplish is that the client would use an https connection to make the request to the function.
So the browser would make a request to https://:8000/Somemodule/Resource and this request would be routed by the af proxy to http://Somemodule:80/Resource .
So my question is, how do I enable https in a function running locally in a docker container, and can the reverse proxy work as described above?
Thanks for any help!
For HTTPS, you primarily need a SSL certificate and reverse proxy like nginx that can do SSL Termination since I believe Azure Functions doesn't support it as part of the runtime itself (which is what the docker container has).
Nginx is a popular and fairly common choice to use for SSL Termination. You would have to configure it for SSL with your domain information and setup your Azure Function as its upstream.
That being said, you could actually just use nginx as your proxy directly too, completely removing the need for Azure Functions, unless you are using it for Functions and Proxies. Your current proxy entries would just become an upstream definition for each module and separate locations (basically path) that would route the requests.
I have an AKS cluster running running Internal nginx ingress + cert-manager which generates lets encrypt for ssl terminations.
I would like to include application gateway as an entry point, where I expect that SSL internet traffic hits Applicatiom Gateway and traffic is forwarded to the nginx ingress, then to my application. I do not mind if SSL offloading is done at Appgw level or on the AKS cluster itself.
One of my biggest headaches is that Application gateway requires a certificate when an https listener is created. Since the certifiate is generated automatically on the AKS cluster I do not see the benefit of supplying an SSL certificate to the Application Gateway neither do I want to go through the extra work of generating a certificate and storing it keyvault etc etc.
What is the neatest way to tackle this problem? Potential solutions I have considered are
Configure Application Gateway to passthrough SSL to the AKS cluster
Somehow configure cert-manager to store the certificate in keyvault
The only options I see are (but I like neither are)
Purchase a certificate and store it in keyvault (however I prefer using Lets Encrypt)
Generate the SSL certificate on a cluster and then write a script which scrapes the certificate and stores it in Azure Key Vault
Any help will be appreciated
As per this tutorial here you can use cert-manager an AKS add-on that automates the creation and management of certificates.
You can also go through this tutorial which uses Azure automation runbook to automate certificate rotation for ApGw.
Since the above hasn't solution hasn't really helped me, I had decided to write an aks cron job which syncs certificates to azure keyvault.
If anyone is interested, I would be able to open source it.
I am working on deploying an API solution on GCP where mutual SSL/TLS is required (server and client side certificates). So for the ingress of the traffic (entry point) I found that kubernetes ingress controller has this possibility (NGINX based). I am interested by cloud endpoints which has ESP (extensible service proxy which is also nginx deployment under kubernetes).
I couldn't find anywhere in the documentation whether mutual SSL/TLS is available for ESP (cloud endpoint), does anyone know the answer for this ?
This might be possible using Istio. Have you come across following article? which seems to suggest how to achieve MTLS for Endpoints.
https://istio.io/docs/examples/platform/endpoints/
ESP supports mTLS. You can specify the certificates files here
proxy_ssl_certificate /etc/nginx/ssl/backend.crt;
proxy_ssl_certificate_key /etc/nginx/ssl/backend.key;
Here is its nginx config
I have created a NodeJS application using http/2 following this example:
Note: this application uses self-signed certificate until now.
We deployed it on GKE, and it is working until now.
Here is how this simple architecture looks like:
Now, we want to start using real certificate, and don`t know where is the right place to put it.
Should we put it in pod (overriding self-signed certificate)?
Should we add a proxy on the top of this architecture to put the certificate in?
In GKE you can use a ingress object to routing external HTTP(S) traffic to your applications in your cluster. With this you have 3 options:
Google-managed certificates
Self-managed certificates shared with GCP
Self-managed certificates as Secret resources
Check this guide for the ingress load balancing
The Client's SSL session terminates at the LB level, the self-signed certificates being used are just to encrypt communication between the LB and the Pods. So if you want the client to use your new valid certificate it needs to be at the LB level.
On a side note, having your application servers communicate with the LoadBalancer over HTTP will give you a performance boost. Since the LB is acting as a reverse proxy anyway.
You can read this article about LoadBalancing it's written by the author of HAProxy
I'm settings up a Service Fabric cluster in Azure and want to run a web API (using .NET Core) over https. I want to use my CA-signed wildcard (*.mydomain.com) certificate to access this API. But I'm a bit confused as to where I use it when I create the cluster, is it the cluster or client certificate? I'm thinking the client certificate, but the documentation states that this is for admin tools (i.e. the Explorer), so I'm unsure on how to proceed.
And yes I've read a ton of posts and resources, but I still find this confusing.
There are three certificate types. Here is a summary overview of them.
The Cluster certificate is used for the Explorer endpoint and is deployed to the primary nodes. So if you add your *.mydomain.com wildcard cert there, and CNAME something (e.g. manage.mydomain.com) to [yourcluster].[region].cloudapp.azure.com, then when you hit your management endpoint that cert will be what will be presented to the web browser.
The Reverse Proxy SSL certificate is deployed to each of the nodes and is used when using the built-in reverse proxy feature of Service Fabric. In this case this is what is being used when you hit https://api.mydomain.com/YourAppName/YourService/Resource (where api.mydomain.com is another CNAME to yourcluster.region.cloudapp.azure.com). This is used as an alternative to running your own reverse proxy or other offloading layer (Application Gateway, IIS, nginx, API Management, etc).
The Client certificates are used in place of Azure Active Directory authentication to the management endpoint. So instead of managing users in AAD (with the _Cluster AAD application and the Admin / Read-Only roles), you manage access by handing out management certificates (Admin or Read-Only) to your trusted users.
You can also have secondaries these certificates to use in certificate rollover situations.
The way we are using it is to have Application gateway configured in front of service fabric cluster, and web certificate is uploaded to Application Gateway (and dns is pointing to application gateway) In that scenario SSL is terminated at application gateway.
Another possibility is to terminate SSL at each node in service fabric cluster, in this scenario you would need to ensure that certificate gets deployed to each of the nodes.
As for cluster vs client certificate dilemma, I am also confused, but I think the answer is neither. Client certificate is not for sure since this certificate is used to identify you as admin when running service fabric admin ps scripts.
I do not think it is cluster certificate either, here is what MS docs say what it is used for:
Cluster and server certificate is required to secure a cluster and prevent unauthorized access to it. It provides cluster security in two ways:
Cluster authentication: Authenticates node-to-node communication for cluster federation. Only nodes that can prove their identity with this certificate can join the cluster.
Server authentication: Authenticates the cluster management endpoints to a management client, so that the management client knows it is talking to the real cluster. This certificate also provides an SSL for the HTTPS management API and for Service Fabric Explorer over HTTPS.
as far as I am reading into it, this certificate is used for internal cluster authentication, and it is also used so your management tools can be asured that they are working with right cluster.