Setup: Kubernetes cluster on AKS with nginx-kubernetes ingress. Azure Application Gateway routing domain with SSL certificate to nginx-kubernetes.
No problems serving everything in Kubernetes.
Now I moved static content to Azure Blob Storage. There's an option to use a custom domain, which works fine, but does not allow to use a custom SSL certificate. The only possible way is to set up a CDN and use the Verizon plan to use custom SSL certificates.
I'd prefer to keep all the routing in the ingress configuration, since some subroutes are directed to different Kubernetes services. Is there a way to mask and rewrite a path to the external blob storage url in nginx-kubernetes? Or is there any other available option that proxies an external url through ingress?
I don't mind having direct blob storage URLs for resources, but the main entry point should use the custom domain.
Not exactly the answer to the question, but the answer to the problem. Unfortunately this isn't documented very well. The solution is to create a service with a type of "ExternalName". According to https://akomljen.com/kubernetes-tips-part-1/ the service should look like this:
kind: Service
apiVersion: v1
metadata:
name: external-service
namespace: default
spec:
type: ExternalName
externalName: full.qualified.domain.name
I just tried it and it works like a charm.
Related
I want to add some custom ports in LoadBalancer on my Azure-OpenShift setup(It might be ARO cluster/Azure VM's with OpenShift installer).
Do we have option to edit/configure Azure loadbalancer for changing custom ports?
If yes, Which is the best way to do. For Example, editing in service manifest or yaml with my custom ports or from AZURE UI etc.
Scenario: I have a customer, where I can say please do these steps for changing ports in loadbalancer and you can have your own Ports which has to present for his Azure-OpenShift setup.
OpenShift cluster has loadbalancers for API and Ingress.
OpenShift 4.x doesn't manager the API loadbalancer so you can change configuration of the loadbalancer from Azure portal. However, on ARO, you can not change the load balancer on Azure portal because all the resources managed by Azure *1.
For the ingress loadbalancer, that is managed by Load Balancer Service. So, you should modify a yaml file on you cluster as cluster administrator. However, the service is deployed by Ingress Operator. If you removed/changed the service content, Operator may roll back your service as it is.
I couldn't find any option on the documentation *2. But you can try to modify the service as you want.
*1: https://learn.microsoft.com/en-us/azure/openshift/openshift-service-definitions#azure-resource-architecture
*2: https://docs.openshift.com/container-platform/4.11/rest_api/operator_apis/ingresscontroller-operator-openshift-io-v1.html
We are trying to set up a replacement kube cluster that has a couple of micro services on it but the majority of code still lives in a cloud service. We are using the app-gateway ingress controller as our routing mechanism. Is it possible to have an app gateway ingress controller route all requests to the external (does not live in the kub cluster) resource except those paths as defined for the micoroservices? And if so, how do we go about setting it up?
this is dotnetcore for linux with helm as the orchestration.
the layout would look something like this...
default: /* => cloud service
microservice 1: /ms1/route1 or /ms1/route2 => internal kube deployment 1
microservice 2: /ms2/route1 => internal kube deployment 2
Is it possible to have an app gateway ingress controller route all
requests to the external (does not live in the kub cluster) resource
except those paths as defined for the micoroservices?
I don't think it's possible to route the requests to the resources that outside the Kubernetes. You can take a look at What is Ingress:
Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.
internet
|
[ Ingress ]
--|-----|--
[ Services ]
I think you can route the outside requests to your microservices or your Kubernetes services through Azure Application Gateway.
it is possible. create a service with selectors or external service and point it to your cloud service and route requests from /* to it.
apiVersion: v1
kind: Service
metadata:
name: name
spec:
type: ExternalName
externalName: url-goes-here
I'm porting an application that was originally developed for the AWS Fargate container service to AKS under Azure. In the AWS implementation an application load balancer is created and placed in front of the UI microservice. This load balancer is configured to use a signed certificate, allowing https access to our back-end.
I've done some searches on this subject and how something similar could be configured in AKS. I've found a lot of different answers to this for a variety of similar questions but none that are exactly what I'm looking for. From what I gather, there is no exact equivalent to the AWS approach in Azure. One thing that's different in the AWS solution is that you create an application load balancer upfront and configure it to use a certificate and then configure an https listener for the back-end UI microservice.
In the Azure case, when you issue the "az aks create" command the load balancer is created automatically. There doesn't seem be be a way to do much configuration, especially as it relates to certificates. My impression is that the default load balancer that is created by AKS is ultimately not the mechanism to use for this. Another option might be an application gateway, as described here. I'm not sure how to adapt this discussion to AKS. The UI pod needs to be the ultimate target of any traffic coming through the application gateway but the gateway uses a different subnet than what is used for the pods in the AKS cluster.
So I'm not sure how to proceed. My question is: Is the application gateway the correct solution to providing https access to a UI running in an AKS cluster or is there another approach I need to use?
You are right, the default Load Balancer created by AKS is a Layer 4 LB and doesn't support SSL offloading. The equivalent of the AWS Application Load Balancer in Azure is the Application Gateway. As of now there is no option in AKS which allows to choose the Application Gateway instead of a classic load balancer, but like alev said, there is an ongoing project that still in preview which will allow to deploy a special ingress controller that will drive the routing rules on an external Application Gateway based on your ingress rules. If you really need something that is production ready, here are your options :
Deploy an Ingress controller like NGINX, Traefik, etc. and use cert-manager to generate your certificate.
Create an Application Gateway and manage your own routing rule that will point to the default layer 4 LB (k8s LoadBalancer service or via the ingress controller)
We implemented something similar lately and we decide to managed our own Application Gateway because we wanted to do the SSL offloading outside the cluster and because we needed the WAF feature of the Application Gateway. We were able to automatically manage the routing rules inside our deployment pipeline. We will probably use the Application Gateway as an ingress project when it will be production ready.
Certificate issuing and renewal are not handled by the ingress, but using cert-manager you can easily add your own CA or use Let's encrypt to automatically issue certificates when you annotate the ingress or service objects. The http_application_routing addon for AKS is perfectly capable of working with cert-manager; can even be further configured using ConfigMaps (addon-http-application-routing-nginx-configuration in kube-system namespace). You can also look at initial support for Application Gateway as ingress here
Question regarding AKS, each time release CD. The Kubernetes will give random IP Address to my services.
I would like to know how to bind the domain to the IP?
Can someone give me some link or article to read?
You have two options.
You can either deploy a Service with type=LoadBalancer which will provision a cloud load balancer. You can then point your DNS entry to that provisioned LoadBalancer with (for example) a CNAME.
More information on this can be found here
Your second option is to use an Ingress Controller with an Ingress Resource. This offers much finer grained access via url parameters. You'll probably need to deploy your ingress controller pod/service with a service Type=LoadBalancer though, to make it externally accessible.
Here's an article which explains how to do ingress on Azure with the nginx-ingress-controller
I have a simple meteor app deployed on kubernetes. I associated an external IP address with the server, so that it's accessible from within the cluster. Now, I am up to exposing it to the internet and securing it (using HTTPS protocol). Can anyone give simple instructions for this section?
In my opinion kube-lego is the best solution for GKE. See why:
Uses Let's Encrypt as a CA
Fully automated enrollment and renewals
Minimal configuration in a single ConfigMap object
Works with nginx-ingress-controller (see example)
Works with GKE's HTTP Load Balancer (see example)
Multiple domains fully supported, including virtual hosting multiple https sites on one IP (with nginx-ingress-controller's SNI support)
Example configuration (that's it!):
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-lego
namespace: kube-lego
data:
lego.email: "your#email"
lego.url: "https://acme-v01.api.letsencrypt.org/directory"
Example Ingress (you can create more of these):
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: site1
annotations:
# remove next line if not using nginx-ingress-controller
kubernetes.io/ingress.class: "nginx"
# next line enable kube-lego for this Ingress
kubernetes.io/tls-acme: "true"
spec:
tls:
- hosts:
- site1.com
- www.site1.com
- site2.com
- www.site2.com
secretName: site12-tls
rules:
...
There are several ways to setup a ssl endpoint, but your solution needs to solve 2 issues: First, you need to get a valid cert and key. Second, you would need to setup a ssl endpoint in your infrastructure.
Have a look at k8s ingress controller. You can provide an ingress controller with a certificate/key secret from the k8s secret store to setup a ssl endpoint. Of course, this requires you to already have a valid certificate and key.
You could have a look at k8s specific solutions for issuing and using certificates like the Kubernetes Letsencrypt Controller, but I have never used them and cannot say how well they work.
Here are some general ideas to issue and use ssl certificates:
1. Getting a valid ssl certificate and key
AWS
If you are running on AWS, the easiest way I can think of is by setting up an ELB, which can issue the ssl cert automatically for you.
LetsEncrypt
You could also have a look at LetsEncrypt to issue free certificates for your domain. Nice thing about it is that you can automate your cert issuing process.
CA
Of course, you could always go the old-fashion way and issue a certificate from a provider that you trust.
2. Setting up the ssl endpoint
AWS
Again, if you have an ELB then it already acts as an endpoint and you are done. Of course your client <-> ELB connection is encrypted, but ELB <-> k8s-cluster is unencrypted.
k8s ingress controller
As mentioned above, depending on the k8s version you use you could also setup a TLS ingress controller.
k8s proxy service
Another option is to setup a service inside your k8s cluster, which terminates the ssl connection and proxies the traffic to your meteor application unencrypted.
You could use nginx as a proxy for this. In this case I suggest you store your certificate's key inside k8s secret store and mount it inside the nginx container. NEVER ship a container which has secrets such as certificate keys stored inside! Of course you still somehow need to send your encrypted traffic to a k8s node - again, there several ways to achieve this... Easiest would be to modify your DNS entry to point to the k8s nodes, but ideally you would use a TCP LB.