I have an application setup on AKS (Azure Kubernetes Service) and I’m currently using Azure Application gateway as ingress resource for my application running on AKS.
Now after setting up ISTIO for my cluster the graphs are coming up fine except one part. Since the Azure APP gateway is unknown to ISTIO it is showing the resource as “unknown”. I even tried launching a virtual service and pointed it to the ingress resource but that didn’t have any effect on the graph. How shall I establish to ISTIO that it is Azure app gateway and not “unknown” resource.
This is because Azure Application gateway is not part of Istio Mesh. Depending on how You have Your Azure Application Gateway configured You might not even get any benefits of using istio.
Getting istio to work with Azure Application Gateway is lot more complicated than it seems.
There is a Github issue that uses istio and Azure Application Gateway at the same time.
With the following statement:
You may wonder why I chose to put the ingress resource into the istio-system namespace. Im doing so because in my understanding the istio-ingress must be the endpoint for each app-gateway redirect. If I would let it redirect to the echo-server service, AGKI(application-gateway-kubernetes-ingress) would point to the ip-address of the deployed pod, which would completely disregard istios servicemesh.
So if don't already have configuration like that and You want to use Istio I suggest setting Istio Ingress Gateway as an endpoint for Your Azure Application Gateway and treat it as traffic comming from outside mesh.
Here is an explanation why Azure Application gateway is "unknown" resource.
In an this article you can find the following statement:
Ingress traffic
Istio expects traffic to go via the the Ingress Gateway. When you see ‘unknown’ traffic it can simply be the case that you use the standard Kubernetes Ingress or an OpenShift route to send traffic from the outside to Istio.
Azure Application gateway uses custom ingress controller:
Application Gateway Ingress Controller (AGIC) allows you to use Application Gateway as the ingress for an Azure Kubernetes Service (AKS) cluster.
The ingress controller runs as a pod within the AKS cluster and consumes Kubernetes Ingress Resources and converts them to an Application Gateway configuration which allows the gateway to load-balance traffic to the Kubernetes pods. The ingress controller only supports Application Gateway V2 SKU.
For more information, see Application Gateway Ingress Controller (AGIC).
According to Kiali documentation:
In some situations you can see a lot of connections from an "Unknown" node to your services in the graph, because some software external to your mesh might be periodically pinging or fetching data. This is typically the case when you setup Kubernetes liveness probes, or have some application metrics pushed or exposed to a monitoring system such as Prometheus. Perhaps you wouldn’t like to see these connections because they make the graph harder to read.
To address Your additional question:
How shall I establish to ISTIO that it is Azure app gateway and not “unknown” resource.
As far as I know there is no way to make Custom (non-istio) Ingress Gateway be part of istio mesh. Leaving Azure Application Gateway labelled as “unknown”.
Hope this helps.
AFAIK, istio needs its own ingress gateway for apps.
Create an istio VirtualService and point it to istio's ingress gateway. The steps to do it are here and here.
Istio's ingress gateway for the app can be seen in the output of kubectl get gateway:
$ kubectl get gateway
NAME AGE
bookinfo-gateway 32s
Related
Our AKS and Application Gateway are in different Vnets. From the AGIC documentation, the connection between AKS and Application Gateway is through a route table, as can be seen here AGIC Github.
However, we have a requirement that the route table should not exist between these two resources. AKS is using Kubenet and we cannot change it.
Is there another way to connect AKS and Application Gateway? Thanks.
To connect AKS and Application Gateway, please try the below:
You can enable AGIC on existing AKS cluster through Azure portal:
Enable the Application Gateway ingress controller and add the created application gateway like below:
If the above suggestion doesn't work in your scenario, then please refer the below links:
Managing traffic to AKS through Azure Application Gateway using Application gateway Ingress Controller by SUDHAKARA RAO SAJJA
Enable ingress controller add-on for existing AKS cluster with existing Azure application gateway | Microsoft Docs
I would need to deploy a second Azure Load Balancer for ingress gateway of an app (to be separated from the main Load Balancer deployed in the istio's default profile).
I have tried the suggestions on GitHub (https://github.com/istio/istio/issues/19263). However, the result was actually an additonal Frontend IP Configuration for the main Load Balancer, and not an additional Load Balancer. This ends up with "ERR_SSL_PROTOCOL_ERROR" error (if curl is used: error:1408F10B:SSL routines:ssl3_get_record:wrong version number), if the same port 443 is used in both istio ingress gateways.
istio version: 1.5.1
Any suggestions on how to deploy an additional Load Balancer for the second ingress gateway? Thanks
This is a tricky configuration as it needs to have an entire new second istio ingress gateway (not just a gateway object). There is an article about this here.
This approach creates new HorizontalPodAutoscaler, Deployment, Gateway, PodDisruptionBudget, Service, ServiceAccount for the second istio ingress gateway based on the default configuration.
After modifying all the names labels You can kubectl apply the manifest to Your istio cluster . As for the Loadbalancer, new one will be attached to new istio-ingress gateway automatically.
Hope it helps.
Is anyone tried to use this bundle? Main question if it's really make sense as long main advantage of App Gateway as K8S Ingress Controller is ability to connect directly to pods, avoiding NodePort schema.
And in case of Istio Ingress Gateway we still have additional hop to pods, so L3 Azure ILB should be also fine?
I'd say that the main advantage of AGIC is not necessarily the ability to connect directly to pods but to be able to use the WAF functionality of the Application Gateway and have Microsoft support, which is sometime needed for big corpo. If you are not planning to use the WAF functionality of the Application Gateway, it don't really make sense to use AGIC instead of a L4 load balancer in front of the Istio Ingress Gateway.
I am in process to integrate Azure API gateway with AKS.
in order to achieve this.
i had
created AKS cluster with advance networking. consider it is in
aks-vnet
created different vnet for API gateway.
peer both vnet i.e. vnet for aks and for api gateway.
created internal load balancer. by
running azure voting app yaml file also added annotation for using
internal load balancer
metadata:
name: azure-vote-front
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
added kubernates internal load balancer public ip in address probe of api gateway.
when i navigate in browser with public ip address of api gateway or dns name associated with api gateway i am getting following error
502 - Web server received an invalid response while acting as a gateway or proxy server.
There is a problem with the page you are looking for, and it cannot be displayed. When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server.
First, there is something wrong described in your case: You should provide add the kubernetes internal load balancer private IP to the application gateway backend pool.
Then I did the test as the steps in Integrate Application Gateway with AKS cluster. As the error shows that you should make the check if the application in the AKS works fine, in your case the application is azure voting app.
If you follow the Azure Kubernetes Service tutorial and push the image to Azure Container Registry, you must grant the AKS permission to access your ACR. See Authenticate with Azure Container Registry from Azure Kubernetes Service.
And on my side, I can access the azure voting app from the public IP of the application gateway. Hope this will help you.
I'm porting an application that was originally developed for the AWS Fargate container service to AKS under Azure. In the AWS implementation an application load balancer is created and placed in front of the UI microservice. This load balancer is configured to use a signed certificate, allowing https access to our back-end.
I've done some searches on this subject and how something similar could be configured in AKS. I've found a lot of different answers to this for a variety of similar questions but none that are exactly what I'm looking for. From what I gather, there is no exact equivalent to the AWS approach in Azure. One thing that's different in the AWS solution is that you create an application load balancer upfront and configure it to use a certificate and then configure an https listener for the back-end UI microservice.
In the Azure case, when you issue the "az aks create" command the load balancer is created automatically. There doesn't seem be be a way to do much configuration, especially as it relates to certificates. My impression is that the default load balancer that is created by AKS is ultimately not the mechanism to use for this. Another option might be an application gateway, as described here. I'm not sure how to adapt this discussion to AKS. The UI pod needs to be the ultimate target of any traffic coming through the application gateway but the gateway uses a different subnet than what is used for the pods in the AKS cluster.
So I'm not sure how to proceed. My question is: Is the application gateway the correct solution to providing https access to a UI running in an AKS cluster or is there another approach I need to use?
You are right, the default Load Balancer created by AKS is a Layer 4 LB and doesn't support SSL offloading. The equivalent of the AWS Application Load Balancer in Azure is the Application Gateway. As of now there is no option in AKS which allows to choose the Application Gateway instead of a classic load balancer, but like alev said, there is an ongoing project that still in preview which will allow to deploy a special ingress controller that will drive the routing rules on an external Application Gateway based on your ingress rules. If you really need something that is production ready, here are your options :
Deploy an Ingress controller like NGINX, Traefik, etc. and use cert-manager to generate your certificate.
Create an Application Gateway and manage your own routing rule that will point to the default layer 4 LB (k8s LoadBalancer service or via the ingress controller)
We implemented something similar lately and we decide to managed our own Application Gateway because we wanted to do the SSL offloading outside the cluster and because we needed the WAF feature of the Application Gateway. We were able to automatically manage the routing rules inside our deployment pipeline. We will probably use the Application Gateway as an ingress project when it will be production ready.
Certificate issuing and renewal are not handled by the ingress, but using cert-manager you can easily add your own CA or use Let's encrypt to automatically issue certificates when you annotate the ingress or service objects. The http_application_routing addon for AKS is perfectly capable of working with cert-manager; can even be further configured using ConfigMaps (addon-http-application-routing-nginx-configuration in kube-system namespace). You can also look at initial support for Application Gateway as ingress here