On Azure I have 2 vnet's (with peerings):
On first I have public LoadBalancer
On second - private AKS cluster with deployments and Nginx Ingress installed. From first vnet Ingress is accessible by private IP and private DNS name
I need to configure LoadBalancer to access Ingress from Internet. Tried to configure LoadBalancer inbound/NAT rules (Public IP -> Ingress IP inside second vnet), but it's still not visible
Is it possible when resources are in different vnet's?
Can it be configured only on LoadBalancer? Or I need to have additional route table/privatelinks between vnet's?
Related
On Azure I placed private AKS cluster with private LoadBalancer (LB).
From any pod (I have only deployment and service with private IP using private LB, no ingress etc) I can't access any site from public internet, and I need to do it avoiding Azure Firewall that is too expensive
Which resources can help me to enable outbound access without making private cluster public?
I don't know how to implement this problem:
I have this private AKS cluster with 4 microservices (.net5) and a frontend. These 4 microservices talk to each other via HTTP requests using their public IP addresses (not good because I want them to have only a private endpoint, like microservicename.api.my-namespace.svc.cluster-domain.example).
Frontend (that has a public DNS and IP) should then be able to request a main API in that private endpoint.
I need to implement a solution to this, and also I feel like that communicating between microservices via HTTP requests on a certain endpoint is not a good design pattern, so I wanted some suggestions on this aspect also.
Many thanks
• You can use the Azure Private Link Service in this case to establish communication between the private AKS cluster and the frontend through the private endpoint that will be exposed in the subnet of the private AKS cluster.
• The Private Link service is supported on Standard Azure Load Balancer only. Basic Azure Load Balancer isn't supported. To use a custom DNS server, add the Azure DNS IP 168.63.129.16 as the upstream DNS server in the custom DNS server.
• You can create a private AKS cluster consisting of the four microservices using the below command with a private DNS zone or a custom private DNS zone: -
‘az aks create -n -g --load-balancer-sku standard --enable-private-cluster --enable-managed-identity --assign-identity --private-dns-zone [system|none]’ – with a private DNS zone (system for default value & none for public DNS zone)
‘az aks create -n -g --load-balancer-sku standard --enable-private-cluster --enable-managed-identity --assign-identity --private-dns-zone --fqdn-subdomain ’
• If the Private DNS Zone is in a different subscription than the AKS cluster, you need to register Microsoft.ContainerServices in both the subscriptions. Additionally, you will need a user assigned identity or service principal with at least the private dns zone contributor and vnet contributor roles. "fqdn-subdomain" can be utilized with "CUSTOM_PRIVATE_DNS_ZONE_RESOURCE_ID" only to provide subdomain capabilities to privatelink..azmk8s.io
• Next, you would need to use a VM that has access to the AKS cluster's Azure Virtual Network (VNet). There are several options for establishing network connectivity to the private cluster such as create a VM in the same Azure Virtual Network (VNet) as the AKS cluster or use a VM in a separate network and set up Virtual network peering. Thus, you can create a private link with a private endpoint for the four microservices in the private AKS cluster and provide connection to the frontend API.
Please refer the below link for more details: -
https://learn.microsoft.com/en-us/azure/aks/private-clusters
I deployed an application in an Azure K8S cluster, using NGINX as gateway, with a public static IP, based on AKS & PUBLIC-IP and on AKS & NGINX.
Now I need to deploy the application in an Azure private cluster, ie, running in a private vnet (see CREATE PRIVATE AKS); attempting to assign a public static IP to NGINX does not work, which can be expected as the load-balancer expects a private IP, not a public IP.
How can I provide inbound access to my app hosted in a private cluster, using NGINX and a public static IP?
Hi you have two ways two achieve that...Depending on your needs (and Azure costs...):
1-Use Azure Application Gateway. For myself I use Terraform. And here you can the see official documentation regarding internal IP address.
Now you can use this one as your new Ingress (and get rid of NGINX) like:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: guestbook
annotations:
kubernetes.io/ingress.class: azure/application-gateway
spec:
rules:
- http:
paths:
- backend:
serviceName: frontend
servicePort: 80
Or you could use NGINX internally as your ingress like explained on option 2.
2- First you must have a Public IP with a Load Balancer associated with it.The backend from that LB must be up to your needs.
But here is the trick...Do not create NGINX with that public IP but with an internal IP and an internal load balancer, you can see how to do that in the following url:
https://learn.microsoft.com/en-us/azure/aks/ingress-internal-ip
And the important thing you must do is the nginx ovveride on the helm parameters:
controller:
service:
loadBalancerIP: 10.240.0.42
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
Of course the internal VNET must be created an the load balancer IP must be a correct one.
And the final trick now that you have NGINX listening behind a private IP is to verify your traffic from the Public IP is redirected to that internal VNET...Of course it depends on how you have infrastructure setup behind that LB that holds the public IP.
As stated in the comment above you can do the same via Application Gateway in Azure. But if you are going to only use AKS you might want to just use Application Gateway as your ingress controller which is already created with the private cluster.
Please follow this to achieve the same https://microsoft.github.io/AzureTipsAndTricks/blog/tip256.html
Based on your description i understand that you want to have ingress traffic through your NGINX ingress controller which has a Loadbalancer service with static IP. If your deployment is correctly configured the a Loadbalancer service should be assigned to your NGINX ingress controller with a public IP. Since i dont know your namespaces, naming of deployments etc try:
kubectl get services --all-namespaces | grep -i loadbalancer
You should be able to find that an nginx loadbalancer service has a public IP. Now since NGINX is your ingress controller this means that you have a Layer 7 loadbalancer as ingress so you need to create an ingress route to your application running in AKS. This is documented here from Azure NGINX ingress but also here Ingress K8s
I created AKS with internal ingress Nginx. This comes up like below in the cluster.
Then I created Azure private DNS Service. In the Azure private DNS service, I created a 'Record set' like
Blockquote
Technically, i should be able to access LoadBalancer External ip with promotion.mydomain.com (as example). Insted of this, I'm having '502 Bad Gateway' error when i hit http://promotion.mydomain.com in the browser. Any advice?
I faced the same issue and have been able to solve it recently.
I created another Ingress but in the desired namespace (mine was default) with the following definition :
(I have enabled tls but you can remove that part)
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
name: my-custom-ingress
spec:
tls:
- hosts:
- foo.mydomain.com
secretName: my-tls-secret
rules:
- host: foo.mydomain.com
http:
paths:
- path: /
backend:
serviceName: my-foo-app-service-nodeport
servicePort: 4444
First of all find the EXTERNAL IP of your nginx ingress and keep it in mind:
kubectl get svc --namespace ingress-basic
Then in the Azure DNS zone you can attach the domain to an Azure Resource :
Open azure portal.
Go in the MC_... resource group created by your AKS cluster.
Find the LoadBalancer resource and click it.
On the LoadBalancer, go into "Frontend IP Configuration". You'll then see a list of public IP with a related ResourceId (example: 11.22.33.44 (xxx-yyyy-bbb))
Find the IP that is corresponding to the LoadBalancer IP you found on the LoadBalancer (before step 1) and memorize the associated object id.
Open you Azure DNS zone and create new domain (or edit one)
Set "Alias Record Set: Yes" then "Alias type: Resource"
Under "Azure Resource" find the resource that has the ResourceId you found in step 5 and select it.
Save
Now it should work.
I see your purpose is to create AKS with internal Ingress Nginx and use the custom DNS. And I see your Ingress external IP is 10.240.0.42. It seems it's a private IP of the subnet which you AKS nodes in.
So I think you need to create An Azure Application Gateway or Azure Load Balancer to route your request from the Internet to your internal Ingress Nginx interface. And the A record also needs to be changed, you need to change the IP into the public IP of the one which you choose from Azure Application Gateway and Azure Load Balancer. I think you know you need to update your custom DNS setting in the DNS server which you DNS in.
When all things are being done. The requests routing path will like this:
Internet ( your custom DNS)
Azure DNS Server
Azure Public IP of the Application Gateway or Load Balancer ( this is what I think you missed)
10.240.0.42 ( ingress Nginx internal IP)
AKS Ingress Nginx
Service
Deployment or Pod
I want to set up a kubernetes cluster with a loadbalancer. Kubernetes will create a load balancer in azure and connect a public ip address with it.
But I don't want to make the api public, it should be exclusive for my api management service.
I tried to direct the load balancer into a vnet with the api service but I found nothing.
So I thought I could just limit the access to the public ip (a whitelist with only the incluced ip of my service) but I found nothing on the internet.
Is it possible to set such rule on a public ip or do I need some extra service for this problem?
With Kubernetes, assuming you have a service defined
Use the following commands:
kubectl get service
kubectl edit svc/<YOUR SERVICE>
change the type from LoadBalancer to ClusterIP
Now you can consume internally the service.