I have an aks cluster with a single nodepool with azure cni network plugin and a subnet of /21 around 2000 ip's dedicated for this single nodepool. This nodepool is configured with 3 availability zones.
Consider I have another cluster with a single nodepool with the configuration same as above but for the nodepool availability zone is set to none.
Workloads are the same in both the nodepools. but will the nodepool deployed in 3 availability zones consume more ip's compared to the one deployed in the availability zone set to None
AZ is for high Availability separated by a physical boundary within the region, Yes, It will consume separate IP's for each Node Pool. Selecting more than one availability zone will create same number of node pool and Consume same number of IP's.
I have tested in my environment with 3 AZ it occupied my 3 Ip's address from the selected subnet.
Related
While migrating a cluster we moved the vnet used by the AKS from one resource group (the one with the nodepool created by the AKS) to a different RG where we created the AKS cluster. This however, resulted in an unexpected state. The original vnet in the nodepool resource group stayed as is while it copied the vnet in to the AKS RG with the same ID. So now we have to vnet with the same name but in two different resource groups. Afterwards when we tried to create a new nodepool we received the following error:
Code="VMScaleSetMustBelongToSameVnetAsLB" Message="VM scale set
references virtual network
/subscriptions/12345/resourceGroups/project-test-k8s-mc-rg/providers/Microsoft.Network/virtualNetworks/AKS-VNET-931
which is different than load balancer virtual network
/subscriptions/12345/resourceGroups/project-test-k8s-rg/providers/Microsoft.Network/virtualNetworks/AKS-VNET-931. VM scale set and load balancer must belong to same virtual network."
The cluster was created with a managed vnet.
We tried searching for ways to change the load balancer created by AKS to use a different vnet, we do not see any options. We cannot afford to recreate the cluster at this stage. So do we have any other options to fix this issue?
There was no direct option to change the load balancer created by AKS to use a different VNet. If the load balancer uses an IP address in a different subnet, ensure the AKS cluster identity also has read access to that subnet. The VM scale set and load balancer must always belong to the same virtual network.
We can modify only address space and subnet only. Found one blog by "Ajay Kumar" refer tutorial for more information.
I have a single AKS cluster.
I would like some workloads being exposed over the internet while other must remain reachable only from specific public IPs.
I can think of two solution that may work, but both of them appears a bit tricky to me :
Usage of Azure CNI and set NSG at pod's CNI level ?
Create an internal and a public node pool and setting NSG at vmss's NIC level ?
Is there a better option I could investigate? Is one of those two points better than the other ?
If you need complete workloads isolation you may consider option of usage Azure CNI and set NSG at pod's CNI level but you should aware that
The subnet assigned to the AKS node pool cannot be a delegated subnet.
AKS doesn't apply Network Security Groups (NSGs) to its subnet and will not modify any of the NSGs associated with that subnet. If you provide your own subnet and add NSGs associated with that subnet, you must ensure the security rules in the NSGs allow traffic within the node CIDR range.
Use Azure Network Policy and define your own virtual network and subnets that can be done only when you create an AKS cluster.
Microsoft also recommends an option to add node pool with a unique subnet to achieve isolation.
This isolation can be supported with separate subnets dedicated to each node pool in the cluster. This can address requirements such as having non-contiguous virtual network address space to split across node pools.
Limitations:
All subnets assigned to node pools must belong to the same virtual network.
System pods must have access to all nodes/pods in the cluster to provide critical functionality such as DNS resolution and tunneling kubectl logs/exec/port-forward proxy.
If you expand your VNET after creating the cluster you must update your cluster before adding a subnet outside the original cidr.
In clusters with Kubernetes version < 1.23.3, kube-proxy will SNAT traffic from new subnets, which can cause Azure Network Policy to drop the packets.
Windows nodes will SNAT traffic to the new subnets until the node pool is reimaged.
Internal load balancers default to one of the node pool subnets (usually the first subnet of the node pool at cluster creation).
Another option is to use Calico Network Policies, an open-source network and network security solution
If you need just some workloads being exposed over the internet while other must remain reachable only from specific public IPs you can run Multiple NGINX Ingress Controllers in AKS and make a whitelisting with annotation nginx.ingress.kubernetes.io/limit-whitelist
https://blog.cpolydorou.net/2022/05/running-multiple-nginx-ingress.html
When deploying AKS cluster into different availability zones ("1,2,3" in our case) the vm scaleset is used for default nodepool deployment (not availability set). Everything is pretty fine there, but the problem is - while using the default nodepool scaleset, it is put into 1 fault domains only, and i did not find a way to change that (despite the fact, that the vm scaleset should be deployed into 5 fault/update domains as per documentation):
Why is it so ? How to put the nodepool into default 5 fault/update domains in addition to 3 availability zones (i mean 5 fault/update domains in each of the 3 availability zones)?
P.S. - You can always deploy AKS cluster's nodepool into availabilitySet, and have 5 update/fault domains, but then the availability zones are not available when using the availabilitySet.
By reference to this doc: Choosing the right number of fault domains for virtual machine scale set, for the regions that support zonal deployment of virtual machine scale sets and this option is selected, the default value of the fault domain count is 1 for each of the zones.
You can also consider aligning the number of scale set fault domains with the number of Managed Disks fault domains.
Following are my queries:
How to simulate Availability Zone Outage on Azure Kubernetes Service to verify incoming traffic switching is done to another zone ?
Is there any way to figure out which Zone or which Node is currently receiving the incoming requests on Azure Kubernetes Service ?
It's not clear what you mean by verify incoming traffic switching. There is no traffic switching, the traffic will go to the pods that are currently available to respond to the request. You are responsible for distributing the replicas across the different Availability Zone in your cluster. if for example you have deploy a Pod with two replicas, one in zone 1 and another in zone 2, the traffic will go to both replicas, if zone 1 goes down, the Load Balancer will only send the traffic to the pod in zone 2 until the pod in zone 1 is back.
One easy way to test it is to stop or restart the VMs in one Availability Zone and check if you experience down time.
Like I said, it is not one zone at a time, you need to deploy replicas in all the Availability Zone if you want zone-resiliency and this it is not done automatically by AKS. One way to do it is by configuring podAntiAffinity on your Pod to make sure the replicas are not deployed on the same node and in the same Availability Zone. In AKS, the nodes have a label topology.kubernetes.io/zone which contains the zone number, you can use that label in the podAntiAffinity rule. The following command will give you the list of nodes with the different Zones :
kubectl get nodes -o custom-columns=NAME:'{.metadata.name}',REGION:'{.metadata.labels.topology\.kubernetes\.io/region}',ZONE:'{metadata.labels.topology\.kubernetes\.io/zone}'
NAME REGION ZONE
aks-nodepool1-34917322-vmss000000 eastus eastus-1
aks-nodepool1-34917322-vmss000001 eastus eastus-2
aks-nodepool1-34917322-vmss000002 eastus eastus-3
When I create kubernetes cluster from kubernetes service of azure in the networking tab , I have the option to create my own virtual network.
Let's say I created a virtual network with 3 subnets , still in the networking tab options , I can only associate 1 of these subnet to my cluster.
Is it a restriction in AKS?
If so, why it allows to create more than 1 subnet in virtual network?
Not sure but you only can specific one subnet when you create an AKS cluster on the Azure portal. It seems a restriction in AKS. Read the prerequisites. At least, you need one subnet, one AKS cluster. Also, AKS supports a single pool for now.
Don't create more than one AKS cluster in the same subnet.
With advanced networking in AKS, you can deploy a AKS cluster in a existing virtual network and define these subnet names and IP address ranges. IP addresses for the pods and the cluster's nodes are assigned from the specified subnet within the virtual network. So you need to plan IP addressing for your cluster. You should consider doing upgrade and scaling operations when you determine the number of IP addresses.
The reason it allows to create more than 1 subnet in virtual network is that essentially you could create many subnets with valid CIDR block in a VNet. You can create VMs in other subnets or create a dedicated Gatewaysubnet used for VPN gateway in the existing VNet. With advanced networking, this existing virtual network often provides connectivity to an on-premises network using Azure ExpressRoute or Site-to-Site VPN.
Also, it's welcome to give feedback on Azure AKS to improve this feature. Hope this helps.