Setup internal and public workload on a single AKS cluster - azure

Question

I have a single AKS cluster.
I would like some workloads being exposed over the internet while other must remain reachable only from specific public IPs.
I can think of two solution that may work, but both of them appears a bit tricky to me :
Usage of Azure CNI and set NSG at pod's CNI level ?
Create an internal and a public node pool and setting NSG at vmss's NIC level ?
Is there a better option I could investigate? Is one of those two points better than the other ?

Answer 1

If you need complete workloads isolation you may consider option of usage Azure CNI and set NSG at pod's CNI level but you should aware that
The subnet assigned to the AKS node pool cannot be a delegated subnet.
AKS doesn't apply Network Security Groups (NSGs) to its subnet and will not modify any of the NSGs associated with that subnet. If you provide your own subnet and add NSGs associated with that subnet, you must ensure the security rules in the NSGs allow traffic within the node CIDR range.
Use Azure Network Policy and define your own virtual network and subnets that can be done only when you create an AKS cluster.
Microsoft also recommends an option to add node pool with a unique subnet to achieve isolation.
This isolation can be supported with separate subnets dedicated to each node pool in the cluster. This can address requirements such as having non-contiguous virtual network address space to split across node pools.
Limitations:
All subnets assigned to node pools must belong to the same virtual network.
System pods must have access to all nodes/pods in the cluster to provide critical functionality such as DNS resolution and tunneling kubectl logs/exec/port-forward proxy.
If you expand your VNET after creating the cluster you must update your cluster before adding a subnet outside the original cidr.
In clusters with Kubernetes version < 1.23.3, kube-proxy will SNAT traffic from new subnets, which can cause Azure Network Policy to drop the packets.
Windows nodes will SNAT traffic to the new subnets until the node pool is reimaged.
Internal load balancers default to one of the node pool subnets (usually the first subnet of the node pool at cluster creation).
Another option is to use Calico Network Policies, an open-source network and network security solution
If you need just some workloads being exposed over the internet while other must remain reachable only from specific public IPs you can run Multiple NGINX Ingress Controllers in AKS and make a whitelisting with annotation nginx.ingress.kubernetes.io/limit-whitelist
https://blog.cpolydorou.net/2022/05/running-multiple-nginx-ingress.html