I have an Azure Kubernetes Cluster Running with Azure CNI (virtual network) as the Network. The cluster is running on 1 subnet of the network.
On another subnet, I have a Virtual Machine running as it has a private IP of 10.1.0.4.
Now I have a pod in the K8S cluster, which is trying to connect with the Virtual Machine. But it's not able to do so.
Also, the ping 10.1.0.4 from inside the pod gives a timeout.
Please help me to figure out, what I am doing wrong so that I can connect the Pod with the VM.
• You cannot directly create communication between an AKS cluster pod and a Virtual Machine as the IP assigned to a pod/node in an AKS cluster is a subset range of the address space of the higher CIDR IP address range assigned while deploying the cluster. And communication within the cluster between the nodes is uninterrupted and possible readily. But the same with resources other AKS is restricted as they are governed by Azure CNI framework policy which directs the Kubernetes cluster to direct traffic outbound of the cluster in a regulated and conditional way.
• Thus, the above said can only be achieved by initiating intermediate services such as an internal load balancer between the AKS and the VMs as the CIDR of the VM and the AKS is different. So, leveraging the Azure plugin to deploy an internal load balancer as a service through AKS is only way through which you can achieve communication between AKS pod and a VM deployed in Azure. Below is a diagram for illustration purposes.
To deploy the internal load balancer through YAML files in AKS for external communication with VMs, kindly refer to the link below for details: -
https://fabriciosanchez-en.azurewebsites.net/implementing-virtual-machine-to-pod-communication-in-azure-kubernetes-service-aks/
Related
I have AZ VM window which installed MSSQL
I created cluster in AZ K8S, then create the pod with image - application run with embedded tomcat using MSSQL connect to private IP of the above VM. The container in the pod start with error: can not connect to that private IP of MSSQL
I can access to that private IP from my local machine (using VPN), so is there any way/config to make pod can connect to that VM using private IP ? since it's same infrastructure, I dont why it cannot connect
(I am newbie with Azure)
Thanks alot
For your requirement, I don't know how do you deploy the VM and the AKS cluster. So I give the solutions for the two situations:
AKS cluster with the network type kubelet:
VM in VNet A and AKS in VNet B
create a service with the internal load balancer for the pod, and then peer the VNet A and B
VM and AKS in the same VNet
create a service with the internal load balancer for the pod
AKS cluster with the network type CNI
VM in VNet A and AKS in VNet B
peer the VNet A and B
VM and the AKS in the same VNet
you don't do anything, it should work
All of the above solutions need you to check the NSG rules between the VM and the AKS cluster pod. You need to allow the inbound traffic to the VM with the MSSQL port.
I am trying to establish an IKEv2 VPN between one VM(subnet: 20.25.0.0/16) and one AKS cluster(subnet: 10.0.0.0/16 - Azure CNI) using strongswan gateway. I need to access some kubernetes services behind of this AKS cluster. With Azure CNI each pod will be assigned an IP address from the POD subnets specified at cluster creation, this subnet is attached in interface eth0 for each node. Already kubernetes services of the type clusterIP will get an IP from service CIDR range specified at cluster creation, but this IP is only available in the cluster is not attached in any interface of the nodes, like POD subnet.
To run the strongswan on K8S it's necessary to mount the kernel modules(/lib/modules), in addition to enable NET_ADMIN capabilities. So the VPN tunnel is established using any of the networks attached on the host(nodes) interface, so I can't established a VPN using service CIDR range specified at cluster creation, since this IPs is known only within the cluster, through personalized routes and is not attached on any host interface. If I try to configure the VPN established with a subnet with the CIDR range of services informed in the creation of the cluster, I get an error stating that the subnet was not found in any of the interfaces.
To get around this, I realized that I can configure a tunnel informing a subnet with a larger range, as long as there is a subnet attached in my interface that is within the wider informed range. For example, I can configure a VPN informing the subnet 10.0.0.0/16, but my subnet for pods and nodes (attached in eth0) is 10.0.0.0/17 and CIDR range for services is 10.0.128.0/17, in this way all traffic 10.0.0.0/16 is routed through the vpn tunnel. In this way, as a workaround I define my services CIDR as a network subsequent to the network of pods and nodes and configure the VPN using a network that overlaps the two.
All 10.0.0.0/16 traffic from one side of the VPN (VM) is correctly routed to inside tunnel. If I try to access a Pod directly, using any IP from the Pods subnet (10.0.0.0/17), everything works fine. The issue is if I try to access a kubernetes service using a IP from CIDR for services(10.0.128.0/17), the traffic is not routed correctly until the K8S services. I can see the request in tcpdump in AKS, but it doesn't arrive in the service. So my question is, how to make a configuration on the strongswan, in which I can access the services on the aks kubernetes cluster?
Below is the current configuration of the strongswan:
PEER-1(VM)
conn %default
authby="secret"
closeaction="restart"
dpdaction="restart"
dpddelay="5"
dpdtimeout="10"
esp="aes256-sha1-modp1536"
ike="aes256-sha1-modp1024"
ikelifetime="1440m"
keyexchange="ikev2"
keyingtries="1"
keylife="60m"
mobike="no"
conn PEER-1
auto=add
leftid=<LEFT-PHASE-1-IP>
left=%any
leftsubnet=20.25.0.0/16
leftfirewall=yes
leftauth=psk
rightid=<RIGHT-PHASE-1-IP>
right=<RIGHT-PHASE-1-IP>
rightsubnet=10.0.0.0/16
rightfirewall=yes
rightauth=psk
PEER-2(AKS)
conn %default
authby="secret"
closeaction="restart"
dpdaction="restart"
dpddelay="5"
dpdtimeout="10"
esp="aes256-sha1-modp1536"
ike="aes256-sha1-modp1024"
ikelifetime="1440m"
keyexchange="ikev2"
keyingtries="1"
keylife="60m"
mobike="no"
conn PEER-2
auto=start
leftid=<LEFT-PHASE-1-IP>
left=%any
leftsubnet=10.0.0.0/16
leftfirewall=yes
leftauth=psk
rightid=<RIGHT-PHASE-1-IP>
right=<RIGHT-PHASE-1-IP>
rightsubnet=20.25.0.0/16
rightfirewall=yes
rightauth=psk
Our Azure kubernetes cluster is configured with Azure CNI for networking which uses the subnet with CIDR: /21.
As we are planning to deploy Istio service mesh and additional sidecars for log shipping, how would those impact the available IPs? Would those consume the IPs? If so, how to avoid the IP congestion?
Kubernetes allocates a single IP per pod, so no matter how many sidecars you have, a single pod will only have a single IP. Basically, you dont need to do anything in this regard
When I create kubernetes cluster from kubernetes service of azure in the networking tab , I have the option to create my own virtual network.
Let's say I created a virtual network with 3 subnets , still in the networking tab options , I can only associate 1 of these subnet to my cluster.
Is it a restriction in AKS?
If so, why it allows to create more than 1 subnet in virtual network?
Not sure but you only can specific one subnet when you create an AKS cluster on the Azure portal. It seems a restriction in AKS. Read the prerequisites. At least, you need one subnet, one AKS cluster. Also, AKS supports a single pool for now.
Don't create more than one AKS cluster in the same subnet.
With advanced networking in AKS, you can deploy a AKS cluster in a existing virtual network and define these subnet names and IP address ranges. IP addresses for the pods and the cluster's nodes are assigned from the specified subnet within the virtual network. So you need to plan IP addressing for your cluster. You should consider doing upgrade and scaling operations when you determine the number of IP addresses.
The reason it allows to create more than 1 subnet in virtual network is that essentially you could create many subnets with valid CIDR block in a VNet. You can create VMs in other subnets or create a dedicated Gatewaysubnet used for VPN gateway in the existing VNet. With advanced networking, this existing virtual network often provides connectivity to an on-premises network using Azure ExpressRoute or Site-to-Site VPN.
Also, it's welcome to give feedback on Azure AKS to improve this feature. Hope this helps.
We have an application that runs on an Ubuntu VM. This application connects to Azure Redis, Azure Postgres and Azure CosmosDB(mongoDB) services.
I am currently working on moving this application to Azure AKS and intend to access all the above services from the cluster. The services will continue to be external and will not reside inside the cluster.
I am trying to understand how the network/firewall of both the services and aks should be configured so that pods inside the cluster can access the above services or any Azure service in general.
I tried the following:
Created a configMap containing the connection params(public ip/address, username/pwd, port, etc) of all the services and used this configMap in the deployment resource.
Hardcoded the connection params of all the services as env vars inside the container image
In the firewall/inbound rules of the services, I added the AKS API ip, individual node ips
None of the above worked. Did I miss anything? What else should be configured?
I tested the setup locally on minikube with all the services running on my local machine and it worked fine.
I am currently working on moving this application to Azure AKS and
intend to access all the above services from the cluster.
I assume that you would like to make all services to access each other and all the services are in AKS cluster? If so, I advise you configure the internal load balancer in AKS cluster.
Internal load balancing makes a Kubernetes service accessible to
applications running in the same virtual network as the Kubernetes
cluster.
You can take a try and follow the following document: Use an internal load balancer with Azure Kubernetes Service (AKS). In the end, good luck to you!
Outbound traffic in azure is SNAT-translated as stated in this article. If you already have a service in your AKS cluster, the outbound connection from all pods in your cluster will come thru the first LoadBalancer type service IP; I strongly suggest you create one for the sole purpose to have a consistent outbound IP. You can also pre-create a Public IP and use it as stated in this article using the LoadBalancerIP spec.
On a side note, rather than a ConfigMap, due to the sensitiveness of the connection string, I'd suggest you create a Secret and pass that down to your Deployment to be mounted or exported as environment variable.