Here in Azure AKS networking using kubenet it is mentioned that IP address range for --dns-cidr, --service-cide and --docker-brige-ip range should be an address space that isn't in use elsewhere in your network environment. I have also created Vnet and this AKS should be in that vnet.
Does this mean, for DNS, Service and Docker bridge, IP address range should be different then VNet IP range?
Pod CIDR, can we have it different then VNet range? As I am using Kubenet pod IP will not be from Vnet subnet.
yes, they should not overlap.
this is virtual pod ip address space, not the one they will get from your vnet (if you would be using Azure CNI). these would be the internal only kubernetes ip addresses. With kubenet they would get routed to the appropriate nodes with UDR and then the node would forward traffic to the appropriate pod.
Related
I am trying to establish an IKEv2 VPN between one VM(subnet: 20.25.0.0/16) and one AKS cluster(subnet: 10.0.0.0/16 - Azure CNI) using strongswan gateway. I need to access some kubernetes services behind of this AKS cluster. With Azure CNI each pod will be assigned an IP address from the POD subnets specified at cluster creation, this subnet is attached in interface eth0 for each node. Already kubernetes services of the type clusterIP will get an IP from service CIDR range specified at cluster creation, but this IP is only available in the cluster is not attached in any interface of the nodes, like POD subnet.
To run the strongswan on K8S it's necessary to mount the kernel modules(/lib/modules), in addition to enable NET_ADMIN capabilities. So the VPN tunnel is established using any of the networks attached on the host(nodes) interface, so I can't established a VPN using service CIDR range specified at cluster creation, since this IPs is known only within the cluster, through personalized routes and is not attached on any host interface. If I try to configure the VPN established with a subnet with the CIDR range of services informed in the creation of the cluster, I get an error stating that the subnet was not found in any of the interfaces.
To get around this, I realized that I can configure a tunnel informing a subnet with a larger range, as long as there is a subnet attached in my interface that is within the wider informed range. For example, I can configure a VPN informing the subnet 10.0.0.0/16, but my subnet for pods and nodes (attached in eth0) is 10.0.0.0/17 and CIDR range for services is 10.0.128.0/17, in this way all traffic 10.0.0.0/16 is routed through the vpn tunnel. In this way, as a workaround I define my services CIDR as a network subsequent to the network of pods and nodes and configure the VPN using a network that overlaps the two.
All 10.0.0.0/16 traffic from one side of the VPN (VM) is correctly routed to inside tunnel. If I try to access a Pod directly, using any IP from the Pods subnet (10.0.0.0/17), everything works fine. The issue is if I try to access a kubernetes service using a IP from CIDR for services(10.0.128.0/17), the traffic is not routed correctly until the K8S services. I can see the request in tcpdump in AKS, but it doesn't arrive in the service. So my question is, how to make a configuration on the strongswan, in which I can access the services on the aks kubernetes cluster?
Below is the current configuration of the strongswan:
PEER-1(VM)
conn %default
authby="secret"
closeaction="restart"
dpdaction="restart"
dpddelay="5"
dpdtimeout="10"
esp="aes256-sha1-modp1536"
ike="aes256-sha1-modp1024"
ikelifetime="1440m"
keyexchange="ikev2"
keyingtries="1"
keylife="60m"
mobike="no"
conn PEER-1
auto=add
leftid=<LEFT-PHASE-1-IP>
left=%any
leftsubnet=20.25.0.0/16
leftfirewall=yes
leftauth=psk
rightid=<RIGHT-PHASE-1-IP>
right=<RIGHT-PHASE-1-IP>
rightsubnet=10.0.0.0/16
rightfirewall=yes
rightauth=psk
PEER-2(AKS)
conn %default
authby="secret"
closeaction="restart"
dpdaction="restart"
dpddelay="5"
dpdtimeout="10"
esp="aes256-sha1-modp1536"
ike="aes256-sha1-modp1024"
ikelifetime="1440m"
keyexchange="ikev2"
keyingtries="1"
keylife="60m"
mobike="no"
conn PEER-2
auto=start
leftid=<LEFT-PHASE-1-IP>
left=%any
leftsubnet=10.0.0.0/16
leftfirewall=yes
leftauth=psk
rightid=<RIGHT-PHASE-1-IP>
right=<RIGHT-PHASE-1-IP>
rightsubnet=20.25.0.0/16
rightfirewall=yes
rightauth=psk
I have queries related to utilization, distribution and pricing of IP addresses on Azure.
Taking an example, 10.0.0.0/27. It says, 10.0.0.0 - 10.0.0.31 (32 addresses).
After expanding I found as..
CIDR Range: 10.0.0.0/27
Netmask: 255.255.255.224
Wildcard Bits: 0.0.0.31
First IP: 10.0.0.0
Last IP: 10.0.0.31
Total Host: 32
I assigned 10.0.0.0/27 for both VNet and Subnet on Azure Portal.
After creating the virtual network, I see available address 27.
1) No virtual machine created yet, then why the available addresses are 27?
2) What is the IP range for 27 available addresses?
While creating Gateway Subnet, I see errors like...
The specified address space overlaps with subnet 'Subnet1' which has a range of '10.0.0.0/27'.
Your subnet is not contained within the address space for this virtual network: 10.0.0.0/27.
3) How to calculate and decide the available gateway subnet address space for a virtual network?
4) For using gateway subnet, is it mandatory or recommended to add another subnet before or after creating gateway subnet?
5) Why address space is required for creating a gateway subnet? Is gateway subnet not a fixed or static IP address for creating connection?
6) In case of virtual network, 10.0.0.0/27, is pricing done on the basis of utilized IP addresses only?
7) In case of virtual network, 10.0.0.0/27, if there are some un-utilized IP addresses, are they blocked to me or my subscription or un-utilized IP addresses can be used by someone else on his azure portal?
I sincerely request to clarify all seven queries.
Q1-Q3:
Read Azure VNet FAQ, Azure reserves 5 IP addresses within each subnet. These are x.x.x.0-x.x.x.3 and the last address of the subnet. So you have 27 available addresses for the address range 10.0.0.0/27. It's address range 10.0.0.4 - 10.0.0.30
Q4: It's not mandatory for creating Gateway subnet order, you only need to calculate the CIDR Range for each subnet including in your current virtual network address range.
Q5: If you need to configure a virtual network gateway. The gateway subnet contains the IP addresses that the virtual network gateway services use. All gateway subnets must be named GatewaySubnet to work properly. Read here. If you don't need a VPN gateway, you don't need to create Gateway subnet in this VNet.
Q6-Q7: Have a look at Azure VNet pricing and VPN gateway pricing.
Azure Virtual Network is free of charge. Every subscription is allowed
to create up to 50 virtual networks across all regions.
Public IP addresses, and reserved IP addresses used on services inside
a virtual network, are charged.
Network appliances such as VPN Gateway and Application Gateway that
are run inside a virtual network are also charged.
For the un-utilized IP addresses, It looks like unassigned private IP address in your on-premise network. They can continue to be assigned to the resources when you deployed some resources in the VNet or subnet.
For more details, you can read this blog: Understanding CIDR Notation when designing Azure Virtual Networks and Subnets
Azure reserves 5 IP addresses within each subnet. These are x.x.x.0-x.x.x.3 and the last address of the subnet.
x.x.x.0 and the last address of the subnet is reserved for protocol conformance.
x.x.x.1-x.x.x.3 is reserved in each subnet for Azure services.
Address space is the super set of subnets. So, your Address space needs to be bigger and should be able to accommodate the IPs which you are defining for the Subnets.
Basic Subnetting would help.
It is not recommended to deploy your workloads on Gateway subnet. So you need other subnets to deploy your workloads.
Gateways does have a Private IP address which it gets from the Gateway subnet. As I mentioned in Point 2, Address space is the super set of subnets.
Only Public IPs are charged. Private IP which you define in Azure VNET is not charged.
Since it is Private IP address, you can create another 10.0.0.0/27 VNET in the same region. VNET provides a isolation and the address which you define is isolated to that VNET.
So as the title suggests, I need to make a load-balanced internal gateway with a VPN. I'm a developer, so networking is not my forte.
I have two identical VMs (VM1 in Availability Zone 1 and VM2 in Availability Zone 2) and I need to share VPN traffic between them. My client has provided a range of 5 addresses that will be configured on their firewall, so I will pick one for them to use and they then need to be oblivious to the internal routing.
My ultimate goal is to allow the client to connect through a VPN to one IP address (in the range they have allocated) and let Azure direct the traffic to VM1 primarily, but failover to VM2 if Availability Zone 1 goes down. The client must be oblivious to which VM they ultimately connect to.
My problem is that I cannot create a configuration where the Load Balancer's static IP is in the address range of the Gateway's VPN P2S address pool. Azure requires the P2S address pool to be outside of the VNet's address space and the Load Balancer needs to use the VNet's Subnet (which obviously is INSIDE the VNet's address space, so I'm stuck.
I can create the GW -> Vnet -> subnet -> VM1/VM2 set up no problem using the client's specified IP range for the P2S VPN, but without a Load Balancer, how do I then direct the traffic between the VMs?
e.g. (IPs are hypothetical)
The Vnet address range is 172.10.0.0/16
The Gateway subnet is 172.10.10.0/24
The Gateway's P2S address pool is 172.5.5.5/29
VM1's IP is 172.10.10.4
VM2's IP is 172.10.10.5
I can create a Load Balancer to use the Vnet (and the VMs in a Backend Pool), but then it's static IP has to fall in the VNet's subnet and thus outside the P2S address pool. So how do I achieve this?
I thought of creating a second VNet and corresponding Gateway and linking the Gateways, but I seemed to end up in the same boat
UPDATE: here is an image of my VNet diagram. I have only added one of the VMs (NSPHiAvail1) for now, but VM2 will be in the same LB backend pool
NSP_Address_Range is the range is a subnet of the VNet and is the range dictated by the client. The load balancer has a frontend IP in this range
Firstly, the Azure load balancer does round-robin load balancing for new incoming TCP connections, you could not use it for failover.
My problem is that I cannot create a configuration where the Load
Balancer's static IP is in the address range of the Gateway's VPN P2S
address pool.
You do not need to add the Load balancer frontend IP in the P2S address pool, the address pool is used for clients connecting to your Azure VNet.
Generally, you could configure P2S VPN gateway, create Gateway subnet and vmsubnet and create an internal standard SKU load balancer in the vmsubnet, then you could add the VMs in the vmsubnet into the backend pool as the backend target of the load balancer and configure the healthpro and load balancer rule for load balancing traffic. If so, you could access the backend VMs from clients via the load balancer frontend private IP.
Moreover, you could know some limitations about internal load balancer.
My problem was the Load Balancer Rules - or lack thereof. Once I had added a rule for port 1433 (SQL Server), I was able to query the DB from my local instance of SSMS
There is another solution that is a LOT simpler than the solution I was trying to implement, BUT it does not work allow for an internal load balancer
Azure Virtual Machine Scale Sets implement as many VMs as I specify and will automatically switch to another zone if one goes down. I have no need for the scalability aspect, so I disabled this and I'm only using the Load balancing aspect.
NB This setup only exposes a PUBLIC IP and you cannot assign an internal load balancer in conjunction with the default public load balancer
Here's some info:
Quickstart: Create a virtual machine scale set in the Azure portal
Create a virtual machine scale set that uses Availability Zones
Networking for Azure virtual machine scale sets
Virtual Machine Scale Sets
The cost is exactly what you'd pay for individual VMs, but the loadbalancing is included. So it's cheaper than the solution I described in my question. Bonus!
When I create kubernetes cluster from kubernetes service of azure in the networking tab , I have the option to create my own virtual network.
Let's say I created a virtual network with 3 subnets , still in the networking tab options , I can only associate 1 of these subnet to my cluster.
Is it a restriction in AKS?
If so, why it allows to create more than 1 subnet in virtual network?
Not sure but you only can specific one subnet when you create an AKS cluster on the Azure portal. It seems a restriction in AKS. Read the prerequisites. At least, you need one subnet, one AKS cluster. Also, AKS supports a single pool for now.
Don't create more than one AKS cluster in the same subnet.
With advanced networking in AKS, you can deploy a AKS cluster in a existing virtual network and define these subnet names and IP address ranges. IP addresses for the pods and the cluster's nodes are assigned from the specified subnet within the virtual network. So you need to plan IP addressing for your cluster. You should consider doing upgrade and scaling operations when you determine the number of IP addresses.
The reason it allows to create more than 1 subnet in virtual network is that essentially you could create many subnets with valid CIDR block in a VNet. You can create VMs in other subnets or create a dedicated Gatewaysubnet used for VPN gateway in the existing VNet. With advanced networking, this existing virtual network often provides connectivity to an on-premises network using Azure ExpressRoute or Site-to-Site VPN.
Also, it's welcome to give feedback on Azure AKS to improve this feature. Hope this helps.
My existing Vnet has a default subnet: 10.0.1.0/24
And I have a VM in it with IP address: 10.0.1.8
Now I want to add a gateway subnet so that I can connect web app to it, and the web app should be able to use the VM's private IP to access it.
But when I try to add a gateway subnet, I can't specify 10.0.1.0/24 because it overlaps with the default. I can't specify 10.0.0.0/24 either because the VM is not within this range.
How do I do it, then?
You cannot add a gateway subnet that uses the same address space (10.0.1.0/24) as VNet.
If you original subnet has a smaller address space (ie: /26, /28) then you can add a gateway subnet using that format.
If you original subnet is using the whole address space (ie: /24) then a work around will be to add a new address space to the VNet and add new subnets on that address to the VNet splitting them up correctly.