I created the GKE Private Cluster via Terraform (google_container_cluster with private = true and region set) and installed the stable/openvpn Helm Chart. My setup is basically the same as described in this article: https://itnext.io/use-helm-to-deploy-openvpn-in-kubernetes-to-access-pods-and-services-217dec344f13 and I am able to see a ClusterIP-only exposed service as described in the article. However, while I am connected to the VPN, kubectl fails due to not being able to reach the master.
I left the OVPN_NETWORK setting as the default (10.240.0.0), and changed the OVPN_K8S_POD_NETWORK and subnet mask setting to the secondary range I chose when I created my private subnet that the Private Cluster lives in.
I even tried adding 10.240.0.0/16 to my master_authorized_networks_config but I'm pretty sure that setting only works on external networks (adding the external IP of a completely different OVPN server allows me to run kubectl when I'm connected to it).
Any ideas what I'm doing wrong here?
Edit: I just remembered I had to set a value for master_ipv4_cidr_block in order to create a Private Cluster. So I added 10.0.0.0/28 to the ovpn.conf file as push "route 10.0.0.0 255.255.255.240" but that didn't help. The docs on this setting states:
Specifies a private RFC1918 block for the master's VPC. The master
range must not overlap with any subnet in your cluster's VPC. The
master and your cluster use VPC peering. Must be specified in CIDR
notation and must be /28 subnet.
but what's the implication for an OpenVPN client on a subnet outside of the cluster? How do I leverage the aforementioned VPC peering?
Figured out what the problem is: gcloud container clusters get-credentials always writes the master's external IP address to ~/.kube/config. So kubectl always talks to that external IP address instead of the internal IP.
To fix: I ran kubectl get endpoints, noted the 10.0.0.x IP and replaced the external IP in ~/.kube/config with it. Now kubectl works fine while connected to the OVPN server inside of the Kube cluster.
You can add --internal-ip to your gcloud command to put automatically the internal ip address to ~/.kube/config file
Related
I am trying to establish an IKEv2 VPN between one VM(subnet: 20.25.0.0/16) and one AKS cluster(subnet: 10.0.0.0/16 - Azure CNI) using strongswan gateway. I need to access some kubernetes services behind of this AKS cluster. With Azure CNI each pod will be assigned an IP address from the POD subnets specified at cluster creation, this subnet is attached in interface eth0 for each node. Already kubernetes services of the type clusterIP will get an IP from service CIDR range specified at cluster creation, but this IP is only available in the cluster is not attached in any interface of the nodes, like POD subnet.
To run the strongswan on K8S it's necessary to mount the kernel modules(/lib/modules), in addition to enable NET_ADMIN capabilities. So the VPN tunnel is established using any of the networks attached on the host(nodes) interface, so I can't established a VPN using service CIDR range specified at cluster creation, since this IPs is known only within the cluster, through personalized routes and is not attached on any host interface. If I try to configure the VPN established with a subnet with the CIDR range of services informed in the creation of the cluster, I get an error stating that the subnet was not found in any of the interfaces.
To get around this, I realized that I can configure a tunnel informing a subnet with a larger range, as long as there is a subnet attached in my interface that is within the wider informed range. For example, I can configure a VPN informing the subnet 10.0.0.0/16, but my subnet for pods and nodes (attached in eth0) is 10.0.0.0/17 and CIDR range for services is 10.0.128.0/17, in this way all traffic 10.0.0.0/16 is routed through the vpn tunnel. In this way, as a workaround I define my services CIDR as a network subsequent to the network of pods and nodes and configure the VPN using a network that overlaps the two.
All 10.0.0.0/16 traffic from one side of the VPN (VM) is correctly routed to inside tunnel. If I try to access a Pod directly, using any IP from the Pods subnet (10.0.0.0/17), everything works fine. The issue is if I try to access a kubernetes service using a IP from CIDR for services(10.0.128.0/17), the traffic is not routed correctly until the K8S services. I can see the request in tcpdump in AKS, but it doesn't arrive in the service. So my question is, how to make a configuration on the strongswan, in which I can access the services on the aks kubernetes cluster?
Below is the current configuration of the strongswan:
PEER-1(VM)
conn %default
authby="secret"
closeaction="restart"
dpdaction="restart"
dpddelay="5"
dpdtimeout="10"
esp="aes256-sha1-modp1536"
ike="aes256-sha1-modp1024"
ikelifetime="1440m"
keyexchange="ikev2"
keyingtries="1"
keylife="60m"
mobike="no"
conn PEER-1
auto=add
leftid=<LEFT-PHASE-1-IP>
left=%any
leftsubnet=20.25.0.0/16
leftfirewall=yes
leftauth=psk
rightid=<RIGHT-PHASE-1-IP>
right=<RIGHT-PHASE-1-IP>
rightsubnet=10.0.0.0/16
rightfirewall=yes
rightauth=psk
PEER-2(AKS)
conn %default
authby="secret"
closeaction="restart"
dpdaction="restart"
dpddelay="5"
dpdtimeout="10"
esp="aes256-sha1-modp1536"
ike="aes256-sha1-modp1024"
ikelifetime="1440m"
keyexchange="ikev2"
keyingtries="1"
keylife="60m"
mobike="no"
conn PEER-2
auto=start
leftid=<LEFT-PHASE-1-IP>
left=%any
leftsubnet=10.0.0.0/16
leftfirewall=yes
leftauth=psk
rightid=<RIGHT-PHASE-1-IP>
right=<RIGHT-PHASE-1-IP>
rightsubnet=20.25.0.0/16
rightfirewall=yes
rightauth=psk
Notice that when I create a Azure Kubernetes, by default it creates the API with an *.azmk8s.io FQDN and it is external facing.
Is there a way to create with local IP instead? If "yes", can this be protected by NSG and Virtual Network to limit connections coming via Jump Server?
If there is any drawback creating to only allow internal IP?
Below is the command I used to create:-
az aks create -g [resourceGroup] -n
[ClusterName] --windows-admin-password [SomePassword]
--windows-admin-username [SomeUserName] --location [Location] --generate-ssh-keys -c 1 --enable-vmss --kubernetes-version 1.14.8 --network-plugin azure
Anyone tried https://learn.microsoft.com/en-us/azure/aks/private-clusters? If that still allows external facing app but private management API?
why not? only the control plane endpoint is different. in all other regards - its a regular AKS cluster.
In a private cluster, the control plane or API server has internal IP
addresses that are defined in the RFC1918 - Address Allocation for
Private Internets document. By using a private cluster, you can ensure
that network traffic between your API server and your node pools
remains on the private network only.
this outlines how to connect to private cluster with kubectl. NSG should work as usual
The question is: How to expose DNS names pointing to the EKS cluster. DNS should only available inside our subnets and accessible with our VPN connection (which essentially means that DNS should point to addresses inside our VPC)
I have an EKS cluster which runs in the 10.0.0.0/16 VPC. Nodes are located inside private subnets, and services are exposed externally with ELB and Ingress Controller.
Since some of the services inside the VPC are only accessible inside of our company we have decided to runa OpenVPN server configured with routing to the 10.0.0.0/16 through the VPN and rest of the traffic going directly to the Internet. Currently public DNS configured in Route53 points to our private addresses which is not ideal. (ex A record for privateservice.example.com -> 10.0.1.1). It is not ideal (existence of the privateservice shouldn't be available in a public DNS) but worked for now.
To resolve the problem of private services in a public DNS I though about running Bind DNS server and configuring our OpenVPN to push this configuration to clients (couldn't get it to work on some client machines but I assume it will work).
However I have no idea how to expose some private services running inside EKS cluster. As mentioned before there are some services in the cluster which are available publicly through ELB. But in the same cluster I have a subset of services which should be available only inside our subnet.
Use private hosted zone of route 53 for private DNS resovling. It should be better using internal ELB to expose services in EKS.
Use route53 resovler for resolving aws managed DNS in on-perms.
I have an VPN between the company network 172.16.0.0/16 and GCP 10.164.0.0/24
On GCP there is a cassandra cluster running with 3 instances. These instances get dynamical local ip adresses - for example 10.4.7.4 , 10.4.6.5, 10.4.3.4.
My issue: from the company network I cannot access 10.4x addresses as the tunnel works only for 10.164.0.0/24.
I tried setting up an LB service on 10.164.0.100 with the cassandra nodes behind. This doesnt work: when I configure that ip adress as seed node on local cluster, it gets an reply from one of the 10.4.x ip addresses, which it doesnt have in its seed list.
I need advice how to setup inter DC sync in this scenario.
IP addresses which K8s assign to Pods and Services are internal cluster-only addresses which are not accessible from outside of the cluster. It is possible by some CNI to create connection between in-cluster addresses and external networks, but I don't think that is a good idea in your case.
You need to expose your Cassandra using Service with NodePort or LoadBalancer type. That is another one answer with a same solution from Kubernetes Github.
If you will add a Service with type NodePort, your Cassandra will be available on a selected port on all Kubernetes nodes.
If you will choose LoadBalancer, Kubernetes will create for you Cloud Load Balancer which will be an entrypoint for Cassandra. Because you have a VPN to your VPC, I think you will need an Internal Load Balancer.
I am running a kubernetes cluster with 1 master (also a node) and 2 nodes on Azure. I am using Ubuntu with Flannel overlay network. So far everything is working well. The only problem I have is exposing the service to the internet.
I am running the cluster on an azure subnet. The master has a NIC attached to it that has a public IP. This means if I run a simple server that listens on port 80, I can reach my server using a domain name (Azure gives an option to have a domain name for a public IP).
I am also able to reach the kubernetes guest book frontend service with some hack. What I did was check all the listening ports on the master and try each port with the public IP. I was able to hit the kubernetes service and get response. Based on my understanding this is directly going to the pod that is running on the master (which is also a node) rather than going through the service IP (which would have load balanced across any of the pods).
My question is how do I map the external IP to the service IP? I know kubernetes has a setting that works only on GCE (which I can't use right now). But is there some neat way of telling etcd/flannel to do this?
If you use the kubectl expose command:
--external-ip="": External IP address to set for the service. The service can be accessed by this IP in addition to its generated service IP.
Or if you create from a json or yaml file, use the spec/externalIPs array.