How to configure routing between Pods on multiple Azure Kubernetes clusters?
Something similar to ip-alias/vpc-native on Google Cloud
In AKS, I think you can create the AKS clusters in the different subnets in the same vnet with an advanced network. For more details about the network, see Choose the appropriate network model. But it's not a perfect solution and there are some limitations for it. For example, the AKS clusters should be in the same region as the vnet.
You can take a look at Gotchas & Solutions Running a Distributed System Across Kubernetes Clusters. As it said, it's hard to communicate from different regions and nothing has yet solved the problem of running a distributed system that spans multiple clusters. It’s still a very hard experience that isn’t really documented.
So, maybe there would be a long time to wait for the perfect solution.
Related
I am having difficulty understanding Azure Availability sets, specifically, what exactly i need to to do ensure my app running on my vm is utilizing Availability sets to be more available.
Lets say i am creating an application that runs on a single VM and i want to make it more resistant to hardware failure.
Option 1:
I create an Availability Set with 2 fault domains and then create a VM on this Availability set.
Is that it?
If there is a hardware failure on the rack hosting my VM, does azure now take care of ensuring the VM stays up and running?
Option 2:
i have to have two servers Vm1 & Vm2, both in the availability set but one on fault domain 1, one on fault domain 2.
i have to then set up a cluster of sorts for my application. In this case the availability set is simply allowing me to be sure that the two servers in my cluster are not on the same hardware, but the plumbing to ensure the application can take advantage of two servers and is highly available is still down to me.
Is option 1 or option 2 the correct way in which Availability Sets work in relation to fault domains?
Appreciate any clarity that can be provided.
Azure deals with hardware failure in two ways, Availability Sets and Availability Zones. AS is all about making sure that your app does not go down even if hardware failure happens within a Data center aka Zone itself. AZs are all about making sure your app does not go down even if the whole data center aka Zone is down. More details here.
Now to understand best practices around availability take a look at the best practices, specifically for VMs can be found here.
A Single VM instance is defined as follows, reference:
"Single Instance" is defined as any single Microsoft Azure Virtual Machine that either is not deployed in an Availability Set or has only one instance deployed in an Availability Set.
So one VM in or not in an availability set does not make any difference, for this you need at least two VMs and which are in an AS using FDs and UDs so Azure will take care of this by making sure that both VMs are running on separate Hardware to avoid your app going down.
One VM in an Availability set is nearly as good as a VM with no Availability set.
If you are placing two or more VMs in an AS and those are identical then you can add a load balancer to distribute traffic.
You can also use AS without a Load balancer if you are not interested in traffic distribution. One scenario can be where you want to switch to a secondary VM only when primary is unavailable.
Also, do understand it is not required to have identical VMs in an AS.
Virtual machine scale set is a good option if you are looking for a high availability solution with VMs.
Need some pointers on how one could achieve "true" multi region setup for AADDS.
As per Microsoft's documentation, AADDS is "designed" to be "single regioned". Although it provides some (arguably) redundancy by spinning up essentially 2 managed domain controllers, it does not take into account performance.
Microsoft recommends (and there isn't really any other way to do this) setting up VPN's or VNET peering in order to access your AADDS from other regions, but this has huge impact over performance, and also over actual redundancy (HA designs should be multi region imo, and AADDS should be HA).
We're deploying Windows VM's in (at the time of writing this question) 10 regions, with AADDS in West Europe. We're seeing huge penalties for our apps that require/rely on LDAP ( >10s in some regions) for even the most basic LDAP queries with quite the small return payload.
Was hoping someone figured out a way to mirror/cache AADDS in a new region, like maybe adding a new worker DC or some black magic, so that VMs and services would connect more locally?
Cheers!
Azure AADDS Multi-Region Support is already a requested feature and is under works currently. However, there is no ETA to share at the moment. You can follow What's new in Azure Active Directory? for updates.
The only option to achieve Geo-redundancy is by deploying ADDS across multiple regions via IaaS VMs, Vnet pairing, and VPN gateways.
Also, for high availability, each Azure AD Domain Services managed domain includes two domain controllers. You don't manage or connect to these domain controllers, they're part of the managed service. If you deploy Azure AD Domain Services into a region that supports Availability Zones, the domain controllers are distributed across zones. In regions that don't support Availability Zones, the domain controllers are distributed across Availability Sets. You have no configuration options or management control over this distribution.
According to Azure AADDS FAQ documentation, they do support a fail-over to another geo location.
You can follow this tutorial page in order to create a replica set for your AADDS deployment.
I'm still experimenting with migrating my service fabric application to kubernetes. In service fabric I have two node types (effectively subnets), and for each service I configure which subnet it will be deployed to. I cannot see an equivalent option in the kubernetes yml file. Is this possible in kubernetes?
First of all, they are not effectively subnets. They could all be in the same subnet. But with AKS you have node pools. similarly to Service Fabric those could be in different subnets (inside the same vnet*, afaik). Then you would use nodeSelectors to assign pods to nodes on the specific node pool.
Same principle would apply if you are creating kubernetes cluster yourself, you would need to label nodes and use nodeSelectors to target specific nodes for your deployments.
In Azure the AKS cluster can be deployed to a specific subnet. If you are looking for deployment level isolation, deploy the two node types to different namespaces in k8s cluster. That way the node types get isolation and can be reached using service name and namespace combination
I want my backend services that access my SQL database in a different subnet to the front-end. This way I can limit access to the DB to backend subnet only.
This is an older way to solve network security. A more modern way is called Zero Trust Networking, see e.g. BeyondCorp at Google or the book Zero Trust Networks.
Limit who can access what
The Kubernetes way to limit what service can access what service is by using Network Policies.
A network policy is a specification of how groups of pods are allowed to communicate with each other and other network endpoints.
NetworkPolicy resources use labels to select pods and define rules which specify what traffic is allowed to the selected pods.
This is a more software defined way to limit access, than the older subnet-based. And the rules are more declarative using Kubernetes Labels instead of hard-to-understand IP-numbers.
This should be used in combination with authentication.
For inspiration, also read We built network isolation for 1,500 services to make Monzo more secure.
In the Security team at Monzo, one of our goals is to move towards a completely zero trust platform. This means that in theory, we'd be able to run malicious code inside our platform with no risk – the code wouldn't be able to interact with anything dangerous without the security team granting special access.
Istio Service Entry
In addition, if using Istio, you can limit access to external services by using Istio Service Entry. It is possible to use custom Network Policies for the same behavior as well, e.g. Cilium.
I am migrating from GCP to Azure. My use case is simple: I have a k8s cluster running some web crawlers which needs to talk to Elastic and Cassandra clusters (not in the k8s cluster) using internal IPs. All of these components can be in the same Azure Region (e.g: East US). I understand from this discussion that VNET peering is the way to go.
This solution did not work for me. I am still unable to reach my Cass/ES cluster from the pods. I believe this solution is outdated, is there some other approach to accomplish this, that I am missing ?
We can use Azure route table to achieve that, then we can ping pod IP address out of your k8s Vnet.
I have answered your another question here, please check it.
If you would like further assistance, please let me know:)
We are currently evaluating azure, to see if we can use it for our stress and production environments.
Our environment is pretty complex, including web servers, mysql servers, hadoop and cassandra servers, as well as monitoring and deployment servers.
To set the stress environment, we need to install the environment, and then load large amounts of data into it, before we can run a stress test. This takes time and effort, and so, since we pay by the hour, we would like to be able to completely shutdown the environment, and start it up again ready to go when we want to run additional stress tests.
Here's our issue - we could not find a way to set a fixed internal ip address for a vm in azure. In AWS it is possible with VPC, but in azure, even if you define a virtual network, there seems to be no way to set a fixed internal ip (at least none that we can find).
This creates several issues for us -
1. Hadoop relies on all nodes in the cluster being able to translate all the modes hostnames to ip addresses.
2. A cassandra cluster that has all the ip addresses in the cluster change at once freaks out. We actually lost data in a test cassandra cluster because of this.
Our questions are:
1. Is there a way to set a fixed internal ip for a vm in azure?
2. If not, did anyone have an experience with running hadoop and cassandra on azure? How did you handle the changing ip addresses when the cluster is shut down?
Any advice on these issues will be much appreciated,
Thanks
Amir
Please note that the portal doesn't always expose all the capabilities of Azure. Some of the features in Azure are only possible through the REST API and PowerShell.
If you take a look at the new release of the PowerShell Cmdlets, you'll notice there is a new option for Static IPs in VNets.
https://github.com/WindowsAzure/azure-sdk-tools