I have an AKS cluster with two nodes. One is a classic VM, and the other one is a Virtual Node. Since I have different subscriptions across my account, I want to deploy a Virtual Node to each of them. So I can track the billing of the usage for every subscription. Is there a way to create a Virtual Node across multiple subscriptions and add them to the AKS cluster?
I have created an AKS cluster with the Virtual Nodes feature enabled. I can see the virtual node, and I can deploy pods on it.
Is there a way to create a Virtual Node across multiple
subscriptions and add them to the AKS cluster? So, I can track the billing of the usage for every subscription.
Azure Kubernetes supports Multi Tenancy deployment where AKS resources can be shared or deployed separately across multiple tenants residing in different subscriptions.
In order to automate the deployment across multiple subscriptions you can make use of IAC infrastructure as a code tools like ARM templates, Bicep, Terraform etc.
Approachs:
1) Vertically partitioned Deployments : -
Here, Tenant A and Tenant B run on same Node pools connected to same AKS cluster and C runs on isolated node pool. Consider these node pools as your virtual nodes backed by Azure Container Instance with virtual kubelet which auto-scales the nodes whenever AKS needs it.
By autoscaling the Virtual nodes, you will have multiple nodes running in your AKS cluster.
2) Horizontal partitioned Deployments :-
In this method you can have a shared AKS cluster across multiple Azure subscriptions and Different Nodes running individually in every subscription or tenant, As in the above diagram Databases are deployed individually across multiple subscriptions.
3) Automated Single Tenant Deployments :-
In this deployment model, you deploy dedicated resources in each subscriptions via a automated code with either bicep, ARM, or terraform via a pipeline. you can refer the above image from this Document.
You can deploy this multi-tenant architecture with the use of Deployment stamps to automate the deployment of virtual nodes backed by ACI across multiple subscriptions.
Deployment stamp pattern helps in deploying multiple Azure resources across multiple tenants and subscriptions. You assign a stamp to multiple resources and deploy those resources in your tenant. Then the copy of the same stamp is deployed in another tenants.
You can enable the Azure virtual Node add on to your AKS by adding the command to add the node to your cluster in the ARM template, IAC deployment in pipeline itself.
You can create either a Bicep or Terraform or ARM Template to deploy your Virtual nodes across multiple subscriptions by running them via a pipeline with git-hub actions or via Azure Devops. You can create different stages or builds for different subscriptions and initiate a parallel build to automate the deployment in Azure DevOps.
You can refer this git-hub repository to deploy a virtual node auto-scaler in your tenant :-
Azure-Samples/virtual-node-autoscale: A sample application to demonstrate Autoscale with AKS Virtual Nodes (github.com)
You can create your own ARM template or github repo with the code files and run it via Azure Devops pipeline with examples: -
ARM Template to deploy AKS: -
Reference: -
Walkthrough: Create Azure Kubernetes Service (AKS) using ARM template – AzureDays
ARM Template with parameter Stamp:-
Reference:
Deployment Stamps pattern - Azure Architecture Center | Microsoft Learn
You can make use of Nested ARM template and use multiple resource in a parent child combination to deploy multiple resources with single template.
You can automate the deployment across multiple subscriptions by using Azure Devops or via Github actions:-
Azure DevOps pipeline -
And to track the billing data you can create an Alert in every subscription to get an email of the Usage consumption of the nodes. Refer here -Azure cost analysis.
Reference : -
Deploy multiple instances of resources - Azure Resource Manager | Microsoft Learn.
Related
If one is running Docker Enterprise with Kubernetes in an on-premises private cloud, is it possible to add clusters in a public cloud like Azure?
On GCP, Anthos is a candidate.
You may have a look on their architecture and see if it fits your needs.
Anthos is advertised in most of the GCP architecture courses and offers integration between GKE and both on-prem(the on-prem cluster must meet some prerequisites or you can use the version provided by Google) and AWS Kubernetes clusters.
Istio is a service mesh and if I understood correctly your requirements, the multiple clusters and multiple networks models could be used.
why don't use rancher for that , you can manage on-premise and GKE AKS EKS or cluster installed in ec2.
it's a great tool for that
This is where Azure Arc can help you to achieve this requirement. However it is in preview stage as of now, i hope soon it will be generally available.
From the DOCS,
You can attach and configure Kubernetes clusters inside or outside of
Azure by using Azure Arc-enabled Kubernetes Preview. When a Kubernetes
cluster is attached to Azure Arc, it will appear in the Azure portal.
It will have an Azure Resource Manager ID and a managed identity.
Clusters are attached to standard Azure subscriptions, are located in
a resource group, and can receive tags just like any other Azure
resource.
cluster API under kubernetes SIG is an open source project which provides declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters.
Cluster API can be extended to support any infrastructure provider (AWS, Azure, vSphere, etc.) or bootstrap provider (kubeadm is default) you need. There is a growing list of supported providers available.
You can use cluster API and build your own custom management plane based on cluster API if vendor provided software is not an option for you.
If you are looking for a vendor provided management plane which can be hosted on prem and can manager life cycle of a on prem kubernetes cluster as well as a cluster on any public cloud provider such as AWS, GCP, Azure then Tanzu Mission Control from VMware is an option. Internally it uses cluster API.
Personally I would not use Anthos or Arc because they seem to be a way to get locked into a specific vendor
A Devops pipeline is linked to one subscription. Is it possible to access resources in multiple subscription from one task in a pipeline?
Is it possible to access resources in multiple subscription from one task in a pipeline?
AFAIK, I do not think it is possible to access resources in multiple subscription from one task.
Just ask you know, we could only link one subscription in one task:
If deployed to multiple environments, we could create a Release pipeline with multiple environments. Then add Deploy task for every environment and choose different Azure Service Connections in different environment. Then you could deploy one app to different Azure environment through one release pipeline.
Hope this helps.
This should be doable with light house configurations.
https://adatum.no/azure/multi-subscription-deployment-with-devops-and-azure-lighthouse
You would have to create a new azure devops SPN. You can then in turn give this SPN account via light house access to the subscription.
Noticed Virtual Network and Virtual Network(Classic) in Azure, Wondering what is the difference between them and when to choose what ?
Azure has two different deployment models for creating and working with resources: Resource Manager and classic.
How you create, configure, and manage your Azure resources is different between these two models. In classic mode, each resource provisioned in Azure is a single management unit. You manage all of the resources individually. The classic mode does not allow grouping of resources, which makes managing Azure resources difficult.
Azure resource management mode adds the concept of the resource group, which is a container for resources that share a common lifecycle. In ARM mode, you can deploy, manage, and monitor all the services for your solution as a group, rather than handling these services individually. Also, you can apply tags to resources to logically organize all the resources in your subscription.
When you interact with Classic mode resources from a command line such as Azure PowerShell, you are using Azure Service Management API calls (ASM). ASM is a traditional way of accessing Azure resources. In the Resource Manager mode, you are using Azure Resource Manager Rest API. When a user sends a request from any of the Azure tools, APIs, or SDKs, Resource Manager receives the request. It authenticates and authorizes the request. Resource Manager sends the request to the Azure service, which takes the requested action. See Azure Resource Manager overview.
The Virtual Network(Classic) is used for the classic models. Virtual Network in Azure is created for Resource Manager models. Currently, Microsoft recommends that you use Resource Manager for all new resources. Mostly, the Virtual Network(Classic) is used when you are creating a VNet for cloud services as far as I can tell. For Resource Manager resources, you just need to create Virtual Network in Azure portal.
There are three scenarios to be aware of:
Cloud Services does not support Resource Manager deployment model.
Virtual machines, storage accounts, and virtual networks support both Resource Manager and classic deployment models.
All other Azure services support Resource Manager
Here is a blog about Azure Classic vs Azure Resource Manager
So which one should I use?
Since both modes are viable options at this point, it is necessary to
pay careful attention to the features that each offer and your
specific requirements. ARM cannot simply be assumed to be the best fit
as it simply may not meet all of your or your customer’s needs.
I want to deploy a web app using CI/CD from VSTS to multiple (customer) Azure accounts. So not multiple subscriptions within the same Azure account, but multiple different Azure accounts owned by customers.
The app is a managed app from the Azure Marketplace so I have full access to the resource group in which the app resides.
Is there an easy way to do this using service connections and the web deploy task or should I be using Powershell to do this and script it myself?
Please create multiple Service Connections in your VSTS which used to connect to these Azure account first.
Then please create a Release pipeline with multiple environments. Then add Deploy task for every environment and choose different Azure Service Connections in different environment. Then you could deploy one app to different Azure environment through one release pipeline.
It turns out that with a Managed App, for every customer deployment a resource group is created that I own and therefore shows up in my Azure account.
So from VSTS I can just use Service Connections to those resource groups, no need to connect to these other Azure accounts.
My understanding so far on PaaS deployment using WebRoles or Worker Roles-
Spinning Web roles or Worker roles will create Cloud service to manage it.
However, in ARM resource group, their is no concept of Cloud service, then how are web & worker roles managed in ARM resource group?
Also I tried adding the webroles via JSON Outline in VS 2015, but no option to add webroles. So not sure if you can deploy webroles via JSON template?
Any information will be of great help.
Cloud Services have been around since the beginning of Azure.
To the deploy or manage them, you have to use the old APIs (Azure Service Management).
As far as I know, they won't be migrated to the new APIs (Azure Resource Manager), but you could use Virtual Machine Scale Sets instead.
Azure Resource Manager Web apps rely on the concept of App Service Plans, rather than cloud services. Basically this is the underlying VM(s) that the service runs on. You can specify the size and number of servers that make up the App Service plan, and then deploy on to those as a single unit.
You create an App service plan, then run web / worker roles inside that (as well as Logic, API, Functions)
As far as Web / Worker roles are concerned, App Service Plans do not draw a distinction between the two. You simply deploy code to it and it will run it however it is packaged. See here
The concept of cloud services simply don't exist within the Azure Resource Management model.
You can find a template for deploying a Web App here