I am looking to write a playbook to resize the azure linux vm disks, has anyone already done that or have experience please provide input.
- name: Update scaleset info using azure_rm_resource
azure_rm_resource:
resource_group: myResourceGroup
provider: compute
resource_type: virtualmachinescalesets
resource_name: myVmss
api_version: "2017-12-01"
body: { body }
Related
To preface, I am new to leveraging Kubernetes in Azure, so I might get some terms wrong. Sorry in advance.
I have been attempting to find a way to create Kubernetes secret objects that are populated with values from Azure key vault (AKV) in Azure Kubernetes services (AKS). The goal is to be able to create an imagePullSecret using values populated from AKV to allow my cluster to pull *images* from an Azure container registry (ACR). I have tested this out with manual creation and got it working, but am open to other solutions to integrating AKS and ACR.
I have found several articles pointing out that I can manage secrets from AKV by using the API secrets-store.csi.x-k8s.io/v1. I attempted to find a way to leverage this, but these appear to be secrets that can be mounted by a pod and not leveraged by the AKS management layer.
Whenever I attempt to search for a way to generate Kubernetes imagePullSecrets, I always find either the API above or manual secret creation. At this point, I am starting to think it is not possible create the imagePullSecrets with a Kubernetes manifest file. If it is possible, a link to documentation would be appreciated. If the mentioned API can achieve the desired goal, I might need help understanding how to leverage it for this task.
TLDR: Can I create a Kubernetes secret object without the need for pods in AKS using AKV?
Alternatively, Is there another integration solution for AKS and ACR that would avoid the need to manual linkage creation?
Edit: Changed secret to image
No, the secret CSI always need a pod to create Kubernetes secrets.
Yes, you can easily attach your ACR to your AKS leverage the Kubelet identity to pull images from your ACR. You just need to export the Identity and add a Role Assignment on your ACR (for an existing ACR and AKS):
export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv)
export ACR_ID=$(az acr show -g <resource group> -n <acr name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "AcrPull" --scope $ACR_ID
For a new AKS simply run :
MYACR=myContainerRegistry
az acr create -n $MYACR -g myContainerRegistryResourceGroup --sku basic
az aks create -n myAKSCluster -g myResourceGroup --generate-ssh-keys --attach-acr $MYACR
This can be also done with IaC like Bicep or Terraform (just create the Role Assignment on the Kubelet Identity)
Here you can also find the documentation.
Let's clarify your objective...
You want to pull images from ACR and in order to do so you need to set up imagePullSecrets in your pods/deployments. I am assuming you want to limit access to the ACR to specific namespaces in your cluster. If you want it cluster wide - then you may just go with direct integration setup with AKS --> ACS
One would not pull secrets from ACR as you indicated in your question, they would pull the secret from Azure Key Vault using the Azure version of the secrets store connector you listed above (https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-driver) - this will set up secrets in your cluster namespace the pods can use for imagePullSecrets to pull container images from ACR.
In short:
set up Azure CSI
use Azure CSI to create secret holding your pull secrets for your ACS. The creation happens on pod deployment
when defining pod spec - set the imagePullSecrets secret name to the one you create in step 2
I would do it this way.
Reconfigure your cluster with the Azure Key Vault provider - you can do this when you create your cluster so you don't have to go through the effort of adding it manually. If you are doing it through the portal it's under the Advanced Tab in the cluster setup pictured below...
or if you are setting up your cluster in terraform like I do then set up the key_vault_secrets_provider section in the resource.
Read the instructions in the link above on how to set up secret pulling from AKV. you will need to mount a volume to your pods to create them. If you haven't done that before it's pretty easy to get the hang of. Once you do, every time you deploy a pod, a kubernetes secret will be created with the value of the secret pulled from AKV.
When you need to assign the image pull secret - you use the secret you created via the mounted volume
your deployment yaml may look something like this..
apiVersion: apps/v1
kind: Deployment
metadata:
name: <your deployment name>
namespace: <namespace name>
<more meta data stuff here>
spec:
<some deployment spec stuff here>
template:
metadata:
<some common metadata for your pods in this deployment>
spec:
imagePullSecrets:
- name: <your secret name that holds the value from AKV here>
containers:
- name: <your container name>
image: <ACR_registry_name_here.repo_name:version>
volumeMounts:
- mountPath: /mnt/<image-pull-secret-mount-for-csi here>
name: <whatever name you want>
readOnly: true
<some other container stuff here>
volumes:
- name: <whatever name you want - must be same as above volume mount>
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: <the name of your secretProviderClass resource - see below>
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: <your secretProviderClass name>
namespace: <namespace name>
spec:
parameters:
keyvaultName: <the name of your Azure Key Vault>
objects: <read instructions in the link on how to create these>
resourceGroup: <the azure resource group name of your key vault>
subscriptionId: <your Azure Subscription GUID>
tenantId: <your Azure tenant GUID>
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: <the managed identity id for your core cluster virtual machines - look in the resource group that is automatically created for your cluster VMs when you create your cluster>
provider: azure
secretObjects:
- <this depends upon the secret type you want - again - read instructions in the link I provided>
there are other resources out there i found that this guy's videos are helpful
https://www.youtube.com/watch?v=S2wzCD5L9Mo&ab_channel=HoussemDellai
good luck - I hope this helps
If your objective is to securely pull an image from a ACR on AKS Cluster
imo, the cleanest would be to grant AcrPull (https://learn.microsoft.com/en-us/azure/container-registry/container-registry-roles?tabs=azure-cli) to your AKS NodePool identity ( this can be done at the point of deployment of the cluster, or grant it later using CLI or the portal)
but as an alternative if you want to read some service principal secret from the Azure key vault and directly want it to be mapped to a key-vault secret --> kubernetes secret , you can use external-secrets.io (https://external-secrets.io/v0.7.0/provider/azure-key-vault/)
My AKS cluster and storage account are in the same Region: East US 2.
I have created secret:
kubectl create secret generic fa-fileshare-secret --from-literal=azurestorageaccountname=fastorage --from-literal=azurestorageaccountkey='OWd7e9Ug' secret/fa-fileshare-secret created
In that storage account I have file share: containershare
I have checked in the Configuration of the secret and values are being matched for account name and key (as this is stated in similar questions which did not help me).
I think VNET for storage account and AKS cluster are different, and also Subscription and Resource group are different (if relevant.)
When I try to execute deployment for my app, I am getting:
Mounting arguments: -t cifs -o actimeo=30,mfsymlinks,file_mode=0777,dir_mode=0777,
<masked> //fastorage.file.core.windows.net/containershare
/var/lib/kubelet/plugins/kubernetes.io/csi/pv/#fa-fileshare-secret#containershare#ads-volume#default/globalmount
Output: mount error(13): Permission denied
In deployment.yaml definition:
........
volumes:
- name: ads-volume
azureFile:
secretName: fa-fileshare-secret
shareName: containershare
readOnly: false
............
What can be the problem (since different region and wrong credentials are not the issue). I am accessing the cluster through the kubectl from remote windows machine.
Thank You AndreyS for confirming you resolve your issue. Here is few more additional details that can help to know cause of your issue.
As Per Microsoft Document here is the possible cause for this error Mount error(13): Permission denied
Cause 1: Kubernetes secret doesn't reference the correct storage account name or
key
Cause 2: AKS's VNET and subnet aren't allowed for the storage account
Cause 3: Connectivity is via a private link but nodes and the private endpoint are in different
VNETs
For mounting the storage file share with AKS Cluster(Pod) you should deploy both the resource in same resource group and same region and also to make sure to both resource in same VNET if not then you have to allow access to your AKS VNET in Storage is set to Selected networks, check if the VNET and subnet of the AKS cluster are added.
It may take a few moments for the changes to take effect. After the VNET and subnet are added, check if the pod status changes from ContainerCreating to Running and mounted the File share as well.
This challenge is regarding Azure and Azure DevOps but I would imagine this happening on similar platforms (AWS, GCP, Github, Gitlab, etc.)
I am currently using Azure DevOps Pipelines but I am facing a problem with interacting with resources behind firewalls (either IP restricted or virtual network restricted). As Azure Pipeline spins up a new VM it requires me to whitelist that given public IP for that newly spun up machine each time I do a run. It is very janky to accommodate for this whitelisting as I am creating Azure Pipelines as submodules for reproducibility purpose extending templates from one project and using it in multiple. Terraform state needs to access configurations on restricted resources, hence throwing access denied messages.
I have looked into the following to solve the challenge and my thoughts about them:
Implementing Azure Virtual Machine scale-set agents instead of Microsoft Hosted agent (https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/scale-set-agents?view=azure-devops).
This would require a long discussion with our security team as they are not fan of using virtual machines in the cloud.
Deploy Atlantis on AKS with a static public IP from the Load Balancer and whitelist this one IP (https://learn.microsoft.com/en-us/azure/aks/static-ip).
This would require some initial setup from my side as I could not find that much documentation on deploying this using Azure (AKS) and Azure DevOps. (found this article: https://engineering.plex.com/posts/deploying-infrastructure-azure/ and this video using Github https://www.youtube.com/watch?v=33j49_n8Zvc&ab_channel=JJAsghar. Terraform Module for deploying Atlantis on AWS: https://github.com/terraform-aws-modules/terraform-aws-atlantis)
Use Terraform Enterprise for infrastructure deployment and whitelist Terraform Enterprise IP range (https://www.terraform.io/docs/cloud/api/ip-ranges.html).
This would require introducing a new platform for provisioning infrastructure.
Make a huge whitelist stage that whitelists each resource from my Terraform Module Registry depending on the module used in the pipeline (very cumbersome to maintain).
This would require too much maintenance and does not seem like the best solution.
What are your thoughts on solving this challenge?
You can use scripts to get the ip of the cloud agents. And dynamically whitelist the ip address for your azure storage account using Azure PowerShel or Azure Cli. See below example:
1, Add Azure Powershell task before Terraform
task in your azure devops pipeline to get the agent's ip address and add whitelist for azure storage account.
- task: AzurePowerShell#5
displayName: 'Azure PowerShell script: InlineScript copy'
inputs:
azureSubscription: 'Microsoft-Azure'
ScriptType: InlineScript
Inline: |
$ip = Invoke-RestMethod http://ipinfo.io/json | Select -exp ip #get agent ip
#add ip to whitelist
Add-AzStorageAccountNetworkRule -ResourceGroupName "myresourcegroup" -AccountName "mystorageaccount" -IPAddressOrRange $ip
azurePowerShellVersion: LatestVersion
2, Add another azure powershell task at the end of your pipeline to remove the whitelist.
- task: AzurePowerShell#5
displayName: 'Azure PowerShell script: InlineScript copy'
inputs:
azureSubscription: 'Microsoft-Azure'
ScriptType: InlineScript
Inline: |
$ip = Invoke-RestMethod http://ipinfo.io/json | Select -exp ip
Remove-AzStorageAccountNetworkRule -ResourceGroupName "myresourcegroup" -AccountName "mystorageaccount" -IPAddressOrRange $ip
azurePowerShellVersion: LatestVersion
Check document here for more information.
The IP ranges for cloud agents changes weekly. You can also check the weekly file and update the whitelist ip address manually. Check here for more information.
I have an user assigned managed identity (UAMI) in my Azure environment, and I'm able to programmatically assign it to whichever VM I choose with the following azure-cli command:
az vm identity assign --ids [Resource ID of VM] --identities [Resource ID of UAMI]
My question now is there a way to do this in Ansible? When I look at the "azure_rm_virtualmachine" module documentation, it doesn't appear there's a way to assign a UAMI at VM creation time; even though there IS functionality to enable a system-managed identity.
I've been looking at the "azure_rm_roleassignment" documentation, but despite every way I try to plug in the IDs for my VM and UAMI, Ansible returns with some sort of error every time. For instance, with the following:
- name: Create a role assignment
azure_rm_roleassignment:
scope: "/subscriptions/{{ subscriptionID }}/"
assignee_object_id: "{{ VMresourceID }}"
role_definition_id:
"{{ UAMIresourceID }}"
Error message is: "Error creating role assignment: Azure Error: InvalidPrincipalId\nMessage: The Principal ID 'XXX' is not valid. Principal ID must be a GUID."
Any ideas? Thanks in advance!
As I see, you set the assignee_object_id with the VM resource ID, I think it's the problem. No matter you choose the system-assigned or user-assigned managed identity, the assignee_object_id should be the object Id of it, not the resource Id. You can get object ID via the CLI command:
az vm identity show -g group_name -n vm_name --query principalId -o tsv
See more details about the assignee_object_id.
It doesn't appear that ansible currently supports a way to assign an UAMI to a VM; as with az vm identity assign --ids [Resource ID of VM] --identities [Resource ID of UAMI] in the CLI. Thus, as a stop-gap, we've opted to just go with system-managed identities for the time being. I've added the following to the VM creation in ansible:
azure_rm_virtualmachine:
vm_identity: SystemAssigned
register: my_server_reg
Then the following to my role assignment:
azure_rm_roleassignment:
scope: "/subscriptions/{{ subscription_id }}/"
assignee_object_id: "{{ my_server_reg.ansible_facts.azure_vm.identity.principalId }}"
role_definition_id:
"/subscriptions/{{ subscription_id }}/providers/Microsoft.Authorization/roleDefinitions/{{ blob_reader_roledef_id }}"
It'll be more cumbersome to manage, but I'm not seeing any other alternatives at this point in time.
I'm trying to deploy a VM from a custom image on Azure. I'm using the portal.Portal VM Creation
When I create a VM from a product in the marketplace, everything goes smoothly. When I capture an image of the VM (going through the deallocation and generalizing process), and then try to deploy a VM from that image, it fails while creating the VM due to a "no plan" error.Error Message
I've been troubleshooting and it seems I need a template and parameters json file to specify how to deploy, but the documentation on this is scarce and I'm unsure how to configure these files and what part of them is the "plan". I can take an existing VM/resource group and download the template.json and parameter.json files, but trying to create a new VM with these files fails.
Does anyone have experience with this or knows where adequate documentation is?
If you use Azure Powershell, you could use the Set-AzureRmVMPlan or Set-AzVMPlan cmdlet sets the Azure Marketplace plan information for a virtual machine. If you are using ARM template, you could refer to Microsoft.Compute virtualMachines template reference.
"plan": {
"name": "string",
"publisher": "string",
"product": "string",
"promotionCode": "string"
},
A Marketplace image in Azure has the following attributes:
Publisher: The organization that created the image. Examples: Canonical, MicrosoftWindowsServer
Offer: The name of a group of related images created by a publisher. Examples: UbuntuServer, WindowsServer
SKU: An instance of an offer, such as a major release of a distribution. Examples: 18.04-LTS, 2019-Datacenter
Version: The version number of an image SKU.
For more information about how to find image information, see Navigating and Selecting Azure Virtual Machine images with PowerShell and the Azure CLI in the Microsoft Azure documentation. You could use az vm image list --output table to see a list of popular VM images in the Azure Marketplace with the Azure CLI
Update
When I deploy a Xeams on Ubuntu in the marketplace, then capture VM image from the VM. I can reproduce this no plan issue when I directly create VM from this Image via the Azure portal. You could create a VM associated with the plan info via Azure CLI.
Find the plan info from the original VM(which was used for capturing custom Image)---Export template. Copy the plan info.
Then you could create a VM from the captured image associated with the plan info.
For example, this works on my side.
az vm create --name vmname --location westus --image Imagename --admin-username username --admin-password userpassword --plan-name xeams-ubuntu --plan-product xeams-ubuntu --plan-publisher apps-4-rent -g resourcegroup
Result:
Note: You can create VMs from an image in any resource group within your subscription. To create a VM in a different resource group than the image, specify the full resource ID to your image. Also, you could create VMs in the same region as the Image.