This definition clearly mentions you can use networkPolicy property as part of the networkProfile and set it to Calico, but that doesnt work. AKS creating just times out with all the nodes being in Not Ready state.
you need enable underlying provider feature:
az feature list --query "[?contains(name, 'Container')].{name:name, type:type}" # example to list all features
az feature register --name EnableNetworkPolicy --namespace Microsoft.ContainerService
az provider register -n Microsoft.ContainerService
after that you can just use REST API\ARM Template to create AKS:
{
"location": "location1",
"tags": {
"tier": "production",
"archv2": ""
},
"properties": {
"kubernetesVersion": "1.12.4", // has to be 1.12.x, 1.11.x doesnt support calico AFAIK
"dnsPrefix": "dnsprefix1",
"agentPoolProfiles": [
{
"name": "nodepool1",
"count": 3,
"vmSize": "Standard_DS1_v2",
"osType": "Linux"
}
],
"linuxProfile": {
"adminUsername": "azureuser",
"ssh": {
"publicKeys": [
{
"keyData": "keydata"
}
]
}
},
"servicePrincipalProfile": {
"clientId": "clientid",
"secret": "secret"
},
"addonProfiles": {},
"enableRBAC": false,
"networkProfile": {
"networkPlugin": "azure",
"networkPolicy": "calico", // set policy here
"serviceCidr": "xxx",
"dnsServiceIP": "yyy",
"dockerBridgeCidr": "zzz"
}
}
}
ps.
Unfortunately, helm doesnt seem to work at the time of writing (I suspect this is because kubectl port-forward which helm relies on doesnt work as well ).
Related
I am trying to get instanceview objects(VirtualMachineScaleSetInstanceViewInner) of all Azure's VirtualMachineScaleSets under a Subscription and this requires both ResourceGroup Name and Vmss name together.
azureResourceManager.virtualMachines().manager().serviceClient().getVirtualMachineScaleSets().getInstanceView(resourceGroupName, virtualMachineScaleSet.name(), Context.NONE);
How do I get specific VirtualMachineScaleSets under a ResourceGroup? I only see AzureResourceManager.ResourceGroups() and AzureResourceManager.virtualMachineScaleSets(), but nothing that gets virtualMachineScaleSets under a ResourceGroup.
Thanks
I have tried to get the VMSS instance in a Resource group by using the below RestApi:
GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/virtualMachineScaleSets/{vmScaleSetName}?api-version=2022-08-01
output:
Body:
{
"name": "rajtestVMSS",
"id": "/subscriptions/*********/resourceGroups/abcaaaa/providers/Microsoft.Compute/virtualMachineScaleSets/rajtestVMSS",
"type": "Microsoft.Compute/virtualMachineScaleSets",
"location": "westus2",
"tags": {
"azsecpack": "nonprod",
"platformsettings.host_environment.service.platform_optedin_for_rootcerts": "true"
},
"sku": {
"name": "Standard_D2s_v3",
"tier": "Standard",
"capacity": 2
},
"properties": {
"singlePlacementGroup": false,
"upgradePolicy": {
"mode": "Manual"
},
"scaleInPolicy": {
"rules": [
"Default"
]
},
"virtualMachineProfile": {
"osProfile": {
"computerNamePrefix": "rajtestvm",
"adminUsername": "rajtest",
"windowsConfiguration": {
"provisionVMAgent": true,
"enableAutomaticUpdates": true,
"enableVMAgentPlatformUpdates": false
------------------
------------------
------------------
You can use the below java code to get the VirtualMachine ScaleSet instanceview for a ResourceGroup.
import com.azure.core.util.Context;
public final class Main {
public static void getAVirtualMachineScaleSet(com.azure.resourcemanager.AzureResourceManager azure) {
azure
.virtualMachines()
.manager()
.serviceClient()
.getVirtualMachineScaleSets()
.getByResourceGroupWithResponse("myResourceGroup", "myVirtualMachineScaleSet", null, Context.NONE);
}
}
Thanks to #XiaofeiCao for the github link to know more about Azure Resource Manager client library for Java.
How can I get the cluster ca certificate from the arm template for some ManagedClusters?
In terraform we use:
azurerm_kubernetes_cluster.aks_cluster.kube_config.0.cluster_ca_certificate
But in ARM I can't find it anywhere.
Can you please give some help?
Thanks
My solution:
"resources": [
{
"type": "Microsoft.Resources/deploymentScripts",
"apiVersion": "2019-10-01-preview",
"name": "get-cluster-ca",
"location": "[resourceGroup().location]",
"identity": {
....
}
},
"kind": "AzureCLI",
"azCliVersion": "2.0.80",
"environmentVariables": [
{
"name": "RESOURCE_GROUP_NAME",
"value": "[resourceGroup().name]"
},
{
"name": "CLUSTER_NAME",
"value": "[variables('cluster_name')]"
}
],
"scriptContent": "#!/bin/bash\r\n\r\naz aks get-credentials --resource-group ${RESOURCE_GROUP_NAME} --name ${CLUSTER_NAME}\r\n\r\naz aks install-cli\r\n\r\necho '{\"ca\":' $(kubectl config view --raw -o json | jq -c '.clusters[0].cluster.\"certificate-authority-data\"') > $AZ_SCRIPTS_OUTPUT_PATH\r\n",
"timeout": "PT30M",
"cleanupPreference": "OnSuccess",
"retentionInterval": "P1D"
}
],
"outputs": {
"result": {
"value": "[reference('get-cluster-ca').outputs.ca]",
"type": "string"
}
}
The following command creates a container in Azure that is mapped to a file share/volume:
az container create -g MyResourceGroup --name myapp --image myimage:latest
--azure-file-volume-share-name myshare --azure-file-volume-account-name mystorageacct
--azure-file-volume-account-key mystoragekey --azure-file-volume-mount-path /mnt/azfile
But I need my container to be mapped to two volumes, not just one. Is this possible?
I do not know if it is possible to do this via azure cli. I do know that you can do this through Azure Resource Manager Templates.
In this example, see how the container group has an array of volumes, while each container can have an array of volume mounts.
{
"type": "Microsoft.ContainerInstance/containerGroups",
"apiVersion": "2018-10-01",
"name": "[parameters('ContainerGroupName')]",
"location": "australiaeast",
"identity": {
"type": "UserAssigned",
"userAssignedIdentities": {
"[variables('managedIdentityId')]": {}
}
},
"dependsOn": [
"[variables('managedIdentityId')]"
],
"properties": {
"containers": [
{
"name": "[parameters('ContainerGroupName')]",
"properties": {
"image": "[parameters('SourceImage')]",
"ports": [{"port": 80},{"port": 443}],
"environmentVariables": [],
"resources": { "requests": { "memoryInGB": 1.5, "cpu": 1 } },
"volumeMounts": [
{
"name": "httpscertificatevolume",
"mountPath": "/https"
},
{
"name": "videofoldervolume",
"mountPath": "[variables('videoFolderMountPath')]"
}
]
}
}
],
"volumes": [{
"name": "httpscertificatevolume",
"azureFile": {
"shareName": "[parameters('HttpsCertificateFileShare')]",
"storageAccountName": "[parameters('StorageAccountName')]",
"storageAccountKey" : "[parameters('StorageAccountKey')]"
}
},
{
"name": "videofoldervolume",
"azureFile": {
"shareName": "[parameters('VideoFileShare')]",
"storageAccountName": "[parameters('StorageAccountName')]",
"storageAccountKey" : "[parameters('StorageAccountKey')]"
}
}
]
}
}
I have been fighting with this for a couple of hours, and I can't work out how to get Kubernetes to configure an Azure Load Balancer when its an IaaS Kubernetes cluster.
This work OOB with AKS, as you'd expect.
Obviously, I am missing where I can input the Service Principal to allow k8s to be able to configure resources in Azure.
The official recommended way is ACS-Engine. Describing it is a bit too much for an answer, but when you define your cluster you are supposed to provide Azure credentials to it:
{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes"
},
"masterProfile": {
"count": 1,
"dnsPrefix": "",
"vmSize": "Standard_D2_v2"
},
"agentPoolProfiles": [
{
"name": "agentpool1",
"count": 3,
"vmSize": "Standard_D2_v2",
"availabilityProfile": "AvailabilitySet"
}
],
"linuxProfile": {
"adminUsername": "azureuser",
"ssh": {
"publicKeys": [
{
"keyData": ""
}
]
}
},
"servicePrincipalProfile": {
"clientId": "CLIENT_ID_GOES_HERE",
"secret": "CLIENT_SECRET_GOES_HERE"
}
}
}
After you provision a cluster with proper input it will work out of the box. Walkthrough
I am developing ARM template to deploy an App Service Environment v2 configured with an Internal Load Balancer (ILB ASE). Is there a way to grab the Virtual IP (VIP) address that the Internal Load Balancer gets from the vnet it is attached to as an output? When I look at the properties of the ASE via PowerShell after it is provisioned, I do not see a property for the IP address, or for the load balancer.
After much research and testing...there is currently no way to do this as an output from the ARM template. Here are the ways that the value can be collected:
Via Resource Explorer...although this is not very helpful for doing it programmatically but it did help me figure out the other 2 ways
Using PowerShell to query the management.azure.com API but you have to publish an app with the appropriate permissions and assign the app to have permissions in the subscription you are trying to query resources from
Using Azure CLI. This method turned out to be the easiest.
I needed this value to fully automate the deployment of an App Gateway sitting in front of an ILB ASE. I use Terraform for deployment automation and I run the Terraform configs from Azure Cloud Shell. I kick off my deployments with a shell script where I dynamically get the storage account key to the storage account where I store state files. I then query the ILB ASE to get the IP address and set it to a variable that I then pass into Terraform
Below is a copy of the shell script I use:
#!/bin/bash
set -eo pipefail
# The block below will grab the access key for the storage account that is used
# to store state files
subscription_name="<my_subscription_name>"
tfstate_storage_resource_group="terraform-state-rg"
tfstate_storage_account="<name_of_statefile_storage_account>"
subscription_id="my_subscription_id>"
ilbase_rg_name="<name_of_resourcegroup_where_ase_is_deployed>"
ilbase_name="<name_of_ase>"
az account set --subscription "$subscription_name"
tfstate_storage_access_key=$(
az storage account keys list \
--resource-group "$tfstate_storage_resource_group" \
--account-name "$tfstate_storage_account" \
--query '[0].value' -o tsv
)
echo ""
echo "Terraform state storage account access key:"
echo $tfstate_storage_access_key
echo ""
# The block below will get the Virtual IP of the ASE Internal Load Balancer
# which will be used to create the App GW
ilbase_virtual_ip=$(
az resource show \
--ids "/subscriptions/$subscription_id/resourceGroups/$ilbase_rg_name/providers/Microsoft.Web/hostingEnvironments/$ilbase_name/capacities/virtualip" \
--query "additionalProperties.internalIpAddress"
)
echo ""
echo "ASE internal load balancer IP:"
echo $ilbase_virtual_ip
echo ""
terraform plan \
-var "tfstate_access_key=$tfstate_storage_access_key" \
-var "ilbase_virtual_ip=$ilbase_virtual_ip"
You can use output like this:
"outputs": {
"privateIp": {
"type": "string",
"value": "[reference(parameters('lbname')).frontendIPConfigurations[0].properties.privateIPAddress]"
}
}
Here is my template, create one Vnet and one internal load balancer:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"vnetName": {
"type": "string",
"defaultValue": "VNet1",
"metadata": {
"description": "VNet name"
}
},
"vnetAddressPrefix": {
"type": "string",
"defaultValue": "10.0.0.0/16",
"metadata": {
"description": "Address prefix"
}
},
"subnet1Prefix": {
"type": "string",
"defaultValue": "10.0.0.0/24",
"metadata": {
"description": "Subnet 1 Prefix"
}
},
"subnet1Name": {
"type": "string",
"defaultValue": "Subnet1",
"metadata": {
"description": "Subnet 1 Name"
}
},
"subnet2Prefix": {
"type": "string",
"defaultValue": "10.0.1.0/24",
"metadata": {
"description": "Subnet 2 Prefix"
}
},
"subnet2Name": {
"type": "string",
"defaultValue": "Subnet2",
"metadata": {
"description": "Subnet 2 Name"
}
},
"lbname": {
"defaultValue": "jasonlbb",
"type": "String"
}
},
"variables": {
"virtualnetworkname" : "vnet1",
"apiVersion": "2015-06-15",
"vnetID": "[resourceId('Microsoft.Network/virtualNetworks',variables('virtualnetworkname'))]",
"subnetRef": "[concat(variables('vnetID'),'/subnets/',parameters('subnet1Name'))]"
},
"resources": [
{
"apiVersion": "2015-06-15",
"type": "Microsoft.Network/virtualNetworks",
"name": "[parameters('vnetName')]",
"location": "[resourceGroup().location]",
"properties": {
"addressSpace": {
"addressPrefixes": [
"[parameters('vnetAddressPrefix')]"
]
},
"subnets": [
{
"name": "[parameters('subnet1Name')]",
"properties": {
"addressPrefix": "[parameters('subnet1Prefix')]"
}
},
{
"name": "[parameters('subnet2Name')]",
"properties": {
"addressPrefix": "[parameters('subnet2Prefix')]"
}
}
]
}
},
{
"apiVersion": "2015-05-01-preview",
"type": "Microsoft.Network/loadBalancers",
"name": "[parameters('lbname')]",
"location": "[resourceGroup().location]",
"dependsOn": [
"[variables('vnetID')]"
],
"properties": {
"frontendIPConfigurations": [
{
"properties": {
"subnet": {
"id": "[variables('subnetRef')]"
},
"privateIPAllocationMethod": "Dynamic"
},
"name": "LoadBalancerFrontend"
}
],
"backendAddressPools": [
{
"name": "BackendPool1"
}
],
"loadBalancingRules": [
{
"properties": {
"frontendIPConfiguration": {
"id": "[concat(resourceId('Microsoft.Network/loadBalancers', parameters('lbname')), '/frontendIpConfigurations/LoadBalancerFrontend')]"
},
"backendAddressPool": {
"id": "[concat(resourceId('Microsoft.Network/loadBalancers', parameters('lbname')), '/backendAddressPools/BackendPool1')]"
},
"probe": {
"id": "[concat(resourceId('Microsoft.Network/loadBalancers', parameters('lbname')), '/probes/lbprobe')]"
},
"protocol": "Tcp",
"frontendPort": 80,
"backendPort": 80,
"idleTimeoutInMinutes": 15
},
"Name": "lbrule"
}
],
"probes": [
{
"properties": {
"protocol": "Tcp",
"port": 80,
"intervalInSeconds": 15,
"numberOfProbes": 2
},
"name": "lbprobe"
}
]
}
}
],
"outputs": {
"privateIp": {
"type": "string",
"value": "[reference(parameters('lbname')).frontendIPConfigurations[0].properties.privateIPAddress]"
}
}
}
Here is the screenshot about the result:
Hope this helps.
If you’re using Terraform, here’s how I got it working. Had to use the external data source in Terraform coupled with Azure CLI and jq to get around the bugs in Azure and the Terraform External Data provider.
# As of writing, the ASE ARM deployment don’t return the IP address of the ILB
# ASE. This workaround querys Azure’s API to get the values we need for use
# elsewhere in the script.
# See this https://stackoverflow.com/a/49436100
data “external” “app_service_environment_ilb_ase_ip_address” {
# This calls the Azure CLI then passes the value to jq to return JSON as a single
# string so that external provider can parse it properly. Otherwise you get an
# error. See this bug https://github.com/terraform-providers/terraform-provider-external/issues/23
program = [“bash”, “-c”, “az resource show --ids ${local.app_service_environment_id}/capacities/virtualip --query ‘{internalIpAddress: internalIpAddress}’ | jq -c”]
# Explicit dependency on the ASE ARM deployment because this command will fail
# if that resource isn’t built yet.
depends_on = [azurerm_template_deployment.ase]
}