provider "azurerm" {
features {}
}
data "azurerm_management_group" "management_group" {
display_name = var.management_group_display_name
}
resource "azurerm_policy_definition" "deployment_policy_definition" {
name = "resources-in-eastus-policy"
policy_type = "Custom"
mode = "All"
display_name = "Allowed to only deploy in East US location"
management_group_id = data.azurerm_management_group.management_group.id
policy_rule = <<POLICY_RULE
{
"if": {
"not": {
"field": "location",
"in": "[parameters('allowedLocations')]"
}
},
"then": {
"effect": "audit"
}
}
POLICY_RULE
parameters = <<PARAMETERS
{
"allowedLocations": {
"type": "Array",
"metadata": {
"description": "The list of allowed locations for resources.",
"displayName": "Allowed locations",
"strongType": "location"
}
}
}
PARAMETERS
}
resource "azurerm_management_group_policy_assignment" "mngmt_grp_dep_pol_assign" {
name = "assign-pol-to-mgmt-grp"
policy_definition_id = azurerm_policy_definition.deployment_policy_definition.id
management_group_id = data.azurerm_management_group.management_group.id
parameters = <<PARAMETERS
{
"allowedLocations": {
"value": [ "eastus" ]
}
}
PARAMETERS
}
Error: creating/updating Policy Definition "resources-in-eastus-policy": policy.DefinitionsClient#CreateOrUpdateAtManagementGroup: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client 'live.com#XXX#gmail.com' with object id '0ab7dad7-dba2-46d9-8cc6-878647e9a5cb' does not have authorization to perform action 'Microsoft.Management/managementGroups/Microsoft.Management/1/Microsoft.Authorization/resources-in-eastus-policy/write' over scope '/providers/Microsoft.Management/managementGroups/providers/Microsoft.Management/managementGroups/1/providers/Microsoft.Authorization/policyDefinitions' or the scope is invalid. If access was recently granted, please refresh your credentials."
Azure Roles added for the owner/user of the azure-cli
The ID of the target management group where I am trying to create and assign the policy under the Tenant Root Group is 1
Error:
The client 'live.com#XXX#gmail.com' with object id
'0ab7daxxxxxxx-xxxxe9a5cb' does not have authorization to perform
action
'Microsoft.Management/managementGroups/Microsoft.Management/1/Microsoft.Authorization/resources-in-eastus-policy/write
As the error mentions the client doesn’t have proper RBAC role to perform policy definition creation on management groups.
Try to assign that ObjectId mentioned in the error , the proper role like Management Group Contributor OR Management Group Reader role.
Note: The principal/user which is deploying ,must have permissions like Contributor to create resources at the tenant scope and to assign that permission one must have Owner role
Also see below table from management-group-access :
From the management group , Go to Access control (IAM), add your client(user/service principal) as an RBAC role
or provide role through powershell:
New-AzRoleAssignment -Scope '/' -RoleDefinitionName 'Owner' -ObjectId <objectidofftheclient>
Then wait for some time for the role to reflect and then try to create policy assignment to management group:
Policy assignment made to management group.
Please make sure if the management group is reflected properly and check the id is correct in terraform, if it is already created in portal. Else import them using terraform import and then perform terraform operations.
I'm trying to create 3 datalakes using terraform by I'm getting a 403 error.
I'm using a admin account with owner roler. I also tried to create an SP and set Blob Reader Role.
Below find my code and the errror
Terraform v1.2.1
on windows_amd64
provider registry.terraform.io/hashicorp/azuread v2.22.0
provider registry.terraform.io/hashicorp/azurerm v3.7.0
resource "azurerm_storage_data_lake_gen2_filesystem" "stg-datalake" {
for_each = toset(["bronze", "silver", "gold"])
name = each.value
storage_account_id = azurerm_storage_account.stg-datalake.id
ace {
scope = "access"
type = "user"
id = azurerm_data_factory.adf.identity[0].principal_id
permissions = "rwx"
}
}
Error:
Error: checking for existence of existing File System "gold" (Account "stgaclientteste"): datalakestore.Client#GetProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: error response cannot be parsed: {"" '\x00' '\x00'} error: EOF
The issue still persists after months, hence I used the workaround below. An ADLS gen2 filesystem is somewhat different than a regular storage container, you need the Storage Blob Data Owner role to create/update the filesystem.
data "azurerm_client_config" "current" {}
# HACK: Role assignment is needed to apply adls gen2 filesystem changes
resource "azurerm_role_assignment" "role_assignment" {
scope = var.storage_account_id
role_definition_name = "Storage Blob Data Owner"
principal_id = data.azurerm_client_config.current.object_id
}
# HACK: Sleep is needed to wait for role assignment to propagate
resource "time_sleep" "role_assignment_sleep" {
create_duration = "60s"
triggers = {
role_assignment = azurerm_role_assignment.role_assignment.id
}
}
resource "azurerm_storage_data_lake_gen2_filesystem" "filesystem" {
name = var.filesystem_name
storage_account_id = var.storage_account_id
depends_on = [time_sleep.role_assignment_sleep]
}
MT
I had to set the permission to the Resource Group where the stgaccount where created.
Setting just to stg account didn't work.
Thanks for your answer!
I'm trying to apply the Azurerm_policy_assignment using Terraform - i'm unable to apply the policy assignment, looks like an identity related issue:
policy.AssignmentsClient#Create: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="MissingSubscription" Message="The request did not have a subscription or a valid tenant level resource provider
The following is the code that I am trying to run:
resource "azurerm_policy_assignment" "policyassignment" {
name = "policy1assignment"
scope = var.policyscope
policy_definition_id = "/providers/Microsoft.Authorization/policySetDefinitions/jfda8af9-198f-asa3-1234-dsa8dfs090fsd"
description = "TestPolicy"
display_name = "TestPolicy"
location = var.location
identity {
type = "SystemAssigned"
}
}
I configured an AKS cluster to use a system-assigned managed identity to access to other Azure resources
resource "azurerm_subnet" "aks" {
name = var.aks_subnet_name
resource_group_name = azurerm_resource_group.main.name
virtual_network_name = module.network.vnet_name
address_prefix = var.aks_subnet
service_endpoints = ["Microsoft.KeyVault"]
}
resource "azurerm_kubernetes_cluster" "aks_main" {
name = module.aks_name.result
depends_on = [azurerm_subnet.aks]
location = azurerm_resource_group.main.location
resource_group_name = azurerm_resource_group.main.name
dns_prefix = "aks-${local.name}"
kubernetes_version = var.k8s_version
addon_profile {
oms_agent {
# For monitoring containers
enabled = var.addons.oms_agent
log_analytics_workspace_id = azurerm_log_analytics_workspace.example.id
}
kube_dashboard {
enabled = true
}
azure_policy {
# If we want to enfore policy definitions in the future
# Check requirements https://learn.microsoft.com/en-ie/azure/governance/policy/concepts/policy-for-kubernetes
enabled = var.addons.azure_policy
}
}
default_node_pool {
name = "default"
orchestrator_version = var.k8s_version
node_count = var.default_node_pool.node_count
vm_size = var.default_node_pool.vm_size
type = "VirtualMachineScaleSets"
availability_zones = var.default_node_pool.zones
# availability_zones = ["1", "2", "3"]
max_pods = 250
os_disk_size_gb = 128
vnet_subnet_id = azurerm_subnet.aks.id
node_labels = var.default_node_pool.labels
enable_auto_scaling = var.default_node_pool.cluster_auto_scaling
min_count = var.default_node_pool.cluster_auto_scaling_min_count
max_count = var.default_node_pool.cluster_auto_scaling_max_count
enable_node_public_ip = false
}
# Configuring AKS to use a system-assigned managed identity to access
identity {
type = "SystemAssigned"
}
network_profile {
load_balancer_sku = "standard"
outbound_type = "loadBalancer"
network_plugin = "azure"
# if non-azure network policies
# https://azure.microsoft.com/nl-nl/blog/integrating-azure-cni-and-calico-a-technical-deep-dive/
network_policy = "calico"
dns_service_ip = "10.0.0.10"
docker_bridge_cidr = "172.17.0.1/16"
service_cidr = "10.0.0.0/16"
}
lifecycle {
ignore_changes = [
default_node_pool,
windows_profile,
]
}
}
I want to use that managed identity (the service principal created inside AKS cluster section code) to give it roles like this Network Contributor over a subnet:
resource "azurerm_role_assignment" "aks_subnet" {
# Giving access to AKS SP identity created to akssubnet by assigning it
# a Network Contributor role
scope = azurerm_subnet.aks.id
role_definition_name = "Network Contributor"
principal_id = azurerm_kubernetes_cluster.aks_main.identity[0].principal_id
# principal_id = azurerm_kubernetes_cluster.aks_main.kubelet_identity[0].object_id
# principal_id = data.azurerm_user_assigned_identity.test.principal_id
# skip_service_principal_aad_check = true
}
But the output I got after terraform apply is:
Error: authorization.RoleAssignmentsClient#Create: Failure responding
to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error.
Status=403 Code="AuthorizationFailed"
Message="The client 'afd5bd09-c294-4597-9c90-e1ee293e5f3a' with object id
'afd5bd09-c294-4597-9c90-e1ee293e5f3a' does not have authorization
to perform action 'Microsoft.Authorization/roleAssignments/write'
over scope '/subscriptions/77dfff95-fbd3-4a15-b97a-b7182939e61a/resourceGroups/rhd-spec-prod-main-6loe4lpkr0hd8/providers/Microsoft.Network/virtualNetworks/rhd-spec-prod-main-wdaht6cn7s3s8/subnets/aks-subnet/providers/Microsoft.Authorization/roleAssignments/8733864c-a5f7-a6a9-a61d-6393989f0ad1'
or the scope is invalid. If access was recently granted, please refresh your credentials."
on aks.tf line 23, in resource "azurerm_role_assignment" "aks_subnet":
23: resource "azurerm_role_assignment" "aks_subnet" {
It seems the service principal is being created does not have enough privileges to perform a role assignment over the subnet, or maybe I have wrong the scope attribute. I am passing there, the aks subnet id.
What am I doing wrong?
UPDATE
Checking the way Managed Identities has role assigneds, looks like we can only assign it roles related with Subscriptions, Resource Groups, Storage services, SQL services, and KeyVault.
Reading here
Before you can use the managed identity, it has to be configured. There are two steps:
Assign a role for the identity, associating it with the subscription that will be used to run Terraform. This step gives the identity permission to access Azure Resource Manager (ARM) resources.
Configure access control for one or more Azure resources. For example, if you use a key vault and a storage account, you will need to configure the vault and container separately.
Before you can create a resource with a managed identity and then assign an RBAC role, your account needs sufficient permissions. You need to be a member of the account Owner role, or have Contributor plus User Access Administrator roles.
Trying to proceed accordingly, I defined this section code:
resource "null_resource" "wait_for_resource_to_be_ready" {
provisioner "local-exec" {
command = "sleep 60"
}
depends_on = [
azurerm_kubernetes_cluster.aks_main
]
}
data "azurerm_subscription" "current" {}
# FETCHING THE IDENTITY CREATED ON AKS CLUSTER
data "azurerm_user_assigned_identity" "test" {
name = "${azurerm_kubernetes_cluster.aks_main.name}-agentpool"
resource_group_name = azurerm_kubernetes_cluster.aks_main.node_resource_group
}
data "azurerm_role_definition" "contributor" {
name = "Network Contributor"
}
resource "azurerm_role_assignment" "aks_subnet" {
# Giving access to AKS SP identity created to akssubnet by assigning it
# a Network Contributor role
# name = azurerm_kubernetes_cluster.aks_main.name
# scope = var.aks_subnet_name # azurerm_subnet.aks.id var.aks_subnet
scope = data.azurerm_subscription.current.id
#role_definition_name = "Network Contributor"
role_definition_id = "${data.azurerm_subscription.current.id}${data.azurerm_role_definition.contributor.id}"
# principal_id = azurerm_kubernetes_cluster.aks_main.identity[0].principal_id
# principal_id = azu rerm_kubernetes_cluster.aks_main.kubelet_identity[0].object_id
principal_id = data.azurerm_user_assigned_identity.test.principal_id
skip_service_principal_aad_check = true
depends_on = [
null_resource.wait_for_resource_to_be_ready
]
}
The terraform workflow try to create the role ...
> terraform_0.12.29 apply "prod_Infrastructure.plan"
null_resource.wait_for_resource_to_be_ready: Creating...
null_resource.wait_for_resource_to_be_ready: Provisioning with 'local-exec'...
null_resource.wait_for_resource_to_be_ready (local-exec): Executing: ["/bin/sh" "-c" "sleep 60"]
null_resource.wait_for_resource_to_be_ready: Still creating... [10s elapsed]
null_resource.wait_for_resource_to_be_ready: Still creating... [20s elapsed]
null_resource.wait_for_resource_to_be_ready: Still creating... [30s elapsed]
null_resource.wait_for_resource_to_be_ready: Still creating... [40s elapsed]
null_resource.wait_for_resource_to_be_ready: Still creating... [50s elapsed]
null_resource.wait_for_resource_to_be_ready: Still creating... [1m0s elapsed]
null_resource.wait_for_resource_to_be_ready: Creation complete after 1m0s [id=8505830187297683728]
azurerm_role_assignment.aks_subnet: Creating...
but finally got the same AuthorizationFailed error this time over the subscription passed.
Error: authorization.RoleAssignmentsClient#Create: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client 'afd5bd09-c294-4597-9c90-e1ee293e5f3a' with object id 'afd5bd09-c294-4597-9c90-e1ee293e5f3a' does not have authorization to perform action 'Microsoft.Authorization/roleAssignments/write' over scope '/subscriptions/77dfff95-fbd3-4a15-b97a-b7182939e61a' or the scope is invalid. If access was recently granted, please refresh your credentials."
on aks.tf line 145, in resource "azurerm_role_assignment" "aks_subnet":
145: resource "azurerm_role_assignment" "aks_subnet" {
Not sure at all how to verify this statement
Before you can create a resource with a managed identity and then assign an RBAC role, your account needs sufficient permissions. You need to be a member of the account Owner role, or have Contributor plus User Access Administrator roles.
By the way, I have the owner role in the subscription I am working with.
UPDATE 2
The object id referenced on both error messages above, belong to a service principal within my tenant.
It is
az ad sp show --id afd5bd09-c294-4597-9c90-e1ee293e5f3a
{
"accountEnabled": "True",
"addIns": [],
"alternativeNames": [],
"appDisplayName": "Product-xxxx-ServicePrincipal-Production",
"appId": "ff9c642c-06b9-47e2-9565-e3f6e782e14f",
"appOwnerTenantId": "xxxxxxxx",
"appRoleAssignmentRequired": false,
"appRoles": [],
"applicationTemplateId": null,
"deletionTimestamp": null,
"displayName": "Product-xxxx-ServicePrincipal-Production",
"errorUrl": null,
"homepage": null,
"informationalUrls": {
"marketing": null,
"privacy": null,
"support": null,
"termsOfService": null
},
"keyCredentials": [],
"logoutUrl": null,
"notificationEmailAddresses": [],
"oauth2Permissions": [],
# THIS IS THE OBJECT ID
"objectId": "afd5bd09-c294-4597-9c90-e1ee293e5f3a",
"objectType": "ServicePrincipal",
"odata.metadata": "https://graph.windows.net/15f996bf-aad1-451c-8d17-9b95d025eafc/$metadata#directoryObjects/#Element",
"odata.type": "Microsoft.DirectoryServices.ServicePrincipal",
"passwordCredentials": [],
"preferredSingleSignOnMode": null,
"preferredTokenSigningKeyEndDateTime": null,
"preferredTokenSigningKeyThumbprint": null,
"publisherName": "xxxxxxx",
"replyUrls": [],
"samlMetadataUrl": null,
"samlSingleSignOnSettings": null,
"servicePrincipalNames": [
"ff9c642c-06b9-47e2-9565-e3f6e782e14f"
],
"servicePrincipalType": "Application",
"signInAudience": "AzureADMyOrg",
"tags": [
"WindowsAzureActiveDirectoryIntegratedApp"
],
"tokenEncryptionKeyId": null
}
Regarding permissions, not sure if it has sufficient, I would say yes, since it is used for multiple stuff in the subscription
What about Users Consent permissions? I don't have anything there
But on the other hand, why the process is trying to assign the role by using this service principal?
I mean, the use of a managed identity, is intended to move away the use of service principals, but perhaps, the workflow procees use this SP just to assign the role to the managed identity and from that in forward the access will be granted by the managed identity (?)
From docs: https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-rest#add-a-role-assignment
To call this API, you must have access to the Microsoft.Authorization/roleAssignments/write operation. Of the built-in roles, only Owner and User Access Administrator are granted access to this operation.
So your service principal must have the role owner or user access administrator. Or you have to create a custom role with sufficient permissions.
Regarding the workflow, I agree. It is quiet counter intuitive.
old answer
There is this bug (?) where azure states that the resource has been created but not all services have access it.
You can have it wait for a minute with something like this:
resource "null_resource" "wait_for_resource_to_be_ready" {
provisioner "local-exec" {
command = "sleep 60"
}
depends_on = [
azurerm_kubernetes_cluster.aks_main
]
}
Add a depends_on statment to your "azurerm_role_assignment" "aks_subnet" resource:
depends_on = [
null_resource.wait_for_resource_to_be_ready
]
Now first your cluster will be created, then terrform will wait for 60 seconds. Then your role_assignment will take place and will hopefully be able to grant the role.
I have written terraform for creating the user, resource group, and roledefinition.
I need to have the scope of resource definition be the resource group that I created.
I don't know how to do that. It would be great if someone could help on this.
########### for creating user ####
# Configure the Azure Provider
provider "azurerm" {
version = "~> 1.30"
subscription_id="723604be-b74b-4473-9d11-1802dbfdb787"
}
provider "azuread" {
version = "~> 0.4"
subscription_id="723604be-b74b-4473-9d11-1802dbfdb787"
}
resource "azuread_user" "test" {
user_principal_name = "user1#catch.whizlabstesting.com"
display_name = "User1"
mail_nickname = "User1"
password = "Muneeshpandi#17"
force_password_change = "false"
}
##### creating resource group #####
resource "azurerm_resource_group" "terraform_rg" {
name = "user1_rgp"
location = "East US"
}
########## creating role definition ##########
data "azurerm_subscription" "primary" {}
resource "azurerm_role_definition" "sql_role" {
name = "sql_role"
scope = "data.azurerm_subscription.primary.id"
description = "This is a custom role to create sql database"
permissions {
actions = ["*"]
not_actions = []
}
assignable_scopes = [
"/subscriptions/723604be-b74b-4473-9d11-1802dbfdb787/resourceGroups/user1_rgp"
]
}
Getting following error while executing above code:
Error: authorization.RoleDefinitionsClient#CreateOrUpdate: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="MissingSubscription" Message="The request did not have a subscription or a valid tenant level resource provider."
How do I make the scope of a custom role be Resourcegroup in azure?
To create a custom role for the resource group, you need to have the permission Microsoft.Authorization/roleDefinitions/write, and to assign the custom role to a user, you need to have the permission Microsoft.Authorization/roleAssignments/write. The simplest way is that you have the Onwer role of the subscription.
And to create an Azure AD user:
To add or delete users you must be a User administrator or Global
administrator.
When you have all the needed permission. Let's focus on your code. You also need to assign the custom role to the user you created with the scope of the resource group. Then you can change the code like this:
resource "azurerm_role_definition" "sql_role" {
name = "sql_role"
scope = data.azurerm_subscription.primary.id
description = "This is a custom role to create sql database"
permissions {
actions = ["*"]
not_actions = []
}
assignable_scopes = [
data.azurerm_subscription.primary.id
]
}
resource "azurerm_role_assignment" "example" {
scope = azurerm_resource_group.terraform_rg.id
role_definition_id = azurerm_role_definition.sql_role.id
principal_id = azuread_user.test.id
}
If you only want the custom available for the resource group, you can change the assignable_scopes with the resource group Id as azurerm_resource_group.terraform_rg.id.