I'm trying to create 3 datalakes using terraform by I'm getting a 403 error.
I'm using a admin account with owner roler. I also tried to create an SP and set Blob Reader Role.
Below find my code and the errror
Terraform v1.2.1
on windows_amd64
provider registry.terraform.io/hashicorp/azuread v2.22.0
provider registry.terraform.io/hashicorp/azurerm v3.7.0
resource "azurerm_storage_data_lake_gen2_filesystem" "stg-datalake" {
for_each = toset(["bronze", "silver", "gold"])
name = each.value
storage_account_id = azurerm_storage_account.stg-datalake.id
ace {
scope = "access"
type = "user"
id = azurerm_data_factory.adf.identity[0].principal_id
permissions = "rwx"
}
}
Error:
Error: checking for existence of existing File System "gold" (Account "stgaclientteste"): datalakestore.Client#GetProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: error response cannot be parsed: {"" '\x00' '\x00'} error: EOF
The issue still persists after months, hence I used the workaround below. An ADLS gen2 filesystem is somewhat different than a regular storage container, you need the Storage Blob Data Owner role to create/update the filesystem.
data "azurerm_client_config" "current" {}
# HACK: Role assignment is needed to apply adls gen2 filesystem changes
resource "azurerm_role_assignment" "role_assignment" {
scope = var.storage_account_id
role_definition_name = "Storage Blob Data Owner"
principal_id = data.azurerm_client_config.current.object_id
}
# HACK: Sleep is needed to wait for role assignment to propagate
resource "time_sleep" "role_assignment_sleep" {
create_duration = "60s"
triggers = {
role_assignment = azurerm_role_assignment.role_assignment.id
}
}
resource "azurerm_storage_data_lake_gen2_filesystem" "filesystem" {
name = var.filesystem_name
storage_account_id = var.storage_account_id
depends_on = [time_sleep.role_assignment_sleep]
}
MT
I had to set the permission to the Resource Group where the stgaccount where created.
Setting just to stg account didn't work.
Thanks for your answer!
Related
I'm trying to deploy a web app with a database on Azure but can't seem to get it to work despite double/triple checking the credentials for the Tenant in Azure. Tried creating new client secrets but doesn't work regardless.
Unable to list provider registration status, it is possible that this is due to invalid credentials or the service principal does not have permission to use the Resource Manager API, Azure error: resources.ProvidersClient#List: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client '########-########-########-########-########' with object id '########-########-########-########-########' does not have authorization to perform action 'Microsoft.Resources/subscriptions/providers/read' over scope '/subscriptions/########-########-########-########-########' or the scope is invalid. If access was recently granted, please refresh your credentials."
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "=3.0.0"
}
}
}
provider "azurerm" {
features {}
subscription_id = var.subscription_id
client_id = var.client_id
client_secret = var.client_secret
tenant_id = var.tenant_id
}
resource "azurerm_resource_group" "example" {
name = "azure-tf-bgapp"
location = "West Europe"
}
resource "azurerm_container_group" "example" {
name = "bgapp-tf"
location = azurerm_resource_group.example.location
resource_group_name = azurerm_resource_group.example.name
ip_address_type = "Public"
dns_name_label = "aci-label"
os_type = "Linux"
container {
name = "bgapp-web"
image = "shekeriev/bgapp-web"
cpu = "0.5"
memory = "1.5"
ports {
port = 80
protocol = "TCP"
}
}
container {
name = "bgapp-web"
image = "shekeriev/bgapp-db"
cpu = "0.5"
memory = "1.5"
environment_variables = {
"MYSQL_ROOT_PASSWORD" = "Password1"
}
}
tags = {
environment = "bgapp"
}
}
I tried in my environment and got below results:
Initially I tried the same code and got same error in my environment.
Console:
The above error occurs due to your (Service principal) doesn't has required permission to do that operation (Authorization).
After assigning a role like Owner to the service principal code worked successfully.
Go to portal -> subscription -> Access control (IAM) -> Add role assignments -> owner -> Add your service principal -> review + create.
After I executed code of terraform it executed perfectly.
Console:
Portal:
I'm trying to create a Synapse Workspace using Terraform. The workspace deploys successfully, but when testing the connection to the WorkSpaceDefaultStorage in Synapse studio I receive the following error:
ADLS Gen2 operation failed for: Storage operation '' on container 'testconnection' get failed with 'Operation returned an invalid status code 'Forbidden''.
The code for the Synapse workspace deployment:
resource "azurerm_storage_account" "sa" {
name = var.storage_account_name
location = azurerm_resource_group.rg.location
resource_group_name = azurerm_resource_group.rg.name
account_tier = "Standard"
account_replication_type = "GRS"
account_kind = "StorageV2"
is_hns_enabled = true
}
resource "azurerm_storage_data_lake_gen2_filesystem" "adlfs" {
name = var.azure_data_lake_name
storage_account_id = azurerm_storage_account.sa.id
}
resource "azurerm_synapse_workspace" "synapseworkspace" {
name = var.synapse_workspace_name
resource_group_name = azurerm_resource_group.rg.name
location = azurerm_resource_group.rg.location
storage_data_lake_gen2_filesystem_id = azurerm_storage_data_lake_gen2_filesystem.adlfs.id
sql_administrator_login = var.synapse_sql_admin_user
sql_administrator_login_password = var.synapse_sql_admin_password
managed_resource_group_name = var.synapse_managed_resource_group_name
aad_admin {
login = var.azure_ad_admin_login
object_id = data.azurerm_client_config.current.object_id
tenant_id = data.azurerm_client_config.current.tenant_id
}
identity {
type = "SystemAssigned"
}
# Add tags
tags = {
source = "terraform"
}
}
resource "azurerm_synapse_firewall_rule" "synapsefirewall" {
name = "AllowAll"
synapse_workspace_id = azurerm_synapse_workspace.synapseworkspace.id
start_ip_address = "0.0.0.0"
end_ip_address = "255.255.255.255"
}
I am assuming the error can easily be fixed using role management in the Azure portal, but solving it using Terraform would be the best option.
Try the following?
I found it's not enough for the app and account to be added as owners.
I would go into your storage account > IAM > Add role assignment, and add the special permissions for this type of request:
Storage Blob Data Contributor
Storage Queue Data Contributor
Cited from: Azure Blob Storage "Authorization Permission Mismatch" error for get request with AD token
With the terraform module azurerm_role_assignment you can assigns a given Principal (User or Group) to a given Role. https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment
For Azure Synapse Workspace specifically, the principal identity can be retrieved using
azurerm_synapse_workspace.synapseworkspace.identity[0].principal_id
And assigned to a role:
# Create storage account
resource "azurerm_storage_account" "sa" {
...
}
# Create synapse workspace
resource "azurerm_synapse_workspace" "synapseworkspace" {
...
}
# Grant Synapse Workspace access to storage as Storage Blob Data Contributor
resource "azurerm_role_assignment" "synapsedatacontributor" {
role_definition_name = "Storage Blob Data Contributor"
scope = azurerm_storage_account.sa.id
principal_id = azurerm_synapse_workspace.synapseworkspace.identity[0].principal_id
}
I’m creating a new Alias and Subscription for an Enrollment Account by using the following terraform code:
data "azurerm_billing_enrollment_account_scope" "example" {
billing_account_name = "1234567890"
enrollment_account_name = "0123456"
}
resource "azurerm_subscription" "example" {
subscription_name = "My Example EA Subscription"
billing_scope_id = data.azurerm_billing_enrollment_account_scope.example.id
}
Next, manages a policy set definition and subscription policy assignment by using the following TF code:
data "azurerm_subscriptions" "availablesubscriptions" {
display_name_prefix = var.subscription_name
}
resource "azurerm_policy_set_definition" "tag_definition" {
name = "${var.subscription_name}-tag-def"
display_name = "${var.subscription_name}-tag-def"
description = "Append the default tags definition"
policy_type = "Custom"
policy_definitions = templatefile("${path.module}/templates/tag_definitions.json",
{ environmentType = var.environment_Type})
metadata = <<METADATA
{ "category" : "Tags" }
METADATA
lifecycle { ignore_changes = [metadata] }
}
resource "azurerm_subscription_policy_assignment" "tag_assignment" {
name = "${var.subscription_name}-tag-assignment"
display_name = "${var.subscription_name}-tag-assignment"
description = "Append the default tags assignment"
subscription_id = data.azurerm_subscriptions.availablesubscriptions.subscriptions[0].subscription_id
policy_definition_id = azurerm_policy_set_definition.tag_definition.id
}
I have configured the release pipeline in Azure DevOps, for creating above infrastructure. But in the release, terraform apply task giving the following error:
Note: The service connection/Active Directory application having the Owner access.
Error: creating Policy Set Definition "XXXX-tag-def": policy.SetDefinitionsClient#CreateOrUpdate: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client 'XXXXXXXXXXXXX' with object id 'XXXXXXXXXX' does not have authorization to perform action 'Microsoft.Authorization/policySetDefinitions/write' over scope '/subscriptions/***' or the scope is invalid. If access was recently granted, please refresh your credentials."
I have written terraform for creating the user, resource group, and roledefinition.
I need to have the scope of resource definition be the resource group that I created.
I don't know how to do that. It would be great if someone could help on this.
########### for creating user ####
# Configure the Azure Provider
provider "azurerm" {
version = "~> 1.30"
subscription_id="723604be-b74b-4473-9d11-1802dbfdb787"
}
provider "azuread" {
version = "~> 0.4"
subscription_id="723604be-b74b-4473-9d11-1802dbfdb787"
}
resource "azuread_user" "test" {
user_principal_name = "user1#catch.whizlabstesting.com"
display_name = "User1"
mail_nickname = "User1"
password = "Muneeshpandi#17"
force_password_change = "false"
}
##### creating resource group #####
resource "azurerm_resource_group" "terraform_rg" {
name = "user1_rgp"
location = "East US"
}
########## creating role definition ##########
data "azurerm_subscription" "primary" {}
resource "azurerm_role_definition" "sql_role" {
name = "sql_role"
scope = "data.azurerm_subscription.primary.id"
description = "This is a custom role to create sql database"
permissions {
actions = ["*"]
not_actions = []
}
assignable_scopes = [
"/subscriptions/723604be-b74b-4473-9d11-1802dbfdb787/resourceGroups/user1_rgp"
]
}
Getting following error while executing above code:
Error: authorization.RoleDefinitionsClient#CreateOrUpdate: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="MissingSubscription" Message="The request did not have a subscription or a valid tenant level resource provider."
How do I make the scope of a custom role be Resourcegroup in azure?
To create a custom role for the resource group, you need to have the permission Microsoft.Authorization/roleDefinitions/write, and to assign the custom role to a user, you need to have the permission Microsoft.Authorization/roleAssignments/write. The simplest way is that you have the Onwer role of the subscription.
And to create an Azure AD user:
To add or delete users you must be a User administrator or Global
administrator.
When you have all the needed permission. Let's focus on your code. You also need to assign the custom role to the user you created with the scope of the resource group. Then you can change the code like this:
resource "azurerm_role_definition" "sql_role" {
name = "sql_role"
scope = data.azurerm_subscription.primary.id
description = "This is a custom role to create sql database"
permissions {
actions = ["*"]
not_actions = []
}
assignable_scopes = [
data.azurerm_subscription.primary.id
]
}
resource "azurerm_role_assignment" "example" {
scope = azurerm_resource_group.terraform_rg.id
role_definition_id = azurerm_role_definition.sql_role.id
principal_id = azuread_user.test.id
}
If you only want the custom available for the resource group, you can change the assignable_scopes with the resource group Id as azurerm_resource_group.terraform_rg.id.
I am trying to create the following resources in Azure using Terraform and Terraform provider for Azure.
Create a storage account for blob storage.
Create an event subscription that will raise events on blob activity.
When running the terraform scripts i get the following error
Error: Error creating/updating EventGrid Event Subscription
"evtFileReceived" (Scope
"/subscriptions/c17cf5ee-d3d7-4f64-b863-f2a4d6948594/resourceGroups/dominos-doodle"):
eventgrid.EventSubscriptionsClient#CreateOrUpdate: Failure sending
request: StatusCode=400 -- Original Error: Code="InvalidRequest"
Message="The specified topic property does not match the expected
topic from the event subscription scope."
How shoud i fix it ?. Google search didn't gave any results.
The script that generated the error is as follows. The step that throwed the error is terraform apply
Obviously one way is to use the ARM templates to achieve this, but i am trying to see if it can be created using native Terraform scripts. I referred to Terraform Docs and created the following.
variable "inp_resource_group_name" { }
variable "inp_geo_location" { }
variable "inp_account_name" { }
variable "inp_az_subscription_id" { }
variable "inp_resource_group_id" { }
resource "azurerm_storage_account" "cave" {
name = var.inp_account_name
resource_group_name = var.inp_resource_group_name
location = var.inp_geo_location
account_tier = "Standard"
account_replication_type = "LRS"
account_kind = "StorageV2"
}
resource "azurerm_storage_container" "validName" {
name = validName"
resource_group_name = var.inp_resource_group_name
storage_account_name = var.inp_account_name
container_access_type = "blob"
}
resource "azurerm_eventgrid_event_subscription" "evtFileReceived" {
name = "evtFileReceived"
scope = var.inp_resource_group_id
topic_name="/subscriptions/${var.inp_az_subscription_id}/resourceGroups/${var.inp_resource_group_name}/providers/Microsoft.Storage/storageAccounts/{var.inp_account_name}"
webhook_endpoint {
url = "https://myendpoint.that.works.well.across.all.osi.layers"
}
}
I had a similar issue and solved it by setting both the scope and topic_name to the storage account id. So in your example, I think this should work;
resource "azurerm_eventgrid_event_subscription" "evtFileReceived" {
name = "evtFileReceived"
scope = azurerm_storage_account.cave.id
topic_name = azurerm_storage_account.cave.id
webhook_endpoint {
url = "https://myendpoint.that.works.well.across.all.osi.layers"
}
}
According to the error message, it indicates that the topic_name property in resource azurerm_eventgrid_event_subscription does not match the expected topic from the event subscription scope.
In this case, the scope should be created at the storage account level as the topic is associated with a storage account resource. It will like this:
resource "azurerm_eventgrid_event_subscription" "evtFileReceived" {
name = "evtFileReceived"
scope = ${azurerm_storage_account.cave.id}
topic_name="/subscriptions/${var.inp_az_subscription_id}/resourceGroups/${var.inp_resource_group_name}/providers/Microsoft.Storage/storageAccounts/{azurerm_storage_account.cave.name}"
webhook_endpoint {
url = "https://myendpoint.that.works.well.across.all.osi.layers"
}
}
Or, refer to this GitHub issue, you could use the scope with the id of the
eventgrid topic.
Realized that the resource group in this case is an art from a topic
type to subscribe and not a reference where to create the subscription
resource. It seems that "topic_name" and "resource_group_name" are
deprecated parameters. Use "scope" instead with the id of the
eventgrid topic.
It will like this:
resource "azurerm_eventgrid_topic" "example" {
name = "my-eventgrid-topic"
location = "${azurerm_resource_group.default.location}"
resource_group_name = "${azurerm_resource_group.default.name}"
}
resource "azurerm_eventgrid_event_subscription" "evtFileReceived" {
name = "evtFileReceived"
scope = "${azurerm_eventgrid_topic.example.id}"
webhook_endpoint {
url = "https://myendpoint.that.works.well.across.all.osi.layers"
}
}
Please let me know if this works or need further help.