I am trying to automate Azure Databricks SQL permissions using Terraform.
I am using Azure Data Lake Gen 2 (Azure storage with hierarchical namespace enabled) as the backend to store the state file.
Terraform , as per my understanding and based on the tests that I have performed only uses the backend resource to store the state file. This operation SHOULD NOT need "owner" permissions. It should be ok to use ACL permissions on the container or a folder that I am trying to write the state file in.
For authentication, I am using a Service Principal (with a secret, for now).
However, though it works when I assign that Service Principal Owner rights on the Data Lake. If I just assign the Service Principal "ACL" permissions on the container or the folder , it fails when doing terraform init.
Error inspecting states in the "azurerm" backend:
Error retrieving keys for Storage Account "datalakename": storage.AccountsClient#ListKeys:
Failure responding to request: StatusCode=403 --
Original Error: autorest/azure: Service returned an error.
Status=403 Code="AuthorizationFailed"
Message="The client 'client guid' with object id 'object id' does not have authorization to perform
action 'Microsoft.Storage/storageAccounts/listKeys/action' over scope '/subscriptions/subscriptionid/resourceGroups/adobe/providers/Microsoft.Storage/storageAccounts/datalakename' or the scope is invalid. If access was recently granted, please refresh your credentials."
My backend configuration looks like this
terraform {
backend "azurerm"{
resource_group_name = "RGname"
storage_account_name = "datalakename"
container_name = "raw"
key = "prod.terraform.tfstate"
subscription_id = "subscrioption id here"
tenant_id = "tenant id here"
}
}
Also, I have the SPN details in the environment variables
export ARM_CLIENT_ID="client id here";
export ARM_CLIENT_SECRET="client secret here";
export ARM_SUBSCRIPTION_ID="subscription id here";
Like I said if I assign the "client id" the RBAC role of owner on the datalake, it works.
But question is why does it try to do a "listing keys of the storage account" action when I am using client credentials for authentication (and not keys) and why does it need an RBAC role to just save a file?
Related
I would like to create a User assigned identity with Terraform, that have the read permission on an already existing azure storage account. How can I grant that with Terraform? I am creating the whole infrastructure with the service principal I'm using right now. So, it's unlikely that it's a permission issue.
# My user assigned identity:
resource "azurerm_user_assigned_identity" "user_assigned_identity" {
name = "${var.resource_prefix}useridentity"
location = var.location
resource_group_name = var.resource_group_name
}
# and the role assignment to this identity
resource "azurerm_role_assignment" "example" {
scope = "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/resource-group-name"
role_definition_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx"
principal_id = azurerm_user_assigned_identity.user_assigned_identity.principal_id
}
But I am having this error:
Error: authorization.RoleAssignmentsClient#Create: Failure responding
to request: StatusCode=403 -- Original Error: autorest/azure: Service
returned an error. Status=403 Code="AuthorizationFailed" Message="The
client 'xxxxxxxxxxxxxxxxxxx' with object id 'xxxxxxxxxxxxxxxxxxx' does
not have authorization to perform action
'Microsoft.Authorization/roleAssignments/write' over scope
'/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/resource-group-name/providers/Microsoft.Authorization/roleAssignments/"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx"'
or the scope is invalid. If access was recently granted, please
refresh your credentials."
Any idea what I'm doing wrong?
For assigning roles to the some user assigned identity using your Service Principal from terraform you need to give the service principal "Owner" permission to to subscription. It is not possible to do from "Contributor" permission. Using contributor access you can create or manage the resources for the subscription but not assign roles.
Testing:
My service principal which I will be using to authenticate from terraform.
Providing Owner access to the above service principal in the subscription.
My Terraform Script:
provider "azurerm"{
client_id = "f6a2f33d-xxxx-xxxxx-xxxxx-xxxx"
subscription_id = "948d4068--xxxx-xxxx-xxxxx-xxxx"
client_secret = "KEa7Q~2673QY.uN.xxxxxxxxxxxx"
tenant_id = "72f988bf-xxxxx-xxxxx-xxxxx-xxxxx"
features{}
}
resource "azurerm_user_assigned_identity" "user_assigned_identity" {
name = "myuseridentity"
location = "East US"
resource_group_name = "ansumantest"
}
# and the role assignment to this identity
resource "azurerm_role_assignment" "example" {
scope = "/subscriptions/948d4068--xxxx-xxxx-xxxxx-xxxxx/resourceGroups/ansumantest"
role_definition_name = "Storage Blob Data Reader"
principal_id = azurerm_user_assigned_identity.user_assigned_identity.principal_id
}
Output:
Validating from Azure Portal:
Note:
Your Service principal must be having the contributor access, giving it owner access should resolve the issue.
To assign roles, you need the Microsoft.Authorization/roleAssignments/write permission, which is included in either the Owner OR User Administrator role
Giving your service principal Owner violates the principle of least privilege, if no other permissions are required, I would suggest to use User Administrator.
I have below terraform script, to create a new service account and make it owner.
The scripts creates the service account, but it will throw an error on assigning role
resource "google_service_account" "pci_api_service_account" {
account_id = "pci-api"
display_name = "Api"
project = var.project_id
}
resource "google_service_account_iam_member" "pci_api_owner_binding" {
# service_account_id = "projects/pcb-poc-pci/serviceAccounts/infra-admin-sa#pcb-poc-pci.iam.gserviceaccount.com"
service_account_id = google_service_account.pci_api_service_account.name
role = "roles/owner"
member = "serviceAccount:${google_service_account.pci_api_service_account.email}"
depends_on = [
google_service_account.pci_api_service_account
]
}
and I already autheticated with infra-admin-sa service account by running
gcloud auth activate-service-account --project=pcb-poc-pci --key-file ~/sa/pcb-poc-pci-test-sa-94aa6c81d650.json
When I run terragrunt apply I get this error for the second script
Error: Error applying IAM policy for service account 'projects/pcb-poc-pci/serviceAccounts/pci-api#pcb-poc-pci.iam.gserviceaccount.com': Error setting IAM policy for service account 'projects/pcb-poc-pci/serviceAccounts/pci-api#pcb-poc-pci.iam.gserviceaccount.com': googleapi: Error 403: Permission iam.serviceAccounts.setIamPolicy is required to perform this operation on service account projects/pcb-poc-pci/serviceAccounts/pci-api#pcb-poc-pci.iam.gserviceaccount.com., forbidden
These are the Roles of that service account
Based on google doc here and the error message, Service Account Admin should be enough, which my service account already have
Not sure what I missed
Solution 1
seems command line was not picking the correct credential/service account although I used gcloud auth activate-service-account command.
so I added this to my script
provider "google" {
credentials = file(var.service_account_file_path)
project = var.project_id
}
and now it's working fine
Solution 2
as per #John Hansley comments below
export GOOGLE_APPLICATION_CREDENTIALS=fullpath.json
then terraform will be picking that service account file and scripts will run successfully.
This method is preferred since less issue in CICD pipeline and other deveopers, to set terraform variables
Been brushing up using Terraform to manage resources in Azure the past week or so.
Great tool.
I've found there is a distinction between using an AZ user account vs service principal.
The goal is to create resources in Azure using a designated service principal and referencing it's secret that is stored within AZ key vault. Moving away from locally stored secret (file, env var, etc).
I can successfully create resources using an authenticated service principal as long as I have my azurerm provider containing the subid, clientid, clientsecret & tenantid, it works great.
Example of what works when I store service principal secret as a var sp_secret in variables.tf (or even works as env var):
provider "azurerm" {
version = "=2.48.0"
features { }
subscription_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
client_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
client_secret = "${var.sp_secret}"
tenant_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}
I have been able to successfully pull the service principals secret from the keyvault and 'output' it, but what I want to do is to pull that secret from kv and use, say as a var inside the provider client_secret value. ex. 'client_secret = "${link to secret sitting in kv}"'
Here is what I am doing to retrieve the SP secret from keyvault and output it:
data "azurerm_client_config" "current" {}
variable "keyvault_name" {
default = "blah-kv"
}
variable "kvrg_name" {
default = "blah-kv-rg"
}
data "azurerm_key_vault" "keyvault" {
name = "${var.keyvault_name}"
resource_group_name = "${var.kvrg_name}"
}
data "azurerm_key_vault_secret" "kv-sp" {
name = "blah-tf-sp-secret"
key_vault_id = "${data.azurerm_key_vault.keyvault.id}"
}
output "secret_value" {
value = "${data.azurerm_key_vault_secret.kv-sp.value}"
}
As mentioned, the above snippet successfully retrieves and outputs the secret. I just want to, instead of output the secret, just set that secret as client_secret value in the azurerm provider reference.
I've tried many variations of client_secret = "${data.azurerm_key_vault_secret.kv-sp.value}", and I get the following error:
Error: Cycle: data.azurerm_key_vault.keyvault, provider["registry.terraform.io/hashicorp/azurerm"], data.azurerm_key_vault_secret.kv-sp
I interpret the above error to indicate a circular reference. I've tried a few things i've picked up in my searching for an answer, but no dice.
Any guidance is appreciated.
Thanks!
As I know. it's impossible to achieve what you expect. When you use the Terraform to manage Azure resources, then you need to have an Azure account or service principle with enough permissions. If you use the service principle, it means you need to configure the provider azurerm with client id and client secret before running the Terraform code. But when you store the secret in the Azure Key Vault, then you need to run the code, and then you can get the secret. It causes cycle dependency.
I try to run terraform with azurerm provider initialized for service principal authorization, with the following in providers.tf (with service principal that has owner access on the subscription and works in other code):
provider "azurerm" {
features {}
subscription_id = "SSSSSSSS-SSSS-SSSS-SSSS-SSSSSSSSSSSS"
client_id = "AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA"
tenant_id = "<MY_TENANT_ID>"
client_secret = "<MY_CLIENT_SECRET>"
}
However, when I run terraform plan or terraform apply, this fails reporting totally different client ID:
Error: Error ensuring Resource Providers are registered.
[.... some lines of useless examples ....]
Original Error: Cannnot register providers: Microsoft.ServiceFabricMesh, Microsoft.ManagedServices, Microsoft.DesktopVirtualization. Errors were: Cannot register provider Microsoft.ServiceFabricMesh with Azure Resource Manager: resources.ProvidersClient#Register: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client 'BBBBBBBB-BBBB-BBBB-BBBB-BBBBBBBBBBBB' with object id 'BBBBBBBB-BBBB-BBBB-BBBB-BBBBBBBBBBBB' does not have authorization to perform action '
Microsoft.ServiceFabricMesh/register/action' over scope '/subscriptions/SSSSSSSS-SSSS-SSSS-SSSS-SSSSSSSSSSSS' or the scope is invalid. If access was recently granted, please refresh your credentials.".
In other words, while I tell terraform to use service principal "AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA", it uses "BBBBBBBB-BBBB-BBBB-BBBB-BBBBBBBBBBBB" instead.
It is obvious to say that nothing in my code refers a service principal with id "BBBBBBBB-BBBB-BBBB-BBBB-BBBBBBBBBBBB".
Setting ARM_* environment variables does not have any effect on this behavior.
Any idea how could that happen?
Update: I'm on terraform 0.14.3 and azurerm 2.41. Also tried azurerm 2.32 - it behaves the same.
First of all, you can output the client id to make sure that you are using the correct service principal that has the appropriate authorization scope on the subscription and tenant you're working with.
data "azurerm_client_config" "current" {
}
output "account_id" {
value = data.azurerm_client_config.current.client_id
}
Also, If you're using service principal credentials, try re-generating your secret key or your service principal. As a workaround, you also could skip the provider registration as
provider "azurerm" {
skip_provider_registration = true
}
At last, try to upgrade your terraform and azurerm provider for a better experience. It might be some bug in some versions, refer to this1 and this2.
I solved this - but still do not know why incorrect service principal id was used by terraform.
Once I registered the groups manually, the issue with 'BBBBBBBB...' principal id happened once more when terraform tried to create a resource in not existing group (there was a typo in RG name). I still can't understand why terraform reported that weird service principal issue instead of missed group, but the issue has gone now.
Thanks everyone who helped me to narrow it!
What specific changes need to be made to the syntax below in order for the terraform azurerm provider to be able to authenticate the service principal that will be created using the following code?
The Problem
A Second Terraform module needs to authenticate to Azure through the azurerm provider with a client_id and client_secret that is created programatically during an earlier, separate process.
The provider block in the Second Terraform module looks like:
provider "azurerm" {
subscription_id = var.subscriptionId
client_id = var.clientId
client_secret = var.clientSecret
tenant_id = var.tenantId
}
The problem arises when the correct values whcih we validated from the earlier preceding process are not accepted as the var.clientId and the var.clientSecret in the provider code block above.
How the Service Principal is Created:
The client_id and client_secret to be used to authenticate to the Second Terraform module are currently created by a First Terraform module which includes the following:
resource "azuread_application" "appReg" {
name = var.appName
}
resource "azuread_service_principal" "example-sp" {
application_id = azuread_application.appReg.application_id
}
resource "azuread_service_principal_password" "example-sp_pwd" {
service_principal_id = azuread_service_principal.example-sp.id
value = "long-random-string"
end_date = "2021-06-02T01:02:03Z"
}
data "azurerm_subscription" "thisSubscription" {
subscription_id = var.subscriptionId
}
resource "azurerm_role_assignment" "example-sp_role_assignment" {
scope = data.azurerm_subscription.thisSubscription.id
role_definition_name = "Contributor"
principal_id = azuread_service_principal.example-sp.id
}
resource "azuread_application_app_role" "example-role" {
application_object_id = azuread_application.appReg.id
allowed_member_types = ["User", "Application"]
description = "Admins can manage roles and perform all task actions"
display_name = "Admin"
is_enabled = true
value = "administer"
}
Terraform reports Apply complete after the above First module is run, and we are also able to confirm in the Azure Portal that the correct Active Directory has a new app registration with name var.appName and with ID equal to what we find in the First modules tfstate file.
The Error Message:
When Terraform tries to apply the Second module using the Service Principal ID and Secret created by the First module, the following error is thrown:
Error:
Error building account:
Error getting authenticated object ID:
Error listing Service Principals:
autorest.DetailedError{
Original:adal.tokenRefreshError{
message:"adal: Refresh request failed.
Status Code = '400'.
Response body: {
\"error\":\"unauthorized_client\",
\"error_description\":\"AADSTS700016:
Application with identifier 'correct-app-id' was not found in the directory 'the-right-ad-id'.
This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant.
You may have sent your authentication request to the wrong tenant.\\r\\n
Trace ID: some-trace-id\\r\\n
Correlation ID: correlation-id-redacted\\r\\n
Timestamp: 2020-12-31 19:02:19Z\",
\"error_codes\":[700016],
\"timestamp\":\"2020-12-31 19:02:19Z\",
\"trace_id\":\"some-trace-id\",
\"correlation_id\":\"correlation-id-redacted\",
\"error_uri\":\"https://login.microsoftonline.com/error?code=700016\"
}",
resp:(*http.Response)(0xc000ac2000)},
PackageType:"azure.BearerAuthorizer",
Method:"WithAuthorization",
StatusCode:400,
Message:"Failed to refresh the Token for request to https://graph.windows.net/the-right-ad-id/servicePrincipals?%24filter=appId+eq+%27correct-app-id%27&api-version=1.6",
ServiceError:[]uint8(nil),
Response:(*http.Response)(0xc000ac2000)
}
The error message does not seem helpful because we validated that the app is registered with the AAD instance.
How can we resolve this problem and programmatically create a client_id and client_secret that will be accepted and usable by the Second module?
As I see there is no problem with your Terraform code. It should work fine. But you got the error that the application was not found in the tenant. So what you need to do is to check if the tenant Id is really right in the second module.
I had the same issue on Deployment Agents for Terraform on Kubernetes. Several types of error can appear when the memory or the CPU is not large enough.
Below are the Terraform recommendations: https://www.terraform.io/docs/enterprise/before-installing/index.html
You have to be careful with the deployment infrastructures that mutualises the resources (K8s, Hypervisor Pool etc.)when several Terraform deployments are in parallel it causes somewhat random errors.
Terraform which does not stop, API AZure / AWS error, tfstate lock etc.