I would like to give access to two users to use Azure Databricks using the below block :
resource "databricks_user" "dbuser" {
display_name = local.name.display_name
user_name = local.name.user_name
workspace_access = true
}
I have locals defined as below :
locals {
name = {
display_name = ["Vincent Casinha",
"Susan Young"
]
user_name = ["vincent.casinha#contoso.com",
"susan.young#contoso.com"]
}
}
While trying to run terraform plan i get the error attached. How do i use the local values properly in the databricks_user block for the argumens display_name & user_name?
locals {
display_name = ["Vincent Casinha", "Susan Young"]
user_name = ["vincent.casinha#contoso.com","susan.young#contoso.com"]
}
#assuming length of display_name and user_name are same
resource "databricks_user" "dbuser" {
count = length(local.display_name)
display_name = local.display_name[count.index]
user_name = local.user_name[count.index]
workspace_access = true
}
#or you can do like below
locals {
name = [
{
display_name = "Vincent Casinha"
user_name = "vincent.casinha#contoso.com"
},
{
display_name = "Susan Young"
user_name = "susan.young#contoso.com"
}
]
}
resource "databricks_user" "dbuser" {
for_each = local.name
display_name = each.value.display_name
user_name = each.value.user_name
workspace_access = true
}
Related
I have the following Azure AD service principal in my terraform module.
module "test_app" {
source = "../../../../../my-adapplication/"
app_owners = ["riker#domain.com","picard#domain.com"]
app_roles = [
{
allowed_member_types = [
"User"
]
description = "Read access to Test app"
display_name = "Test App Read"
is_enabled = true
value = "TestApp.Read"
id = random_uuid.test_app_read.result
},
{
allowed_member_types = [
"User"
]
description = "Write access to Test app"
display_name = "Test App Write"
is_enabled = true
value = "TestApp.Write"
id = random_uuid.test_app_write.result
},
{
allowed_member_types = [
"User"
]
description = "Admin access to Test app"
display_name = "Test App Admin"
is_enabled = true
value = "TestApp.Admin"
id = random_uuid.test_app_admin.result
}
]
app_role_assignments = [
{
app_role_id = random_uuid.test_app_read.result #"TestApp.Read"
principal_object_id = data.azuread_group.group_role_read.object_id
},
{
app_role_id = random_uuid.test_app_write.result #"TestApp.Write"
principal_object_id = data.azuread_group.group_role_write.object_id
},
{
app_role_id = random_uuid.test_app_admin.result #"TestApp.Admin"
principal_object_id = data.azuread_group.group_role_write.object_id
}
]
}
resource "random_uuid" "test_app_read" {
}
resource "random_uuid" "test_app_write" {
}
resource "random_uuid" "test_app_admin" {
}
data "azuread_group" "group_role_read" {
display_name = "group-role-read"
}
data "azuread_group" "group2_role_read" {
display_name = "group2-role-read"
}
data "azuread_group" "group_role_write" {
display_name = "group-role-write"
}
data "azuread_group" "group_role_admin" {
display_name = "group-role-admin"
}
The my-adapplication module file looks like this:
resource "azuread_application" "app" {
...
...
dynamic "app_role" {
for_each = var.app_roles
content {
id = app_role.value["id"]
allowed_member_types = app_role.value["allowed_member_types"]
description = app_role.value["description"]
display_name = app_role.value["display_name"]
enabled = app_role.value["is_enabled"]
value = app_role.value["value"]
}
}
}
resource "azuread_service_principal" "sp" {
application_id = azuread_application.app.application_id
}
resource "azuread_app_role_assignment" "role" {
for_each = { for a in var.app_role_assignments : a.app_role_id => a }
app_role_id = each.value["app_role_id"]
principal_object_id = each.value["principal_object_id"]
resource_object_id = azuread_service_principal.sp.object_id
}
The issue I am having is related to the app_role_assignments. If I pass in only a single principal_object_id it works. However if I pass in multiple principal_object_ids it doesn't work. For example TestApp.Read below:
app_role_assignments = [
{
app_role_id = random_uuid.test_app_read.result #"TestApp.Read"
principal_object_id = [data.azuread_group.group_role_read.object_id,data.azuread_group.group2_role_read.object_id]
},
{
app_role_id = random_uuid.test_app_write.result #"TestApp.Write"
principal_object_id = data.azuread_group.group_role_write.object_id
},
{
app_role_id = random_uuid.test_app_admin.result #"TestApp.Admin"
principal_object_id = data.azuread_group.group_role_write.object_id
}
]
The error received is:
Error: Incorrect attribute value type
│
│ on .terraform/modules/test_app/main.tf line 116, in resource "azuread_app_role_assignment" "role":
│ 116: principal_object_id = each.value["principal_object_id"]
│ ├────────────────
│ │ each.value["principal_object_id"] is tuple with 2 elements
│
│ Inappropriate value for attribute "principal_object_id": string required.
╵
How do I get terraform to loop over this principal_object_id list? I guess I am after a loop inside a loop. Is there a better way of doing this than the way I am above?
Is it possible to do this using for_each so I don't have the problems with list order changing if i use count/for.
Many thanks in advance.
You have to re-organize your app_role_assignments and then flatten it. If you want principal_object_id to have more then one value, it should always be a list, even for a single element:
app_role_assignments = [
{
app_role_id = random_uuid.test_app_read.result #"TestApp.Read"
principal_object_id = [data.azuread_group.group_role_read.object_id,data.azuread_group.group2_role_read.object_id]
},
{
app_role_id = random_uuid.test_app_write.result #"TestApp.Write"
principal_object_id = [data.azuread_group.group_role_write.object_id]
},
{
app_role_id = random_uuid.test_app_admin.result #"TestApp.Admin"
principal_object_id = [data.azuread_group.group_role_write.object_id]
}
]
then you can flatten is as:
locals {
app_role_assignments_flat = merge([
for val in var.app_role_assignments: {
for principal_object_id in val["principal_object_id"]:
"${val.app_role_id}-${principal_object_id}" => {
app_role_id = val.app_role_id
principal_object_id = principal_object_id
}
}
]...) # please do NOT remove the dots
}
then
resource "azuread_app_role_assignment" "role" {
for_each = local.app_role_assignments_flat
app_role_id = each.value["app_role_id"]
principal_object_id = each.value["principal_object_id"]
resource_object_id = azuread_service_principal.sp.object_id
}
I need some assistance to undertand the various forms of logging in to Databricks. I am using Terraform to provision Azure Databricks
I would like to know the difference in the two codes below
When i use option 1, i get the error as shown
Option 1:
required_providers {
azuread = "~> 1.0"
azurerm = "~> 2.0"
azuredevops = { source = "registry.terraform.io/microsoft/azuredevops", version = "~> 0.0" }
databricks = { source = "registry.terraform.io/databrickslabs/databricks", version = "~> 0.0" }
}
}
provider "random" {}
provider "azuread" {
tenant_id = var.project.arm.tenant.id
client_id = var.project.arm.client.id
client_secret = var.secret.arm.client.secret
}
provider "databricks" {
host = azurerm_databricks_workspace.db-workspace.workspace_url
azure_use_msi = true
}
resource "azurerm_databricks_workspace" "db-workspace" {
name = module.names-db-workspace.environment.databricks_workspace.name_unique
resource_group_name = module.resourcegroup.resource_group.name
location = module.resourcegroup.resource_group.location
sku = "premium"
public_network_access_enabled = true
custom_parameters {
no_public_ip = true
virtual_network_id = module.virtualnetwork["centralus"].virtual_network.self.id
public_subnet_name = module.virtualnetwork["centralus"].virtual_network.subnets["db-sub-1-public"].name
private_subnet_name = module.virtualnetwork["centralus"].virtual_network.subnets["db-sub-2-private"].name
public_subnet_network_security_group_association_id = module.virtualnetwork["centralus"].virtual_network.nsgs.associations.subnets["databricks-public-nsg-db-sub-1-public"].id
private_subnet_network_security_group_association_id = module.virtualnetwork["centralus"].virtual_network.nsgs.associations.subnets["databricks-private-nsg-db-sub-2-private"].id
}
tags = local.tags
}
Databricks Cluster Creation
resource "databricks_cluster" "dbcselfservice" {
cluster_name = format("adb-cluster-%s-%s", var.project.name, var.project.environment.name)
spark_version = var.spark_version
node_type_id = var.node_type_id
autotermination_minutes = 20
autoscale {
min_workers = 1
max_workers = 7
}
azure_attributes {
availability = "SPOT_AZURE"
first_on_demand = 1
spot_bid_max_price = 100
}
depends_on = [
azurerm_databricks_workspace.db-workspace
]
}
Databricks Workspace RBAC Permission
resource "databricks_group" "db-group" {
display_name = format("adb-users-%s", var.project.name)
allow_cluster_create = true
allow_instance_pool_create = true
depends_on = [
resource.azurerm_databricks_workspace.db-workspace
]
}
resource "databricks_user" "dbuser" {
count = length(local.display_name)
display_name = local.display_name[count.index]
user_name = local.user_name[count.index]
workspace_access = true
depends_on = [
resource.azurerm_databricks_workspace.db-workspace
]
}
Adding Members to Databricks Admin Group
resource "databricks_group_member" "i-am-admin" {
for_each = toset(local.email_address)
group_id = data.databricks_group.admins.id
member_id = databricks_user.dbuser[index(local.email_address, each.key)].id
depends_on = [
resource.azurerm_databricks_workspace.db-workspace
]
}
data "databricks_group" "admins" {
display_name = "admins"
depends_on = [
# resource.databricks_cluster.dbcselfservice,
resource.azurerm_databricks_workspace.db-workspace
]
}
The error that i get while TF apply is below :
Error: User not authorized
with databricks_user.dbuser[1],
on resources.adb.tf line 80, in resource "databricks_user" "dbuser":
80: resource "databricks_user" "dbuser"{
Error: User not authorized
with databricks_user.dbuser[0],
on resources.adb.tf line 80, in resource "databricks_user" "dbuser":
80: resource "databricks_user" "dbuser"{
Error: cannot refresh AAD token: adal:Refresh request failed. Status Code = '500'. Response body: {"error":"server_error", "error_description":"Internal server error"} Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.core.windows.net%2F
with databricks_group.db-group,
on resources.adb.tf line 80, in resource "databricks_group" "db-group":
71: resource "databricks_group" "db-group"{
Is the error coming because of this block below ?
provider "databricks" {
host = azurerm_databricks_workspace.db-workspace.workspace_url
azure_use_msi = true
}
I just need to login automatically when i click on the URL from the portal. So what shall i use for that? And why do we need to provide two times databricks providers, once under required_providers and again in provider "databricks"?
I have seen if i don't provide the second provider i get the error :
"authentication is not configured for provider"
As mentioned in the comments , If you are using Azure CLI authentication i.e. az login using your username and password , then you can use the below code :
terraform {
required_providers {
databricks = {
source = "databrickslabs/databricks"
version = "0.3.11"
}
}
}
provider "azurerm" {
features {}
}
provider "databricks" {
host = azurerm_databricks_workspace.example.workspace_url
}
resource "azurerm_databricks_workspace" "example" {
name = "DBW-ansuman"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
sku = "premium"
managed_resource_group_name = "ansuman-DBW-managed-without-lb"
public_network_access_enabled = true
custom_parameters {
no_public_ip = true
public_subnet_name = azurerm_subnet.public.name
private_subnet_name = azurerm_subnet.private.name
virtual_network_id = azurerm_virtual_network.example.id
public_subnet_network_security_group_association_id = azurerm_subnet_network_security_group_association.public.id
private_subnet_network_security_group_association_id = azurerm_subnet_network_security_group_association.private.id
}
tags = {
Environment = "Production"
Pricing = "Standard"
}
}
data "databricks_node_type" "smallest" {
local_disk = true
depends_on = [
azurerm_databricks_workspace.example
]
}
data "databricks_spark_version" "latest_lts" {
long_term_support = true
depends_on = [
azurerm_databricks_workspace.example
]
}
resource "databricks_cluster" "dbcselfservice" {
cluster_name = "Shared Autoscaling"
spark_version = data.databricks_spark_version.latest_lts.id
node_type_id = data.databricks_node_type.smallest.id
autotermination_minutes = 20
autoscale {
min_workers = 1
max_workers = 7
}
azure_attributes {
availability = "SPOT_AZURE"
first_on_demand = 1
spot_bid_max_price = 100
}
depends_on = [
azurerm_databricks_workspace.example
]
}
resource "databricks_group" "db-group" {
display_name = "adb-users-admin"
allow_cluster_create = true
allow_instance_pool_create = true
depends_on = [
resource.azurerm_databricks_workspace.example
]
}
resource "databricks_user" "dbuser" {
display_name = "Rahul Sharma"
user_name = "example#contoso.com"
workspace_access = true
depends_on = [
resource.azurerm_databricks_workspace.example
]
}
resource "databricks_group_member" "i-am-admin" {
group_id = databricks_group.db-group.id
member_id = databricks_user.dbuser.id
depends_on = [
resource.azurerm_databricks_workspace.example
]
}
Output:
If you are using Service Principal as authentication , then you can use something like below:
terraform {
required_providers {
databricks = {
source = "databrickslabs/databricks"
version = "0.3.11"
}
}
}
provider "azurerm" {
subscription_id = "948d4068-xxxx-xxxx-xxxx-e00a844e059b"
tenant_id = "72f988bf-xxxx-xxxx-xxxx-2d7cd011db47"
client_id = "f6a2f33d-xxxx-xxxx-xxxx-d713a1bb37c0"
client_secret = "inl7Q~Gvdxxxx-xxxx-xxxxyaGPF3uSoL"
features {}
}
provider "databricks" {
host = azurerm_databricks_workspace.example.workspace_url
azure_client_id = "f6a2f33d-xxxx-xxxx-xxxx-d713a1bb37c0"
azure_client_secret = "inl7Q~xxxx-xxxx-xxxxg6ntiyaGPF3uSoL"
azure_tenant_id = "72f988bf-xxxx-xxxx-xxxx-2d7cd011db47"
}
resource "azurerm_databricks_workspace" "example" {
name = "DBW-ansuman"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
sku = "premium"
managed_resource_group_name = "ansuman-DBW-managed-without-lb"
public_network_access_enabled = true
custom_parameters {
no_public_ip = true
public_subnet_name = azurerm_subnet.public.name
private_subnet_name = azurerm_subnet.private.name
virtual_network_id = azurerm_virtual_network.example.id
public_subnet_network_security_group_association_id = azurerm_subnet_network_security_group_association.public.id
private_subnet_network_security_group_association_id = azurerm_subnet_network_security_group_association.private.id
}
tags = {
Environment = "Production"
Pricing = "Standard"
}
}
data "databricks_node_type" "smallest" {
local_disk = true
depends_on = [
azurerm_databricks_workspace.example
]
}
data "databricks_spark_version" "latest_lts" {
long_term_support = true
depends_on = [
azurerm_databricks_workspace.example
]
}
resource "databricks_cluster" "dbcselfservice" {
cluster_name = "Shared Autoscaling"
spark_version = data.databricks_spark_version.latest_lts.id
node_type_id = data.databricks_node_type.smallest.id
autotermination_minutes = 20
autoscale {
min_workers = 1
max_workers = 7
}
azure_attributes {
availability = "SPOT_AZURE"
first_on_demand = 1
spot_bid_max_price = 100
}
depends_on = [
azurerm_databricks_workspace.example
]
}
resource "databricks_group" "db-group" {
display_name = "adb-users-admin"
allow_cluster_create = true
allow_instance_pool_create = true
depends_on = [
resource.azurerm_databricks_workspace.example
]
}
resource "databricks_user" "dbuser" {
display_name = "Rahul Sharma"
user_name = "example#contoso.com"
workspace_access = true
depends_on = [
resource.azurerm_databricks_workspace.example
]
}
resource "databricks_group_member" "i-am-admin" {
group_id = databricks_group.db-group.id
member_id = databricks_user.dbuser.id
depends_on = [
resource.azurerm_databricks_workspace.example
]
}
And why do we need to provide
two times databricks providers, once under required_providers and
again in provider "databricks"?
The required_providers is used to download and initialize the required providers from the source i.e. Terraform Registry . But the Provider Block is used for further configuration of that downloaded provider like describing client_id, features block etc. which can be used for authentication or other configuration.
The azure_use_msi option is primarily intended for use from CI/CD pipelines that are executed on machines with managed identity assigned to them. All possible authentication options are described in the documenation, but simplest way is to use authentication via Azure CLI, so you just need to leave host parameter in the provider block. If you don't have Azure CLI on that machine, you can use combination of host + personal access token instead.
if you're running that code from the machine with assigned managed identity, then you need to make sure that this identity is either added into workspace, or it has Contributor access to it - see Azure Databricks documentation for more details.
I'm trying to write a Terraform module whose purpose is to add members (users and service principals) to Azure AD groups. Here's my code:
# This is our input which contains the display names of users, SPNs and groups
memberships_service_principals = {
"spn1" = "group1"
"spn2" = "group2"
}
memberships_users = {
"user1" = "group1"
"user1" = "group2"
"user2" = "group2"
}
# Here we get each SPN's group object ID
data "azuread_group" "groups_service_principals" {
for_each = var.memberships_service_principals
display_name = each.value
}
# Here we get each SPN's object ID
data "azuread_service_principal" "service_principals" {
for_each = var.memberships_service_principals
display_name = each.key
}
# Here we get each user's group object ID
data "azuread_group" "groups_users" {
for_each = var.memberships_users
display_name = each.value
}
# Here we get each user's object ID
data "azuread_user" "users" {
for_each = var.memberships_users
user_principal_name = each.key
}
# Here we construct and merge two maps of groups => object IDs
locals {
service_principal_object_ids = { for k, v in var.memberships_service_principals : data.azuread_service_principal.service_principals["${k}"].object_id => data.azuread_group.groups_service_principals["${k}"].object_id }
user_object_ids = { for k, v in var.memberships_users : data.azuread_user.users["${k}"].object_id => data.azuread_group.groups_users["${k}"].object_id }
member_object_ids = merge(local.service_principal_object_ids, local.user_object_ids)
}
# Here we configure the memberships by passing the group and SPN/user object IDs
resource "azuread_group_member" "group_members" {
for_each = var.member_object_ids
group_object_id = each.value
member_object_id = each.key
}
Unfortunately, the above code doesn't work because a map in Terraform can have only unique keys. This rule gets violated in these cases:
When a user/SPN is specified more than once in order to be member of more than one group
Even if the map gets inverted (so as to have groups as keys instead of users/SPNs), group names and IDs are also not unique
I tried to define my input as follows but I still have no idea how to come up with a map of unique keys which to pass to create the memberships via 'resource "azuread_group_member"':
locals {
test_input1 = [
{principal = "user1", groups = ["group1", "group2"]},
{principal = "user2", groups = ["group2"]}
]
test_input2 = [
{ "user1" = ["group1", "group2"] },
{ "user2" = ["group2"] }
]
}
Any help and/or ideas will be greatly appreciated!
I don't fully understand your setup, but you could create a map from your variables with unique keys as follows:
variable "memberships_users" {
default = {
"user1" = "group1"
"user1" = "group2"
"user2" = "group2"
}
}
locals {
service_users = merge([
for k1,v1 in var.memberships_service_principals:
{
for k2,v2 in var.memberships_users:
"${k1}-${k2}" => {
memberships_service_principal = k1
memberships_service_principal_group = v1
memberships_user = k2
memberships_user_group = v2
}
}
]...)
}
which gives:
service_users = {
"spn1-user1" = {
"memberships_service_principal" = "spn1"
"memberships_service_principal_group" = "group1"
"memberships_user" = "user1"
"memberships_user_group" = "group2"
}
"spn1-user2" = {
"memberships_service_principal" = "spn1"
"memberships_service_principal_group" = "group1"
"memberships_user" = "user2"
"memberships_user_group" = "group2"
}
"spn2-user1" = {
"memberships_service_principal" = "spn2"
"memberships_service_principal_group" = "group2"
"memberships_user" = "user1"
"memberships_user_group" = "group2"
}
"spn2-user2" = {
"memberships_service_principal" = "spn2"
"memberships_service_principal_group" = "group2"
"memberships_user" = "user2"
"memberships_user_group" = "group2"
}
}
Then you could use that:
# SAMPLE CODE. Its not clear what you wish to accomplish,
# thus some changes probably will be required!
resource "azuread_group_member" "group_members" {
for_each = local.service_users
group_object_id = data.azuread_group.groups_service_principals[each.value.memberships_service_principal].object_id
member_object_id =data.azuread_group.groups_users[each.value.memberships_user].object_id
}
I have maps of variables like this:
users.tfvars
users = {
"testterform" = {
path = "/"
force_destroy = true
email_address = "testterform#example.com"
group_memberships = [ "test1" ]
tags = { department : "test" }
ssh_public_key = "ssh-rsa AAAAB3NzaC1yc2EAAA4l7"
}
"testterform2" = {
path = "/"
force_destroy = true
email_address = "testterform2#example.com"
group_memberships = [ "test1" ]
tags = { department : "test" }
ssh_public_key = ""
}
I would like to upload ssh key only if ssh_public_key not empty for the user. But don't understand how to check this
#main.tf
resource "aws_iam_user" "this" {
for_each = var.users
name = each.key
path = each.value["path"]
force_destroy = each.value["force_destroy"]
tags = merge(each.value["tags"], { Provisioner : var.provisioner, EmailAddress : each.value["email_address"] })
}
resource "aws_iam_user_group_membership" "this" {
for_each = var.users
user = each.key
groups = each.value["group_memberships"]
depends_on = [ aws_iam_user.this ]
}
resource "aws_iam_user_ssh_key" "this" {
for_each = var.users
username = each.key
encoding = "SSH"
public_key = each.value["ssh_public_key"]
depends_on = [ aws_iam_user.this ]
}
It sounds like what you need here is a derived "users that have non-empty SSH keys" map. You can use the if clause of a for expression to derive a new collection from an existing one while filtering out some of the elements:
resource "aws_iam_user_ssh_key" "this" {
for_each = {
for name, user in var.users : name => user
if user.ssh_public_key != ""
}
username = each.key
encoding = "SSH"
public_key = each.value.ssh_public_key
depends_on = [aws_iam_user.this]
}
The derived map here uses the same keys and values as the original var.users, but is just missing some of them. That means that the each.key results will correlate and so you'll still get the same username value you were expecting, and your instances will have addresses like aws_iam_user_ssh_key.this["testterform"].
You can use a for loop to exclude those blanks.
For example, you can do it on local:
variable "users" {
default = {
"testterform" = {
path = "/"
force_destroy = true
tags = { department : "test" }
ssh_public_key = "ssh-rsa AAAAB3NzaC1yc2EAAA4l7"
}
"testterform2" = {
path = "/"
force_destroy = true
tags = { department : "test" }
ssh_public_key = ""
}
}
}
locals {
public_key = flatten([
for key, value in var.users :
value.ssh_public_key if ! contains([""], value.ssh_public_key)
])
}
output "myout" {
value = local.public_key
}
that will output:
myout = [
"ssh-rsa AAAAB3NzaC1yc2EAAA4l7",
]
As you can see the empty ones have been removed, and you can add other stuff you want to exclude on that contains array.
Then you can use that local.public_key in the for_each for your ssh keys
So I have created a google_bigquery module to create datasets and set access.
The module iterates over a map of list of maps. It uses the each.key to create the datasets then iterates over the list of maps to create the dynamic access.
The module works as in:
It has no errors nor warning
It deploys the resources
It populates the remote statefile appropriately.
The issue is that everytime I ran terraform it wants to re-apply the same changes, over and over again.
Clearly something is not right but not sure what.
here is the code
main.tf
locals {
env = basename(path.cwd)
project = basename(abspath("${path.cwd}/../.."))
project_name = coalesce(var.project_name, format("%s-%s", local.project, local.env))
}
data "google_compute_zones" "available" {
project = local.project_name
region = var.region
}
provider "google" {
project = local.project_name
region = var.region
version = "~> 2.0" #until 3.0 goes out of beta
}
terraform {
required_version = ">= 0.12.12"
}
resource "google_bigquery_dataset" "main" {
for_each = var.datasets
dataset_id = upper("${each.key}_${local.env}")
location = var.region
delete_contents_on_destroy = true
dynamic "access" {
for_each = flatten([ for k, v in var.datasets : [
for i in each.value : {
role = i.role
user_by_email = i.user_by_email
group_by_email = i.group_by_email
dataset_id = i.dataset_id
project_id = i.project_id
table_id = i.table_id
}]])
content {
role = lookup(access.value,"role", "")
user_by_email = lookup(access.value,"user_by_email","")
group_by_email = lookup(access.value,"group_by_email","")
view {
dataset_id = lookup(access.value,"dataset_id","")
project_id = lookup(access.value,"project_id","")
table_id = lookup(access.value,"table_id", "")
}
}
}
access {
role = "READER"
special_group = "projectReaders"
}
access {
role = "OWNER"
group_by_email = "Group"
}
access {
role = "OWNER"
user_by_email = "ServiceAccount"
}
access {
role = "WRITER"
special_group = "projectWriters"
}
}
variables.tf
variable "region" {
description = ""
default = ""
}
variable "env" {
default = ""
}
variable "project_name" {
default = ""
}
variable "owner_group" {
description = ""
default = ""
}
variable "owner_sa" {
description = ""
default = ""
}
variable "datasets" {
description = "A map of objects, including dataset_isd abd access"
type = map(list(map(string)))
}
terraform.tfvars
datasets = {
dataset01 = [
{
role = "WRITER"
user_by_email = "email_address"
group_by_email = ""
dataset_id = ""
project_id = ""
table_id = ""
},
{
role = ""
user_by_email = ""
group_by_email = ""
dataset_id ="MY_OTHER_DATASET"
project_id ="my_other_project"
table_id ="my_test_view"
}
]
dataset02 = [
{
role = "READER"
user_by_email = ""
group_by_email = "group"
dataset_id = ""
project_id = ""
table_id = ""
},
{
role = ""
user_by_email = ""
group_by_email = ""
dataset_id ="MY_OTHER_DATASET"
project_id ="my_other_project"
table_id ="my_test_view_2"
}
]
}
So the problem is that the dynamic block (the way I wrote it) can generate this output
+ access {
+ role = "WRITER"
+ special_group = "projectWriters"
+ view {}
}
this is applied, no errors, but it will want to re-apply it over and over
The issue seems to be that the provider API response doesn't include the empty view{}
Any suggestion how I could make the view block conditional on the values of it being not null?
I fixed the problem. I changed the module slightly and the variable type.
I have split the roles and the views into their own lists of maps within the parent map of datasets.
There are conditionals in each block so the dynamic block is only applied if the roles exists or views exists.
Also realized the dynamic block was iterating on the wrong iterator.
The dynamic block was iterating on var.datasets which was causing the permissions assigned to each dataset to be applied to all datasets. So now it has been changed to iterate on each.value (from the resource for_each).
Here is the new code that works
MAIN.TF
resource "google_bigquery_dataset" "main" {
for_each = var.datasets
dataset_id = upper("${each.key}_${local.env}")
location = var.region
delete_contents_on_destroy = true
dynamic "access" {
for_each = flatten([for i in each.value : [
for k, v in i : [
for l in v :
{
role = l.role
user_by_email = l.user_by_email
group_by_email = l.group_by_email
special_group = l.special_group
}]
if k == "roles"
]])
content {
role = access.value["role"]
user_by_email = access.value["user_by_email"]
group_by_email = access.value["group_by_email"]
special_group = access.value["special_group"]
}
}
dynamic "access" {
for_each = flatten([for i in each.value : [
for k, v in i : [
for l in v :
{
dataset_id = l.dataset_id
project_id = l.project_id
table_id = l.table_id
}]
if k == "views"
]])
content {
view {
dataset_id = access.value["dataset_id"]
project_id = access.value["project_id"]
table_id = access.value["table_id"]
}
}
}
}
VARIABLES.TF
variable "datasets" {
description = "A map of objects, including datasets IDs, roles and views"
type = map(list(map(list(map(string)))))
default = {}
}
continued....
Terraform.tfvars
datasets = {
dataset01 = [
{
roles = [
{
role="WRITER"
user_by_email="email_address"
group_by_email=""
special_group=""
}
]
views = [
{
dataset_id="MY_OTHER_DATASET"
project_id="my_other_project"
table_id="my_test_view"
}
]
}
]
dataset02 = [
{
roles = [
{
role="READER"
user_by_email=""
group_by_email="group"
special_group=""
}
]
views=[
{
dataset_id="MY_OTHER_DATASET"
project_id="my_other_project"
table_id="my_test_view_2"
}
]
}
]
}