Terraform flatten and looping issue inside modules - azure

I'm trying to create a module in Terraform for creating Azure resources and facing some issues. This module creates a resource group, subnet, vnet and Role bindings. I see that the below code creates the resources twice because of the loop. Does the for_each loop work in such a way that the entire resource or module block will be executed each time it loops? I'm new to Terraform and come from a Java background.
Also, ideally would like to use the flatten inside the module without locals possibly, any way to do that? Code is below.
locals {
groupsbyrole = flatten([
for roleName, groupList in var.testproject1_role_assignments : [
for groupName in groupList : {
role_name = roleName
group_name = groupName
}
]
])
}
module "testproject1" {
source = "C:\\Users\\ebo1h8h\\Documents\\Project\\Automation\\Terraform\\Code\\Azure\\modules\\sandbox-module"
short_name = "testproj"
# Resource Group Variables
az_rg_location = "eastus"
az_tags = {
Environment = "Sandbox"
CostCenter = "Department"
ResourceOwner = "Vikram"
Project = "testproj"
Role = "Resource Group"
}
address_space = ["10.0.0.0/16"]
subnet_prefixes = ["10.0.1.0/24"]
subnet_names = ["a-npe-snet01-sbox"]
vnet_location = var.az_rg_location
for_each = {
for group in local.groupsbyrole : "${group.role_name}.${group.group_name}}" => group
}
principal_id = each.value.group_name
role_definition_name = each.value.role_name
}
And here is the role_assignments variable
variable "testproject1_role_assignments" {
type = map(list(string))
default = {
"Contributor" = ["prod-azure-contrib-sbox", "gcp-org-network-engineering"],
"Owner" = ["gcp-org-cloud-delivery"]
}
}
The above code creates 12 resources when it should be only 6. The only was I was able to get around this is have the resource "azurerm_role_assignment" "role_assignment" as a separate module. Ideally, I want to pass the role assignments variable in each of the module to be created so that it creates a set of resources.
Any pointers on how to achieve that?
Thanks,

The docs state
If a resource or module block includes a for_each argument whose value is a map or a set of strings, Terraform will create one instance for each member of that map or set.
So in your scenario you are creating 3 instances of the module, whereas it sounds like you want to pass in the local.groupsbyrole object as a variable in the module and only attach the for_each to the resources you want multiple instances of.
Sidenote: You could simplify your local by adding group like below:
locals {
groupsbyrole = flatten([
for roleName, groupList in var.testproject1_role_assignments : [
for groupName in groupList : {
role_name = roleName
group_name = groupName
group = "${roleName}.${groupName}"
}
]
])
}
Tip: I find adding an output to see the shape of the object whilst developing can also be useful
output "test_output" {
value = local.groupsbyrole
}
Then when you run plan you will see your object
test_output = [
+ {
+ group = "Contributor.prod-azure-contrib-sbox"
+ group_name = "prod-azure-contrib-sbox"
+ role_name = "Contributor"
},
+ {
+ group = "Contributor.gcp-org-network-engineering"
+ group_name = "gcp-org-network-engineering"
+ role_name = "Contributor"
},
+ {
+ group = "Owner.gcp-org-cloud-delivery"
+ group_name = "gcp-org-cloud-delivery"
+ role_name = "Owner"
},
]

Related

terraform - use count and for_each together

I'm creating multiple Google Cloud Projects using a module which creates a custom service account for each one and provides that as an output eg
module "app_projects" {
count = var.number_application_projects
etc
}
I then have a list of IAM roles I want to assign to each Project Service account like this:
sa_roles = ["roles/run.developer","roles/appengine.deployer"]
resource "google_project_iam_member" "proj_sa_iam_roles" {
count = var.number_application_projects
for_each = toset(var.sa_roles)
project = module.app_projects[count.index].project_id
role = each.value
member = "serviceAccount:${module.app_projects[count.index].service_account_email}"
}
This runs into the error about "count" and "for_each" being mutually-exclusive, and I can't for the life of me figure out what to use instead, any help would be appreciated!
You can probably use setproduct, but the standard method is using flatten.
They even have a section tailored for similar use cases. Flattening nested structures for for_each.
Here is an example, that doesn't use your exact resource, but should be instructive and you can actually run it to test it out.
modules/app-project/variables.tf
variable "name" {}
modules/app-project/outputs.tf
output "name" {
value = var.name
}
modules/member/variables.tf
variable "project" {}
variable "role" {}
modules/member/outputs.tf
output "project_role" {
value = "${var.project}-${var.role}"
}
main.tf
locals {
roles = ["rad", "tubular"]
}
module "app_project" {
source = "./modules/app-project"
count = 2
name = "app-project-${count.index}"
}
module "project_role" {
source = "./modules/member"
for_each = { for pr in flatten([for p in module.app_project[*] : [
for r in local.roles : {
app_project_name = p.name
role = r
}]
]) : "${pr.app_project_name}-${pr.role}" => pr }
project = each.value.app_project_name
role = each.value.role
}
output "project_roles" {
value = values(module.project_role)[*].project_role
}
terraform plan output
Changes to Outputs:
+ project_roles = [
+ "app-project-0-rad",
+ "app-project-0-tubular",
+ "app-project-1-rad",
+ "app-project-1-tubular",
]
In your case specifically, I think something like this would work:
resource "google_project_iam_member" "proj_sa_iam_roles" {
for_each = { for i, pr in flatten([for p in module.app_project[*] : [
for r in var.sa_roles : {
app_project = p
role = r
}]
]) : "${i}-${pr.role}" => pr }
project = each.value.app_project.project_id
role = each.value.role
member = "serviceAccount:${each.value.app_project.service_account_email}"
}

Terraform Invalid for_each argument local will be known only after apply

I would like to create an AWS account with SSO Account Assignments in the same first terraform run without hit the for_each limitation with dynamic values that cannot be predicted during plan.
I've tried to separate the aws_organizations_account resource from aws_ssoadmin_account_assignment in completely separate TF module and also I tried to use depends_on between those resources and modules.
What is the simplest and correct way to fix this issue?
Terraform v1.2.4
AWS SSO Account Assignments Module
Closed Pull Request that did not fix this issue
main.tf file (aws module)
resource "aws_organizations_account" "account" {
name = var.aws_account_name
email = "${var.aws_account_name}#gmail.com"
tags = {
Name = var.aws_account_name
}
parent_id = var.aws_org_folder_id
}
data "aws_identitystore_group" "this" {
for_each = local.group_list
identity_store_id = local.identity_store_id
filter {
attribute_path = "DisplayName"
attribute_value = each.key
}
}
data "aws_identitystore_user" "this" {
for_each = local.user_list
identity_store_id = local.identity_store_id
filter {
attribute_path = "UserName"
attribute_value = each.key
}
}
data "aws_ssoadmin_instances" "this" {}
locals {
assignment_map = {
for a in var.account_assignments :
format("%v-%v-%v-%v", aws_organizations_account.account.id, substr(a.principal_type, 0, 1), a.principal_name, a.permission_set_name) => a
}
identity_store_id = tolist(data.aws_ssoadmin_instances.this.identity_store_ids)[0]
sso_instance_arn = tolist(data.aws_ssoadmin_instances.this.arns)[0]
group_list = toset([for mapping in var.account_assignments : mapping.principal_name if mapping.principal_type == "GROUP"])
user_list = toset([for mapping in var.account_assignments : mapping.principal_name if mapping.principal_type == "USER"])
}
resource "aws_ssoadmin_account_assignment" "this" {
for_each = local.assignment_map
instance_arn = local.sso_instance_arn
permission_set_arn = each.value.permission_set_arn
principal_id = each.value.principal_type == "GROUP" ? data.aws_identitystore_group.this[each.value.principal_name].id : data.aws_identitystore_user.this[each.value.principal_name].id
principal_type = each.value.principal_type
target_id = aws_organizations_account.account.id
target_type = "AWS_ACCOUNT"
}
main.tf (root)
module "sso_account_assignments" {
source = "./modules/aws"
account_assignments = [
{
permission_set_arn = "arn:aws:sso:::permissionSet/ssoins-0000000000000000/ps-31d20e5987f0ce66",
permission_set_name = "ReadOnlyAccess",
principal_type = "GROUP",
principal_name = "Administrators"
},
{
permission_set_arn = "arn:aws:sso:::permissionSet/ssoins-0000000000000000/ps-955c264e8f20fea3",
permission_set_name = "ReadOnlyAccess",
principal_type = "GROUP",
principal_name = "Developers"
},
{
permission_set_arn = "arn:aws:sso:::permissionSet/ssoins-0000000000000000/ps-31d20e5987f0ce66",
permission_set_name = "ReadOnlyAccess",
principal_type = "GROUP",
principal_name = "Developers"
},
]
}
The important thing about a map for for_each is that all of the keys must be made only of values that Terraform can "see" during the planning step.
You defined local.assignment_map this way in your example:
assignment_map = {
for a in var.account_assignments :
format("%v-%v-%v-%v", aws_organizations_account.account.id, substr(a.principal_type, 0, 1), a.principal_name, a.permission_set_name) => a
}
I'm not personally familiar with the aws_organizations_account resource type, but I'm guessing that aws_organizations_account.account.id is an attribute whose value gets decided by the remote system during the apply step (once the object is created) and so this isn't a suitable value to use as part of a for_each map key.
If so, I think the best path forward here is to use a different attribute of the resource that is defined statically in your configuration. If var.aws_account_name has a static value defined in your configuration (that is, it isn't derived from an apply-time attribute of another resource) then it might work to use the name attribute instead of the id attribute:
assignment_map = {
for a in var.account_assignments :
format("%v-%v-%v-%v", aws_organizations_account.account.name, substr(a.principal_type, 0, 1), a.principal_name, a.permission_set_name) => a
}
Another option would be to remove the organization reference from the key altogether. From what you've shared it seems like there is only one account and so all of these keys would end up starting with exactly the same account name anyway, and so that string isn't contributing to the uniqueness of those keys. If that's true then you could drop that part of the key and just keep the other parts as the unique key:
assignment_map = {
for a in var.account_assignments :
format(
"%v-%v-%v",
substr(a.principal_type, 0, 1),
a.principal_name,
a.permission_set_name,
) => a
}

Create users, groups and group_member using terraform

I have the following csv file :
first_name,last_name,department,job_title
Michael,Scott,TF_TEST1,Manager
Jim,Halpert,TF_TEST1,Engineer
Pam,Beesly,TF_TEST2,Engineer
I want to create all those users and set them member of the group corresponding to their departement, like :
User
Group
Michael Scott
TF_TEST1
Jim Halpert
TF_TEST1
Pam Beesly
TF_TEST2
Here is what I have so far :
# Configure the Azure Active Directory Provider
provider "azuread" {}
# Retrieve domain information
data "azuread_domains" "default" {
only_initial = true
}
locals {
domain_name = data.azuread_domains.default.domains.0.domain_name
users = csvdecode(file("${path.module}/users.csv"))
groups = toset(local.users[*].department)
}
resource "azuread_user" "users" {
for_each = { for user in local.users : user.first_name => user }
user_principal_name = format(
"%s.%s#%s",
lower(each.value.first_name),
lower(each.value.last_name),
local.domain_name
)
password = format(
"%s%s%s!",
lower(each.value.last_name),
substr(lower(each.value.first_name), 0, 1),
length(each.value.first_name)
)
force_password_change = true
display_name = "${each.value.first_name} ${each.value.last_name}"
department = each.value.department
job_title = each.value.job_title
}
resource "azuread_group" "groups" {
for_each = local.groups
display_name = each.key
security_enabled = true
assignable_to_role = true
}
Users and groups get created just fine.
However I can't figure a way of adding those users inside their corresponding groups.
I feel like I should itarate through my azuread_user.users and azuread_group.groups to make the binding using a group_member resources but no chance.
Or maybe that would be easier using the members = [] property from group resource ?
Any help will be appreciated :)
As per our discussion from the comments, you can achieve what you want by using a combination of values built-in function [1] and if instead of the ternary operator:
resource "azuread_group" "groups" {
for_each = local.groups
display_name = each.key
security_enabled = true
assignable_to_role = true
members = [ for u in values(azuread_user.users) : u.id if u.department == each.key ]
}
[1] https://www.terraform.io/language/functions/values

Adding Entities to Vault Namespaces,Groups, or Policies Terraform

I'm having an issue with the Vault Terraform. I am able to create Entities, Namespaces, Groups, and policies but linking them together is not happening for me. I can get the policy added to the group just fine, but adding members to that group I cannot.
Here's what I have so far:
# module.users returns vault_identity_entity.entity.id
data "vault_identity_entity" "user_lookup" {
for_each = toset([for user in local.groups : user.name])
entity_name = each.key
depends_on = [
module.users
]
}
# module.devops_namespace returns vault_namespace.namespace.path
resource "vault_identity_group" "devops" {
depends_on = [
vault_policy.policy
]
name = "devops_users"
namespace = module.devops_namespace.vault_namespace
member_entity_ids = [for user in data.vault_identity_entity.user_lookup : jsondecode(user.data_json).id]
}
resource "vault_identity_group_policies" "default" {
policies = [vault_policy.gitlab_policy.name]
exclusive = false
group_id = vault_identity_group.devops.id
}
What I need to do is create a namespace and add users and a policy to that namespace.
Any help would be appreciated, thanks!
resource "vault_policy" "namespace" {
depends_on = [module.namespace]
name = "namespace"
policy = file("policies/namespace.hcl")
namespace = "devops"
}
resource "vault_identity_group" "devops" {
depends_on = [
module.users
]
name = "devops_users"
namespace = module.devops_namespace.vault_namespace
policies = [vault_policy.gitlab_policy.name]
member_entity_ids = [for user in module.users : user.entity_id]
}
By referring the users the module created I was able to achieve the correct result.
Since the module created the users from locals and the data resource was trying to pull down the same users, the extra data resource section wasn't needed.
Thank you Marko E!

terraform - Iterate over two linked resources

I’m trying to write some code which would take an input structure like this:
projects = {
"project1" = {
namespaces = ["mynamespace1"]
},
"project2" = {
namespaces = ["mynamespace2", "mynamespace3"]
}
}
and provision multiple resources with for_each which would result in this:
resource "rancher2_project" "project1" {
provider = rancher2.admin
cluster_id = module.k8s_cluster.cluster_id
wait_for_cluster = true
}
resource "rancher2_project" "project2" {
provider = rancher2.admin
cluster_id = module.k8s_cluster.cluster_id
wait_for_cluster = true
}
resource "rancher2_namespace" "mynamespace1" {
provider = rancher2.admin
project_id = rancher2_project.project1.id
depends_on = [rancher2_project.project1]
}
resource "rancher2_namespace" "mynamespace2" {
provider = rancher2.admin
project_id = rancher2_project.project2.id
depends_on = [rancher2_project.project2]
}
resource "rancher2_namespace" "mynamespace3" {
provider = rancher2.admin
project_id = rancher2_project.project2.id
depends_on = [rancher2_project.project2]
}
namespaces are dependent on Projects and the generate id needs to be passed into namespace.
Is there any good way of doing this dynamically ? We might have a lot of Projects/namespaces.
Thanks for any help and advise.
The typical answer for systematically generating multiple instances of a resource based on a data structure is resource for_each. The main requirement for resource for_each is to have a map which contains one element per resource instance you want to create.
In your case it seems like you need one rancher2_project per project and then one rancher2_namespace for each pair of project and namespaces. Your current data structure is therefore already sufficient for the rancher2_project resource:
resource "rancher2_project" "example" {
for_each = var.projects
provider = rancher2.admin
cluster_id = module.k8s_cluster.cluster_id
wait_for_cluster = true
}
The above will declare two resource instances with the following addresses:
rancher2_project.example["project1"]
rancher2_project.example["project2"]
You don't currently have a map that has one element per namespace, so it will take some more work to derive a suitable value from your input data structure. A common pattern for this situation is flattening nested structures for for_each using the flatten function:
locals {
project_namespaces = flatten([
for pk, proj in var.projects : [
for nsk in proj.namespaces : {
project_key = pk
namespace_key = ns
project_id = rancher2_project.example[pk].id
}
]
])
}
resource "rancher2_namespace" "example" {
for_each = {
for obj in local.project_namespaces :
"${obj.project_key}.${obj.namespace_key}" => obj
}
provider = rancher2.admin
project_id = each.value.project_id
}
This produces a list of objects representing all of the project and namespace pairs, and then the for_each argument transforms it into a map using compound keys that include both the project and namespace keys to ensure that they will all be unique. The resulting instances will therefore have the following addresses:
rancher2_namespace.example["project1.mynamespace1"]
rancher2_namespace.example["project2.mynamespace2"]
rancher2_namespace.example["project2.mynamespace3"]
This seems to work too:
resource "rancher2_namespace" "example" {
count = length(local.project_namespaces)
provider = rancher2.admin
name = local.project_namespaces[count.index].namespace_name
project_id = local.project_namespaces[count.index].project_id
}

Resources