terraform - use count and for_each together - terraform

I'm creating multiple Google Cloud Projects using a module which creates a custom service account for each one and provides that as an output eg
module "app_projects" {
count = var.number_application_projects
etc
}
I then have a list of IAM roles I want to assign to each Project Service account like this:
sa_roles = ["roles/run.developer","roles/appengine.deployer"]
resource "google_project_iam_member" "proj_sa_iam_roles" {
count = var.number_application_projects
for_each = toset(var.sa_roles)
project = module.app_projects[count.index].project_id
role = each.value
member = "serviceAccount:${module.app_projects[count.index].service_account_email}"
}
This runs into the error about "count" and "for_each" being mutually-exclusive, and I can't for the life of me figure out what to use instead, any help would be appreciated!

You can probably use setproduct, but the standard method is using flatten.
They even have a section tailored for similar use cases. Flattening nested structures for for_each.
Here is an example, that doesn't use your exact resource, but should be instructive and you can actually run it to test it out.
modules/app-project/variables.tf
variable "name" {}
modules/app-project/outputs.tf
output "name" {
value = var.name
}
modules/member/variables.tf
variable "project" {}
variable "role" {}
modules/member/outputs.tf
output "project_role" {
value = "${var.project}-${var.role}"
}
main.tf
locals {
roles = ["rad", "tubular"]
}
module "app_project" {
source = "./modules/app-project"
count = 2
name = "app-project-${count.index}"
}
module "project_role" {
source = "./modules/member"
for_each = { for pr in flatten([for p in module.app_project[*] : [
for r in local.roles : {
app_project_name = p.name
role = r
}]
]) : "${pr.app_project_name}-${pr.role}" => pr }
project = each.value.app_project_name
role = each.value.role
}
output "project_roles" {
value = values(module.project_role)[*].project_role
}
terraform plan output
Changes to Outputs:
+ project_roles = [
+ "app-project-0-rad",
+ "app-project-0-tubular",
+ "app-project-1-rad",
+ "app-project-1-tubular",
]
In your case specifically, I think something like this would work:
resource "google_project_iam_member" "proj_sa_iam_roles" {
for_each = { for i, pr in flatten([for p in module.app_project[*] : [
for r in var.sa_roles : {
app_project = p
role = r
}]
]) : "${i}-${pr.role}" => pr }
project = each.value.app_project.project_id
role = each.value.role
member = "serviceAccount:${each.value.app_project.service_account_email}"
}

Related

Terraform flatten and looping issue inside modules

I'm trying to create a module in Terraform for creating Azure resources and facing some issues. This module creates a resource group, subnet, vnet and Role bindings. I see that the below code creates the resources twice because of the loop. Does the for_each loop work in such a way that the entire resource or module block will be executed each time it loops? I'm new to Terraform and come from a Java background.
Also, ideally would like to use the flatten inside the module without locals possibly, any way to do that? Code is below.
locals {
groupsbyrole = flatten([
for roleName, groupList in var.testproject1_role_assignments : [
for groupName in groupList : {
role_name = roleName
group_name = groupName
}
]
])
}
module "testproject1" {
source = "C:\\Users\\ebo1h8h\\Documents\\Project\\Automation\\Terraform\\Code\\Azure\\modules\\sandbox-module"
short_name = "testproj"
# Resource Group Variables
az_rg_location = "eastus"
az_tags = {
Environment = "Sandbox"
CostCenter = "Department"
ResourceOwner = "Vikram"
Project = "testproj"
Role = "Resource Group"
}
address_space = ["10.0.0.0/16"]
subnet_prefixes = ["10.0.1.0/24"]
subnet_names = ["a-npe-snet01-sbox"]
vnet_location = var.az_rg_location
for_each = {
for group in local.groupsbyrole : "${group.role_name}.${group.group_name}}" => group
}
principal_id = each.value.group_name
role_definition_name = each.value.role_name
}
And here is the role_assignments variable
variable "testproject1_role_assignments" {
type = map(list(string))
default = {
"Contributor" = ["prod-azure-contrib-sbox", "gcp-org-network-engineering"],
"Owner" = ["gcp-org-cloud-delivery"]
}
}
The above code creates 12 resources when it should be only 6. The only was I was able to get around this is have the resource "azurerm_role_assignment" "role_assignment" as a separate module. Ideally, I want to pass the role assignments variable in each of the module to be created so that it creates a set of resources.
Any pointers on how to achieve that?
Thanks,
The docs state
If a resource or module block includes a for_each argument whose value is a map or a set of strings, Terraform will create one instance for each member of that map or set.
So in your scenario you are creating 3 instances of the module, whereas it sounds like you want to pass in the local.groupsbyrole object as a variable in the module and only attach the for_each to the resources you want multiple instances of.
Sidenote: You could simplify your local by adding group like below:
locals {
groupsbyrole = flatten([
for roleName, groupList in var.testproject1_role_assignments : [
for groupName in groupList : {
role_name = roleName
group_name = groupName
group = "${roleName}.${groupName}"
}
]
])
}
Tip: I find adding an output to see the shape of the object whilst developing can also be useful
output "test_output" {
value = local.groupsbyrole
}
Then when you run plan you will see your object
test_output = [
+ {
+ group = "Contributor.prod-azure-contrib-sbox"
+ group_name = "prod-azure-contrib-sbox"
+ role_name = "Contributor"
},
+ {
+ group = "Contributor.gcp-org-network-engineering"
+ group_name = "gcp-org-network-engineering"
+ role_name = "Contributor"
},
+ {
+ group = "Owner.gcp-org-cloud-delivery"
+ group_name = "gcp-org-cloud-delivery"
+ role_name = "Owner"
},
]

Terraform Invalid for_each argument local will be known only after apply

I would like to create an AWS account with SSO Account Assignments in the same first terraform run without hit the for_each limitation with dynamic values that cannot be predicted during plan.
I've tried to separate the aws_organizations_account resource from aws_ssoadmin_account_assignment in completely separate TF module and also I tried to use depends_on between those resources and modules.
What is the simplest and correct way to fix this issue?
Terraform v1.2.4
AWS SSO Account Assignments Module
Closed Pull Request that did not fix this issue
main.tf file (aws module)
resource "aws_organizations_account" "account" {
name = var.aws_account_name
email = "${var.aws_account_name}#gmail.com"
tags = {
Name = var.aws_account_name
}
parent_id = var.aws_org_folder_id
}
data "aws_identitystore_group" "this" {
for_each = local.group_list
identity_store_id = local.identity_store_id
filter {
attribute_path = "DisplayName"
attribute_value = each.key
}
}
data "aws_identitystore_user" "this" {
for_each = local.user_list
identity_store_id = local.identity_store_id
filter {
attribute_path = "UserName"
attribute_value = each.key
}
}
data "aws_ssoadmin_instances" "this" {}
locals {
assignment_map = {
for a in var.account_assignments :
format("%v-%v-%v-%v", aws_organizations_account.account.id, substr(a.principal_type, 0, 1), a.principal_name, a.permission_set_name) => a
}
identity_store_id = tolist(data.aws_ssoadmin_instances.this.identity_store_ids)[0]
sso_instance_arn = tolist(data.aws_ssoadmin_instances.this.arns)[0]
group_list = toset([for mapping in var.account_assignments : mapping.principal_name if mapping.principal_type == "GROUP"])
user_list = toset([for mapping in var.account_assignments : mapping.principal_name if mapping.principal_type == "USER"])
}
resource "aws_ssoadmin_account_assignment" "this" {
for_each = local.assignment_map
instance_arn = local.sso_instance_arn
permission_set_arn = each.value.permission_set_arn
principal_id = each.value.principal_type == "GROUP" ? data.aws_identitystore_group.this[each.value.principal_name].id : data.aws_identitystore_user.this[each.value.principal_name].id
principal_type = each.value.principal_type
target_id = aws_organizations_account.account.id
target_type = "AWS_ACCOUNT"
}
main.tf (root)
module "sso_account_assignments" {
source = "./modules/aws"
account_assignments = [
{
permission_set_arn = "arn:aws:sso:::permissionSet/ssoins-0000000000000000/ps-31d20e5987f0ce66",
permission_set_name = "ReadOnlyAccess",
principal_type = "GROUP",
principal_name = "Administrators"
},
{
permission_set_arn = "arn:aws:sso:::permissionSet/ssoins-0000000000000000/ps-955c264e8f20fea3",
permission_set_name = "ReadOnlyAccess",
principal_type = "GROUP",
principal_name = "Developers"
},
{
permission_set_arn = "arn:aws:sso:::permissionSet/ssoins-0000000000000000/ps-31d20e5987f0ce66",
permission_set_name = "ReadOnlyAccess",
principal_type = "GROUP",
principal_name = "Developers"
},
]
}
The important thing about a map for for_each is that all of the keys must be made only of values that Terraform can "see" during the planning step.
You defined local.assignment_map this way in your example:
assignment_map = {
for a in var.account_assignments :
format("%v-%v-%v-%v", aws_organizations_account.account.id, substr(a.principal_type, 0, 1), a.principal_name, a.permission_set_name) => a
}
I'm not personally familiar with the aws_organizations_account resource type, but I'm guessing that aws_organizations_account.account.id is an attribute whose value gets decided by the remote system during the apply step (once the object is created) and so this isn't a suitable value to use as part of a for_each map key.
If so, I think the best path forward here is to use a different attribute of the resource that is defined statically in your configuration. If var.aws_account_name has a static value defined in your configuration (that is, it isn't derived from an apply-time attribute of another resource) then it might work to use the name attribute instead of the id attribute:
assignment_map = {
for a in var.account_assignments :
format("%v-%v-%v-%v", aws_organizations_account.account.name, substr(a.principal_type, 0, 1), a.principal_name, a.permission_set_name) => a
}
Another option would be to remove the organization reference from the key altogether. From what you've shared it seems like there is only one account and so all of these keys would end up starting with exactly the same account name anyway, and so that string isn't contributing to the uniqueness of those keys. If that's true then you could drop that part of the key and just keep the other parts as the unique key:
assignment_map = {
for a in var.account_assignments :
format(
"%v-%v-%v",
substr(a.principal_type, 0, 1),
a.principal_name,
a.permission_set_name,
) => a
}

Iterate over list of maps

I'm trying to iterate over a simple list of maps. Here's a segment of what my module code looks like:
resource "helm_release" "nginx-external" {
count = var.install_ingress_nginx_chart ? 1 : 0
name = "nginx-external"
repository = "https://kubernetes.github.io/ingress-nginx"
chart = "ingress-nginx"
version = var.nginx_external_version
namespace = "default"
lint = true
values = [
"${file("chart_values/nginx-external.yaml")}"
]
dynamic "set" {
for_each = { for o in var.nginx_external_overrides : o.name => o }
content {
name = each.value.name
value = each.value.value
}
}
}
variable "nginx_external_overrides" {
description = "A map of maps to override customizations from the default chart/values file."
type = any
}
And here's a snippet of how I'm trying to call it from terragrunt:
nginx_external_overrides = [
{ name = "controller.metrics.enabled", value = "false" }
]
When trying to use this in a dynamic block, I'm getting:
Error: each.value cannot be used in this context
A reference to "each.value" has been used in a context in which it
unavailable, such as when the configuration no longer contains the value in
its "for_each" expression. Remove this reference to each.value in your
configuration to work around this error.
Ideally, I would be able to pass any number of maps in nginx_external_overrides to override the settings in the yaml being passed, but am struggling to do so. Thanks for the help.
If you are using for_each in dynamic blocks, you can't use each. Instead, in your case, it should be set:
dynamic "set" {
for_each = { for o in var.nginx_external_overrides : o.name => o }
content {
name = set.value.name
value = set.value.value
}
}

Accessing the module from within for_each(avoiding cycle)

I am creating modules using for_each loop. I want to access a previous module from within the module block to get a variable but it stops me from doing so because of the cycle.
locals{
deployment_plan = ["a", "b", "c"]
}
module "tier" {
source = "./modules/deployment"
for_each = { for tier,data in local.tier_config : tier => data }
tier_cfg = each.value
predecessors = [module.tier[local.deployment_plan[index(local.deployment_plan, each.key) - 1]].last_release_phase] : []
}
I see an error when assigning predecessor for the module as I am accessing module in cycle. Although I want to access the previous module.
You can't access module.tier before it is created. Thus you get the error. In your case you have to create 3 modules, for each of your deployment_plan:
locals{
deployment_plan = ["a", "b", "c"]
}
module "tier_a" {
source = "./modules/deployment"
for_each = { for tier,data in local.tier_config : tier => data }
tier_cfg = each.value
predecessors = []
}
module "tier_b" {
source = "./modules/deployment"
for_each = { for tier,data in local.tier_config : tier => data }
tier_cfg = each.value
predecessors = [module.tier_a.last_release_phase]
}
module "tier_c" {
source = "./modules/deployment"
for_each = { for tier,data in local.tier_config : tier => data }
tier_cfg = each.value
predecessors = [module.tier_b.last_release_phase]
}

Get resources based on a value created using count

I am using Terraform v12.19 with the aws provider v2.34.0.
Imagine, I have a resource generated with a count value:
resource "aws_iam_role" "role" {
count = length(var.somevariable)
name = var.somevariable[count.index]
}
Later on, I want to reference one specific resource instance in that way, e. g.:
resource "aws_iam_role_policy_attachment" "polatt" {
role = aws_iam_role.role["TheRoleNameIWant"].id
policy_arn = "arn:aws:iam::aws:policy/..."
}
I don't know the index, I can just rely on the name, provided by the variable. Thats because the values of the variable are provided by an external source and the order could change...
Any ideas how to do this?
You should be able to accomplish this using the index terraform function.
Here's a minimal example using null_resources to test it out
locals {
role_names = [
"role-a",
"role-b",
"role-c",
"role-d",
]
target_role_name = "role-c"
}
resource "null_resource" "hi" {
count = length(local.role_names)
}
output "target_resource" {
value = null_resource.hi[index(local.role_names, local.target_role_name)].id
}
output "all_resources" {
value = [for r in null_resource.hi : r.id]
}
This outputs, for example
all_resources = [
"4350570701002192774",
"9173388682753384584",
"1634695740603384613",
"2098863759573339880",
]
target_resource = 1634695740603384613
So your example, I suppose, would look like
resource "aws_iam_role_policy_attachment" "polatt" {
role = aws_iam_role.role[index(var.somevariable, "TheRoleNameIWant")].id
policy_arn = "arn:aws:iam::aws:policy/..."
}
Update
Your comment below mentions that you actually have a more complicated data structure than just a list of names. I just wanted to mention that you can derive names from your JSON structure.
Assuming you have something like the following
variable "role_values" {
value = [
{
name = "foo",
other = "details",
fields = 3
},
{
name = "bar",
other = "yet more details",
fields = 3
}
]
}
you could derive just the names by using a local and the newer for loops TF 0.12 offers
locals {
role_names = [for role in var.role_values: role.name]
}
That way you don't have to store the names twice.

Resources