for_each in terraform nested block - terraform

I've searched quite a bit and don't think I've found the answer I really need. I'm trying to loop through a nested block and am successful in doing this if all of the attributes are on the same root object. This is great if I want to loop over the entire set of attributes. However this situation is a bit different. I need to loop over an entire set of attributes and also a sub-set.
In this Terragrunt example, you can see the desired inputs since we want to loop over the escalation policy entirely as well as loop the rule and its targets so that we can create many escalation policies with many rules/targets in them.
/// PagerDuty Escalation Policies
create_escalation_policy = true
escalation_policies = [
{
name = "TEST Engineering Escalation 1"
description = "My TEST engineering escalation policy 1"
teams = ["111N1CV"]
num_loops = 2
rule = [
{
escalation_delay_in_minutes = 15
target = {
type = "user_reference"
id = "ABCB8F3"
}
},
{
escalation_delay_in_minutes = 15
target = {
type = "user_reference"
id = "NBCB1A1"
}
}
}
]
However, after quite a bit of trial and error, I'm able to loop over the entire escalation policy but not if we have values inside of rule = { which returns a generic error that Terraform can't find those attributes in the object which I have confirmed is the root object instead of the nested one. This was validated by simply moving those attributes out to the root of the object input block.
│ Error: Unsupported attribute
│
│ on main.tf line 121, in resource "pagerduty_escalation_policy" "this":
│ 121: id = rule.value.id
│ ├────────────────
│ │ rule.value is object with 5 attributes
│
│ This object does not have an attribute named "id".
For reference, here is the variable for var.escalation_policies
variable "escalation_policies" {
description = "A list of escalation policies and rules for a given PagerDuty service."
type = any
}
and the resource
resource "pagerduty_escalation_policy" "this" {
for_each = {
for key in var.escalation_policies : key.name => {
name = key.name
description = key.description
num_loops = key.num_loops
teams = key.teams
}
if var.create_escalation_policy == true
}
name = each.value.name
description = each.value.description
num_loops = each.value.num_loops
teams = each.value.teams
dynamic "rule" {
for_each = {
for k, v in var.escalation_policies : k => v }
content {
escalation_delay_in_minutes = rule.value.escalation_delay_in_minutes
target {
type = rule.value.type
id = rule.value.id
}
}
}
}

With your current example the dynamic "rule" block has a for expression that isn't really doing anything useful:
{ for k, v in var.escalation_policies : k => v }
This expression is strange in two ways:
Taking the expression alone, it's unusual to project k, v directly to k => v because that doesn't really change anything about the key or the value. Since your source var.escalation_policies is a list rather than a map this is changing the data type of the result and making Terraform convert the integer indices to strings instead, but otherwise the elements are the same as var.escalation_policies.
Considering the context, this is also unusual because it's repeating the nested block based on the same collection as the containing resource: there will be one pagerduty_escalation_policy.this instance per var.escalation_policy element and then each one will have one nested rule block for each of your escalation policies.
To get a useful result the for_each in your dynamic block should use a different collection as the basis for its repetition. I think in your case you're intending to use the nested lists inside the rule attributes of each of your policies, but your outermost for_each expression doesn't include the rules so you'll first need to update that:
resource "pagerduty_escalation_policy" "this" {
for_each = {
for policy in var.escalation_policies : policy.name => {
name = policy.name
description = policy.description
num_loops = policy.num_loops
teams = policy.teams
rules = policy.rule
}
if var.create_escalation_policy == true
}
# ...
}
This means that each.value will now include an additional attribute rules which has the same value as the corresponding attribute in each element of var.escalation_policies.
You can then refer to that rules attribute in your dynamic block:
dynamic "rule" {
for_each = each.value.rules
content {
escalation_delay_in_minutes = rule.value.escalation_delay_in_minutes
target {
type = rule.value.target.type
id = rule.value.target.id
}
}
}
This tells Terraform to generate a dynamic rule block for each element of each.value.rules, which is the rules attribute for the current policy.
Inside the content block rule.value is the current rule object, so you can refer to attributes like escalation_delay_in_minutes and target from that object.

id is a key within the target object and not within rule:
id = rule.value.target.id
Note also that a for expression which iterates through key-value pairs within a map and outputs the exact same structure is extraneous and can be removed for efficiency and readability:
dynamic "rule" {
for_each = var.escalation_policies
...
}

Related

variable not defined vs variable set to null

I try to pass some parameters to the google pub sub terraform module where they use a code block like this
for_each = var.create_topic ? { for i in var.push_subscriptions : i.name => i if try(i.dead_letter_topic, "") != "" } : {}
When I pass variables for the push_subscription like:
push_subscriptions = [
{
name = "push"
push_endpoint = "https://example.com/push"
dead_letter_topic = null
},
]
I will get an error with:
on .terraform/modules/pubsub/main.tf line 62, in resource "google_pubsub_topic_iam_member" "push_topic_binding":
│ 62: topic = each.value.dead_letter_topic
│
│ The argument "topic" is required, but no definition was found.
When I completely remove the dead_letter_topic variable it works fine.
Im wondering why this is the case? I thought (and read) when something is null then terraform threads it like it does not exist? So in my opinion both options should result in the same outcome.
Your code is trying to create a google_pubsub_topic_iam_member by passing the value of dead_letter_topic as the topic value for that resource. The topic value of that resource is a required value that you have to set. If dead_letter_topic is null, then you are trying to create a google_pubsub_topic_iam_member that has a null or empty string for a topic.
Terraform treats optional attributes as "unset" when you pass a null value. You can't do that at all for required attributes.
I believe the logic in your for_each is flawed. I think part of the problem may be that you are turning a list into a map, instead of a set.

How to concatenate strings in Terraform output with for loop?

I have multiple aws_glue_catalog_table resources and I want to create a single output that loops over all resources to show the S3 bucket location of each one. The purpose of this is to test if I am using the correct location (because it is a concatenation of variables) for each resource in Terratest. I cannot use aws_glue_catalog_table.* or aws_glue_catalog_table.[] because Terraform does not allow to reference a resource without specifying its name.
So I created a variable "table_names" with r1, r2, rx. Then, I can loop over the names. I want to create the string aws_glue_catalog_table.r1.storage_descriptor[0].location dynamically, so I can check if the location is correct.
resource "aws_glue_catalog_table" "r1" {
name = "r1"
database_name = var.db_name
storage_descriptor {
location = "s3://${var.bucket_name}/${var.environment}-config/r1"
}
...
}
resource "aws_glue_catalog_table" "rX" {
name = "rX"
database_name = var.db_name
storage_descriptor {
location = "s3://${var.bucket_name}/${var.environment}-config/rX"
}
}
variable "table_names" {
description = "The list of Athena table names"
type = list(string)
default = ["r1", "r2", "r3", "rx"]
}
output "athena_tables" {
description = "Athena tables"
value = [for n in var.table_names : n]
}
First attempt: I tried to create an output "athena_tables_location" with the syntax aws_glue_catalog_table.${table} but does does.
output "athena_tables_location" {
// HOW DO I ITERATE OVER ALL TABLES?
value = [for t in var.table_names : aws_glue_catalog_table.${t}.storage_descriptor[0].location"]
}
Second attempt: I tried to create a variable "table_name_locations" but IntelliJ already shows an error ${t} in the for loop [for t in var.table_names : "aws_glue_catalog_table.${t}.storage_descriptor[0].location"].
variable "table_name_locations" {
description = "The list of Athena table locations"
type = list(string)
// THIS ALSO DOES NOT WORK
default = [for t in var.table_names : "aws_glue_catalog_table.${t}.storage_descriptor[0].location"]
}
How can I list all table locations in the output and then test it with Terratest?
Once I can iterate over the tables and collect the S3 location I can do the following test using Terratest:
athenaTablesLocation := terraform.Output(t, terraformOpts, "athena_tables_location")
assert.Contains(t, athenaTablesLocation, "s3://rX/test-config/rX",)
It seems like you have an unusual mix of static and dynamic here: you've statically defined a fixed number of aws_glue_catalog_table resources but you want to use them dynamically based on the value of an input variable.
Terraform doesn't allow dynamic references to resources because its execution model requires building a dependency graph between all of the objects, and so it needs to know which exact resources are involved in a particular expression. However, you can in principle build your own single value that includes all of these objects and then dynamically choose from it:
locals {
tables = {
r1 = aws_glue_catalog_table.r1
r2 = aws_glue_catalog_table.r2
r3 = aws_glue_catalog_table.r3
# etc
}
}
output "table_locations" {
value = {
for t in var.table_names : t => local.tables[t].storage_descriptor[0].location
}
}
With this structure Terraform can see that output "table_locations" depends on local.tables and local.tables depends on all of the relevant resources, and so the evaluation order will be correct.
However, it also seems like your table definitions are systematic based on var.table_names and so could potentially benefit from being dynamic themselves. You could achieve that using the resource for_each feature to declare multiple instances of a single resource:
variable "table_names" {
description = "Athena table names to create"
type = set(string)
default = ["r1", "r2", "r3", "rx"]
}
resource "aws_glue_catalog_table" "all" {
for_each = var.table_names
name = each.key
database_name = var.db_name
storage_descriptor {
location = "s3://${var.bucket_name}/${var.environment}-config/${each.key}"
}
...
}
output "table_locations" {
value = {
for k, t in aws_glue_catalog_table.all : k => t.storage_descriptor[0].location
}
}
In this case aws_glue_catalog_table.all represents all of the tables together as a single resource with multiple instances, each one identified by the table name. for_each resources appear in expressions as maps, so this will declare resource instances with addresses like this:
aws_glue_catalog_table.all["r1"]
aws_glue_catalog_table.all["r2"]
aws_glue_catalog_table.all["r3"]
...
Because this is already a map, this time we don't need the extra step of constructing the map in a local value, and can instead just access this map directly to build the output value, which will be a map from table name to storage location:
{
r1 = "s3://BUCKETNAME/ENVNAME-config/r1"
r2 = "s3://BUCKETNAME/ENVNAME-config/r2"
r3 = "s3://BUCKETNAME/ENVNAME-config/r3"
# ...
}
In this example I've assumed that all of the tables are identical aside from their names, which I expect isn't true in practice but I was going only by what you included in the question. If the tables do need to have different settings then you can change var.table_names to instead be a variable "tables" whose type is a map of object type where the values describe the differences between the tables, but that's a different topic kinda beyond the scope of this question, so I won't get into the details of that here.

Terraform: Create block only if variable matches certain values

I'm trying to create a module that creates interconnect-attachments, but some parts are only defined if the attachment is using ipsec encryption and if it's not, that block must not exist in the resource else it causes an error (even if it only contains a value set to null.)
I've tried using a dynamic, but I can't quite get the layout right to have it work:
resource "google_compute_interconnect_attachment" "interconnect-attachment" {
project = var.project
region = var.region
name = var.name
edge_availability_domain = var.availability_domain
type = var.type
router = google_compute_router.router.name
encryption = var.encryption
dynamic "ipsec_internal_addresses" {
for_each = var.encryption != "IPSEC" ? [] : [1]
content {
var.address
}
}
}
Essentially, if var.encryption is set to IPSEC then i want the following block included:
ipsec_internal_addresses = [
var.address,
]
The other issue is it appears a dynamic block expects some kind of assignment to happen, but the terraform examples just have the value inside the ipsec_internal_addresses so I'm unsure how to to achieve this.
ipsec_internal_addresses is not a block in the google_compute_interconnect_attachment resource. It is an argument. Therefore, you can use the normal pattern for specifying optional arguments where the conditional returns a null type if you do not want to specify a value. Using your conditional and variables:
ipsec_internal_addresses = var.encryption == "IPSEC" ? [var.address] : null
which will return and assign your [var.address] to ipsec_internal_addresses when var.encryption equals the string IPSEC. Otherwise, it will return null and the ipsec_internal_addresses argument will be ignored.

Terraform: Conditional creation of a resource based on a variable in .tfvars

I have resources defined in .tf files that are generic to several applications. I populate many of the fields via a .tfvars file. I need to omit some of the resources entirely based on variables in the .tfvars.
For example if I have a resource like:
resource "cloudflare_record" "record" {
zone_id = "${data.cloudflare_zones.domain.zones[0].id}"
name = "${var.subdomain}"
value = "${var.origin_server}"
type = "CNAME"
ttl = 1
proxied = true
}
But then I declare something like cloudflare = false in my .tfvars file I'd like to be able to do something like this:
if var.cloudflare {
resource "cloudflare_record" "record" {
zone_id = "${data.cloudflare_zones.domain.zones[0].id}"
name = "${var.subdomain}"
value = "${var.origin_server}"
type = "CNAME"
ttl = 1
proxied = true
}
}
I've looked at dynamic blocks but that looks like you can only use those to edit fields and blocks within a resource. I need to be able to ignore an entire resource.
Add a count parameter with a ternary conditional using the variable declared in .tfvars like this:
resource "cloudflare_record" "record" {
count = var.cloudflare ? 1 : 0
zone_id = "${data.cloudflare_zones.domain.zones[0].id}"
name = "${var.subdomain}"
value = "${var.origin_server}"
type = "CNAME"
ttl = 1
proxied = true
}
In this example var.cloudflare is a boolean declared in the .tfvars file. If it is true a count of 1 record will be created. If it is false a count of 0 record will be created.
After the count apply the resource becomes a group, so later in the reference use 0-index of the group:
cloudflare_record.record[0].some_field
Expanding on #Joel Guerra's answer, after you use count to determine whether to deploy the resource or not, you can use the one() function to refer to the resource without an index (i.e. without having to use [0]).
For example, after defining the resource like below
resource "cloudflare_record" "record" {
count = var.cloudflare ? 1 : 0
}
Define a local variable like below
locals {
cloudflare_record_somefield = one(cloudflare_record.record[*].some_field)
}
Now instead of cloudflare_record.record[0].some_field, you can use
local.cloudflare_record_somefield
If the count is 0 (e.g. var.cloudflare is false and the resource wasn't created) then local.cloudflare_record_somefield would return null (instead of returning an error when indexing using [0]).
Reference: https://developer.hashicorp.com/terraform/language/functions/one
An issue i'm seeing this with is if the resource your trying to create is already using a for_each then you can't use both count and for_each in the resource. I'm still trying to find an answer on this will update if I find something better.

cloudflare terraform provider firewall creation with loop

I am trying to work around a constraint where firewall creation is split into 2 sections, creating a filter and creating the rule based on the filter. Filter creation exposes a filter id that should be used in the fw rule creation. I cant wrap my head on how to properly iterate through the map that has values for filter and rule and include newly created filter. if i just use a simple map with name and expression, things work, but if i add rule priority things break
here is my map
variable "fw_allowfilters1" {
description = "list of expressions for firewall to be included in the allow rules"
type = map(object({
fvalue = string
priority = number
}))
default = {
"office_filter1" = [
{
fvalue = "ip.geoip.asnum eq 111111"
priority = 1
}
]
"office_filter2" = [
{
fvalue = "ip.src eq 8.8.8.8"
priority = 3
}
]
}
}
now here is my code for both filter and FW
resource "cloudflare_filter" "allow-filters1" {
for_each = var.fw_allowfilters1
zone_id = var.zoneid
expression = each.value.fvalue
description = each.key
//description = [for o in var.fw_allowfilters1: "Filter_${var.fw_allowfilters1.name}"]
//expression = [for o in var.fw_allowfilters1: var.fw_allowfilters1.value]
}
resource "cloudflare_firewall_rule" "whitelist-rule" {
for_each = cloudflare_filter.allow-filters1
action = "allow"
filter_id = tostring(each.value.id)
zone_id = var.zoneid
description = [for p in var.fw_allowfilters1.name: p.name ]
priority = [for p in var.fw_allowfilters1.priority: p.priority ]
}
now if i dont include priority, i can do the for_each on the filter output in firewall creation, using id output from the resource and key for descirption ( cf tf provider uses description as a name) however, if i need to add the key, i need to iterate through the map with values plus the id that is output of the filter creation and I am not sure how to properly map it. code currently does not work.
so i figured it out and it was not easy:) using locals helped me create proper iterators:
resource "cloudflare_filter" "filters1" {
for_each = var.fw_rules
zone_id = var.zoneid
description = "Filter_${tostring(each.key)}"
expression = tostring(each.value[0])
}
locals {
filterids = [for f in cloudflare_filter.filters1 : f.id] //putting filter
IDs into a separate list for concat later
fwvalues = (values(var.fw_rules)) // putting values from the map of fwvalues into
a separate list to use the index position of a particular value as an interator when
creating commong object that has both filterid and fwvalues
fwkeys = (keys(var.fw_rules)) //putting keys into a separate list
//iterating over all elements in the allowfilters1, combining existing lists in the
variable with the ID value and readding the key as an element in the list
withid3 = {for p in var.fw_rules : local.fwkeys[index(local.fwvalues, p.*)] =>
concat(p, list(local.filterids[index(local.fwvalues,
p.*)]),list(local.fwkeys[index(local.fwvalues, p.*)]))} //working version
}
resource "cloudflare_firewall_rule" "fw-rules" {
for_each = local.withid3
action = each.value[2]
filter_id = each.value[4]
paused = each.value[3]
zone_id = var.zoneid
description = "FW-${tostring(each.value[2])}-${tostring(each.key)}"
priority = each.value[1]
}
where varilable is this:
// the syntax is the following: name of the rule(try to be precise = [ expression, priority,action, disabled - boolean] - all values should be strings, make sure to terminate the quotes correctly
allowed values for the action are: block, challenge, js_challenge, allow, log, bypass
list has to be maintained according to the rule priority
variable "fw_rules" {
description = "list of expressions for firewall to be included in therules"
type = map
default = {
office_allow = ["putexpressionhere","1","allow","false"],
office_allow1 = ["putexpressionhere1","2","deny","false"]
}

Resources