Using terraform's for_each with data that doesn't have a unique key - terraform

Question

When using terraform's for_each you have to specify a unique id to be used as a way to link the generated recource with its source definition.
I'd like to use a natural index for this, rather than an arbitrary unique value. In this case I'm working with DNS, so the natural index would be the DNS record name (FQDN)... Only that isn't always unique; i.e. you can have multipe A records for example.com to allow load balancing, or you may have multiple TXT records for providing verification to multiple vendors.
Is there a way to combine the natural index with a calculated value to provide a unique value; e.g. so we have the natural index followed by a 1 if it's the first time this value's seen, a 2 for the first duplicate, etc?
Specific Requirement / Context
I'm working on migrating our DNS records to be managed via IaC using Terraform/Terragrunt (this is for scenarios where the entries are manually managed, rather than those where the related service is also under IaC).
I'm hoping to hold the record data in CSVs (or similar) to avoid those managing the records day to day from requiring familiarity with TF/TG; instead allowing them to just update the data, and have the pipeline take care of the rest.
The CSV format would be something like this:
myId
RecordName
Type
Value
1
A
1.2.3.4
2
A
2.3.4.5
3
test
A
3.4.5.6
4
test
A
4.5.6.7
5
www
cname
example.com
Note: I'm considering each DNS Zone would have a folder with its name, and a CSV formatted as above which gives the records for that zone; so the above would be in the /example.com/ folder, and thus we'd have 2 A records for example.com, 2 for test.example.com and one CName for www.example.com which pointed to example.com.
locals {
instances = csvdecode(file("myDnsRecords.csv"))
}
resource aws_route53_zone zone {
name = var.domainname
provider = aws
}
resource aws_route53_record route53_entry {
for_each = {for inst in local.instances : inst.myId => inst}
name = "${each.value.RecordName}${each.value.RecordName == "" ? "" : "."}${var.domainname}"
type = each.value.Type
zone_id = aws_route53_zone.zone.zone_id
ttl = 3600
records = [each.value.Value]
}
I don't want the myId column though; as that doesn't add value / has no real relationship to the records; so if we were to remove/insert a record early in the CSV and renumber the following records it would result in a number of changes being required to records which hadn't really changed, just because their related "index" had changed.
I also don't want those working with these CSVs to have to manually manage such fields; i.e. I could provide another column and ask that they populate this as below... but that's asking for human error and adding complexity:
myId
RecordName
Type
Value
1
A
1.2.3.4
2
A
2.3.4.5
test1
test
A
3.4.5.6
test2
test
A
4.5.6.7
www1
www
cname
example.com
Question
Is there a way I can use a for_each loop with CSV data such as below, whilst working around the unique constraint?
RecordName
Type
Value
A
1.2.3.4
A
2.3.4.5
test
A
3.4.5.6
test
A
4.5.6.7
www
cname
example.com

Answer 1

You can add unique keys to the data structure:
locals {
instances = csvdecode(file("myDnsRecords.csv"))
instance_map = zipmap(range(0,length(local.instances)), local.instances)
}
resource "..." "..." {
for_each = local.instance_map
...
}

Answer 2

Terraform's for expressions when constructing a mapping have a "grouping mode" where it allows duplicate keys in return for the values of the map all being lists of potentially-multiple values that all had the same key.
I would therefore start by using that to project the CSV data into a map(list(map(string))) value where the keys are build from the record name and type, like this:
locals {
records_raw = csvdecode(file("${path.module}/myDnsRecords.csv"))
records_grouped = tomap({
for row in local.records_raw :
"${row.RecordName} ${row.RecordType}" => row...
})
}
The resulting data structure would be shaped like this:
records_grouped = tomap({
" A" = tolist([
{ RecordName = "", Type = "A", Value = "1.2.3.4" },
{ RecordName = "", Type = "A", Value = "2.3.4.5" },
])
"test A" = tolist([
{ RecordName = "test", Type = "A", Value = "3.4.5.6" },
{ RecordName = "test", Type = "A", Value = "4.5.6.7" },
])
"www CNAME" = tolist([
{ RecordName = "www", Type = "CNAME", Value = "example.com" },
])
})
Collecting the records with common keys into lists means that we now have a list index for each one that's unique only within the records with the common key.
So now we can project this one more time into a flat map of maps (map(map(string))) by incorporating those list indices into the map keys:
locals {
records = tomap(merge([
for group_key, group in local.records_grouped : {
for idx, record in group :
"${group_key} ${idx}" => group
}
]...))
}
This should produce a data structure like the following:
records = tomap({
" A 0" = { RecordName = "", Type = "A", Value = "1.2.3.4" }
" A 1" = { RecordName = "", Type = "A", Value = "2.3.4.5" }
"test A 0" = { RecordName = "test", Type = "A", Value = "3.4.5.6" }
"test A 1" = { RecordName = "test", Type = "A", Value = "4.5.6.7" }
"www CNAME 0" = { RecordName = "www", Type = "CNAME", Value = "example.com" }
})
That data structure is suitably-shaped for a for_each expression, so finally:
resource "aws_route53_record" "example" {
for_each = local.records
name = "${each.value.RecordName}${each.value.RecordName == "" ? "" : "."}${var.domainname}"
type = each.value.Type
zone_id = aws_route53_zone.zone.zone_id
ttl = 3600
records = [each.value.Value]
}
This will produce instance unique instance keys for each entry in the source CSV file while keeping all of the distinct (name, type) pairs separated so that you can add new ones without disturbing any existing records:
aws_route53_record.example[" A 0"]
aws_route53_record.example[" A 1"]
aws_route53_record.example["test A 0"]
...etc
You mentioned wanting a separate instance for each row in your CSV file but I also wanted to note that the aws_route53_record resource type is already designed to manage multiple records with the same name and type together, and so I think it would actually be fine to leave the records grouped together. (The name `aws_route53_record is a bit of a misnomer because each instance of this resource type manages a record set, not just a single record.)
Here's a variation that works that way:
locals {
records_raw = csvdecode(file("${path.module}/myDnsRecords.csv"))
record_groups = tomap({
for row in local.records_raw :
"${row.RecordName} ${row.RecordType}" => row...
})
recordsets = tomap({
for group_key, group in local.record_groups : group_key => {
name = group[0].Name
type = group[0].Type
values = group[*].Value
}
})
}
resource "aws_route53_record" "example" {
for_each = local.recordsets
name = "${each.value.name}${each.value.name == "" ? "" : "."}${var.domainname}"
type = each.value.type
zone_id = aws_route53_zone.zone.zone_id
ttl = 3600
records = each.value.values
}
This time the final map has one element per recordset instead of one element per record, after grouping all of the individual records together using their names and types. Now you don't need any synthetic indices at all because the name and type pair is the natural unique identifier for a Route53 recordset.

Answer 3

Does this get what you're looking for?
Dataset
RecordName,Type,Value
,A,1.2.3.4
,A,2.3.4.5
test,A,3.4.5.6
test,A,4.5.6.7
www,cname,example.com
main.tf
locals {
records = [for pref in {for _, key in distinct([for i, v in csvdecode(file("myDnsRecords.csv")): v.RecordName]): key => [for r in csvdecode(file("myDnsRecords.csv")): r if key == r.RecordName]}: {for i, r in pref: ("${r.RecordName}_${i}") => r}]
}
output "test" {
value = local.records
}
Output
Changes to Outputs:
+ test = [
+ {
+ _0 = {
+ RecordName = ""
+ Type = "A"
+ Value = "1.2.3.4"
}
+ _1 = {
+ RecordName = ""
+ Type = "A"
+ Value = "2.3.4.5"
}
},
+ {
+ test_0 = {
+ RecordName = "test"
+ Type = "A"
+ Value = "3.4.5.6"
}
+ test_1 = {
+ RecordName = "test"
+ Type = "A"
+ Value = "4.5.6.7"
}
},
+ {
+ www_0 = {
+ RecordName = "www"
+ Type = "cname"
+ Value = "example.com"
}
},
]
You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.