Terraform: creating multiple instances with for_each - terraform

Question

I'd love some help with Terraform's count/for_each functions.
The goal is to read multiple json files(current two) into a list of maps and create specific amount of aws_instances with specific naming convention.
Configurations
cat test_service_1.json
{
"instance_name": "front",
"instance_count": "3",
"instance_type": "t2.micro",
"subnet_type": "private",
"elb": "yes",
"data_volume": ["no", "0"]
}
cat test_service_2.json
{
"instance_name": "back",
"instance_count": "3",
"instance_type": "t2.micro",
"subnet_type": "private",
"elb": "yes",
"data_volume": ["no", "0"]
}
cat main.tf
locals {
services = [jsondecode(file("${path.module}/test_service_1.json")),
jsondecode(file("${path.module}/test_service_2.json"))]
}
resource "aws_instance" "test_instance" {
ami = "amzn-ami-hvm-2018.03.0.20200206.0-x86_64-gp2"
instance_type = "t2.micro"
tags = merge(
map("Name", "prod-app-?"),
map("env", "prod")
)
}
Eventually I want the code to go over both json files and create:
prod-front-1
prod-front-2
prod-front-3
prod-back-1
prod-back-2
prod-back-3
I can do that with [count.index +1] but I don't know how to loop through more than one map.

Answer 1

When using resource for_each our task is always to write an expression that produces a map where there is one element per instance we want to create. In this case, that seems to be an expression that can expand a single object containing a count to instead be multiple objects of the number given in the count.
The building blocks we can use to do this in Terraform are:
for expressions to project one collection value into another.
The range function to generate sequences of integers given a count.
The flatten function to turn multiple nested lists into a single flat list.
Let's take this step by step. We'll start with your existing expression to load the data from the JSON files:
locals {
services = [
jsondecode(file("${path.module}/test_service_1.json")),
jsondecode(file("${path.module}/test_service_2.json")),
]
}
The result of this expression is a list of objects, one per file. Next, we'll expand each of those objects into a list of objects whose length is given in instance_count:
locals {
service_instance_groups = [
for svc in local.services : [
for i in range(1, svc.instance_count+1) : {
instance_name = "${svc.instance_name}-${i}"
instance_type = svc.instance_type
subnet_type = svc.subnet_type
elb = svc.elb
data_volume = svc.data_volume
}
]
]
}
The result of this one is a list of lists of objects, each of which will have a unique instance_name value due to concatenating the value i to the end.
To use for_each though we we will need a flat collection with one element per instance, so we'll use the flatten function to achieve that:
locals {
service_instances = flatten(local.service_instance_groups)
}
Now we have a list of objects again, but with six elements (three from each of the two input objects) instead of two.
Finally, we need to project that list to be a map whose keys are the unique identifiers Terraform will use to track the instances. I usually prefer to do this final step directly inside the for_each argument because this result is specific to that use-case and unlikely to be used anywhere else in the module:
resource "aws_instance" "test_instance" {
for_each = {
for inst in local.service_instances : inst.instance_name => inst
}
ami = "amzn-ami-hvm-2018.03.0.20200206.0-x86_64-gp2"
instance_type = each.value.instance_type
tags = {
Name = "prod-app-${each.key}"
Env = "prod"
}
}
This should result in Terraform planning to create instances with addresses like aws_instance.test_instance["front-2"].
I wrote each of the above steps out separately to explain what each one was achieving, but in practice I'd usually do the service_instance_groups and service_instances steps together in a single expression, because that intermediate service_instance_groups result isn't likely to be reused elsewhere. Bringing that all together into a single example, then:
locals {
services = [
jsondecode(file("${path.module}/test_service_1.json")),
jsondecode(file("${path.module}/test_service_2.json")),
]
service_instances = flatten([
for svc in local.services : [
for i in range(1, svc.instance_count+1) : {
instance_name = "${svc.instance_name}-${i}"
instance_type = svc.instance_type
subnet_type = svc.subnet_type
elb = svc.elb
data_volume = svc.data_volume
}
]
])
}
resource "aws_instance" "test_instance" {
for_each = {
for inst in local.service_instances : inst.instance_name => inst
}
ami = "amzn-ami-hvm-2018.03.0.20200206.0-x86_64-gp2"
instance_type = each.value.instance_type
tags = {
Name = "prod-app-${each.key}"
Env = "prod"
}
}
As a bonus, beyond what you were asking about here, if you give those JSON files systematic names and group them together into a subdirectory then you could use Terraform's fileset function to to automatically pick up any new files added in that directory later, without changing the Terraform configuration. For example:
locals {
services = [
for fn in fileset("${path.module}", "services/*.json") :
jsondecode(file("${path.module}/${fn}"))
]
}
The above will produce a list containing an object for each of the files in the services subdirectory that have names ending in .json.