Writing DRY code with terraform for loop? - terraform

I am very new to terraform and had a task dropped upon me to create 2 AWS KMS keys.
So I am doing this:
resource "aws_kms_key" "ebs_encryption_key" {
description = "EBS encryption key"
... omitted for brevity ...
tags = merge(map(
"Name", format("%s-ebs-encryption-key", var.name_prefix),
"component", "kms",
"dataclassification","low",
), var.extra_tags)
}
resource "aws_kms_alias" "ebs_encryption_key" {
name = format("alias/%s-ebs-encryption-key", var.name_prefix)
target_key_id = aws_kms_key.ebs_encryption_key.key_id
}
# Repeated code!
resource "aws_kms_key" "rds_encryption_key" {
description = "RDS encryption key"
... omitted for brevity ...
tags = merge(map(
"Name", format("%s-rds-encryption-key", var.name_prefix),
"component", "kms",
"dataclassification","low",
), var.extra_tags)
}
resource "aws_kms_alias" "rds_encryption_key" {
name = format("alias/%s-rds-encryption-key", var.name_prefix)
target_key_id = "${aws_kms_key.rds_encryption_key.key_id}"
}
As you can see the only difference between the two blocks of code is "ebs" and "rds"?
How could I use a for loop to avoid repeating the code blocks?

This seems like it could be a candidate for a small module that encapsulates the details of declaring a key and an associated alias, since a key and an alias are typically declared together in your system.
The module itself would look something like this:
variable "name" {
type = string
}
variable "description" {
type = string
}
variable "tags" {
type = map(string)
}
resource "aws_kms_key" "main" {
description = var.description
# ...
tags = var.tags
}
resource "aws_kms_alias" "main" {
name = "alias/${var.name}"
target_key_id = aws_kms_key.main.key_id
}
output "key_id" {
value = aws_kms_key.main.key_id
}
output "alias_name" {
value = aws_kms_alias.main.name
}
(As written here this module feels a little silly because there's not really much here that isn't derived only from the variables, but I'm assuming that the interesting stuff you want to avoid repeating is in "omitted for brevity" in your example, which would go in place of # ... in my example.)
Your calling module can then include a module block that uses for_each to create two instances of the module, systematically setting the arguments to populate its input variables:
module "kms_key" {
for_each = {
kms = "KMS"
ebs = "EBS"
}
name = "${var.name_prefix}-${each.key}-encryption-key"
description = "${each.value} Encryption Key"
tags = merge(
var.extra_tags,
{
Name = "${var.name_prefix}-${each.key}-encryption-key"
component = "kms"
dataclassification = "low"
},
)
}
Since the for_each map here has the keys kms and ebs, the result of this will be to declare resource instances which should have the following addresses in the plan:
module.kms_key["kms"].aws_kms_key.main
module.kms_key["kms"].aws_kms_alias.main
module.kms_key["ebs"].aws_kms_key.main
module.kms_key["ebs"].aws_kms_alias.main
Since they are identified by the map keys, you can add new keys to that map in future to create new key/alias pairs without disturbing the existing ones.
If you need to use the key IDs or alias names elsewhere in your calling module then you can access them via the outputs exposed in module.kms_key elsewhere in that calling module:
module.kms_key["kms"].key_id
module.kms_key["kms"].alias_name
module.kms_key["ebs"].key_id
module.kms_key["ebs"].alias_name

Related

Terraform: Easier way to run a flexible number of objects

Currently I have a powershell script that reads a yaml config file with all the objects I need created and creates a .tfvars file which contains all the variables, maps, lists of maps etc.
It would be something like the following:
global_tags = {
Provisioner = "Terraform"
}
resource_groups = {
myrg1 = {
location = "uksouth",
tags = {
ResourceType = "resourcegroup"
}
}
}
storage_accounts = {
mystorage1 = {
resource_group_name = "myrg1",
location = "uksouth",
account_tier = "Standard",
account_replication_type = "GRS",
tags = {
ResourceType = "storageaccount"
}
containers_list = [
{ name = "test_private_x", access_type = "private" },
{ name = "test_blob_x", access_type = "blob" },
{ name = "test_container_x", access_type = "container" }
]
}
The idea is to then pump each list of maps into each module to create the resources, e.g. main.tf would be just:
module "resourcegroup" {
source = "./modules/azure-resourcegroup"
resource_groups = var.resource_groups
global_tags = var.global_tags
}
module "storageaccount" {
source = "./modules/azure-storageaccount"
depends_on = [module.resourcegroup]
storage_accounts = var.storage_accounts
global_tags = var.global_tags
}
Also, an example of a simple module would be:
resource "azurerm_resource_group" "rg" {
for_each = var.resource_groups
name = each.key
location = each.value.location
tags = lookup(each.value,"tags",null) == null ? var.global_tags : merge(var.global_tags,each.value.tags)
}
The issue is that writing a complex module, say around storage account, isn't too bad if you are just feeding in all the params, but feeding in a list of maps and writing a module to read that list and create multiple flattened lists to perform say 15 different calls (to create containers, shares, network rules etc.) is very complex.
Obviously the reason I want to use for_each loops in the modules is so that my main.tf doesn't have to call the module multiple times with hard coded values for say 50 storage accounts.
Just wondering if I am missing an obvious way to create complicated multiples of each resource type ?
I appreciate I could do separate modules for containers, shares etc and break the complex maps down into simpler ones to pass to the additional modules, but I was trying to just have 1 storage account module that could handle anything and be fed by a complex list of maps so main.tf did not need editing, I could just control the config completely via a .tfvars file

How to create a single dynamic block for a module from a map or object definition?

If I want to define a lambda function with a VPC config. I can do it like this:
resource "aws_lambda_function" "lambda" {
function_name = "..."
...
vpc_config {
subnet_ids = ["..."]
security_group_ids = ["..."]
}
}
I would like to create the lambda in a terraform module and define the vpc_config in the module definition. I can define the module like this:
resource "aws_lambda_function" "lambda" {
function_name = "..."
...
dynamic "vpc_config" {
for_each = var.vpc_configs
content {
subnet_ids = vpc_config.value["subnet_ids"]
security_group_ids = vpc_config.value["security_group_ids"]
}
}
}
variable "vpc_configs" {
type = list(object({
subnet_ids = list(string)
security_group_ids = list(string)
}))
default = []
}
And then use it:
module "my_lambda" {
source = "./lambda"
...
vpc_configs = [
{
subnet_ids = ["..."]
security_group_ids = ["..."]
}
]
}
However, since there is only one vpc_config block allowed there is no point in defining the variable as a list. I would prefer the following syntax:
module "my_lambda" {
source = "./lambda"
...
vpc_config = {
subnet_ids = ["..."]
security_group_ids = ["..."]
}
# or:
#vpc_config {
# subnet_ids = ["..."]
# security_group_ids = ["..."]
#}
}
However, I can't figure out if it is possible to define a variable like this and then use it in a dynamic block. I defined it as a list in the first place because I don't always need a VPC config and this way I can simply leave the list empty and no VPC config will be created. Is there a way to create an optional vpc_config block through a simple map or object definition?
dynamic blocks work by generating one block for each element in a collection, if any, whereas you want to define a variable that is an optional non-collection value. Therefore the key to this problem is to translate from a single value that might be null (representing absence) into a list of zero or one elements.
Due to how commonly this arises, Terraform has a concise way to represent that conversion using the splat operator, [*]. If you apply it to a value that isn't a list, then it will implicitly convert it into a list of zero or one elements, depending on whether the value is null.
The example in the documentation I just linked to shows a practical example of this pattern. The following is essentially the same approach, but adapted to use the resource type that you are using in your question:
variable "vpc_config" {
type = object({
subnet_ids = list(string)
security_group_ids = list(string)
})
default = null
}
resource "aws_lambda_function" "lambda" {
function_name = "..."
...
dynamic "vpc_config" {
for_each = var.vpc_config[*]
content {
subnet_ids = vpc_config.value.subnet_ids
security_group_ids = vpc_config.value.security_group_ids
}
}
}
The default value of var.vpc_config is null, so if the caller doesn't set it then that is the value it will take.
var.vpc_config[*] will either return an empty list or a list containing one vpc_config object, and so this dynamic block will generate either zero or one vpc_config blocks depending on the "null-ness" of var.vpc_config.
so you are wanting a conditional dynamic block
you could possibly get away with it by doing a check similar to the one on the object below
dynamic "vpc_config"{
for_each = length(var.vpc_config) > 0 ? {config=var.vpc_config}: {}
content{
...
}
}
if no vpc_config is passed in the module then the input variable should default to something like an empty object {}, that way the dynamic conditional check will still work if no config is passed
Turns out it doesn't seem to be possible what I want to do (building an optional type safe configuration through an object definition without having to nest it in a list).
Instead I now use the lambda module provided by Terraform:
module "email_lambda" {
source = "terraform-aws-modules/lambda/aws"
version = "3.3.1"
function_name = "${var.stack_name}-email"
handler = "pkg.email.App::handleRequest"
runtime = "java11"
architectures = ["x86_64"]
memory_size = 512
timeout = 30
layers = [aws_lambda_layer_version.lambda_layer.arn]
create_package = false
local_existing_package = "../email/target/email.jar"
environment_variables = {
# https://aws.amazon.com/blogs/compute/optimizing-aws-lambda-function-performance-for-java/
JAVA_TOOL_OPTIONS = "-XX:+TieredCompilation -XX:TieredStopAtLevel=1"
}
vpc_subnet_ids = module.vpc.private_subnets
vpc_security_group_ids = [aws_security_group.lambda_security_group.id]
attach_policies = true
policies = [
"arn:aws:iam::aws:policy/service-role/AWSLambdaSQSQueueExecutionRole",
]
number_of_policies = 1
attach_policy_json = true
policy_json = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "SESBulkTemplatedPolicy"
Effect = "Allow"
Resource = [...]
Action = [
"ses:SendEmail",
"ses:SendRawEmail",
"ses:SendTemplatedEmail",
"ses:SendBulkTemplatedEmail",
]
}
]
})
}
As one can see in this configuration I had to set the VPC parameters individually and in case of the policy I had to specify a boolean parameter to tell Terraform that the configuration was set (I even had to specify the length of the provided list). Looking at the source code of the module reveals that there may not be a better way how to achieve this in the most up to date version of Terraform.

Whats the right way to create multiple elements using terraform variables?

I am creating AWS SQS queues using Terraform. for each service, i need to create two queues, one normal queue and one error queue. The settings for each are mostly the same, but i need to create the error queue first so i can pass its ARN to the normal queue as part of its redrive policy. Instead of creating 10 modules there has to be a better way to loop through replacing just the names. So programming logic... foreach queue in queue_prefixes, create error module, then regular module. Im sure im just not searching right or asking the right question.
sandbox/main.tf
provider "aws" {
region = "us-west-2"
}
module "hfd_sqs_error_sandbox" {
source = "../"
for_each = var.queue_prefixes
name= each.key+"_Error"
}
module "hfd_sqs_sandbox" {
source = "../"
name=hfd_sqs_error_sandbox.name
redrive_policy = jsonencode({
deadLetterTargetArn = hfd_sqs_error_sandbox_this_sqs_queue_arn,
maxReceiveCount = 3
})
}
variables.tf
variable "queue_prefixes" {
description = "Create these queues with the enviroment prefixed"
type = list(string)
default = [
"Clops",
"Document",
"Ledger",
"Log",
"Underwriting",
"Wallet",
]
}
You may want to consider adding a wrapper module that creates both Normal Queue and Dead-Letter Queue. That would make creating resources in order much easier.
Consider this example (with null resources for easy testing):
Root module creating all queues:
# ./main.tf
locals {
queue_prefixes = [
"Queue_Prefix_1",
"Queue_Prefix_2",
]
}
module queue_set {
source = "./modules/queue_set"
for_each = toset(local.queue_prefixes)
name = each.key
}
Wrapper module creating a set of 2 queues: normal + dlq:
# ./modules/queue_set/main.tf
variable "name" {
type = string
}
module dlq {
source = "../queue"
name = "${var.name}_Error"
}
module queue {
source = "../queue"
name = var.name
redrive_policy = module.dlq.id
}
Individual queue resource suitable to create both types of queues:
# ./modules/queue/main.tf
variable "name" {
type = string
}
variable "redrive_policy" {
type = string
default = ""
}
resource "null_resource" "queue" {
provisioner "local-exec" {
command = "echo \"Created queue ${var.name}, redrive policy: ${var.redrive_policy}\""
}
# this is irrelevant to the question, it's just to make null resource change every time
triggers = {
always_run = timestamp()
}
}
output "id" {
value = null_resource.queue.id
}
Now if we run this stack, we can see the resources created in the correct order:

terraform - Iterate over two linked resources

I’m trying to write some code which would take an input structure like this:
projects = {
"project1" = {
namespaces = ["mynamespace1"]
},
"project2" = {
namespaces = ["mynamespace2", "mynamespace3"]
}
}
and provision multiple resources with for_each which would result in this:
resource "rancher2_project" "project1" {
provider = rancher2.admin
cluster_id = module.k8s_cluster.cluster_id
wait_for_cluster = true
}
resource "rancher2_project" "project2" {
provider = rancher2.admin
cluster_id = module.k8s_cluster.cluster_id
wait_for_cluster = true
}
resource "rancher2_namespace" "mynamespace1" {
provider = rancher2.admin
project_id = rancher2_project.project1.id
depends_on = [rancher2_project.project1]
}
resource "rancher2_namespace" "mynamespace2" {
provider = rancher2.admin
project_id = rancher2_project.project2.id
depends_on = [rancher2_project.project2]
}
resource "rancher2_namespace" "mynamespace3" {
provider = rancher2.admin
project_id = rancher2_project.project2.id
depends_on = [rancher2_project.project2]
}
namespaces are dependent on Projects and the generate id needs to be passed into namespace.
Is there any good way of doing this dynamically ? We might have a lot of Projects/namespaces.
Thanks for any help and advise.
The typical answer for systematically generating multiple instances of a resource based on a data structure is resource for_each. The main requirement for resource for_each is to have a map which contains one element per resource instance you want to create.
In your case it seems like you need one rancher2_project per project and then one rancher2_namespace for each pair of project and namespaces. Your current data structure is therefore already sufficient for the rancher2_project resource:
resource "rancher2_project" "example" {
for_each = var.projects
provider = rancher2.admin
cluster_id = module.k8s_cluster.cluster_id
wait_for_cluster = true
}
The above will declare two resource instances with the following addresses:
rancher2_project.example["project1"]
rancher2_project.example["project2"]
You don't currently have a map that has one element per namespace, so it will take some more work to derive a suitable value from your input data structure. A common pattern for this situation is flattening nested structures for for_each using the flatten function:
locals {
project_namespaces = flatten([
for pk, proj in var.projects : [
for nsk in proj.namespaces : {
project_key = pk
namespace_key = ns
project_id = rancher2_project.example[pk].id
}
]
])
}
resource "rancher2_namespace" "example" {
for_each = {
for obj in local.project_namespaces :
"${obj.project_key}.${obj.namespace_key}" => obj
}
provider = rancher2.admin
project_id = each.value.project_id
}
This produces a list of objects representing all of the project and namespace pairs, and then the for_each argument transforms it into a map using compound keys that include both the project and namespace keys to ensure that they will all be unique. The resulting instances will therefore have the following addresses:
rancher2_namespace.example["project1.mynamespace1"]
rancher2_namespace.example["project2.mynamespace2"]
rancher2_namespace.example["project2.mynamespace3"]
This seems to work too:
resource "rancher2_namespace" "example" {
count = length(local.project_namespaces)
provider = rancher2.admin
name = local.project_namespaces[count.index].namespace_name
project_id = local.project_namespaces[count.index].project_id
}

Get resources based on a value created using count

I am using Terraform v12.19 with the aws provider v2.34.0.
Imagine, I have a resource generated with a count value:
resource "aws_iam_role" "role" {
count = length(var.somevariable)
name = var.somevariable[count.index]
}
Later on, I want to reference one specific resource instance in that way, e. g.:
resource "aws_iam_role_policy_attachment" "polatt" {
role = aws_iam_role.role["TheRoleNameIWant"].id
policy_arn = "arn:aws:iam::aws:policy/..."
}
I don't know the index, I can just rely on the name, provided by the variable. Thats because the values of the variable are provided by an external source and the order could change...
Any ideas how to do this?
You should be able to accomplish this using the index terraform function.
Here's a minimal example using null_resources to test it out
locals {
role_names = [
"role-a",
"role-b",
"role-c",
"role-d",
]
target_role_name = "role-c"
}
resource "null_resource" "hi" {
count = length(local.role_names)
}
output "target_resource" {
value = null_resource.hi[index(local.role_names, local.target_role_name)].id
}
output "all_resources" {
value = [for r in null_resource.hi : r.id]
}
This outputs, for example
all_resources = [
"4350570701002192774",
"9173388682753384584",
"1634695740603384613",
"2098863759573339880",
]
target_resource = 1634695740603384613
So your example, I suppose, would look like
resource "aws_iam_role_policy_attachment" "polatt" {
role = aws_iam_role.role[index(var.somevariable, "TheRoleNameIWant")].id
policy_arn = "arn:aws:iam::aws:policy/..."
}
Update
Your comment below mentions that you actually have a more complicated data structure than just a list of names. I just wanted to mention that you can derive names from your JSON structure.
Assuming you have something like the following
variable "role_values" {
value = [
{
name = "foo",
other = "details",
fields = 3
},
{
name = "bar",
other = "yet more details",
fields = 3
}
]
}
you could derive just the names by using a local and the newer for loops TF 0.12 offers
locals {
role_names = [for role in var.role_values: role.name]
}
That way you don't have to store the names twice.

Resources