I have a list variable "test" in variables.tf. I am trying to use this list variable inside my zone.tf .
I do not want to use list indexes here infact I want to run a loop to get all the values of the list from list variable dynamically. How can I accomplish this ? Any help is much appreciated.
I have tried to use count in test.tf inside resource resource "aws_route53_record" but it creates multiple record sets which I do not want as I just need a single record set with multiple records
resource "aws_route53_record" "test" {
zone_id = "${data.aws_route53_zone.dns.zone_id}"
name = "${lower(var.environment)}xyz"
type = "CAA"
ttl = 300
count = "${length(var.test)}"
records = [
"0 issue \"${element(var.test, count.index)}\"",
]
}
variables.tf :-
variable "test" {
type = "list"
default = ["godaddy.com", "Namecheap.org"]
}
zone.tf :-
resource "aws_route53_record" "test" {
zone_id = "${data.aws_route53_zone.dns.zone_id}"
name = "${lower(var.environment)}xyz"
type = "CAA"
ttl = 300
records = [
"0 issue \"${var.test[0]}\"",
"0 issue \"${var.test[1]}\"",
]
}
Expecting to get the one record set with two records.
Actual :- getting Two record sets with two records.
so if I am understanding correctly you want to associate two records with your zone, but right now when using count you are getting two zones with one record.
This is because by specifying county terraform will create the resource that has the count attribute on it equal to the number of count.
The issue is fundamentally that right now you have a list variable and are trying to pass it to where a list is expected by extracting each individual element of the list to put back element by element into the list attribute.
Rather than go through that additional work an easier solution would be to just add the additional parts of the string, the "0 issue" part to the definition of the variable and then just pass the whole list object in as shown below
variable "test" {
type = "list"
default = ["0 issue godaddy.com", "0 issue Namecheap.org"]
}
zone.tf :-
resource "aws_route53_record" "test" {
zone_id = "${data.aws_route53_zone.dns.zone_id}"
name = "${lower(var.environment)}xyz"
type = "CAA"
ttl = 300
records = ["${var.test}"]
}
This will then pass the list in for that attribute and terraform will take care of the marshaling and unmarshaling and handling of the list. I hope this answers your question.
Related
The problem I am trying to solve is I need to identify one of the Azure subnets in a virtual network by part of it's name. This is so I can then later retrieve it's CIDR. I only know part beforehand such as "mgmt-1" or "egress-1". The actual name of the subnet is much longer but will end in something like that. This was my process:
I have the vnet name so I pull all subnets:
data "azurerm_virtual_network" "this" {
name = local.vnet
resource_group_name = "myrg"
}
Now what I wish I could do is this:
locals {
mgmt_index = index(data.azurerm_virtual_network.this.subnets, "*mgmt-1")
mgmt_subnet = data.azurerm_virtual_network.this.subnets[local.mgmt_index]
}
However index wants an exact match, not a regex. Is this possible to do? Perhaps a better way?
Thank you,
It is not possible to directly look up a list item using a regex match, but you can use for expressions to apply arbitrary filters to a collection when constructing a new collection:
locals {
mgmt_subnets = toset([
for s in data.azurerm_virtual_network.this.subnets : s
if can(regex(".*?mgmt-1", s.name))
])
}
In principle an expression like the above could match more than one object, so I wrote this to produce a set of objects that match.
If you expect that there will never be more than one object whose name matches the pattern then you can use Terraform's one function to assert that and then Terraform will check to confirm that there's no more than one element (returning an error if not) and then return that one value.
locals {
mgmt_subnet = one([
for s in data.azurerm_virtual_network.this.subnets : s
if can(regex(".*?mgmt-1", s.name))
])
}
If the condition doesn't match any of the subnet objects then in the first case you'll have an empty set and in the second case you'll have the value null.
I have multiple aws_glue_catalog_table resources and I want to create a single output that loops over all resources to show the S3 bucket location of each one. The purpose of this is to test if I am using the correct location (because it is a concatenation of variables) for each resource in Terratest. I cannot use aws_glue_catalog_table.* or aws_glue_catalog_table.[] because Terraform does not allow to reference a resource without specifying its name.
So I created a variable "table_names" with r1, r2, rx. Then, I can loop over the names. I want to create the string aws_glue_catalog_table.r1.storage_descriptor[0].location dynamically, so I can check if the location is correct.
resource "aws_glue_catalog_table" "r1" {
name = "r1"
database_name = var.db_name
storage_descriptor {
location = "s3://${var.bucket_name}/${var.environment}-config/r1"
}
...
}
resource "aws_glue_catalog_table" "rX" {
name = "rX"
database_name = var.db_name
storage_descriptor {
location = "s3://${var.bucket_name}/${var.environment}-config/rX"
}
}
variable "table_names" {
description = "The list of Athena table names"
type = list(string)
default = ["r1", "r2", "r3", "rx"]
}
output "athena_tables" {
description = "Athena tables"
value = [for n in var.table_names : n]
}
First attempt: I tried to create an output "athena_tables_location" with the syntax aws_glue_catalog_table.${table} but does does.
output "athena_tables_location" {
// HOW DO I ITERATE OVER ALL TABLES?
value = [for t in var.table_names : aws_glue_catalog_table.${t}.storage_descriptor[0].location"]
}
Second attempt: I tried to create a variable "table_name_locations" but IntelliJ already shows an error ${t} in the for loop [for t in var.table_names : "aws_glue_catalog_table.${t}.storage_descriptor[0].location"].
variable "table_name_locations" {
description = "The list of Athena table locations"
type = list(string)
// THIS ALSO DOES NOT WORK
default = [for t in var.table_names : "aws_glue_catalog_table.${t}.storage_descriptor[0].location"]
}
How can I list all table locations in the output and then test it with Terratest?
Once I can iterate over the tables and collect the S3 location I can do the following test using Terratest:
athenaTablesLocation := terraform.Output(t, terraformOpts, "athena_tables_location")
assert.Contains(t, athenaTablesLocation, "s3://rX/test-config/rX",)
It seems like you have an unusual mix of static and dynamic here: you've statically defined a fixed number of aws_glue_catalog_table resources but you want to use them dynamically based on the value of an input variable.
Terraform doesn't allow dynamic references to resources because its execution model requires building a dependency graph between all of the objects, and so it needs to know which exact resources are involved in a particular expression. However, you can in principle build your own single value that includes all of these objects and then dynamically choose from it:
locals {
tables = {
r1 = aws_glue_catalog_table.r1
r2 = aws_glue_catalog_table.r2
r3 = aws_glue_catalog_table.r3
# etc
}
}
output "table_locations" {
value = {
for t in var.table_names : t => local.tables[t].storage_descriptor[0].location
}
}
With this structure Terraform can see that output "table_locations" depends on local.tables and local.tables depends on all of the relevant resources, and so the evaluation order will be correct.
However, it also seems like your table definitions are systematic based on var.table_names and so could potentially benefit from being dynamic themselves. You could achieve that using the resource for_each feature to declare multiple instances of a single resource:
variable "table_names" {
description = "Athena table names to create"
type = set(string)
default = ["r1", "r2", "r3", "rx"]
}
resource "aws_glue_catalog_table" "all" {
for_each = var.table_names
name = each.key
database_name = var.db_name
storage_descriptor {
location = "s3://${var.bucket_name}/${var.environment}-config/${each.key}"
}
...
}
output "table_locations" {
value = {
for k, t in aws_glue_catalog_table.all : k => t.storage_descriptor[0].location
}
}
In this case aws_glue_catalog_table.all represents all of the tables together as a single resource with multiple instances, each one identified by the table name. for_each resources appear in expressions as maps, so this will declare resource instances with addresses like this:
aws_glue_catalog_table.all["r1"]
aws_glue_catalog_table.all["r2"]
aws_glue_catalog_table.all["r3"]
...
Because this is already a map, this time we don't need the extra step of constructing the map in a local value, and can instead just access this map directly to build the output value, which will be a map from table name to storage location:
{
r1 = "s3://BUCKETNAME/ENVNAME-config/r1"
r2 = "s3://BUCKETNAME/ENVNAME-config/r2"
r3 = "s3://BUCKETNAME/ENVNAME-config/r3"
# ...
}
In this example I've assumed that all of the tables are identical aside from their names, which I expect isn't true in practice but I was going only by what you included in the question. If the tables do need to have different settings then you can change var.table_names to instead be a variable "tables" whose type is a map of object type where the values describe the differences between the tables, but that's a different topic kinda beyond the scope of this question, so I won't get into the details of that here.
I have resources defined in .tf files that are generic to several applications. I populate many of the fields via a .tfvars file. I need to omit some of the resources entirely based on variables in the .tfvars.
For example if I have a resource like:
resource "cloudflare_record" "record" {
zone_id = "${data.cloudflare_zones.domain.zones[0].id}"
name = "${var.subdomain}"
value = "${var.origin_server}"
type = "CNAME"
ttl = 1
proxied = true
}
But then I declare something like cloudflare = false in my .tfvars file I'd like to be able to do something like this:
if var.cloudflare {
resource "cloudflare_record" "record" {
zone_id = "${data.cloudflare_zones.domain.zones[0].id}"
name = "${var.subdomain}"
value = "${var.origin_server}"
type = "CNAME"
ttl = 1
proxied = true
}
}
I've looked at dynamic blocks but that looks like you can only use those to edit fields and blocks within a resource. I need to be able to ignore an entire resource.
Add a count parameter with a ternary conditional using the variable declared in .tfvars like this:
resource "cloudflare_record" "record" {
count = var.cloudflare ? 1 : 0
zone_id = "${data.cloudflare_zones.domain.zones[0].id}"
name = "${var.subdomain}"
value = "${var.origin_server}"
type = "CNAME"
ttl = 1
proxied = true
}
In this example var.cloudflare is a boolean declared in the .tfvars file. If it is true a count of 1 record will be created. If it is false a count of 0 record will be created.
After the count apply the resource becomes a group, so later in the reference use 0-index of the group:
cloudflare_record.record[0].some_field
Expanding on #Joel Guerra's answer, after you use count to determine whether to deploy the resource or not, you can use the one() function to refer to the resource without an index (i.e. without having to use [0]).
For example, after defining the resource like below
resource "cloudflare_record" "record" {
count = var.cloudflare ? 1 : 0
}
Define a local variable like below
locals {
cloudflare_record_somefield = one(cloudflare_record.record[*].some_field)
}
Now instead of cloudflare_record.record[0].some_field, you can use
local.cloudflare_record_somefield
If the count is 0 (e.g. var.cloudflare is false and the resource wasn't created) then local.cloudflare_record_somefield would return null (instead of returning an error when indexing using [0]).
Reference: https://developer.hashicorp.com/terraform/language/functions/one
An issue i'm seeing this with is if the resource your trying to create is already using a for_each then you can't use both count and for_each in the resource. I'm still trying to find an answer on this will update if I find something better.
I have a file users.tf file that creates the admin users for aws. It does so by defining a list (e.g. users = ["bob", "john", "tom"])
and then iterating over them with the aws_iam_user resource using the count functionality in terraform, like so:
resource "aws_iam_user" "user" {
count = length(local.users)
name = local.users[count.index]
}
the issue here, is that if I remove the first element of the array ("bob" from the above example), what terraform will suggest to do after issuing terraform plan, instead of deleting bob, is to change bob to john, change john to tom, and delete tom.
like so:
# aws_iam_user.user[0] will be updated in-place
~ resource "aws_iam_user" "user" {
arn = "arn:aws:iam::5555555:user/bob"
force_destroy = false
id = "bob"
~ name = "bob" -> "john"
path = "/"
tags = {}
unique_id = "BLABLABLA11111"
}
# aws_iam_user.user[1] will be updated in-place
~ resource "aws_iam_user" "user" {
arn = "arn:aws:iam::5555555:user/john"
force_destroy = false
id = "john"
~ name = "john" -> "tom"
path = "/"
tags = {}
unique_id = "BLABLABLA22222"
}
# aws_iam_user.user[2] will be destroyed
- resource "aws_iam_user" "user" {
- arn = "arn:aws:iam::5555555:user/tom" -> null
- force_destroy = false -> null
- id = "tom" -> null
- name = "tom" -> null
- path = "/" -> null
- tags = {} -> null
- unique_id = "BLABLABLA3333" -> null
this will result in john getting the arn of bob, and tom getting the arn of john. which is undesirable.
I tried using the very new feature (released 19 hours prior to the writing of this question) of for_each loop instead of count, and defining the keys as the original index numbers, hoping that terraform will consider them as the same resource.
yeah well, no such luck:
...
# aws_iam_user.user[1] will be destroyed
...
# aws_iam_user.user["1"] will be created
...
I will summarize my question:
Is there any way to delete a resource (specifically aws_iam_user), when that resource was created by iterating over a list, such that all the remaining resources stay the way they were?
What you have seen here is the situation that the count documentation warns about in its final paragraph:
Note that the separate resource instances created by count are still identified by their index, and not by the string values in the given list. This means that if an element is removed from the middle of the list, all of the indexed instances after it will see their subnet_id values change, which will cause more remote object changes than were probably intended. The practice of generating multiple instances from lists should be used sparingly, and with due care given to what will happen if the list is changed later.
Fortunately, this is the very problem the for_each feature is intended to solve. Though, in order to use it effectively it's important to choose meaningful unique keys in the map you pass to for_each:
resource "aws_iam_user" "user" {
for_each = { for name in local.users : name => name }
name = each.value
}
This will cause Terraform to track instance identifiers like aws_iam_user.user["john"] rather than aws_iam_user.user[1].
You have existing count-based instances in your state though, so it'll take some migration steps to get there. Unfortunately Terraform doesn't have enough information to automatically correlate your existing index-based addresses with the new name-based ones, but by using your existing list with a separate one-off script you can tell Terraform how to translate these by running a command like this for each entry in the list:
terraform state mv 'aws_iam_user.user[1]' 'aws_iam_user.user["john"]'
After that, Terraform will track these objects by name and thus adding and removing names will affect only the objects relating to the names you changed.
If you aren't ready to do a full switch to for_each right now, you can use a similar strategy with a one-off script to "heal" the hole created by removing an item from your list:
# First, Terraform must "forget" the user that you removed
terraform state rm 'aws_iam_user.user[0]'
# Then renumber the subsequent items to correlate with their new
# positions in the list.
terraform state mv 'aws_iam_user.user[1]' 'aws_iam_user.user[0]'
terraform state mv 'aws_iam_user.user[2]' 'aws_iam_user.user[1]'
# etc, etc
This will of course be a rather tedious, error-prone process to do manually if you have more than a small number of users, so better to write a small program to generate the script.
I need to create 6 subnets with below cidr value but it's order has been changed while creating it with terraform.
private_subnets = {
"10.1.80.0/27" = "x"
"10.1.80.32/27" = "x"
"10.1.80.64/28" = "y"
"10.1.80.80/28" = "y"
"10.1.80.96/27" = "z"
"10.1.80.128/27" = "z"
}
Terraform is creating with 10.1.80.0/27 , 10.1.80.128/27,10.1.80.32/27,10.1.80.64/28,10.1.80.80/28,10.1.80.96/27 order
Module of terraform:
resource "aws_subnet" "private" {
vpc_id = "${var.vpc_id}"
cidr_block = "${element(keys(var.private_subnets), count.index)}"
availability_zone = "${element(var.availability_zones, count.index)}"
count = "${length(var.private_subnets)}"
tags {
Name = "${lookup(var.private_subnets, element(keys(var.private_subnets), count.index))}
}
}
Updated Answer:
Thanks to the discussion in the comments, I revise my answer:
You are assuming an order within a dictionary. This is not intended behaviour. As from your example, one can see that terraform orders the keys alphabetically internally, i.e., you can "think" of your variable as
private_subnets = {
"10.1.80.0/27" = "x"
"10.1.80.128/27" = "z"
"10.1.80.32/27" = "x"
"10.1.80.64/28" = "y"
"10.1.80.80/28" = "y"
"10.1.80.96/27" = "z"
}
You are running into problems, because you are having mismatches with your other variable var.availability_zones where you assume the index to be sorted the same as for var.private_subnets.
Relying on the above ordering (alphabetically), is not a good solution, since it may change with any version of terraform (order of keys is not guaranteed).
Hence, I propose to use a list of maps:
private_subnets = [
{
"cidr" = "10.1.80.0/27"
"name" = "x"
"availability_zone" = 1
},
{
"cidr" = "10.1.80.32/27"
"name" = "x"
"availability_zone" = 2
},
…
]
I encoded the availability zone as index of your var.availability_zones list. However, you could also consider using the availability zone directly.
The adaption of your code is straightforward: Get (element(…)) the list element to get the map and then lookup(…) the desired key.
Old Answer (not applicable here):
Before Terraform creates any resources, it creates a graphstructure to represent all the objects it wants to track (create, update, delete) and the dependencies upon one another.
In your example, 6 different aws_subnet objects are created in the graph which do not depend on each other (there is no variable in one subnet dependent on another subnet).
When Terraform now tries to create the attributes, it does so concurrently in (potentially) multiple threads and creates resources potentially simultaniously, if they do not depend on each other.
This is why you might see very different orders of execution within multiple runs of terraform.
Note that this is a feature, since if you have many resources to be created that have no dependency on each other, they all are created simultaneously saving a lot of time with long-running creation operations.
A solution to your problem is to explicitly model the dependencies you are thinking of. Why should one subnet be created before the other? And if so, how can you make them dependent (e.g. via depends_on parameter)?
Answering this questions should bring you into the right direction to model your code according to your required layout.