How can I output a data source that uses count? - terraform

I want to output each VM created and their UUID e.g
data "vsphere_virtual_machine" "vms" {
count = "${length(var.vm_names)}"
name = "${var.vm_names[count.index]}"
datacenter_id = "12345"
}
output "vm_to_uuid" {
# value = "${data.vsphere_virtual_machine.newvms[count.index].name}"
value = "${data.vsphere_virtual_machine.newvms[count.index].id}"
}
Example output I'm looking for:
"vm_to_uuids":[
{
"name":"node1",
"id":"123456",
},
{
"name":"node2",
"id":"987654",
}
]

Use the wildcard attribute in the expression given for the output value to get the list of ids for the created VMs. e.g.
output "vm_to_uuids" {
value = "${data.vsphere_virtual_machine.*.id}"
}
The required syntax provided in your question is one exemption where to prefer function over form.
Writing a terraform configuration that provides that isn't straightforward.
Perhaps, I suggest to adopt other simpler ways to output this same information.
Names mapped to ids can be output:
output "vm_to_uuids" {
value = "${zipmap(
data.vsphere_virtual_machine.*.name,
data.vsphere_virtual_machine.*.id)}"
}
A map of names and ids can be output in a columnar manner:
output "vm_to_uuids" {
value = "${map("name",
data.vsphere_virtual_machine.*.name,
"id",
data.vsphere_virtual_machine.*.id)}"
}
A list of names and ids can be output in a columnar manner:
output "vm_to_uuids" {
value = "${list(
data.vsphere_virtual_machine.*.name,
data.vsphere_virtual_machine.*.id)}"
}

One thing you could do (if you wanted exactly that output), is use formatlist(format, args, ...)
data "vsphere_virtual_machine" "vms" {
count = "${length(var.vm_names)}"
name = "${var.vm_names[count.index]}"
datacenter_id = "12345"
}
output "vm_to_uuid" {
value = "${join(",", formatlist("{\"name\": \"%s\", \"id\": \"%s\"}", data.vsphere_virtual_machine.newvms.*.name, data.vsphere_virtual_machine.newvms.*.id))}"
}
Haven't tested the code, but you get the idea. Especially the quote escape is just a guess, but that's easy to figure out from here.
What happens is you take two lists (names and ID's) and format dict strings from each entry, after which you join them together using comma separation.

Related

Terraform output from several resources generated with count

I have these outputs:
output "ec2_id" {
value = aws_instance.ec2instance[*].id
}
output "ec2_name" {
value = aws_instance.ec2instance[*].tags["Name"]
}
output "ec2_mgmt_eip" {
value = aws_eip.eip_mgmt_ec2instance[*].public_ip
}
I want to make an output like:
"<instanceName>: <instanceID> -> <publicIP>"
(all data in same line for same ec2 instance).
In any non-declarative language i can use something like for (var i=0; i<length(myarray);i++) and use "i" as index for each list, in every index concatenate in a new string, but in terraform I can't find how to do it.
Thanks!
Even though you got the answer in the comments, I will add an example. So, the thing you want does exist in terraform as it also has for loops [1]. A for loop along with the right syntax will give you a desired output, which is a map in terraform:
output "ec2_map" {
value = { for i in aws_instance.ec2instance: i.tags.Name => "${i.id}:${i.public_ip}" }
}
The output you said you want is quite similar to this. Also, there is no concept of "same line" in terraform. In this case, since this is a map, the keys will be instance names and value will be a combination of instance id and the public IP, but that will be a string.
[1] https://www.terraform.io/language/expressions/for

Retrieving elements from a list of objects based on a criteria

I have data source that returns a list of objects containing id, name, type.
data " data_source" "some_source" {
filter = ["env:a"]
...
}
I have a another resource that requires a set of ids
resource "another_rerouce" "bar" {
...
set_of_ids = [for i in data.data_source.some_source.objects : i.id]
...
}
Now what I require is only take the ids of the objects which has for e.g. type as live or pending etc. Is there a way i can incorporate this requirement inside [for i in data.data_source.some_source.objects : i.id]?
I am using Terraform v1.2.3.
I would say the answer would be different depending on the number of values you want to check for, i.e., if it is only live and pending, you could use the suggested solution from the comments (h/t: Matt Schuchard):
set_of_ids = [ for i in data.data_source.some_source.objects :
i.id if (i.type == "live" || i.type == "pending")
]
Alternatively, if there will be more than two values, you could create a local variable of type list(string) and assign all the values that are acceptable:
locals {
acceptable_types = ["live", "pending"]
}
Then, in the for loop, you could do the following:
set_of_ids = [ for i in data.data_source.some_source.objects :
i.id if contains(local.acceptable_types, i.type)
]
Here you would use the contains built-in function [1] to check if the type is inside of the acceptable/allowed types list.
[1] https://www.terraform.io/language/functions/contains

How to concatenate strings in Terraform output with for loop?

I have multiple aws_glue_catalog_table resources and I want to create a single output that loops over all resources to show the S3 bucket location of each one. The purpose of this is to test if I am using the correct location (because it is a concatenation of variables) for each resource in Terratest. I cannot use aws_glue_catalog_table.* or aws_glue_catalog_table.[] because Terraform does not allow to reference a resource without specifying its name.
So I created a variable "table_names" with r1, r2, rx. Then, I can loop over the names. I want to create the string aws_glue_catalog_table.r1.storage_descriptor[0].location dynamically, so I can check if the location is correct.
resource "aws_glue_catalog_table" "r1" {
name = "r1"
database_name = var.db_name
storage_descriptor {
location = "s3://${var.bucket_name}/${var.environment}-config/r1"
}
...
}
resource "aws_glue_catalog_table" "rX" {
name = "rX"
database_name = var.db_name
storage_descriptor {
location = "s3://${var.bucket_name}/${var.environment}-config/rX"
}
}
variable "table_names" {
description = "The list of Athena table names"
type = list(string)
default = ["r1", "r2", "r3", "rx"]
}
output "athena_tables" {
description = "Athena tables"
value = [for n in var.table_names : n]
}
First attempt: I tried to create an output "athena_tables_location" with the syntax aws_glue_catalog_table.${table} but does does.
output "athena_tables_location" {
// HOW DO I ITERATE OVER ALL TABLES?
value = [for t in var.table_names : aws_glue_catalog_table.${t}.storage_descriptor[0].location"]
}
Second attempt: I tried to create a variable "table_name_locations" but IntelliJ already shows an error ${t} in the for loop [for t in var.table_names : "aws_glue_catalog_table.${t}.storage_descriptor[0].location"].
variable "table_name_locations" {
description = "The list of Athena table locations"
type = list(string)
// THIS ALSO DOES NOT WORK
default = [for t in var.table_names : "aws_glue_catalog_table.${t}.storage_descriptor[0].location"]
}
How can I list all table locations in the output and then test it with Terratest?
Once I can iterate over the tables and collect the S3 location I can do the following test using Terratest:
athenaTablesLocation := terraform.Output(t, terraformOpts, "athena_tables_location")
assert.Contains(t, athenaTablesLocation, "s3://rX/test-config/rX",)
It seems like you have an unusual mix of static and dynamic here: you've statically defined a fixed number of aws_glue_catalog_table resources but you want to use them dynamically based on the value of an input variable.
Terraform doesn't allow dynamic references to resources because its execution model requires building a dependency graph between all of the objects, and so it needs to know which exact resources are involved in a particular expression. However, you can in principle build your own single value that includes all of these objects and then dynamically choose from it:
locals {
tables = {
r1 = aws_glue_catalog_table.r1
r2 = aws_glue_catalog_table.r2
r3 = aws_glue_catalog_table.r3
# etc
}
}
output "table_locations" {
value = {
for t in var.table_names : t => local.tables[t].storage_descriptor[0].location
}
}
With this structure Terraform can see that output "table_locations" depends on local.tables and local.tables depends on all of the relevant resources, and so the evaluation order will be correct.
However, it also seems like your table definitions are systematic based on var.table_names and so could potentially benefit from being dynamic themselves. You could achieve that using the resource for_each feature to declare multiple instances of a single resource:
variable "table_names" {
description = "Athena table names to create"
type = set(string)
default = ["r1", "r2", "r3", "rx"]
}
resource "aws_glue_catalog_table" "all" {
for_each = var.table_names
name = each.key
database_name = var.db_name
storage_descriptor {
location = "s3://${var.bucket_name}/${var.environment}-config/${each.key}"
}
...
}
output "table_locations" {
value = {
for k, t in aws_glue_catalog_table.all : k => t.storage_descriptor[0].location
}
}
In this case aws_glue_catalog_table.all represents all of the tables together as a single resource with multiple instances, each one identified by the table name. for_each resources appear in expressions as maps, so this will declare resource instances with addresses like this:
aws_glue_catalog_table.all["r1"]
aws_glue_catalog_table.all["r2"]
aws_glue_catalog_table.all["r3"]
...
Because this is already a map, this time we don't need the extra step of constructing the map in a local value, and can instead just access this map directly to build the output value, which will be a map from table name to storage location:
{
r1 = "s3://BUCKETNAME/ENVNAME-config/r1"
r2 = "s3://BUCKETNAME/ENVNAME-config/r2"
r3 = "s3://BUCKETNAME/ENVNAME-config/r3"
# ...
}
In this example I've assumed that all of the tables are identical aside from their names, which I expect isn't true in practice but I was going only by what you included in the question. If the tables do need to have different settings then you can change var.table_names to instead be a variable "tables" whose type is a map of object type where the values describe the differences between the tables, but that's a different topic kinda beyond the scope of this question, so I won't get into the details of that here.

Terraform dynamically generate strings using data sources output

Is there any way I can feed the Terraform data source output to another Terraform file as input
The scenario is, I have a terraform code to fetch the private IP addresses (here 3 IPs 10.1.1.1,10.1.1.2,10.1.1.3) for particular tags(here jenkins) using data source
data "aws_instances" "mytag" {
filter {
name = "tag:Application"
values = ["jenkins"]
}
}
output "output from aws" {
value = data.aws_instances.mytag_private_ips
}
Whenever, I do the terraform apply, the on the pattern section in the
below metric-filter code should be able to fetch the resultant IPs from the above code and make them available in the live state ( aws console )
resource "aws_cloudwatch_log_metric_filter" "test" {
name = "test-metric-filter"
pattern = "[w1,w2,w3,w4!=\"*<IP1>*\"&&w4!=\"*<IP2>*\"&&w4!=\"*<IP3>*\",w5=\"*admin*\"]"
log_group_name = var.test_log_group_name
metric_transformation {
name ="test-metric-filter"
namespace = "General"
}
}
So, the final result of metric pattern in the aws console should be like below
[w1,w2,w3,w4!="*10.1.1.1*"&&w4!="*10.1.1.2*"&&w4!="*10.1.1.3*",w5="*admin*"]
The end goal is whenever if the new IPs are generated, it will get populated to pattern (in aws-console) without changing the metric-filter code.
Any help is appreciated, as I could not find any precise document on terraform allows us to dynamically generate strings using data sources
Not sure why you need two files for something this simple...
Here is what I would do:
provider "aws" {
region = "us-east-1"
}
data "aws_instances" "test" {
filter {
name = "architecture"
values = ["x86_64"]
}
}
resource "aws_cloudwatch_log_metric_filter" "test" {
name = "test-metric-filter"
pattern = "[w1,w2,w3,w4!=\"*${data.aws_instances.test.private_ips[0]}*\",w5=\"*admin*\"]"
log_group_name = "test_log_group_name"
metric_transformation {
name = "test-metric-filter"
namespace = "General"
value = 1
}
}
And a terraform plan will show
Terraform will perform the following actions:
# aws_cloudwatch_log_metric_filter.test will be created
+ resource "aws_cloudwatch_log_metric_filter" "test" {
+ id = (known after apply)
+ log_group_name = "test_log_group_name"
+ name = "test-metric-filter"
+ pattern = "[w1,w2,w3,w4!=\"*172.31.70.170*\",w5=\"*admin*\"]"
+ metric_transformation {
+ name = "test-metric-filter"
+ namespace = "General"
+ unit = "None"
+ value = "1"
}
}
Plan: 1 to add, 0 to change, 0 to destroy.
Concatenating strings is easy: "foo ${var.bar} 123"
and on this case our private_ips is an array so we need the [x]
For more complex concatenations look into the format function:
https://www.terraform.io/docs/language/functions/format.html
I did changed the filter to be able to test on my environment and also used a shorter pattern than yours, but that is the basis for what you need, just add more of make changes to suit your needs.
What you are looking for is string interpolation in Terraform.
I believe you would want to do the following:
pattern = "[w1,w2,w3,w4!=\"*${data.aws_instances.mytag.private_ips[0]}*\"&&w4!=\"*${data.aws_instances.mytag.private_ips[1]}*\"&&w4!=\"*${data.aws_instances.mytag.private_ips[2]}*\",w5=\"*admin*\"]"
I suggest being careful with this statement, because it will fail if you don't have at least 3 instances. You would want to have something dynamic instead.

Any way to conditionalize variable in jsonencoded data?

Say I have the simplified following snippet to create a task definition as json.
...
task_container_definitions = jsonencode([{
name : var.name,
image : "${var.image}:${var.tag}",
cpu : var.cpu,
memory : var.memory,
}])
...
Say I want to add a variable to optionally create an additional definition so it looks something like this:
variable "another_definition" {
type = any
default = {}
}
...
task_container_definitions = jsonencode([{
name : var.name,
image : "${var.image}:${var.tag}",
cpu : var.cpu,
memory : var.memory,
},
var.another_definition
])
And define it as follows.
another_definition = {
name = "another_container"
image = "another_container"
cpu = 10
memory = 512
essential = true
}
I am able to get this to to output as expected as long as the variable is defined.
...
+ {
+ cpu = 10
+ essential = true
+ image = "another_container"
+ memory = 512
+ name = "another_container"
},
But if the variable is not defined, I see empty {} added to the output when I do a terraform plan, which is not what I expect. I have tried using null as well as the default but get an error.
...
+ {},
Is there a way to toggle this variable off so that if it is not defined then it doesn't show up in the outputted json definition? Is there a better approach than what I am attempting?
I was a little confused at first as to what you were asking, thinking that you were asking for the functionality of the merge function, and I mention that only in case I was right the first time, but I think I now understand your problem as that you want this task_container_definitions to have either one or two elements, depending on whether var.another_definition is set.
There's no single function for that particular situation, but I think we can combine some language features together to get that result.
First, let's decide that the variable being set means that it has a non-null value, and thus its default value should be null to represent the "unset" case:
variable "another_definition" {
type = any
default = null
validation {
# The time constraint above is looser than we really
# want, so this validation rule also enforces that
# the caller can't set this to something inappropriate,
# like a single string or a list.
condition = (
var.another_definition != null ?
can(keys(var.another_definition)) :
true
)
error_message = "Additional task container definition must be an object."
}
}
In Terraform it's a pretty common situation to need to convert between a value that might be null and a list that might have zero or one elements, or vice-versa, and so Terraform has some language features to help with that. In this case we can use a splat expression to concisely represent that. Let's see how that looks in terraform console first just to give a sense of what we're achieving with this:
$ terraform console
> null[*]
[]
> "hello"[*]
[
"hello",
]
> { object = "example" }[*]
[
{
"object" = "example"
},
]
Notice that when I applied the [*] operator to null it returned an empty list, but when I applied it to these other values it converted them to a single-element list. This is how the [*] operator behaves when you apply it to something that isn't a list; see the splat operator docs if you want to learn about the different behavior for lists, which isn't really relevant here because of the validation rule I added above which prevents the var.another_definition value from being a list.
Another tool we have in our Terraform toolbox here is the concat function, which takes one or more lists and returns a single list with the input elements all concatenated together in the given order. We can use this to combine your predefined list that's populated from var.name, var.cpu, etc with the zero-or-one element list created by [*], in order to create a list with their one or two elements:
locals {
task_container_definitions = concat(
[
name = var.name
image = "${var.image}:${var.tag}"
cpu = var.cpu
memory = var.memory
],
var.another_definition[*],
)
task_container_definitions_json = jsonencode(local.task_container_definitions)
}
If any of the arguments to concat are empty lists then they are effectively ignored altogether, because they contribute no elements to the result, and so this achieves (what I hope is) the desired result, by making the "other definition" appear in the result only when it's set to something other than null.

Resources