Using terraform I wish to refer to the content of a list of files (ultimately I want to zip them up using the archive_file provider, but in the context of this post that isn't important). These files all live within the same directory so I have two variables:
variable "source_root_dir" {
type = "string"
description = "Directory containing all the files"
}
variable "source_files" {
type = "list"
description = "List of files to be added to the cloud function. Locations are relative to source_root_dir"
}
I want to use the template data provider to refer to the content of the files. Given the number of files in source_files can vary I need to use a count to carry out the same operation on all of them.
Thanks to the information provided at https://stackoverflow.com/a/43195932/201657 I know that I can refer to the content of a single file like so:
provider "template" {
version = "1.0.0"
}
variable "source_root_dir" {
type = "string"
}
variable "source_file" {
type = "string"
}
data "template_file" "t_file" {
template = "${file("${var.source_root_dir}/${var.source_file}")}"
}
output "myoutput" {
value = "${data.template_file.t_file.rendered}"
}
Notice that that contains nested string interpolations. If I run:
terraform init && terraform apply -var source_file="foo" -var source_root_dir="./mydir"
after creating file mydir/foo of course then this is the output:
Success!
Now I want to combine that nested string interpolation syntax with my count. Hence my terraform project now looks like this:
provider "template" {
version = "1.0.0"
}
variable "source_root_dir" {
type = "string"
description = "Directory containing all the files"
}
variable "source_files" {
type = "list"
description = "List of files to be added to the cloud function. Locations are relative to source_root_dir"
}
data "template_file" "t_file" {
count = "${length(var.source_files)}"
template = "${file("${"${var.source_root_dir}"/"${element("${var.source_files}", count.index)}"}")}"
}
output "myoutput" {
value = "${data.template_file.t_file.*.rendered}"
}
yes it looks complicated but syntactically, its correct (at least I think it is). However, if I run init and apply:
terraform init && terraform apply -var source_files='["foo", "bar"]' -var source_root_dir='mydir'
I get errors:
Error: data.template_file.t_file: 2 error(s) occurred:
* data.template_file.t_file[0]: __builtin_StringToInt: strconv.ParseInt: parsing "mydir": invalid syntax in:
${file("${"${var.source_root_dir}"/"${element("${var.source_files}", count.index)}"}")}
* data.template_file.t_file1: __builtin_StringToInt: strconv.ParseInt: parsing "mydir": invalid syntax in:
${file("${"${var.source_root_dir}"/"${element("${var.source_files}", count.index)}"}")}
My best guess is that its interpreting the / as a division operation hence its attempting to parse the value mydir in source_root_dir as an int.
I've played around with this for ages now and can't figure it out. Can someone figure out how to use nested string interpolations together with a count in order to refer to the content of multiple files using the template provider?
OK, I think I figured it out. formatlist to the rescue
provider "template" {
version = "1.0.0"
}
variable "source_root_dir" {
type = "string"
description = "Directory containing all the files"
}
variable "source_files" {
type = "list"
description = "List of files to be added to the cloud function. Locations are relative to source_root_dir"
}
locals {
fully_qualified_source_files = "${formatlist("%s/%s", var.source_root_dir, var.source_files)}"
}
data "template_file" "t_file" {
count = "${length(var.source_files)}"
template = "${file(element("${local.fully_qualified_source_files}", count.index))}"
}
output "myoutput" {
value = "${data.template_file.t_file.*.rendered}"
}
when applied:
terraform init && terraform apply -var source_files='["foo", "bar"]' -var source_root_dir='mydir'
outputs:
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
myoutput = [
This is the content of foo
,
This is the content of bar
]
Related
We want to load a file from a git repo and place it into a parameter store. The file contains configuration data that is custom to each of several organizational-accounts, which are being constructed with Terraform and are otherwise identical. The data will be stored in AWS SM Parameter Store.
For example the Terraform code to store a string as a parameter is:
resource "aws_ssm_parameter" "parameter_config" {
name = "config_object"
type = "String"
value = "long json string with configuration data"
}
I know there is a file() operator (reference) from Terraform and I know that TF can load files from remote git repos, but I'm not sure if I can bring all this together.
There are a few ways that you can do this.
The first would be to use the github provider with the github_repository_file data source:
terraform {
required_providers {
github = {
source = "integrations/github"
version = "5.12.0"
}
}
}
provider "github" {
token = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
owner = "org-name"
}
data "github_repository_file" "config" {
repository = "my-repo"
branch = "master"
file = "config-file"
}
resource "aws_ssm_parameter" "parameter_config" {
name = "config_object"
type = "String"
value = data.github_repository_file.config.content
}
You could also do this with the http provider:
data "http" "config" {
url = "https://raw.githubusercontent.com/my-org/my-repo/master/config-file"
}
resource "aws_ssm_parameter" "parameter_config" {
name = "config_object"
type = "String"
value = data.http.config.response_body
}
Keep in mind that you may get multiline string delimiters when using the http data source method. (e.g.: <<EOT ... EOT)
variable "static_policies" {
type = list(tuple([string, string]))
description = "Map, where key is the auth method type and the value is the path to static policy file"
default = [
["app", "ROOT_PATH_STATIC/global/demo.hcl"]
]
}
locals {
root_path_static = "${path.root}/${var.static_policy_folder}/"
root_path_template = "${path.root}/${var.templated_policies_folder}/"
static_policies_replaced = [for policy in var.static_policies :
[
element(policy, 0), replace(element(policy, 1), "ROOT_PATH_STATIC", "${local.root_path_static}")
[
]
}
My Terraform module has a list of tuples with some strings. I am trying to iterate through each of them and do a find and replace of a substring, with a path based on where the module is executed. This is because Terraform can’t use interpolation in a variable. I want to produce the same list but with a substring replaced with a path. Any help appreciated
I need to set additional variables in my value.yaml (link to jaeger https://github.com/jaegertracing/helm-charts/blob/main/charts/jaeger/values.yaml#L495) helm chart via terraform + terragrunt. In values.yaml, the code looks like this:
spark:
extraEnv: []
It is necessary that it be like this:
spark:
extraEnv:
- name: JAVA_OPTS
value: "-Xms4g -Xmx4g"
Terraform uses this dynamic block:
dynamic "set" {
for_each = var.extraEnv
content {
name = "spark.extraEnv [${set.key}]"
value = set.value
}
}
The variable is defined like this:
variable "extraEnv" {
type = map
}
From terragrunt I pass the value of the variable:
extraEnv = {
"JAVA_OPTS" = "-Xms4g -Xmx4g"
}
And I get this error:
Error: failed parsing key "spark.extraEnv [JAVA_OPTS]" with value -Xms4g -Xmx4g, error parsing index: strconv.Atoi: parsing "JAVA_OPTS": invalid syntax
on main.tf line 16, in resource "helm_release" "jaeger":
16: resource "helm_release" "jaeger" {
Tell me how to use the dynamic block correctly in this case. I suppose that in this case you need to use a list of maps, but I do not understand how to use this in a dynamic block.
UPD:
I solved my problem in a different way.
In values, defined the list 'spark.extraEnv' using yamlencode.
values = [
"${file("${path.module}/values.yaml")}",
yamlencode({
spark = {
extraEnv = var.spark_extraEnv
}
})
]
in variables.tf
variable "spark_extraEnv" {
type = list(object({
name = string
value = string
}))
}
And in terragrunt passed the following variable value:
spark_extraEnv = [
{
name = "JAVA_OPTS"
value = "-Xms4g -Xmx4g"
}
]
I landed here while I was looking for setting extraEnv for a different chart. Finally figured answer for the above question as well:
set {
name = "extraEnv[0].name"
value = "JAVA_OPTS"
}
set {
name = "extraEnv[0].value"
value = "-Xms4g -Xmx4g"
}
Terraform Version
Terraform v0.12.1
Terraform Configuration Files
main.tf in my root provider:
provider "google" {}
module "organisation_info" {
source = "../../modules/organisation-info"
top_level_domain = "smoothteam.fi"
region = "us-central1"
}
module "stack_info" {
source = "../../modules/stack-info"
organisation_info = "${module.organisation_info}"
}
Here's module 'organisation-info':
variable "top_level_domain" {}
variable "region" {}
data "google_organization" "organization" {
domain = "${var.top_level_domain}"
}
locals {
organization_id = "${data.google_organization.organization.id}"
ns = "${replace("${var.top_level_domain}", ".", "-")}-"
}
output "organization_id" {
value = "${local.organization_id}"
}
output "ns" {
value = "${local.ns}"
}
Then module 'stack-info':
variable "organisation_info" {
type = any
description = "The organisation-scope this environment exists in."
}
module "project_info" {
source = "../project-info"
organisation_info = "${var.organisation_info}"
name = "${local.project}"
}
locals {
# Use the 'default' workspace for the 'staging' stack.
name = "${terraform.workspace == "default" ? "staging" : terraform.workspace}"
# In the 'production' stack, target the production project. Otherwise, target the staging project.
project = "${local.name == "production" ? "production" : "staging"}"
}
output "project" {
value = "${module.project_info}" # COMMENTING THIS OUTPUT REMOVES THE CYCLE.
}
And finally, the 'project-info' module:
variable "organisation_info" {
type = any
}
variable "name" {}
data "google_project" "project" {
project_id = "${local.project_id}"
}
locals {
project_id = "${var.organisation_info.ns}${var.name}"
}
output "org" {
value = "${var.organisation_info}"
}
Debug Output
After doing terraform destroy -auto-approve, I get:
Error: Cycle: module.stack_info.module.project_info.local.project_id, module.stack_info.output.project, module.stack_info.module.project_info.data.google_project.project (destroy), module.organisation_info.data.google_organization.organization (destroy), module.stack_info.var.organisation_info, module.stack_info.module.project_info.var.organisation_info, module.stack_info.module.project_info.output.org
And terraform graph -verbose -draw-cycles -type=plan-destroy gives me this graph:
Source:
digraph {
compound = "true"
newrank = "true"
subgraph "root" {
"[root] module.organisation_info.data.google_organization.organization" [label = "module.organisation_info.data.google_organization.organization", shape = "box"]
"[root] module.stack_info.module.project_info.data.google_project.project" [label = "module.stack_info.module.project_info.data.google_project.project", shape = "box"]
"[root] provider.google" [label = "provider.google", shape = "diamond"]
"[root] module.organisation_info.data.google_organization.organization" -> "[root] module.stack_info.module.project_info.data.google_project.project"
"[root] module.organisation_info.data.google_organization.organization" -> "[root] provider.google"
"[root] module.stack_info.module.project_info.data.google_project.project" -> "[root] provider.google"
}
}
Expected Behavior
The idea is to use modules at the org, project and stack levels to set up naming conventions that can be re-used across all resources. Organisation-info loads organisation info, project-info about projects, and stack-info determines which project to target based on current workspace.
I have omitted a bunch of other logic in the modules in order to keep them clean for this issue.
According to terraform there are no cycles, and destroy should work fine.
Actual Behavior
We get the cycle I posted above, even though terraform shows no cycles.
Steps to Reproduce
Set up the three modules, organisation-info, project-info, and stack-info as shown above.
Set up a root provider as shown above.
terraform init
terraform destroy (it doesn't seem to matter if you've applied first)
Additional Context
The weird thing is that if I comment out this output in stack-info, the cycle stops:
output "project" {
value = "${module.project_info}" # IT'S THIS... COMMENTING THIS OUT REMOVES THE CYCLE.
}
This seems really weird... I neither understand why outputting a variable should make a difference, nor why I'm getting a cycle error when there's no cycle.
Oddly, terraform plan -destroy does not reveal the cycle, only terraform destroy.
My spidey sense tells me evil is afoot.
Appreciate anyone who can tell me what's going on, whether this is a bug, and perhaps how to work around.
In my case the same cycle error was because the out of 3 key:pair one of the key:pair expected by map type variable inside variables.tf of a module not declared in main.tf nor in anywhere.
Am trying to create cross account role using terraform while giveing the policy name as input getting error Error: Error asking for user input: Cannot parse value for variable policy_arns
data "aws_iam_policy_document" "cross_account" {
statement {
effect = "Allow"
principals {
type = "AWS"
identifiers = ["${var.principal_arns}"]
}
actions = ["sts:AssumeRole"]
}
}
resource "aws_iam_role" "cross_account_role" {
name = "${var.name}"
assume_role_policy = "${data.aws_iam_policy_document.cross_account_assume_role_policy.json}"
}
resource "aws_iam_role_policy_attachment" "cross_account_role" {
count = "${length(var.policy_arns)}"
role = "${aws_iam_role.cross_account_role.name}"
policy_arn = "${element(var.policy_arns, count.index)}"
}
variable "name" {
type = "string"
description = "Name of the role being created."
}
variable "principal_arns" {
type = "list"
description = "ARNs of accounts, groups, or users with the ability to assume this role."
}
variable "policy_arns" {
type = "list"
description = "List of ARNs of policies to be associated with the created IAM role"
}
Variables passed with either -var foo=bar command line flag or as the environment variable TF_VAR_foo=bar can only be literal strings:
Variables specified via the -var command line flag will be literal strings "true" and "false", so care should be taken to explicitly use "0" or "1".
Variables specified with the TF_VAR_ environment variables will be literal string values, just like -var.
If you want to be able to use list variables then you will either need to define these ahead of time in a terraform.tfvars file or other vars file or you could use the split() function to take a separated string and turn it into a list:
variable "string_list" {
type = "string"
}
locals {
list_list = "${split(",", var.string_list)}"
}
output "list_list" {
value = ["${local.list_list}"]
}