I have a module a in terraform which creates a text file , i need to use that text file in another module b, i am using locals to pull the content of that text file like below in module b
locals {
ports = split("\n", file("ports.txt") )
}
But the terraform expects this file to be present at the start itself, throws error as below
Invalid value for "path" parameter: no file exists at
path/ports.txt; this function works only with files
that are distributed as part of the configuration source code, so if this file
will be created by a resource in this configuration you must instead obtain
this result from an attribute of that resource.
What am i missing here? Any help on this would be appreciated. Is there any depends_on for locals, how can i make this work
Modules are called from within other modules using module blocks. Most arguments correspond to input variables defined by the module. To reference the value from one module, you need to declare the output in that module, then you can call the output value from other modules.
For example, I suppose you have a text file in module a.
.tf file in module a
output "textfile" {
value = file("D:\\Terraform\\modules\\a\\ports.txt")
}
.tf file in module b
variable "externalFile" {
}
locals {
ports = split("\n", var.externalFile)
}
# output "b_test" {
# value = local.ports
# }
.tf file in the root module
module "a" {
source = "./modules/a"
}
module "b" {
source = "./modules/b"
externalFile = module.a.textfile
depends_on = [module.a]
}
# output "module_b_output" {
# value = module.b.b_test
# }
For more reference, you could read https://www.terraform.io/docs/language/modules/syntax.html#accessing-module-output-values
As the error message reports, the file function is only for files that are included on disk as part of your configuration, not for files generated dynamically during the apply phase.
I would typically suggest avoiding writing files to local disk as part of a Terraform configuration, because one of Terraform's main assumptions is that any objects you manage with Terraform will persist from one run to the next, but that could only be true for a local file if you always run Terraform in the same directory on the same computer, or if you use some other more complex approach such as a network filesystem. However, since you didn't mention why you are writing a file to disk I'll assume that this is a hard requirement and make a suggestion about how to do it, even though I would consider it a last resort.
The hashicorp/local provider includes a data source called local_file which will read a file from disk in a similar way to how a more typical data source might read from a remote API endpoint. In particular, it will respect any dependencies reflected in its configuration and defer reading the file until the apply step if needed.
You could coordinate this between modules then by making the output value which returns the filename also depend on whichever resource is responsible for creating the file. For example, if the file were created using a provisioner attached to an aws_instance resource then you could write something like this inside the module:
output "filename" {
value = "D:\\Terraform\\modules\\a\\ports.txt"
depends_on = [aws_instance.example]
}
Then you can pass that value from one module to the other, which will carry with it the implicit dependency on aws_instance.example to make sure the file is actually created first:
module "a" {
source = "./modules/a"
}
module "b" {
source = "./modules/b"
filename = module.a.filename
}
Then finally, inside the module, declare that input variable and use it as part of the configuration for a local_file data resource:
variable "filename" {
type = string
}
data "local_file" "example" {
filename = var.filename
}
Elsewhere in your second module you can then use data.local_file.example.content to get the contents of that file.
Notice that dependencies propagate automatically aside from the explicit depends_on in the output "filename" block. It's a good practice for a module to encapsulate its own behaviors so that everything needed for an output value to be useful has already happened by the time a caller uses it, because then the rest of your configuration will just get the correct behavior by default without needing any additional depends_on annotations.
But if there is any way you can return the data inside that ports.txt file directly from the first module instead, without writing it to disk at all, I would recommend doing that as a more robust and less complex approach.
Related
I'm trying to something fairly simple, but can't seem to get my head around it. I have the following structure:
- terragrunt.hcl
-----dummy/
---------main.tf
---------terragrunt.hcl
I'm looking to set some common variables at the root level, and use them in main.tf. Howe would I go about declaring the varibale in the root terragrunt level, and have them available downstream?
I've tried setting them as inputs in the root, but then have to explicitly declare "variables" at the dummy level for the inputs to get picked up. I'm looking to somehow define these things at the root level and not repeat variable declarations at dummy/ level. Is this doable?
You can indeed do this documented here:
https://terragrunt.gruntwork.io/docs/reference/built-in-functions/#read_terragrunt_config
You can merge all inputs defined in some file above any module.
From the docs:
read_terragrunt_config(config_path, [default_val]) parses the terragrunt config at the given path and serializes the result into a map that can be used to reference the values of the parsed config. This function will expose all blocks and attributes of a terragrunt config.
For example, suppose you had a config file called common.hcl that contains common input variables:
inputs = {
stack_name = "staging"
account_id = "1234567890"
}
You can read these inputs in another config by using read_terragrunt_config, and merge them into the inputs:
locals {
common_vars = read_terragrunt_config(find_in_parent_folders("common.hcl"))
}
inputs = merge(
local.common_vars.inputs,
{
# additional inputs
}
)
This function also takes in an optional second parameter which will be returned if the file does not exist:
locals {
common_vars = read_terragrunt_config(find_in_parent_folders("i-dont-exist.hcl", "i-dont-exist.hcl"), {inputs = {}})
}
inputs = merge(
local.common_vars.inputs, # This will be {}
{
# additional inputs
}
)
Per the Terragrunt documentation: "Currently you can only reference locals defined in the same config file. Terragrunt does not automatically include locals defined in the parent config of an include block into the current context."
However, one way you can do this is as follows:
Create a file containing the common variables (e.g. myvars.hcl)
Load that in the child terragrunt:
locals {
myvars = read_terragrunt_config(find_in_parent_folders("myvars.hcl"))
foo = local.myvars.locals.foo
}
Hope that helps!
Other tools like Ansible has directory hierarchy where child can refer to or override the value of a variable set at the parent level.
Terraform does not have such a mechanism and each directory having tf files is a separate Terraform module. So directory hierarchy cannot be used to pass/inherit/reference Terraform variables.
Perhaps better to let the idea of "downstream or upstream" go.
One way to define common variables and share them among other modules is Data-only Modules . Extension of this and make the common variable world-wide available is using Terraform registry although it is not the intended use.
Terraform registry AWS VPC example terraform-aws-vpc/examples/complete-vpc/main.tf has the code below which seems to me a circular dependency.
data "aws_security_group" "default" {
name = "default"
vpc_id = module.vpc.vpc_id
}
module "vpc" {
source = "../../"
name = "complete-example"
...
# VPC endpoint for SSM
enable_ssm_endpoint = true
ssm_endpoint_private_dns_enabled = true
ssm_endpoint_security_group_ids = [data.aws_security_group.default.id] # <-----
...
data.aws_security_group.default refers to "module.vpc.vpc_id" and module.vpc refers to "data.aws_security_group.default.id".
Please explain why this does not cause an error and how come module.vpc can refer to data.aws_security_group.default.id?
In the Terraform language, a module creates a separate namespace but it is not a node in the dependency graph. Instead, each of the module's Input Variables and Output Values are separate nodes in the dependency graph.
For that reason, this configuration contains the following dependencies:
The data.aws_security_group.default resource depends on module.vpc.vpc_id, which is specifically the output "vpc_id" block in that module, not the module as a whole.
The vpc module's variable "ssm_endpoint_security_group_ids" variable depends on the data.aws_security_group.default resource.
We can't see the inside of the vpc module in your question here, but the above is okay as long as there is no dependency connection between output "vpc_id" and variable "ssm_endpoint_security_group_ids" inside the module.
I'm assuming that such a connection does not exist, and so the evaluation order of objects here would be something like this:
aws_vpc.example in module.vpc is created (I just made up a name for this because it's not included in your question)
The output "vpc_id" in module.vpc is evaluated, referring to module.vpc.aws_vpc.example, and producing module.vpc.vpc_id.
data.aws_security_group.default in the root module is read, using the value of module.vpc.vpc_id.
The variable "ssm_endpoint_security_group_ids" for module.vpc is evaluated, referring to data.aws_security_group.default.
aws_vpc_endpoint.example in module.vpc is created, including a reference to var.ssm_endpoint_security_group_ids.
Notice that in all of the above I'm talking about objects in modules, not modules themselves. The modules serve only to create separate namespaces for objects, and then the separate objects themselves (which includes individual variable and output blocks) are what participate in the dependency graph.
Normally this design detail isn't visible: Terraform normally just uses it to potentially optimize concurrency by beginning work on part of a module before the whole module is ready to process. In some interesting cases like this though, you can also intentionally exploit this design so that an operation for the calling module can be explicitly sandwiched between two operations for the child module.
Another reason why we might make use of this capability is when two modules naturally depend on one another, such as in an experimental module I built that hides some of the tricky details of setting up VPC peering connections:
locals {
vpc_nets = {
us-west-2 = module.vpc_usw2
us-east-1 = module.vpc_use1
}
}
module "peering_usw2" {
source = "../../modules/peering-mesh"
region_vpc_networks = local.vpc_nets
other_region_connections = {
us-east-1 = module.peering_use1.outgoing_connection_ids
}
providers = {
aws = aws.usw2
}
}
module "peering_use1" {
source = "../../modules/peering-mesh"
region_vpc_networks = local.vpc_nets
other_region_connections = {
us-west-2 = module.peering_usw2.outgoing_connection_ids
}
providers = {
aws = aws.use1
}
}
(the above is just a relevant snippet from an example in the module repository.)
In the above case, the peering-mesh module is carefully designed to allow this mutual referencing, internally deciding for each pair of regional VPCs which one will be the peering initiator and which one will be the peering accepter. The outgoing_connection_ids output refers only to the aws_vpc_peering_connection resource and the aws_vpc_peering_connection_accepter refers only to var.other_region_connections, and so the result is a bunch of concurrent operations to create aws_vpc_peering_connection resources, followed by a bunch of concurrent operations to create aws_vpc_peering_connection_accepter resources.
terraform has adjusted its authorization
in main.tf [for sql config] I now have:
resource "google_sql_database_instance" "master" {
name = "${random_id.id.hex}-master"
region = "${var.region}"
database_version = "POSTGRES_9_6"
# allow direct access from work machines
ip_configuration {
authorized_networks = "${var.authorized_networks}"
require_ssl = "${var.sql_require_ssl}"
ipv4_enabled = true
}
}
where
in variables.tf I have
variable "authorized_networks" {
description = "The networks that can connect to cloudsql"
type = "list"
default = [
{
name = "work"
value = "xxx.xxx.xx.xxx/32"
}
]
}
where xxx.xxx.xx.xxx is the ip address I would like to allow. However, I prefer not to put this in my variables.tf file, but rather in a non-source controlled .tfvars file.
for variables that have a simple value, this is easy, but it is not clear to me how to do it with the nested structure. Replacing xxx.xxx.xx.xxx by a variable [e.g. var.work_ip] leads to an error
variables may not be used here
any insights?
If you omit the default argument in your main configuration altogether, you will mark variable "authorized_networks" as a required input variable, which Terraform will then check to ensure that it is set by the caller.
If this is a root module variable, then you can provide the value for it in a .tfvars file using the following syntax:
authorized_networks = [
{
name = "work"
value = "xxx.xxx.xx.xxx/32"
}
]
If this file is being generated programmatically by some wrapping automation around Terraform, you can also write it into a .tfvars.json file and use JSON syntax, which is often easier to construct robustly in other languages:
{
"authorized_networks": [
{
"name": "work",
"value": "xxx.xxx.xx.xxx/32"
}
]
}
You can either specify this file explicitly on the command line using the -var-file option, or you can give it a name ending in .auto.tfvars or .auto.tfvars.json in the current working directory when you run Terraform and Terraform will then find and load it automatically.
A common reason to keep something out of version control is because it's a dynamic setting configured elsewhere in the broader system rather than a value fixed in version control. If that is true here, then an alternative strategy is to save that setting in a configuration data store that Terraform is able to access via data sources and then write your Terraform configuration to retrieve that setting directly from the place where it is published.
For example, if the network you are modelling here were a Google Cloud Platform subnetwork, and it has either a fixed name or one that can be derived systematically in Terraform, you could retrieve this setting using the google_compute_subnetwork data source:
data "google_compute_subnetwork" "work" {
name = "work"
}
Elsewhere in configuration, you can then use data.google_compute_subnetwork.work.ip_cidr_range to access the CIDR block definition for this network.
The major Terraform providers have a wide variety of data sources like this, including ones that retrieve specific first-class objects from the target platform and also more generic ones that access configuration stores like AWS Systems Manager Parameter Store or HashiCorp Consul. Accessing the necessary information directly or publishing it "online" in a configuration store can be helpful in a larger system to efficiently integrate subsystems.
I defined a aws_cloudwatch_event_target in terraform to fire an event to lambda from cloudwatch. The input field is the event parameter for example:
resource "aws_cloudwatch_event_target" "data" {
rule = "${aws_cloudwatch_event_rule.scheduler.name}"
target_id = "finance_producer_cloudwatch"
arn = "${aws_lambda_function.finance_data_producer.arn}"
input = "{\"test\": [\"111\"]}"
}
I wonder how I can load the input json data from an external file.
The answer here depends on a few different questions:
Is this file a static part of your configuration, checked in to version control alongside your .tf files, or is it dynamically generated as part of the apply process?
Do you want to use the file contents literally, or do you need to substitute values into it from elsewhere in the Terraform configuration?
These two questions form a matrix of four different answers:
| Literal Content Include Values from Elsewhere
-------------|----------------------------------------------------------
Static File | file(...) function templatefile(...) function
Dynamic File | local_file data source template_file data source
I'll describe each of these four options in more detail below.
A common theme in all of these examples will be references to path.module, which evaluates to the path where the current module is loaded from. Another way to think about that is that it is the directory containing the current .tf file. Accessing files in other directories is allowed, but in most cases it's appropriate to keep things self-contained in your module by keeping the data files and the configuration files together.
Terraform strings are sequences of unicode characters, so Terraform can only read files containing valid UTF-8 encoded text. For JSON that's no problem, but worth keeping in mind for other file formats that might not conventionally be UTF-8 encoded.
The file function
The file function reads the literal content of a file from disk as part of the initial evaluation of the configuration. The content of the file is treated as if it were a literal string value for validation purposes, and so the file must exist on disk (and usually, in your version control) as a static part of your configuration, as opposed to being generated dynamically during terraform apply.
resource "aws_cloudwatch_event_target" "data" {
rule = aws_cloudwatch_event_rule.scheduler.name
target_id = "finance_producer_cloudwatch"
arn = aws_lambda_function.finance_data_producer.arn
input = file("${path.module}/input.json")
}
This is the most common and simplest option. If the file function is sufficient for your needs then it's the best option to use as a default choice.
The templatefile function
The templatefile function is similar to the file function, but rather than just returning the file contents literally it instead parses the file contents as a string template and then evaluates it using a set of local variables given in its second argument. This is useful if you need to pass some data from elsewhere in the Terraform configuration, as in this example:
resource "aws_cloudwatch_event_target" "data" {
rule = aws_cloudwatch_event_rule.scheduler.name
target_id = "finance_producer_cloudwatch"
arn = aws_lambda_function.finance_data_producer.arn
input = templatefile("${path.module}/input.json.tmpl", {
instance_id = aws_instance.example.id
})
}
In input.json.tmpl you can use the Terraform template syntax to substitute that variable value:
{"instance_id":${jsonencode(instance_id)}}
In cases like this where the whole result is JSON, I'd suggest just generating the whole result using jsonencode, since then you can let Terraform worry about the JSON escaping etc and just write the data structure in Terraform's object syntax:
${jsonencode({
instance_id = instance_id
})}
As with file, because templatefile is a function it gets evaluated during initial decoding of the configuration and its result is validated as a literal value. The template file must therefore also be a static file that is distributed as part of the configuration, rather than a dynamically-generated file.
The local_file data source
Data sources are special resource types that read an existing object or compute a result, rather than creating and managing a new object. Because they are resources, they can participate in the dependency graph and can thus make use of objects (including local files) that are created by other resources in the same Terraform configuration during terraform apply.
The local_file data source belongs to the local provider and is essentially the data source equivalent of the file function.
In the following example, I'm using var.input_file as a placeholder for any reference to a file path that is created by some other resource in the same configuration. In a real example, that is most likely to be a direct reference to an attribute of a resource.
data "local_file" "input" {
filename = var.input_file
}
resource "aws_cloudwatch_event_target" "data" {
rule = aws_cloudwatch_event_rule.scheduler.name
target_id = "finance_producer_cloudwatch"
arn = aws_lambda_function.finance_data_producer.arn
input = data.local_file.input.content
}
The template_file data source
NOTE: Since I originally wrote this answer, the provider where template_file was implemented has been declared obsolete and no longer maintained, and there is no replacement. In particular, the provider was archived prior to the release of Apple Silicon and so there is no available port for macOS on that architecture.
The Terraform team does not recommend rendering of dynamically-loaded templates, because it pushes various errors that could normally be detected at plan time to be detected only during apply time instead.
I've retained this content as I originally wrote it in case it's useful, but I would suggest treating this option as a last resort.
The template_file data source is the data source equivalent of the templatefile function. It's similar in usage to local_file though in this case we populate the template itself by reading it as a static file, using either the file function or local_file as described above depending on whether the template is in a static file or a dynamically-generated one, though if it were a static file we'd prefer to use the templatefile function and so we'll use the local_file data source here:
data "local_file" "input_template" {
filename = var.input_template_file
}
data "template_file" "input" {
template = data.local_file.input_template.content
vars = {
instance_id = aws_instance.example.id
}
}
resource "aws_cloudwatch_event_target" "data" {
rule = aws_cloudwatch_event_rule.scheduler.name
target_id = "finance_producer_cloudwatch"
arn = aws_lambda_function.finance_data_producer.arn
input = data.template_file.input.rendered
}
The templatefile function was added in Terraform 0.12.0, so you may see examples elsewhere of using the template_file data source to render static template files. That is an old pattern, now deprecated in Terraform 0.12, because the templatefile function makes for a more direct and readable configuration in most cases.
One quirk of the template_file data source as opposed to the templatefile function is that the data source belongs to the template provider rather than to Terraform Core, and so which template features are available in it will depend on which version of the provider is installed rather than which version of Terraform CLI is installed. The template provider is likely to lag behind Terraform Core in terms of which template language features are available, which is another reason to prefer the templatefile function where possible.
Other Possibilities
This question was specifically about reading data from a file, but for completeness I also want to note that for small JSON payloads it can sometimes be preferable to inline them directly in the configuration as a Terraform data structure and convert to JSON using jsonencode, like this:
resource "aws_cloudwatch_event_target" "data" {
rule = aws_cloudwatch_event_rule.scheduler.name
target_id = "finance_producer_cloudwatch"
arn = aws_lambda_function.finance_data_producer.arn
input = jsonencode({
instance_id = aws_instance.example.id
})
}
Writing the data structure inline as a Terraform expression means that a future reader can see directly what will be sent without needing to refer to a separate file. However, if the data structure is very large and complicated then it can hurt overall readability to include it inline because it could overwhelm the other configuration in the same file.
Which option to choose will therefore depend a lot on the specific circumstances, but always worth considering whether the indirection of a separate file is the best choice for readability.
Terraform also has a yamlencode function (experimental at the time of writing) which can do similarly for YAML-formatted data structures, either directly inside a .tf file or in an interpolation sequence in an external template.
You can use the file() operator to pull data from an external file:
input = "${file("myjson.json")}"
Just make sure myjson.json exists on disk in the same directory as the rest of your Terraform files.
I would use the data template_file resource. Like so...
data "template_file" "my_file" {
template = "${file("${path.module}/my_file.json")}"
vars = {
var_to_use_in_file = "${var.my_value}"
}
}
Then in your resource block....
resource "aws_cloudwatch_event_target" "data" {
rule = "${aws_cloudwatch_event_rule.scheduler.name}"
target_id = "finance_producer_cloudwatch"
arn = "${aws_lambda_function.finance_data_producer.arn}"
input = "${data.template_file.my_file.rendered}"
}
I'm trying to create a module in Terraform that can be instantiated multiple times with different variable inputs. Within the module, how do I reference resources when their names depend on an input variable? I'm trying to do it via the bracket syntax ("${aws_ecs_task_definition[var.name].arn}") but I just guessed at that.
(Caveat: I might be going about this in completely the wrong way)
Here's my module's (simplified) main.tf file:
variable "name" {}
resource "aws_ecs_service" "${var.name}" {
name = "${var.name}_service"
cluster = ""
task_definition = "${aws_ecs_task_definition[var.name].arn}"
desired_count = 1
}
resource "aws_ecs_task_definition" "${var.name}" {
family = "ecs-family-${var.name}"
container_definitions = "${template_file[var.name].rendered}"
}
resource "template_file" "${var.name}_task" {
template = "${file("task-definition.json")}"
vars {
name = "${var.name}"
}
}
I'm getting the following error:
Error loading Terraform: Error downloading modules: module foo: Error loading .terraform/modules/af13a92c4edda294822b341862422ba5/main.tf: Error reading config for aws_ecs_service[${var.name}]: parse error: syntax error
I was fundamentally misunderstanding how modules worked.
Terraform does not support interpolation in resource names (see the relevant issues), but that doesn't matter in my case, because the resources of each instance of a module are in the instance's namespace. I was worried about resource names colliding, but the module system already handles that.
The picture below shows what is going on.
The terraform documentation does not make their use of "NAME" clear versus the "name" values that are used for the actual resources created by the infrastructure vender (like, AWS or Google Cloud).
Additionally, it isn't always "name=, but sometimes, say, "endpoint= or even "resource_group_name= or whatever.
And there are a couple of ways to generate multiple "name" values -- using count, variables, etc., or inside tfvar files and running terraform apply -var-file=foo.tfvars