I've set up a terraform project with the following folder structure:
modules/environments/[$environment,all]/regions/[$region,all]
modules/resources/[$resource]
environments/[$environment]/[$region]
In modules/resources I have folders like api, load balancer, etc. In modules/environments/[$environment,all]/regions/[$region,all], I import the required modules from modules/resources/[$resource]. Given the environment is prod, in eu-west-1 for example, I import modules from modules/environments/[prod,all]/regions/[eu-west-1,all].
If I add a new variable, I have to update the variables.tf file in all the affected places, and in every main.tf, where I assign the variables to the modules. This is a lot of effort.
It would be a much easier scenario if I wouldn't have to assign the variables in the "module" {} level, but it would automatically get the variables from the .tfvars file that is being used when planning or applying, but it's not the case.
Is there any workoaround for this?
Related
I'm writing a terraform module that will select a value from a map, based on a key passed to it.
Nothing unusual.
The values in that map are secret however, and do not change based on who is calling the module.
A simple way I thought to keep those secrets secret would be to define the variable as a map in variables.tf in the module, put the keys/values in terraform.tfvars in the module, .gitignore terraform.tfvars, and encrypt it to terraform.tfvars.gpg or something.
But that doesn't work, because I have no default for the variable in the module terraform is expecting the variable to be set by the caller.
I can define the variable in the caller without a value, add it to the call, and either specify manually --var-file or include a terraform.tfvars in the caller. But that means the user having to remember a magic --var-file invocation or the duplication of terraform.tfvars everywhere my new module is used.
Remembering magic strings and duplication are both not good options.
Is it possible for a module to use its own tfvars to fill in variables not passed to it?
There is no way to use an automatic .tfvars file with a non-root module. Child modules always get all of their values from the calling module block (with default values inserted where appropriate); .tfvars is only for assigning values to root module variables.
Another option with similar characteristics to what you're describing is to use a data file in either JSON or YAML format inside the module directory, and load it in using the file function and one of the decoding functions. For example:
locals {
# This file must be created before using this module by
# decrypting secrets.yaml.gpg in this directory.
secrets = yamldecode(file("${path.module}/secrets.yaml"))
}
If the caller neglects to decrypt into the .gitignored file before using the module then the file call will fail with a "file not found" error message.
I'm not totally sure I understand but taking a stab. Are you using AWS? A fun solution I've used in the past is SSM parameters.
data "aws_ssm_parameter" "foo" {
name = "foo"
}
...
value = data.aws_ssm_parameter.foo.value
...
The SSM param could be created outside of tf and looked up in your module (and policies granting access depending on caller via IAM, or whatever).
I am going to be managing the CDN configurations of dozens of applications through Terraform. I have a set of .tf files holding all the default constant settings that are common among all configurations and then each application has its own .tfvars file to hold its unique settings.
If I run something like terraform apply --var-file=app1.tfvars --var-file=app2.tfvars --var-file=app3.tfvars then only the last file passed in is used.
Even if this did work it will become unmanageable when I extend this to more sites.
What is the correct way to incorporate multiple .tfvars files that populate a common set of .tf files?
Edit: I should add that the .tfvar files define the same variables but with different values. I need to declare state of the resources defined in the .tf files once for each .tfvar file.
I found the best way to handle this case (without any 3rd party tools is to use) Terraform workspaces and create a separate workspace for each .tfvars file. This way I can use the same common .tf files and simply swap to a different workspace with terraform workspace select <workspace name> before running terraform apply --var-file=<filename> with each individual .tfvars file.
This should work using process substitution:
terraform apply -var-file=<(cat app1.tfvars app2.tfvars app3.tfvars)
Best Way may be the use of TerraGrunt https://terragrunt.gruntwork.io/ from GruntWork, which is a thin wrapper around Terraform, you can use the HCL configuration file to define your requirements.
Sample terragrunt.hcl configuration:
terraform {
extra_arguments "conditional_vars" {
commands = [
"apply",
"plan",
"import",
"push",
"refresh"
]
required_var_files = [
"${get_parent_terragrunt_dir()}/terraform.tfvars"
]
optional_var_files = [
"${get_parent_terragrunt_dir()}/${get_env("TF_VAR_env", "dev")}.tfvars",
"${get_parent_terragrunt_dir()}/${get_env("TF_VAR_region", "us-east-1")}.tfvars",
"${get_terragrunt_dir()}/${get_env("TF_VAR_env", "dev")}.tfvars",
"${get_terragrunt_dir()}/${get_env("TF_VAR_region", "us-east-1")}.tfvars"
]
}
You can pass down tfvars, also you can get more features from terragrunt by better organise your Terraform Layout, and use configuration file for passing tfvars from different locations.
This question already has answers here:
What is the difference between variables.tf and terraform.tfvars?
(5 answers)
Closed 3 years ago.
I've been researching this but can't find the distinction. A variables.tf file can store variable defaults/values, like a terraform.tfvars file.
What's the difference between these two and the need for one over the other? My understanding is if you pass in the var file as an argument in terraform via the command line.
There is a thread about this already and the only benefit seems to be passing in the tfvars file as an argument, as you can "potentially" do assignment of variables in a variable.tf file.
Is this the correct thinking?
The distinction between these is of declaration vs. assignment.
variable blocks (which can actually appear in any .tf file, but are in variables.tf by convention) declare that a variable exists:
variable "example" {}
This tells Terraform that this module accepts an input variable called example. Stating this makes it valid to use var.example elsewhere in the module to access the value of the variable.
There are several different ways to assign a value to this input variable:
Include -var options on the terraform plan or terraform apply command line.
Include -var-file options to select one or more .tfvars files to set values for many variables at once.
Create a terraform.tfvars file, or files named .auto.tfvars, which are treated the same as -var-file arguments but are loaded automatically.
For a child module, include an expression to assign to the variable inside the calling module block.
A variable can optionally be declared with a default value, which makes it optional. Variable defaults are used for situations where there's a good default behavior that would work well for most uses of the module/configuration, while still allowing that behavior to be overridden in exceptional cases.
The various means for assigning variable values are for dealing with differences. What that means will depend on exactly how you are using Terraform, but for example if you are using the same configuration multiple times to deploy different "copies" of the same infrastructure (environments, etc) then you might choose to have a different .tfvars file for each of these copies.
Because terraform.tfvars and .auto.tfvars are automatically loaded without any additional options, they behave similarly to defaults, but the intent of these is different. When running Terraform in automation, some users have their automation generate a terraform.tfvars file or .auto.tfvars just before running Terraform in order to pass in values the automation knows, such as what environment the automation is running for, etc.
The difference between the automatically-loaded .tfvars files and variable defaults is more clear when dealing with child modules. .tfvars files (and -var, -var-file options) only apply to the root module variables, while variable defaults apply when that module is used as a child module too, meaning that variables with defaults can be omitted in module blocks.
A variables.tf file is used to define the variables type and optionally set a default value.
A terraform.tfvars file is used to set the actual values of the variables.
You could set default values for all your variables and not use tfvars files at all.
Actually the objective of splitting between the definitions and the values, is to allow the definition of a common infrastructure design, and then apply specific values per environment.
Using multiple tfvars files that you give as an argument allows you to set different values per environment : secrets, VM size, number of instances, etc.
Using Terraform v0.11.11, when invoking terraform plan, supply a .tfvars file that is NOT in the current directory e.g a common 'global' file for global constants
seems to just ignore it when I try ../ in an var-file option
-var-file="../$(ENV).global.tfvars"
What options do I have to achieve a similar result if this is not possible (event in the v0.12 is this still the case)?
I have a root level variable, whose value is set at run time via a tfvar file. The tfvar file used and the var value can vary. Now I want to use this variable inside a module, the terraform way to do this is to set it up as a module variable and pass the root var when creating the module.
Except, I have this module used in our infrastructure several hundred times. Is repeating this variable 100s of times the only way to do this? Can the module access root namespace to grab the variable value?
I'm half tempted to use an external data script in the module to fetch the value instead, except, I wont know which tfvars file will be in effect at runtime..
Unfortunately, the only Terraform supported way is to pass the value into the module as a variable. Terraform has made me a copy/paste expert.
The only other approach that comes to mind that Terraform supports is making creative use of the External Data Source.