We are trying to create Terraform modules for below activities in AWS, so that we can use them where ever that is required.
VPC creation
Subnets creation
Instance creation etc.
But while creating these modules we have to define the provider in all above listed modules. So we decided to create one more module for provider so that we can call that provider module in other modules (VPC, Subnet, etc.).
Issue in above approach is that it is not taking provider value, and asking for the user input for region.
Terraform configuration is as follow:
$HOME/modules/providers/main.tf
provider "aws" {
region = "${var.region}"
}
$HOME/modules/providers/variables.tf
variable "region" {}
$HOME/modules/vpc/main.tf
module "provider" {
source = "../../modules/providers"
region = "${var.region}"
}
resource "aws_vpc" "vpc" {
cidr_block = "${var.vpc_cidr}"
tags = {
"name" = "${var.environment}_McD_VPC"
}
}
$HOME/modules/vpc/variables.tf
variable "vpc_cidr" {}
variable "environment" {}
variable "region" {}
$HOME/main.tf
module "dev_vpc" {
source = "modules/vpc"
vpc_cidr = "${var.vpc_cidr}"
environment = "${var.environment}"
region = "${var.region}"
}
$HOME/variables.tf
variable "vpc_cidr" {
default = "192.168.0.0/16"
}
variable "environment" {
default = "dev"
}
variable "region" {
default = "ap-south-1"
}
Then when running terraform plan command at $HOME/ location it is not taking provider value and instead asking for the user input for region.
I need help from the Terraform experts, what approach we should follow to address below concerns:
Wrap provider in a Terraform module
Handle multiple region use case using provider module or any other way.
I knew a long time back that it wasn't possible to do this because Terraform built a graph that required a provider for any resource before it included any dependencies and it didn't used to be possible to force a dependency on a module.
However since Terraform 0.8 it is now possible to set a dependency on modules with the following syntax:
module "network" {
# ...
}
resource "aws_instance" "foo" {
# ...
depends_on = ["module.network"]
}
However, if I try that with your setup by changing modules/vpc/main.tf to look something like this:
module "aws_provider" {
source = "../../modules/providers"
region = "${var.region}"
}
resource "aws_vpc" "vpc" {
cidr_block = "${var.vpc_cidr}"
tags = {
"name" = "${var.environment}_McD_VPC"
}
depends_on = ["module.aws_provider"]
}
And run terraform graph | dot -Tpng > graph.png against it it looks like the graph doesn't change at all from when the explicit dependency isn't there.
This seems like it might be a potential bug in the graph building stage in Terraform that should probably be raised as an issue but I don't know the core code base well enough to spot where the change needs to be.
For our usage we use symlinks heavily in our Terraform code base, some of which is historic from before Terraform supported other ways of doing things but could work for you here.
We simply define the provider in a single .tf file (such as environment.tf) along with any other generic config needed for every place you would ever run Terraform (ie not at a module level) and then symlink this into each location. That allows us to define the provider in a single place with overridable variables if necessary.
Step 1
Add region alias in the main.tf file where you gonna execute the terraform plan.
provider "aws" {
region = "eu-west-1"
alias = "main"
}
provider "aws" {
region = "us-east-1"
alias = "useast1"
}
Step 2
Add providers block inside your module definition block
module "lambda_edge_rule" {
providers = {
aws = aws.useast1
}
source = "../../../terraform_modules/lambda"
tags = var.tags
}
Step 3
Define "aws" as providers inside your module. ( source = ../../../terraform_modules/lambda")
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 2.7.0"
}
}
}
resource "aws_lambda_function" "lambda" {
function_name = "blablabla"
.
.
.
.
.
.
.
}
Note: Terraform version v1.0.5 as of now.
Related
We want to deploy services into several regions.
Looks like because of the aws provider, we can't just use count or for_each, as the provider can't be interpolated. Thus I need to set this up manually:
resource "aws_instance" "app-us-west-1" {
provider = aws.us-west-1
#other stuff
}
resource "aws_instance" "app-us-east-1" {
provider = aws.us-east-1
#other stuff
}
I would like when running this to create a file which contains all the IPs created (for an ansible inventory).
I was looking at this answer:
https://stackoverflow.com/a/61788089/169252
and trying to adapt it for my case:
resource "local_file" "app-hosts" {
content = templatefile("${path.module}/templates/app_hosts.tpl",
{
hosts = aws_instance[*].public_ip
}
)
filename = "app-hosts.cfg"
}
And then setting up the template accordingly.
But this fails:
Error: Invalid reference
on app.tf line 144, in resource "local_file" "app-hosts":
122: hosts = aws_instance[*].public_ip
A reference to a resource type must be followed by at least one attribute
access, specifying the resource name
I am suspecting that I can't just reference all the aws_instance defined as above like this. Maybe to refer to all aws_instance in this file I need to use a different syntax.
Or maybe I need to use a module somehow. Can someone confirm this?
Using terraform v0.12.24
EDIT: The provider definitions use alias and it's all in the same app.tf, which I was naively assuming to be able to apply in one go with terraform apply (did I mention I am a beginner with terraform?):
provider "aws" {
alias = "us-east-1"
region = "us-east-1"
}
provider "aws" {
alias = "us-west-1"
region = "us-west-1"
}
My current workaround is to not do a join but simply listing them all individually:
{
host1 = aws_instance.app-us-west-1.public_ip
host2 = aws_instance.app-us-east-1.public_ip
# more hosts
}
Given multiple provider:
provider "aws" {
alias = "eu-west-1"
region = "eu-west-1"
}
provider "aws" {
alias = "ap-northeast-1"
region = "ap-northeast-1"
}
I have an instance definition which will load an ami based on the region.
I was wanting to assing the provider alias inside the for_each, but I have serious doubts this works:
resource "aws_instance" "test" {
for_each = var.aws_amis
ami = var.aws_amis[each.key]
provider = aws.each.key # <------ THIS LOOKS ODD; terraform init does not complain but probably will fail later...
}
Looks like I can not do that...and will have to use explicit instances for every region (I just show 2 here but I was wanting several more)
I'm new at terraform and I created a custom azure policies on module structure.
each policy represents a custom module.
One of the modules that I have created is enabling diagnostics logs for any new azure resource created.
but, I need a storage account for that. (before enabling the diagnostics settings how can I implement "depends_on"? or any other methods?
I want to create first the storage account and then the module of diagnostics settings.
on the main.tf (where calling all the other modules) or inside the resource (module)?
Thanks for the help!! :)
this below code represents the main.tf file:
//calling the create storage account name
module "createstorageaccount" {
source = "./modules/module_create_storage_account"
depends_on = [
"module_enable_diagnostics_logs"
]
}
this one represents the create storage account module
resource "azurerm_resource_group" "management" {
name = "management-rg"
location = "West Europe"
}
resource "azurerm_storage_account" "test" {
name = "diagnostics${azurerm_resource_group.management.name}"
resource_group_name = "${azurerm_resource_group.management.name}"
location = "${azurerm_resource_group.management.location}"
account_tier = "Standard"
account_replication_type = "LRS"
tags = {
environment = "diagnostics"
}
}
depends_on = [
"module_enable_diagnostics_logs"
]
In most cases, the necessary dependencies just occur automatically as a result of your references. If the configuration for one resource refers directly or indirectly to another, Terraform automatically infers the dependency between them without the need for explicit depends_on.
This works because module variables and outputs are also nodes in the dependency graph: if a child module resource refers to var.foo then it indirectly depends on anything that the value of that variable depends on.
For the rare situation where automatic dependency detection is insufficient, you can still exploit the fact that module variables and outputs are nodes in the dependency graph to create indirect explicit dependencies, like this:
variable "storage_account_depends_on" {
# the value doesn't matter; we're just using this variable
# to propagate dependencies.
type = any
default = []
}
resource "azurerm_storage_account" "test" {
name = "diagnostics${azurerm_resource_group.management.name}"
resource_group_name = "${azurerm_resource_group.management.name}"
location = "${azurerm_resource_group.management.location}"
account_tier = "Standard"
account_replication_type = "LRS"
tags = {
environment = "diagnostics"
}
# This resource depends on whatever the variable
# depends on, indirectly. This is the same
# as using var.storage_account_depends_on in
# an expression above, but for situations where
# we don't actually need the value.
depends_on = [var.storage_account_depends_on]
}
When you call this module, you can set storage_account_depends_on to any expression that includes the objects you want to ensure are created before the storage account:
module "diagnostic_logs" {
source = "./modules/diagnostic_logs"
}
module "storage_account" {
source = "./modules/storage_account"
storage_account_depends_on = [module.diagnostic_logs.logging]
}
Then in your diagnostic_logs module you can configure indirect dependencies for the logging output to complete the dependency links between the modules:
output "logging" {
# Again, the value is not important because we're just
# using this for its dependencies.
value = {}
# Anything that refers to this output must wait until
# the actions for azurerm_monitor_diagnostic_setting.example
# to have completed first.
depends_on = [azurerm_monitor_diagnostic_setting.example]
}
If your relationships can be expressed by passing actual values around, such as by having an output that includes the id, I'd recommend preferring that approach because it leads to a configuration that is easier to follow. But in rare situations where there are relationships between resources that cannot be modeled as data flow, you can use outputs and variables to propagate explicit dependencies between modules too.
module dependencies are now supported in Terraform 13, this is currently at the release candidate stage.
resource "aws_iam_policy_attachment" "example" {
name = "example"
roles = [aws_iam_role.example.name]
policy_arn = aws_iam_policy.example.arn
}
module "uses-role" {
# ...
depends_on = [aws_iam_policy_attachment.example]
}
Using depends_on at resource level is different from using depends_on at inter-module level i found very simple way to do to it at module level
module "eks" {
source = "../modules/eks"
vpc_id = module.vpc.vpc_id
vpc_cidr = [module.vpc.vpc_cidr_block]
public_subnets = flatten([module.vpc.public_subnets])
private_subnets_id = flatten([module.vpc.private_subnets])
depends_on = [module.vpc]
}
i created dependencies directly with module simple as simplest no complex relation reequired
This code will create an EC2 instance with name EC2 Instance:
provider "aws" {
region = "eu-west-1"
}
module ec2 {
source = "./ec2_instance"
name = "EC2 Instance"
}
However, if I try and use the random_pet resource the Instance name becomes an empty string.
provider "aws" {
region = "eu-west-1"
}
resource "random_pet" "server" {
length = 4
}
module ec2 {
source = "./ec2_instance"
name = "${random_pet.server.id}"
}
How come?
I'm using the random_pet.server.id code from https://www.terraform.io/docs/providers/random/r/pet.html
UPDATE: by using an output I was able to debug this.
Terraform does not seem to show the value of this variable during a plan. However, when doing an apply it did successfully populate this variable (and therefore name the instance). The question then becomes why does it not work in plan but does in apply?
I have TF templates whose purpose is to create multiple copies of the same cloud infrastructure. For example you have multiple business units inside a big organization, and you want to build out the same basic networks. Or you want an easy way for a developer to spin up the stack that he's working on. The only difference between "tf apply" invokations is the variable BUSINESS_UNIT, for example, which is passed in as an environment variable.
Is anyone else using a system like this, and if so, how do you manage the state files ?
You should use a Terraform Module. Creating a module is nothing special: just put any Terraform templates in a folder. What makes a module special is how you use it.
Let's say you put the Terraform code for your infrastructure in the folder /terraform/modules/common-infra. Then, in the templates that actually define your live infrastructure (e.g. /terraform/live/business-units/main.tf), you could use the module as follows:
module "business-unit-a" {
source = "/terraform/modules/common-infra"
}
To create the infrastructure for multiple business units, you could use the same module multiple times:
module "business-unit-a" {
source = "/terraform/modules/common-infra"
}
module "business-unit-b" {
source = "/terraform/modules/common-infra"
}
module "business-unit-c" {
source = "/terraform/modules/common-infra"
}
If each business unit needs to customize some parameters, then all you need to do is define an input variable in the module (e.g. under /terraform/modules/common-infra/vars.tf):
variable "business_unit_name" {
description = "The name of the business unit"
}
Now you can set this variable to a different value each time you use the module:
module "business-unit-a" {
source = "/terraform/modules/common-infra"
business_unit_name = "a"
}
module "business-unit-b" {
source = "/terraform/modules/common-infra"
business_unit_name = "b"
}
module "business-unit-c" {
source = "/terraform/modules/common-infra"
business_unit_name = "c"
}
For more information, see How to create reusable infrastructure with Terraform modules and Terraform: Up & Running.
There's two ways of doing this that jump to mind.
Firstly, you could go down the route of using the same Terraform configuration folder that you apply and simply pass in a variable when running Terraform (either via the command line or through environment variables). You'd also want to have the same wrapper script that calls Terraform to configure your state settings to make them differ.
This might end up with something like this:
variable "BUSINESS_UNIT" {}
variable "ami" { default = "ami-123456" }
resource "aws_instance" "web" {
ami = "${var.ami}"
instance_type = "t2.micro"
tags {
Name = "web"
Business_Unit = "${var.BUSINESS_UNIT}"
}
}
resource "aws_db_instance" "default" {
allocated_storage = 10
engine = "mysql"
engine_version = "5.6.17"
instance_class = "db.t2.micro"
name = "${var.BUSINESS_UNIT}"
username = "foo"
password = "bar"
db_subnet_group_name = "db_subnet_group"
parameter_group_name = "default.mysql5.6"
}
Which creates an EC2 instance and an RDS instance. You would then call that with something like this:
#!/bin/bash
if [ "$#" -ne 1 ]; then
echo "Illegal number of parameters - specify business unit as positional parameter"
fi
business_unit=$1
terraform remote config -backend="s3" \
-backend-config="bucket=${business_unit}" \
-backend-config="key=state"
terraform remote pull
terraform apply -var 'BUSINESS_UNIT=${business_unit}'
terraform remote push
As an alternative route you might want to consider using modules to wrap your Terraform configuration.
So instead you might have something that now looks like:
web-instance/main.tf
variable "BUSINESS_UNIT" {}
variable "ami" { default = "ami-123456" }
resource "aws_instance" "web" {
ami = "${var.ami}"
instance_type = "t2.micro"
tags {
Name = "web"
Business_Unit = "${var.BUSINESS_UNIT}"
}
}
db-instance/main.tf
variable "BUSINESS_UNIT" {}
resource "aws_db_instance" "default" {
allocated_storage = 10
engine = "mysql"
engine_version = "5.6.17"
instance_class = "db.t2.micro"
name = "${var.BUSINESS_UNIT}"
username = "foo"
password = "bar"
db_subnet_group_name = "db_subnet_group"
parameter_group_name = "default.mysql5.6"
}
And then you might have different folders that call these modules per business unit:
business-unit-1/main.tf
variable "BUSINESS_UNIT" { default = "business-unit-1" }
module "web_instance" {
source = "../web-instance"
BUSINESS_UNIT = "${var.BUSINESS_UNIT}"
}
module "db_instance" {
source = "../db-instance"
BUSINESS_UNIT = "${var.BUSINESS_UNIT}"
}
and
business-unit-2/main.tf
variable "BUSINESS_UNIT" { default = "business-unit-2" }
module "web_instance" {
source = "../web-instance"
BUSINESS_UNIT = "${var.BUSINESS_UNIT}"
}
module "db_instance" {
source = "../db-instance"
BUSINESS_UNIT = "${var.BUSINESS_UNIT}"
}
You still need a wrapper script to manage state configuration as before but going this route enables you to provide a rough template in your modules and then hard code certain extra configuration by business unit such as the instance size or the number of instances that are built for them.
This is rather popular use case. To archive this you can let developers to pass variable from command-line or from tfvars file into resource to make different resources unique:
main.tf:
resource "aws_db_instance" "db" {
identifier = "${var.BUSINESS_UNIT}"
# ... read more in docs
}
$ terraform apply -var 'BUSINESS_UNIT=unit_name'
PS: We do this often to provision infrastructure for specific git branch name, and since all resources are identifiable and are located in separate tfstate files, we can safely destroy them when we don't need them.