Terraform: Easier way to run a flexible number of objects - terraform

Currently I have a powershell script that reads a yaml config file with all the objects I need created and creates a .tfvars file which contains all the variables, maps, lists of maps etc.
It would be something like the following:
global_tags = {
Provisioner = "Terraform"
}
resource_groups = {
myrg1 = {
location = "uksouth",
tags = {
ResourceType = "resourcegroup"
}
}
}
storage_accounts = {
mystorage1 = {
resource_group_name = "myrg1",
location = "uksouth",
account_tier = "Standard",
account_replication_type = "GRS",
tags = {
ResourceType = "storageaccount"
}
containers_list = [
{ name = "test_private_x", access_type = "private" },
{ name = "test_blob_x", access_type = "blob" },
{ name = "test_container_x", access_type = "container" }
]
}
The idea is to then pump each list of maps into each module to create the resources, e.g. main.tf would be just:
module "resourcegroup" {
source = "./modules/azure-resourcegroup"
resource_groups = var.resource_groups
global_tags = var.global_tags
}
module "storageaccount" {
source = "./modules/azure-storageaccount"
depends_on = [module.resourcegroup]
storage_accounts = var.storage_accounts
global_tags = var.global_tags
}
Also, an example of a simple module would be:
resource "azurerm_resource_group" "rg" {
for_each = var.resource_groups
name = each.key
location = each.value.location
tags = lookup(each.value,"tags",null) == null ? var.global_tags : merge(var.global_tags,each.value.tags)
}
The issue is that writing a complex module, say around storage account, isn't too bad if you are just feeding in all the params, but feeding in a list of maps and writing a module to read that list and create multiple flattened lists to perform say 15 different calls (to create containers, shares, network rules etc.) is very complex.
Obviously the reason I want to use for_each loops in the modules is so that my main.tf doesn't have to call the module multiple times with hard coded values for say 50 storage accounts.
Just wondering if I am missing an obvious way to create complicated multiples of each resource type ?
I appreciate I could do separate modules for containers, shares etc and break the complex maps down into simpler ones to pass to the additional modules, but I was trying to just have 1 storage account module that could handle anything and be fed by a complex list of maps so main.tf did not need editing, I could just control the config completely via a .tfvars file

Related

Understanding Terraform for_each loop iteration

I am learning terraform and trying to understand the for_each loop iteration in terraform.
I am iterating through a loop for creating RGs in Azure cloud and what I want to understand is the difference between accessing the value of an instance using . or [""].
So for example, below is my tfvar file:
resource_groups = {
resource_group_1 = {
name = "terraform-apply-1"
location = "eastus2"
tags = {
created_by = "vivek89#test.com"
}
},
resource_group_2 = {
name = "terraform-apply-2"
location = "eastus2"
tags = {
created_by = "vivek89#test.com"
}
},
resource_group_3 = {
name = "terraform-apply-3"
location = "eastus2"
tags = {
created_by = "vivek89#test.com"
contact_dl = "vivek89#test.com"
}
}
}
and below is my terraform main.tf file:
resource "azurerm_resource_group" "terraformRG" {
for_each = var.resource_groups
name = each.value.name
location = each.value.location
tags = each.value.tags
}
I am confused with the expression in for_each in RG creation block. Both the below codes works and create RGs:
name = each.value.name
name = each.value["name"]
I want to understand the difference between the two and which one is correct.
They are equivalent as explained in the docs:
Map/object attributes with names that are valid identifiers can also be accessed using the dot-separated attribute notation, like local.object.attrname. In cases where a map might contain arbitrary user-specified keys, we recommend using only the square-bracket index notation (local.map["keyname"]).
The main difference is that dot notation requires key attributes to be valid identifiers. In contrast, the square-bracket notation works with any identifiers.

Unable to Create Terraform Resource Group when using modules

I am optimizing my terraform code by using modules. When i create a resource group module it works perfectly well but it creates two resource groups
i.e.
Temp-AppConfiguration-ResGrp
Temp-AppServices-ResGrp
instead it should only create
Temp-AppConfiguration-ResGrp
Code Resourcegroup.tf.
resource "azurerm_resource_group" "resource" {
name = "${var.environment}-${var.name_apptype}-ResGrp"
location = var.location
tags = {
environment = var.environment
}
}
output "resource_group_name" {
value = "${var.environment}-${var.name_apptype}-ResGrp"
}
output "resource_group_location" {
value = var.location
}
Variable.tf
variable "name_apptype" {
type = string
default = "AppServices"
}
variable "environment" {
type = string
default = "Temp"
}
variable "location" {
type = string
default = "eastus"
}
Main.tf
module "resourcegroup" {
source = "../Modules"
name_apptype = "AppConfiguration"
}
I want to pass name_apptype in main.tf when calling resource group module. So that i don't need to update variable.tf every time.
Any suggestions
where i am doing wrong. Plus i am also unable to output the value, i need it so that i could pass resource group name in the next module i want to create.
Thanks
You need to do that in the Main.tf
module "resourcegroup" {
source = "../Modules"
name_apptype = "AppConfiguration"
}
module "resourcegroup-appservices" {
source = "../Modules"
name_apptype = "AppServices"
}
These create a 2 resources groups with the values that you need, additionally you can remove the default value from the name_apptype variable.
If you want to create with the same module both resource groups you need to use count to iterate over an array of names

terraform - Iterate over two linked resources

I’m trying to write some code which would take an input structure like this:
projects = {
"project1" = {
namespaces = ["mynamespace1"]
},
"project2" = {
namespaces = ["mynamespace2", "mynamespace3"]
}
}
and provision multiple resources with for_each which would result in this:
resource "rancher2_project" "project1" {
provider = rancher2.admin
cluster_id = module.k8s_cluster.cluster_id
wait_for_cluster = true
}
resource "rancher2_project" "project2" {
provider = rancher2.admin
cluster_id = module.k8s_cluster.cluster_id
wait_for_cluster = true
}
resource "rancher2_namespace" "mynamespace1" {
provider = rancher2.admin
project_id = rancher2_project.project1.id
depends_on = [rancher2_project.project1]
}
resource "rancher2_namespace" "mynamespace2" {
provider = rancher2.admin
project_id = rancher2_project.project2.id
depends_on = [rancher2_project.project2]
}
resource "rancher2_namespace" "mynamespace3" {
provider = rancher2.admin
project_id = rancher2_project.project2.id
depends_on = [rancher2_project.project2]
}
namespaces are dependent on Projects and the generate id needs to be passed into namespace.
Is there any good way of doing this dynamically ? We might have a lot of Projects/namespaces.
Thanks for any help and advise.
The typical answer for systematically generating multiple instances of a resource based on a data structure is resource for_each. The main requirement for resource for_each is to have a map which contains one element per resource instance you want to create.
In your case it seems like you need one rancher2_project per project and then one rancher2_namespace for each pair of project and namespaces. Your current data structure is therefore already sufficient for the rancher2_project resource:
resource "rancher2_project" "example" {
for_each = var.projects
provider = rancher2.admin
cluster_id = module.k8s_cluster.cluster_id
wait_for_cluster = true
}
The above will declare two resource instances with the following addresses:
rancher2_project.example["project1"]
rancher2_project.example["project2"]
You don't currently have a map that has one element per namespace, so it will take some more work to derive a suitable value from your input data structure. A common pattern for this situation is flattening nested structures for for_each using the flatten function:
locals {
project_namespaces = flatten([
for pk, proj in var.projects : [
for nsk in proj.namespaces : {
project_key = pk
namespace_key = ns
project_id = rancher2_project.example[pk].id
}
]
])
}
resource "rancher2_namespace" "example" {
for_each = {
for obj in local.project_namespaces :
"${obj.project_key}.${obj.namespace_key}" => obj
}
provider = rancher2.admin
project_id = each.value.project_id
}
This produces a list of objects representing all of the project and namespace pairs, and then the for_each argument transforms it into a map using compound keys that include both the project and namespace keys to ensure that they will all be unique. The resulting instances will therefore have the following addresses:
rancher2_namespace.example["project1.mynamespace1"]
rancher2_namespace.example["project2.mynamespace2"]
rancher2_namespace.example["project2.mynamespace3"]
This seems to work too:
resource "rancher2_namespace" "example" {
count = length(local.project_namespaces)
provider = rancher2.admin
name = local.project_namespaces[count.index].namespace_name
project_id = local.project_namespaces[count.index].project_id
}

Terraform: How can I have one common output for all declared modules?

I have got something like this in my terraform file:
main.tf
module "airflow_tenant_one" {
source = "../modules/airflow_tenant"
name = "one-airflow"
project = var.project
cluster_name = var.cluster_name
region = var.region
kubernetes_endpoint = var.kubernetes_endpoint
tenant_domain = "one-airflow.${var.domain_name}"
}
module "airflow_tenant_two" {
source = "../modules/airflow_tenant"
name = "two-airflow"
project = var.project
cluster_name = var.cluster_name
region = var.region
kubernetes_endpoint = var.kubernetes_endpoint
tenant_domain = "two-airflow.${var.domain_name}"
}
How can I do one common output for all declared modules?
Currently, it looks like:
outputs.tf
output "chart_name_one" {
description = "The name of the chart"
value = module.airflow_tenant_one.chart_name
}
output "chart_name_two" {
description = "The name of the chart"
value = module.airflow_tenant_two.chart_name
}
I asked because in a future it is possible that I will add more modules in my main.tf file. Will be better to have one output declaration for all of them.
The module blocks you shared seem like they are all systematically configured based on some client names, so you can potentially do this using a single module block with for_each if you are using Terraform 0.13.0 or later.
locals {
tenants = toset([
"one",
"two",
])
}
module "airflow_tenant" {
for_each = local.tenants
name = "${each.key}-airflow"
project = var.project
cluster_name = var.cluster_name
region = var.region
kubernetes_endpoint = var.kubernetes_endpoint
tenant_domain = "${each.key}-airflow.${var.domain_name}"
}
output "tenant_chart_names" {
value = {
for name, tenant in module.airflow_tenant : name => tenant.chart_name
}
}
The above will cause there to be one instance of the airflow_tenant module per element of local.tenants, with addresses like this:
module.airflow_tenant["one"]
module.airflow_tenant["two"]
The for_each makes the module behave as a map of instances when you refer to it elsewhere, which is why we're able to project that map in output "tenant_chart_names" to derive a map from tenant name to chart names.
You can add and remove elements of local.tenants over time, in which case Terraform will understand that as an intent to either create a instances or destroy an instances of all of the objects described inside that module.
You can read more about this feature in Multiple Instances of a Module.

How do I make terraform skip that block while creating multiple resources in loop from a CSV file?

Hi I am trying to create a Terraform script which will take inputs from the user in the form of a CSV file and create multiple Azure resources.
For example if the user wants to create: ResourceGroup>Vnet>Subnet in bulk, he will provide input in CSV format as below:
resourcegroup,RG_location,RG_tag,domainname,DNS_Zone_tag,virtualnetwork,VNET_location,addressspace
csvrg1,eastus2,Terraform RG,test.sd,Terraform RG,csvvnet1,eastus2,10.0.0.0/16,Terraform VNET,subnet1,10.0.0.0/24
csvrg2,westus,Terraform RG2,test2.sd,Terraform RG2,csvvnet2,westus,172.0.0.0/8,Terraform VNET2,subnet1,171.0.0.0/24
I have written the following working main.tf file:
# Configure the Microsoft Azure Provider
provider "azurerm" {
version = "=1.43.0"
subscription_id = var.subscription
tenant_id = var.tenant
client_id = var.client
client_secret = var.secret
}
#Decoding the csv file
locals {
vmcsv = csvdecode(file("${path.module}/computelanding.csv"))
}
# Create a resource group if it doesn’t exist
resource "azurerm_resource_group" "myterraformgroup" {
count = length(local.vmcsv)
name = local.vmcsv[count.index].resourcegroup
location = local.vmcsv[count.index].RG_location
tags = {
environment = local.vmcsv[count.index].RG_tag
}
}
# Create a DNS Zone
resource "azurerm_dns_zone" "dnsp-private" {
count = 1
name = local.vmcsv[count.index].domainname
resource_group_name = local.vmcsv[count.index].resourcegroup
depends_on = [azurerm_resource_group.myterraformgroup]
tags = {
environment = local.vmcsv[count.index].DNS_Zone_tag
}
}
To be continued....
The issue I am facing here what is in the second resource group, the user don't want a resource type, suppose the user want to skip the DNS zone in the resource group csvrg2. How do I make terraform skip that block ?
Edit: What I am trying to achieve is "based on some condition in the CSV file, not to create azurerm_dns_zone resource for the resource group csvrg2"
I have provided an example of the CSV file, how it may look like below:
resourcegroup,RG_location,RG_tag,DNS_required,domainname,DNS_Zone_tag,virtualnetwork,VNET_location,addressspace
csvrg1,eastus2,Terraform RG,1,test.sd,Terraform RG,csvvnet1,eastus2,10.0.0.0/16,Terraform VNET,subnet1,10.0.0.0/24
csvrg2,westus,Terraform RG2,0,test2.sd,Terraform RG2,csvvnet2,westus,172.0.0.0/8,Terraform VNET2,subnet1,171.0.0.0/24
you had already the right thought in your mind using the depends_on function. Although, you're using a count inside, which causes from my understanding, that once the first resource[0] is created, Terraform sees the dependency as solved and goes ahead as well.
I found this post with a workaround which you might be able to try:
https://github.com/hashicorp/terraform/issues/15285#issuecomment-447971852
That basically tells us to create a null_resource like in that example:
variable "instance_count" {
default = 0
}
resource "null_resource" "a" {
count = var.instance_count
}
resource "null_resource" "b" {
depends_on = [null_resource.a]
}
In your example, it might look like this:
# Create a resource group if it doesn’t exist
resource "azurerm_resource_group" "myterraformgroup" {
count = length(local.vmcsv)
name = local.vmcsv[count.index].resourcegroup
location = local.vmcsv[count.index].RG_location
tags = {
environment = local.vmcsv[count.index].RG_tag
}
}
# Create a DNS Zone
resource "azurerm_dns_zone" "dnsp-private" {
count = 1
name = local.vmcsv[count.index].domainname
resource_group_name = local.vmcsv[count.index].resourcegroup
depends_on = null_resource.example
tags = {
environment = local.vmcsv[count.index].DNS_Zone_tag
}
}
resource "null_resource" "example" {
...
depends_on = [azurerm_resource_group.myterraformgroup[length(local.vmcsv)]]
}
or depending on your Terraform version (0.12+ which you're using guessing your syntax)
# Create a resource group if it doesn’t exist
resource "azurerm_resource_group" "myterraformgroup" {
count = length(local.vmcsv)
name = local.vmcsv[count.index].resourcegroup
location = local.vmcsv[count.index].RG_location
tags = {
environment = local.vmcsv[count.index].RG_tag
}
}
# Create a DNS Zone
resource "azurerm_dns_zone" "dnsp-private" {
count = 1
name = local.vmcsv[count.index].domainname
resource_group_name = local.vmcsv[count.index].resourcegroup
depends_on = [azurerm_resource_group.myterraformgroup[length(local.vmcsv)]]
tags = {
environment = local.vmcsv[count.index].DNS_Zone_tag
}
}
I hope that helps.
Greetings

Resources