Terraform - Reference for_each from a module output - terraform

I am trying to use Terraform to create multiple storage containers, and then upload multiple blobs to each container.
I have the part of creating multiple containers working, but can't figure out how to reference the for_each output of each container when uploading the blobs.
Storage Container Module (Works)
resource "azurerm_storage_container" "azure" {
for_each = toset(var.storage_containers)
name = each.value
storage_account_name = var.storage_account_name
container_access_type = var.storage_account_container_access_type
}
output "azurerm_storage_container_name" {
value = toset(keys(azurerm_storage_container.azure))
}
Child Module (Works)
module "storage_container" {
source = "C:/TerraformModules/modules/azurerm/azurerm_storage_container"
storage_account_name = module.storage_account.azurerm_storage_account_name
storage_containers = var.STORAGE_CONTAINER_NAMES
tags = var.TAGS
}
Code to upload blob (doesn't work for trying to upload into each container)
**In a variables.tf file**
variable "STORAGE_CONTAINER_DEFAULT_BLOBS" {
description = "The default blobs in each storage container"
type = list(string)
}
**In a vars.auto.tfvars file**
STORAGE_CONTAINER_DEFAULT_BLOBS = ["one", "two", "three"]
**In a main.tf file**
resource "azurerm_storage_blob" "storage_blob" {
for_each = toset(var.STORAGE_CONTAINER_DEFAULT_BLOBS)
name = each.value
storage_account_name = module.storage_account.azurerm_storage_account_name
storage_container_name = module.storage_container[each.value].azurerm_storage_container_name
type = "Block"
source_content = "blob file"
}
If I were to set the container name in storage_container_name, it works and the container gets each blob. But I'm not able to reference the container from the module.
I have this error:
Error: Invalid index
on storage_blobs.tf line 5, in resource "azurerm_storage_blob" "storage_blob":
5: storage_container_name = module.storage_container[each.value].azurerm_storage_container_name
|----------------
| each.value is "two"
| module.storage_container is object with 1 attribute "azurerm_storage_container_name"
The given key does not identify an element in this collection value.
What I need to achieve:
resource "azurerm_storage_blob" "storage_blob" {
for_each = toset(var.STORAGE_CONTAINER_DEFAULT_BLOBS)
name = each.value
storage_account_name = module.storage_account.azurerm_storage_account_name
storage_container_name = # How to reference the storage accounts created with the `storage_container ` module? #
type = "Block"
source_content = "blob file"
}

storage_container_name takes only one value, not a list of values. So if you have 3 STORAGE_CONTAINER_DEFAULT_BLOBS and n number of var.storage_containers you have to iterate n*3 times in azurerm_storage_blob.
Thus, you can try the following with setproduct:
resource "azurerm_storage_blob" "storage_blob" {
for_each = {for idx, val in setproduct(module.storage_container.azurerm_storage_container_name, var.STORAGE_CONTAINER_DEFAULT_BLOBS): idx=>val}
name = each.value[1]
storage_account_name = module.storage_account.azurerm_storage_account_name
storage_container_name = each.value[0]
type = "Block"
source_content = "blob file"
}

Related

Terraform Azurerm: Create blob if not exists

I got Terrafrom code that creates storage account, container and block blob. Is it possible to configure that block blob is created only if it doesn't already exist?
In case of re-running terraform I wouldn't like to replace blob if it is already there as the content might have been manually modified and i would like to keep it.
Any tips? Only alternative I could think of is running powershell/bash script during further deployment steps that would create file if needed, but I am curious if this can be done just with Terraform.
locals {
storage_account_name_teast = format("%s%s", local.main_pw_prefix_short, "teast")
}
resource "azurerm_storage_account" "teaststorage" {
name = local.storage_account_name_teast
resource_group_name = azurerm_resource_group.main.name
location = var.location
account_tier = var.account_tier
account_replication_type = var.account_replication_type
allow_nested_items_to_be_public = false
min_tls_version = "TLS1_2"
network_rules {
default_action = "Deny"
bypass = [
"AzureServices"
]
virtual_network_subnet_ids = []
ip_rules = local.ip_rules
}
tags = var.tags
}
resource "azurerm_storage_container" "teastconfig" {
name = "config"
storage_account_name = azurerm_storage_account.teaststorage.name
container_access_type = "private"
}
resource "azurerm_storage_blob" "teastfeaturetoggle" {
name = "featureToggles.json"
storage_account_name = azurerm_storage_account.teaststorage.name
storage_container_name = azurerm_storage_container.teastconfig.name
type = "Block"
source = "vars-pr-default-toggles.json"
}
After scanning through terraform plan I figured out it was forcing a blob replacement because of:
content_md5 = "9a95db04fb1ff3abcd7ff81fcfb96307" -> null # forces replacement
I added lifecycle hook to blob resource to prevent it:
resource "azurerm_storage_blob" "teastfeaturetoggle" {
name = "featureToggles.json"
storage_account_name = azurerm_storage_account.teaststorage.name
storage_container_name = azurerm_storage_container.teastconfig.name
type = "Block"
source = "vars-pr-default-toggles.json"
lifecycle {
ignore_changes = [
content_md5,
]
}
}

terraform nested for each loop in azure storage account

I would want to create multiple storage acounts and inside each of those sotrage accounts some containers.
If I would want 3 storage account i would always want to create container-a and container-b in those 3 storage accounts
So for example would be. Storage account list ["sa1","sa2","sa3"].
resource "azurerm_storage_account" "storage_account" {
count = length(var.list)
name = var.name
resource_group_name = module.storage-account-resource-group.resource_group_name[0]
location = var.location
account_tier = var.account_tier
account_kind = var.account_kind
then container block
resource "azurerm_storage_container" "container" {
depends_on = [azurerm_storage_account.storage_account]
count = length(var.containers)
name = var.containers[count.index].name
container_access_type = var.containers[count.index].access_type
storage_account_name = azurerm_storage_account.storage_account[0].name
container variables:
variable "containers" {
type = list(object({
name = string
access_type = string
}))
default = []
description = "List of storage account containers."
}
list variable
variable "list" {
type = list(string)
description = "the env to deploy. ['dev','qa','prod']"
This code will create only one container in the first storage account "sa1" but not in the others two "sa2" and "sa3". I read I need to use 2 for each to iterate in both list of storage account and continaers, but not sure how should be the code for it.
It would be better to use for_each:
resource "azurerm_storage_account" "storage_account" {
for_each = toset(var.list)
name = var.name
resource_group_name = module.storage-account-resource-group.resource_group_name[0]
location = var.location
account_tier = var.account_tier
account_kind = var.account_kind
}
then you need an equivalent of a double for loop, which you can get using setproduct:
locals {
flat_list = setproduct(var.list, var.containers)
}
and then you use local.flat_list for containers:
resource "azurerm_storage_container" "container" {
for_each = {for idx, val in local.flat_list: idx => val}
name = each.value.name[1].name
container_access_type = each.value.name[1].access_type
storage_account_name = azurerm_storage_account.storage_account[each.value[0]].name
}
p.s. I haven't run the code, thus it may require some adjustments, but the idea remains valid.

Finding Ways to Merge Resource Tags

Hello Terraform Experts,
I inherited some old Terraform code for deploying resources to Azure. One of the main components that I see in most of the modules is to merge the Resource Group tags with additional tags that go on individual resources. The Resource Group tags are outputs as a map of tags. For example:
output "resource_group_tags_map" {
value = { for r in azurerm_resource_group.this : r.name => r.tags }
description = "map of rg tags."
}
and then a resource like vnets merges the RG tags with additional specific tags for the vnet given the name of the RG in a variable.
# merge Resource Group tags with Tags for VNET
# this is going to break if we change RGs
locals {
tags = merge(var.net_additional_tags, data.azurerm_resource_group.this.tags)
This works just fine if we can set the resource group in a single variable. It assumes that the resource(s) being deployed will go into one RG. However, this is not the case anymore and we somehow need to build in a way for any RG to be chosen when deploying a resource. The code below shows how the original concept works.
locals {
tags = merge(var.net_additional_tags, data.azurerm_resource_group.this.tags)
# - Virtual Network
# -
resource "azurerm_virtual_network" "this" {
for_each = var.virtual_networks
name = each.value["name"]
location = data.azurerm_resource_group.this.location
resource_group_name = var.resource_group_name
address_space = each.value["address_space"]
dns_servers = lookup(each.value, "dns_servers", null)
tags = local.tags
}
looking for help therefore to work around this. Say we create 100 vnets and each one of them goes into a different RG, we couldn't create 100 different resource group variables to capture that as it would become too cumbersome.
Here is my example with Key Vault
resource "azurerm_key_vault" "this" {
for_each = var.key_vaults
name = each.value["name"]
location = each.value["location"]
resource_group_name = each.value["resource_group_name"]
sku_name = each.value["sku_name"]
access_policy = var.access_policies
enabled_for_deployment = each.value["enabled_for_deployment"]
enabled_for_disk_encryption = each.value["enabled_for_disk_encryption"]
enabled_for_template_deployment = each.value["enabled_for_template_deployment"]
enable_rbac_authorization = each.value["enable_rbac_authorization"]
purge_protection_enabled = each.value["purge_protection_enabled"]
soft_delete_retention_days = each.value["soft_delete_retention_days"]
tags = merge(each.value["tags"], )
In the tags argument, we need to somehow merge the tags entered for this instance of Key Vault with the resource group tags that the user chose to place the key vault in. I thought of something like this, but clearly the syntax is wrong.
merge(each.value["tags"], data.azurerm_resource_group[each.key][each.value["resource_group_name"].tags)
Thanks for your input.
UPDATE:
│ Error: Invalid index
│
│ on Modules\keyvault\main.tf line 54, in resource "azurerm_key_vault" "this":
│ 54: tags = merge(each.value["tags"], data.azurerm_resource_group.this["${each.value.resource_group_name}"].tags)
│ ├────────────────
│ │ data.azurerm_resource_group.this is object with 1 attribute "keyvault1"
│ │ each.value.resource_group_name is "Terraform1"
│
│ The given key does not identify an element in this collection value.
Solution code posted below using a map and locals.
SOLUTION
Variables.tf
variable "key_vaults" {
description = "Key Vaults and their properties."
type = map(object({
name = string
location = string
resource_group_name = string
sku_name = string
tenant_id = string
enabled_for_deployment = bool
enabled_for_disk_encryption = bool
enabled_for_template_deployment = bool
enable_rbac_authorization = bool
purge_protection_enabled = bool
soft_delete_retention_days = number
tags = map(string)
}))
default = {}
}
# soft_delete_retention_days numeric value can be between 7 and 90. 90 is default
Main.tf for KeyVault module
data "azurerm_resource_group" "this" {
# read from local variable, index is resource_group_name
for_each = local.rgs_map
name = each.value.name
}
# use data azurerm_client_config to get tenant_id, not from config
data "azurerm_client_config" "current" {}
# -
# - Setup key vault
# - transform variables to locals to make sure the correct index will be used: resource group name and key vault name
locals {
rgs_map = {
for n in var.key_vaults :
n.resource_group_name => {
name = n.resource_group_name
}
}
kvs_map = {
for n in var.key_vaults :
n.name => {
name = n.name
location = n.location
resource_group_name = n.resource_group_name
sku_name = n.sku_name
tenant_id = data.azurerm_client_config.current.tenant_id # n.tenant_id
enabled_for_deployment = n.enabled_for_deployment
enabled_for_disk_encryption = n.enabled_for_disk_encryption
enabled_for_template_deployment = n.enabled_for_template_deployment
enable_rbac_authorization = n.enable_rbac_authorization
purge_protection_enabled = n.purge_protection_enabled
soft_delete_retention_days = n.soft_delete_retention_days
tags = merge(n.tags, data.azurerm_resource_group.this["${n.resource_group_name}"].tags)
}
}
}
resource "azurerm_key_vault" "this" {
for_each = local.kvs_map # use local variable, other wise keyvault1 will be used in stead of kv-eastus2-01 as index
name = each.value["name"]
location = each.value["location"]
resource_group_name = each.value["resource_group_name"]
sku_name = each.value["sku_name"]
tenant_id = each.value["tenant_id"]
enabled_for_deployment = each.value["enabled_for_deployment"]
enabled_for_disk_encryption = each.value["enabled_for_disk_encryption"]
enabled_for_template_deployment = each.value["enabled_for_template_deployment"]
enable_rbac_authorization = each.value["enable_rbac_authorization"]
purge_protection_enabled = each.value["purge_protection_enabled"]
soft_delete_retention_days = each.value["soft_delete_retention_days"]
tags = each.value["tags"]
}

Accessing specific storage account id created using terraform module

I am creating an infrastructure with terraform modules. Some of the common and repeatitive infra are created using module
and other resources are created independently outside of the module. The structure of my code is described as below.
-terraform\module\storage.tf
-terraform\main.tf
-terraform\mlws.tf
This is my code for /module/storage.tf where I am createing a storage account like this
resource "azurerm_storage_account" "storage" {
name = var.storage_account_name
resource_group_name = var.rg_name
location = var.location
account_tier = "Standard"
account_replication_type = "GRS"
min_tls_version = "TLS1_2"
}
module "m1" {
source = "./modules"
storage_account_name = "m1storage"
rg_name = "rg1"
location = "USCentral"
}
module "m2" {
source = "./modules"
storage_account_name = "m2storage"
rg_name = "rg2"
location = "USCentral"
}
module "m3" {
source = "./modules"
storage_account_name = "m3storage"
rg_name = "rg3"
location = "USCentral"
}
resource "azurerm_machine_learning_workspace" "mlws" {
name = "mlws"
location = ""USCentral"
resource_group_name = "mlws-rg1"
application_insights_id = azurerm_application_insights.mlops_appins.id
key_vault_id = data.azurerm_key_vault.kv.id
storage_account_id = **<Mandatory to be filled>**
container_registry_id = azurerm_container_registry.acr.id
identity {
type = "SystemAssigned"
}
depends_on = [
module.m2
]
}
The code for storage account is under \terraform\module\storage.tf, The code for calling the module is under \terraform\main.tf, The code for machine learning workspace is under \terraform\mlws.tf.
Since my mlws.tf code is outside the module but it need to be associated with storage account id created under module m2 in above code.
I am struggling to fetch the id of "m2storage" storage account. Can you please provide solution on how can I access the id of specific storage account created through module and attach it with my code which is outside the module.
This is how it normally works. You run module m2 and it should give output something like this (should include storage_account_id):
output "storage_account_id" {
description = "M2 storage account id."
value = m2.storage_account.storage_account_id
}
Now you have the output and you want to use it you will refer to it as:
resource "azurerm_machine_learning_workspace" "mlws" {
name = "mlws"
location = ""USCentral"
resource_group_name = "mlws-rg1"
application_insights_id = azurerm_application_insights.mlops_appins.id
key_vault_id = data.azurerm_key_vault.kv.id
storage_account_id = module.m2.storage_account_id
container_registry_id = azurerm_container_registry.acr.id
identity {
type = "SystemAssigned"
}
depends_on = [
module.m2
]
}
Let me know if you need more help.

Could not read output attribute from remote state datasource

I am new to terraform so I will attempt to explain with the best of my ability. Terraform will not read in the variable/output from the statefile and use that value in another file.
I have tried searching the internet for everything I could find to see if anyone how has had this problem and how they fixed it.
###vnet.tf
#Remote State pulling data from bastion resource group state
data "terraform_remote_state" "network" {
backend = "azurerm"
config = {
storage_account_name = "terraformstatetracking"
container_name = "bastionresourcegroups"
key = "terraform.terraformstate"
}
}
#creating virtual network and putting that network in resource group created by bastion.tf file
module "quannetwork" {
source = "Azure/network/azurerm"
resource_group_name = "data.terraform_remote_state.network.outputs.quan_netwk"
location = "centralus"
vnet_name = "quan"
address_space = "10.0.0.0/16"
subnet_prefixes = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
subnet_names = ["subnet1", "subnet2", "subnet3"]
tags = {
environment = "quan"
costcenter = "it"
}
}
terraform {
backend "azurerm" {
storage_account_name = "terraformstatetracking"
container_name = "quannetwork"
key = "terraform.terraformstate"
}
}
###resourcegroups.tf
# Create a resource group
#Bastion
resource "azurerm_resource_group" "cm" {
name = "${var.prefix}cm.RG"
location = "${var.location}"
tags = "${var.tags}"
}
#Bastion1
resource "azurerm_resource_group" "network" {
name = "${var.prefix}network.RG"
location = "${var.location}"
tags = "${var.tags}"
}
#bastion2
resource "azurerm_resource_group" "storage" {
name = "${var.prefix}storage.RG"
location = "${var.location}"
tags = "${var.tags}"
}
terraform {
backend "azurerm" {
storage_account_name = "terraformstatetracking"
container_name = "bastionresourcegroups"
key = "terraform.terraformstate"
}
}
###outputs.tf
output "quan_netwk" {
description = "Quan Network Resource Group"
value = "${azurerm_resource_group.network.id}"
}
When running the vnet.tf code it should read in the output from the outputs.tf which is stored in the azure backend storage account statefile file and use that value for the resource_group_name in the quannetwork module. Instead it creates a resource group named data.terraform_remote_state.network.outputs.quan_netwk. Any help would be greatly appreciated.
First, you need to input a string for the resource_group_name in your module quannetwork, not the resource group Id.
Second, if you want to quote something in the remote state, do not just put it in the Double quotes, the right format below:
resource_group_name = "${data.terraform_remote_state.network.outputs.quan_netwk}"

Resources