In my module I want to enable or disable a gcp bucket lifecycle_rule based on if a variable is set, so basically I want some buckets to have rule enabled and other buckets ignore the rule setting. But terraform does not have support enabled for a GCP lifecycle_rule that could easily do so (check the AWS link below).
AWS: https://www.terraform.io/docs/providers/aws/r/s3_bucket.html#enabled-1
GCP:
https://www.terraform.io/docs/providers/google/r/storage_bucket.html#lifecycle_rule
I've tried conditionally setting the rule's action type and storage_class to ""
resource "google_storage_bucket" "bucket_ageoff" {
count = "${var.enable_ageoff}"
name = "${local.bucket_name}"
project = "${local.project_id}"
location = "${var.region}"
storage_class = "${local.bucket_storage_class}"
encryption {
default_kms_key_name = "${google_kms_crypto_key.bucket_key.self_link}"
}
logging {
log_bucket = "${google_storage_bucket.log_bucket.name}"
}
labels {
"environment" = "${var.env}"
}
lifecycle_rule {
action {
type = "Delete"
}
condition {
age = "${var.ageoff_days}"
}
}
lifecycle_rule {
action {
type = "${var.coldline_days != "" ? "SetStorageClass" : ""}"
storage_class = "${var.coldline_days != "" ? "COLDLINE" : ""}"
}
condition {
age = "${var.coldline_days != "" ? var.coldline_days : "0"}"
}
}
}
}
but gcp api gives this error:
1 error(s) occurred:
module.log_archive_bucket.google_storage_bucket.bucket_ageoff: 1 error(s) occurred:
google_storage_bucket.bucket_ageoff: googleapi: Error 400: Invalid argument, invalid
Below is what the code actually trying to do:
...
lifecycle_rule {
//enabled is supported by TF for AWS but not GCP
//enabled = "${var.coldline_days != "" ? true : false }"
action {
type = "SetStorageClass"
storage_class = "COLDLINE"
}
condition {
age = "${var.coldline_days != "" ? var.coldline_days : "0" }"
}
}
}
Error: module.log_archive_bucket.google_storage_bucket.bucket_ageoff: lifecycle_rule.1: invalid or unknown key: enabled
Any idea to implement the above requirement is appreciated.
Related
I am trying to define SES rule sets with an order defined by a collection of rules in a variable.
I have tried the solution in https://github.com/hashicorp/terraform-provider-aws/issues/24067 to use the after property of the resource, and It does create the first rule, and fails when creating the second, and all subsequent rules, because the first rule does not exist yet (the one with after=null). I guess it needs some time to finalize. depends_on does not work with dynamic dependencies, as far as i know, so this will not make it either.
If I re-run the apply, then the second rule is created, but all the other rules fail.
my recipients_by_position map is indexed by 0-padded position (i.e. "01", "02", etc):
This is my code
locals {
recipients = {
"mail1-recipient1" = {
short_name = "mail1"
domain = "mail1.domain.com"
recipient = "recipient1"
position = 1
target_bucket = "bucket1"
}
"mail1-recipient2" = {
short_name = "mail1"
domain = "mail1.domain.com"
recipient = "recipient2"
position = 2
target_bucket = "bucket1"
}
"mail2-recipient1" = {
short_name = "mail2"
domain = "mail2.domain.com"
recipient = "recipient1"
position = 3
target_bucket = "bucket2"
}
}
spec_by_domain = {
"mail1.domain.com" = {
irrelevant ={}
}
"mail2.domain.com" = {
irrelevant ={}
}
}
recipients_by_position = {for r in local.recipients: "${format("%02s",r.position)}" => r}
}
resource "aws_ses_domain_identity" "domains" {
for_each = local.spec_by_domain
domain = each.key
}
resource "aws_ses_receipt_rule_set" "main" {
rule_set_name = "new-rules"
}
# store it in S3
resource "aws_ses_receipt_rule" "store" {
for_each = local.recipients_by_position
after = each.value.position == 1 ? null : "${format("%02s",each.value.position - 1)}"
# name = "${each.value.short_name}-store_on_s3-${each.value.recipient}"
name = each.key
rule_set_name = aws_ses_receipt_rule_set.main.rule_set_name
recipients = ["${each.value.recipient}#${each.value.domain}"]
enabled = true
scan_enabled = true
s3_action {
bucket_name = aws_s3_bucket.mailboxes[each.value.domain].bucket
object_key_prefix = each.value.recipient
position = 1
}
}
apply fails with a bunch of
Error: Error creating SES rule: RuleDoesNotExist: Rule does not exist: xx
with xx from 01 to whatever number of rules were defined
Updated with a more illustrative example.
My end goal is to have Terraform create instances of a resource generated with the for_each meta argument in a specific sequence. HCL is known to be a declarative language and when Terraform applies a configuration it can create resources randomly unless you use the depends_on argument or refer from one resource (instance) to another. However, the depends_on argument does not take values that are "calculated", so I don't know how to use it in modules.
For this reason, in order to force Terraform to create instances of a resource in a specific sequence, I decided to try to make the value of a certain argument in an instance it creates "calculated" based on the values of the same argument from another instance.
Below you can find a more practical example based on using one of the providers, but the question is more general and pertains to Terraform as such.
Let's take a test module that instantiates the cloudflare_page_rule resource:
# Module is placed to module\main.tf
terraform {
experiments = [module_variable_optional_attrs]
}
terraform {
required_providers {
cloudflare = {
source = "cloudflare/cloudflare"
version = ">= 3.10.0"
}
}
}
variable "zone" {
type = string
description = "The DNS zone name which will be added, e.g. example.com."
}
variable "page_rules" {
type = list(object({
page_rule_name = string
target = string
actions = object({
forwarding_url = optional(object({
url = string
status_code = number
}))
})
priority = optional(number)
status = optional(string)
depends_on = optional(string)
}))
description = "Zone's page rules."
default = []
}
//noinspection HILUnresolvedReference
locals {
page_rule_dependencies = { for p in var.page_rules : p.page_rule_name => p.depends_on if p.depends_on != null }
}
# https://registry.terraform.io/providers/cloudflare/cloudflare/latest/docs/resources/zone
resource "cloudflare_zone" "this" {
zone = var.zone
}
# https://registry.terraform.io/providers/cloudflare/cloudflare/latest/docs/resources/page_rule
//noinspection HILUnresolvedReference
resource "cloudflare_page_rule" "this" {
for_each = var.page_rules != null ? { for p in var.page_rules : p.page_rule_name => p } : {}
zone_id = cloudflare_zone.this.id
target = each.value.target
actions {
//noinspection HILUnresolvedReference
forwarding_url {
status_code = each.value.actions.forwarding_url.status_code
url = each.value.actions.forwarding_url.url
}
}
priority = each.value.depends_on != null ? cloudflare_page_rule.this[local.page_rule_dependencies[each.key]].priority + 1 : each.value.priority
status = each.value.status
}
output "page_rule_dependencies" {
value = local.page_rule_dependencies
}
And a configuration that is used to create resources:
terraform {
required_version = ">= 0.15.0"
required_providers {
cloudflare = {
source = "cloudflare/cloudflare"
version = ">= 3.10.1"
}
}
}
variable "cloudflare_api_token" {
type = string
sensitive = true
}
provider "cloudflare" {
api_token = var.cloudflare_api_token
}
module "acme_com" {
source = "./module"
zone = "acme.com"
page_rules = [
{
page_rule_name = "page_rule_1"
target = "acme.com/url1"
actions = {
forwarding_url = {
status_code = 301
url = "https://www.example.com/url1"
}
}
priority = 1
},
{
page_rule_name = "page_rule_2"
target = "acme.com/url2"
actions = {
forwarding_url = {
status_code = 301
url = "https://www.example.com/url2"
}
}
priority = 2
depends_on = "page_rule_1"
},
{
page_rule_name = "page_rule_3"
target = "acme.com/url3"
actions = {
forwarding_url = {
status_code = 301
url = "https://www.example.com/url3"
}
}
priority = 3
depends_on = "page_rule_2"
}
]
}
output "page_rule_dependencies" {
value = module.acme_com.page_rule_dependencies
}
In this particular example, I've added the depends_on argument to the page_rules variable (don't confuse this argument with the depends_on meta argument). For the value of the depends_on argument, I specified the name of a page_fule on which another page_fule depends.
Next, I created a local variable page_rule_dependencies, the value of which, after calculations, is the following (you can check this yourself by replacing the priority = each.value.depends_on != null ? cloudflare_page_rule.this[local.page_rule_dependencies[each.key]].priority + 1 : each.value.priority construct with priority = each.value.priority and executing terraform apply):
page_rule_dependencies = {
"page_rule_2" = "page_rule_1"
"page_rule_3" = "page_rule_2"
}
Further, in the priority = each.value.depends_on != null ? cloudflare_page_rule.this[local.page_rule_dependencies[each.key]].priority + 1 : each.value.priority construct, I refer to the values of the local variable, thereby forming a "reference" to the page_fule instance, on which the current instance depends:
When creating page_rule_1, the value of its argument priority = 1.
When creating page_rule_2, the value of its argument priority = cloudflare_page_rule.this["page_rule_1"].priority + 1.
When creating page_rule_3, the value of its argument priority = cloudflare_page_rule.this["page_rule_2"].priority + 1.
However, I get an Error: Cycle: module.acme_com.cloudflare_page_rule.this["page_rule_3"], module.acme_com.cloudflare_page_rule.this["page_rule_2"], module.acme_com.cloudflare_page_rule.this["page_rule_1"] error.
Either I'm doing something wrong, or it's some kind of Terraform limitation/bug. Is there a way to get rid of this error?
P.S. Resulting graph after terraform graph -draw-cycles | dot -Tsvg > graph.svg or terraform graph -draw-cycles -type=plan | dot -Tsvg > graph-plan.svg (the same result):
P.P.S. I use Terraform v1.1.7.
On my root module, I am declaring two modules (paired_regions_network and paired_regions_app), that both iterate a set of regions.
module "paired_regions_network" {
source = "./modules/network"
application_hostname = module.paired_regions_app[each.key].website_hostname
...
for_each = ( var.environment == "TEST" || var.environment == "PROD") ? { region1 = var.paired_regions.region1 } : { }
}
module "paired_regions_app" {
source = "./modules/multi-region"
wag_public_ip = module.paired_regions_network[each.key].wag_public_ip
...
for_each = (var.environment == "TEST" || var.environment == "PROD") ? var.paired_regions : { region1 = var.paired_regions.region1 }
}
output "network_outputs" {
value = module.paired_regions_network
}
output "app_outputs" {
value = module.paired_regions_app
}
The iterated regions are declared as follows:
variable "paired_regions" {
description = "The paired regions"
default = {
region1 = {
...
},
region2 = {
...
}
}
}
From the paired_regions_network module I want to have access to the output coming from the paired_regions_app module, namely the website_hostname value, which I want to assign to the application_hostname parameter, of the paired_regions_network module, as shown above.
output "website_hostname" {
value = azurerm_app_service.was_app.default_site_hostname
description = "The hostname of the website"
}
And from the paired_regions_app module I want to have access to the output coming from the paired_regions_network module, namely the wag_public_ip value, which I want to assign to the parameter with the same name, of the paired_regions_app module, as shown above.
output "wag_ip_address" {
value = azurerm_public_ip.network_ip.ip_address
description = "The Public IP address that will be used by the app gateway"
}
But this causes a dependency cycle, that I can't get rid off. The error is the following:
Error: Cycle: ...
Can I pass the output between the two modules, without causing the dependency cycle?
As per #Marcin's advice, I was able to overcome the issue by creating a third module containing only the Public IP resource. So, the paired_regions_app module would depend on the new module instead of depending on the paired_regions_network module. The paired_regions_network would then depend on both the other two modules. Besides removing the output from the paired_regions_network module, the code changes are as follows:
Root Module
module "paired_regions_ips" {
source = "./modules/public-ip"
...
for_each = ( var.environment == "TEST" || var.environment == "PROD") ? { region1 = var.paired_regions.region1 } : { }
}
module "paired_regions_app" {
source = "./modules/multi-region"
wag_public_ip = length(module.paired_regions_ips) > 0 ? (lookup(module.paired_regions_ips, each.key, "") != "" ? join("/", ["${module.paired_regions_ips[each.key].ip_obj.ip_address}", "32"]) : "" ) : ""
...
for_each = (var.environment == "TEST" || var.environment == "PROD") ? var.paired_regions : { region1 = var.paired_regions.region1 }
}
module "paired_regions_network" {
source = "./modules/network"
wag_public_ip_id = module.paired_regions_ips[each.key].ip_obj.id
application_hostname = module.paired_regions_app[each.key].website_hostname
...
for_each = ( var.environment == "TEST" || var.environment == "PROD") ? { region1 = var.paired_regions.region1 } : { }
}
output "network_outputs" {
value = module.paired_regions_ips
}
output "app_outputs" {
value = module.paired_regions_app
}
The new module
output "ip_obj" {
value = azurerm_public_ip.network_ip
description = "The Public IP address"
}
Some remarks:
Because the paired_regions_ips module has different conditions on the for_each loop when compared to the paired_regions_app module, I had to add some logic when fetching the output from the latter
The new module outputs a public IP object, so that I have access to both its ID (from the paired_regions_network module) and to the IP address (from the paired_regions_app module)
I want to create a dynamic block that will able to dynamically create envs for docker container inside kubernetes using terraform.
I already tried creating a var of list and iterate over the envs but I am getting syntax error
Error: Reference to "count" in non-counted context
on kubernetes/kubernetes.main.tf line 68, in resource "kubernetes_deployment" "kube_deployment":
This is due to usage of count out of resource block.
I am looking now to create multiple envs like this
...
env {
name = "NAME"
value = "VALUE"
}
env {
name = "NAME"
value = "VALUE"
}
.
.
.
is there anyway to create this iteration or any hacks to create dynamic envs in container block. I understand that dynamic blocks are only inside resource, data, provider, and provisioner.
I was previously using helm to do this kind of templating but now I want to fully move to terraform.
I would love any directions to solve such issue.
Thanks
resource "kubernetes_deployment" "kube_deployment" {
metadata {
name = var.deployment_name
labels = {
App = var.deployment_name
}
}
spec {
replicas = 1
selector {
match_labels = {
App = var.deployment_name
}
}
template {
metadata {
labels = {
App = var.deployment_name
}
}
spec {
container {
image = var.container_image
name = var.container_name
env {
name = "NAME"
value = "VALUE"
}
port {
container_port = var.container_port
}
}
}
}
}
}
It was actually possible even if inside nested block of type resource, data, provider, and provisione..
here is a working code
resource "kubernetes_deployment" "kube_deployment" {
metadata {
name = var.deployment_name
labels = {
App = var.deployment_name
}
}
spec {
replicas = 1
selector {
match_labels = {
App = var.deployment_name
}
}
template {
metadata {
labels = {
App = var.deployment_name
}
}
spec {
container {
image = var.container_image
name = var.container_name
dynamic "env" {
for_each = var.envs
content {
name = env.value.name
value = env.value.value
}
}
port {
container_port = var.container_port
}
}
}
}
}
}
I am trying to add redrive policies to existing queues.
I have managed to define a list like this:
variable "sqsq_primary" {
type = "list"
default = [
{
name = "PrimaryQueue1"
maxReceiveCount = -1
deadLetterQueue = ""
},
{
name = "PrimaryQueue2"
maxReceiveCount = 5
deadLetterQueue = "PrimaryQueue2_DL"
},
{
name = "PrimaryQueue3"
maxReceiveCount = 20
deadLetterQueue = "PrimaryQueue3_DL"
}
]
}
I have defined a list of DL queues like this:
variable "sqsq_primary_dl" {
type = "list"
default = [
"PrimaryQueue2_DL",
"PrimaryQueue3_DL"
]
}
In my module I define resources like this:
resource "aws_sqs_queue" "q" {
count = "${length(var.sqsq_primary)}"
name = "${lookup(var.sqsq_primary[count.index], "name")}-${var.environment}"
## Conditionally Sets A Redrive Policy ##
redrive_policy = "${lookup(var.sqsq_primary[count.index], "deadLetterQueue") != "" ? "{\"deadLetterTargetArn\":\"arn:aws:sqs:${var.region}:${var.acc_number}:${lookup(var.sqsq_primary[count.index], "deadLetterQueue")}-${var.environment}\",\"maxReceiveCount\":${lookup(var.sqsq_primary[count.index], "maxReceiveCount")}}" : ""}"
depends_on = ["aws_sqs_queue.qdl"]
}
resource "aws_sqs_queue" "qdl" {
count = "${length(var.sqsq_primary_dl)}"
name = "${element(var.sqsq_primary_dl, count.index)}-${var.environment}"
}
This works. However, I don't like the duplicated information which is the names of the DL queues.
So the question is, how could I get rid of the second list? How could I iterate in the second resource over the first list instead and only create a DL queue if deadLetterQueue != "" ?
Thanks for your help!
I think you may have encountered a limitation of terraform interpolation. Unless you deconstruct your list of maps to separate maps, the best is probably below.
If you keep your definitions for queues with no dl at the bottom and use a static value for minus maths on the dl resource count, the plan stays the same as before.
As a side note, it's dead letter not dead leater.
variable "sqsq_primary" {
type = "list"
default = [
{
name = "PrimaryQueue2"
maxReceiveCount = 5
deadLeaterQueue = "PrimaryQueue2_DL"
},
{
name = "PrimaryQueue3"
maxReceiveCount = 20
deadLeaterQueue = "PrimaryQueue3_DL"
},
{
name = "PrimaryQueue1"
maxReceiveCount = -1
deadLeaterQueue = ""
}
]
}
resource "aws_sqs_queue" "q" {
count = "${length(var.sqsq_primary)}"
name = "${lookup(var.sqsq_primary[count.index], "name")}-${var.environment}"
## Conditionally Sets A Redrive Policy ##
redrive_policy = "${lookup(var.sqsq_primary[count.index], "deadLeaterQueue") != "" ? "{\"deadLetterTargetArn\":\"arn:aws:sqs:${var.region}:${var.acc_number}:${lookup(var.sqsq_primary[count.index], "deadLeaterQueue")}-${var.environment}\",\"maxReceiveCount\":${lookup(var.sqsq_primary[count.index], "maxReceiveCount")}}" : ""}"
depends_on = ["aws_sqs_queue.qdl"]
}
resource "aws_sqs_queue" "qdl" {
count = "${length(var.sqsq_primary) - 1}"
name = "${lookup(var.sqsq_primary[count.index], "deadLeaterQueue")-var.environment}"
}
My colleague has come up with a solution that seems slightly more flexible than the one provided by #henry-dobson.
We have also refactored it so now it doesn't require the deadLeaterQueue value - we conform to a naming standard now, so the resulting names of the DL queues are different from the ones in the question.
variable "sqsq_primary" {
type = "list"
default = [
{
name = "PrimaryQueue1"
maxReceiveCount = 0
},
{
name = "PrimaryQueue2"
maxReceiveCount = 5
},
{
name = "PrimaryQueue3"
maxReceiveCount = 20
}
]
}
data "empty_data_source" "deadletterq" {
count = "${length(var.sqsq_primary)}"
inputs = {
dl = "${lookup(var.sqsq_primary[count.index], "maxReceiveCount", "") > 0 ? "${replace(lookup(var.sqsq_primary[count.index], "name"),"Queue","DeadLetterQueue")}" : ""}"
}
}
resource "aws_sqs_queue" "q" {
count = "${length(var.sqsq_primary)}"
name = "${lookup(var.sqsq_primary[count.index], "name")}-${var.environment}"
## Conditionally Sets A Redrive Policy ##
redrive_policy = "${lookup(var.sqsq_primary[count.index], "maxReceiveCount") > 0 ? "{\"deadLetterTargetArn\":\"arn:aws:sqs:${var.region}:${var.acc_number}:${replace(lookup(var.sqsq_primary[count.index], "name"),"Queue","DeadLetterQueue")}-${var.environment}\",\"maxReceiveCount\":${lookup(var.sqsq_primary[count.index], "maxReceiveCount")}}" : ""}"
depends_on = ["aws_sqs_queue.qdl"]
}
resource "aws_sqs_queue" "qdl" {
count = "${length(compact(data.empty_data_source.deadletterq.*.outputs.dl))}"
name = "${element(compact(data.empty_data_source.deadletterq.*.outputs.dl), count.index)}-${var.environment}"
}