I have a remote state attribute called subnets which is stored in: data.terraform_remote_state.alb.subnets
Depending on what I'm deploying, this attribute either exists or doesn't exist.
When I try to create an ECS cluster, it requires an input of the subnet groups in which I would like to either use:
data.terraform_remote_state.alb.subnets
or
var.vpc_subnets (the subnets of the VPC)
Unfortunately, because of the way the interpolation works, it needed to be hacked together:
"${split(",", length(var.vpc_subnets) == 0 ? join(",",data.terraform_remote_state.alb.subnets) : join(",",var.vpc_subnets))}"
(Refering to: https://github.com/hashicorp/terraform/issues/12453)
However, because Terraform does not seem to 'lazily' evaluate ternary operators, it throws me the following error even if var.vpc_subnets is NOT zero:
Resource 'data.terraform_remote_state.alb' does not have attribute 'subnets' for variable 'data.terraform_remote_state.alb.subnets'
How can I properly handle remote state resources that could be undefined?
EDIT: Typo: Subnet->Subnets
Managed to figure it out.
When using Terraform Remote State, you have the ability to set a default: https://www.terraform.io/docs/providers/terraform/d/remote_state.html
This works in my situation when my data "terraform_remote_state.alb.subnets does not return a value. I can preset the variable to be "" and use locals to do a check for this variable.
Will it be subnet or subnets?
Suppose you have below data source:
data "terraform_remote_state" "alb" {
backend = "s3"
config {
name = "alb"
}
}
You need check the remote state attribute have any outputs with name subnet or not. Or the key name is subnets, you need confirm by yourself.
Related
There is a bug in the current version of the official aws_elasticsearch_domain resource which prevents you from creating new OpenSearch domains (and also changing existing domains).
To circumvent this, I'm working on a module that creates an OpenSearch domain via a CloudFormation stack using the Terraform aws_cloudformation_stack resource.
I want to pass the CloudFormation template via the jsonencode() function, so that I can work in HCL instead of JSON.
Some of the resource parameters are conditional, e.g. CustomEndpoint. You're only allowed to provide a value if CustomEndpointEnabled is set to true. Otherwise the parameter has to be omitted completely.
In Terraform one would usually do this via a conditional statement, like this:
DomainEndpointOptions = {
CustomEndpoint = var.custom_endpoint_enabled ? var.custom_endpoint : null
CustomEndpointCertificateArn = var.custom_endpoint_enabled ? var.custom_endpoint_certificate_arn : null
CustomEndpointEnabled = var.custom_endpoint_enabled
EnforceHTTPS = true
TLSSecurityPolicy = "Policy-Min-TLS-1-2-2019-07"
}
jsonencode() encodes the null value which is the expected behaviour. But the AWS API will throw an error since null is an unexpected value.
So my question is: What is the correct way of omitting conditional values inside jsonencode() blocks?
I have the following use-case: I'm using a combination of the Azure DevOps pipelines and Terraform to synchronize our TAP for Grafana (v7.4). Intention is that we can tweak and tune our dashboards on Test, and push the changes to Acceptance (and Production) via the pipelines.
I've got one pipeline that pulls in the state of the Test environment and writes it to a set of json files (for the dashboards) and a single json array (for the folders).
The second pipeline should use these resources to synchronize the Acceptance environment.
This works flawlessly for the dashboards, but I'm hitting a snag putting the dashboards in the right folder dynamically. Here's my latest working code:
resource "grafana_folder" "folders" {
for_each = toset(var.grafana_folders)
title = each.key
}
resource "grafana_dashboard" "dashboards" {
for_each = fileset(path.module, "../dashboards/*.json")
config_json = file("${path.module}/${each.key}")
}
The folder resources pushes the folders based on a variable list of names that I pass via variables. This generates the folders correctly.
The dashboard resource pushes the dashboards correctly, based on all dashboard files in the specified folder.
But now I'd like to make sure the dashboards end up in the right folder. The provider specifies that I need to do this based on the folder UID, which is generated when the folder is created. So I'd like to take the output from the grafana_folder resource and use it in the grafana_dashboard resource. I'm trying the following:
resource "grafana_folder" "folders" {
for_each = toset(var.grafana_folders)
title = each.key
}
resource "grafana_dashboard" "dashboards" {
for_each = fileset(path.module, "../dashboards/*.json")
config_json = file("${path.module}/${each.key}")
folder = lookup(transpose(grafana_folder.folders), "Station_Details", "Station_Details")
depends_on = [grafana_folder.folders]
}
If I read the Grafana Provider github correctly, the grafana_folder resource should output a map of [uid, title]. So I figured if I transpose that map, and (by way of test) lookup a folder title that I know exists, I can test the concept.
This gives the following error:
on main.tf line 38, in resource "grafana_dashboard" "dashboards":
38: folder = lookup(transpose(grafana_folder.folders),
"Station_Details", "Station_Details")
Invalid value for "default" parameter: the default value must have the
same type as the map elements.
Both Uid and Title should be strings, so I'm obviously overlooking something.
Does anyone have an inkling where I'm going wrong and/or have suggestions on how I can do this (better)?
I think the problem this error is trying to report is that grafana_folder.folders is a map of objects, and so passing it to transpose doesn't really make sense but seems to be succeeding because Terraform has found some clever way to do automatic type conversions to produce some result, but then that result (due to the signature of transpose) is a map of lists rather than a map of strings, and so "Station_Details" (a string, rather than a list) isn't a valid fallback value for that lookup.
My limited familiarity with folders in Grafana leaves me unsure as to what to suggest instead, but I expect the final expression will look something like the following:
folder = grafana_folder.folders[SOMETHING].id
SOMETHING here will be an expression that allows you to know for a given dashboard which folder key it ought to belong to. I'm not seeing an answer to that from what you shared in your question, but just as a placeholder to make this a complete answer I'll suggest that one option would be to make a local map from dashboard filename to folder name:
locals {
# a local value probably isn't actually the right answer
# here, but I'm just showing it as a placeholder for one
# possible way to map from dashboard filename to folder
# name. These names should all be elements of
# var.grafana_folders in order for this to work.
dashboard_folders = {
"example1.json" = "example-folder"
"example2.json" = "example-folder"
"example3.json" = "another-folder"
}
}
resource "grafana_dashboard" "dashboards" {
for_each = fileset("${path.module}/dashboards", "*.json")
config_json = file("${path.module}/dashboards/${each.key}")
folder = grafana_folder.folders[local.dashboard_folders[each.key]].id
}
I have a very frustrating Terraform issue, I made some changes to my terraform script which failed when I applied the plan. I've gone through a bunch of machinations and probably made the situation worse as I ended up manually deleting a bunch of AWS resources in trying to resolve this.
So now I am unable to use Terraform at all (refresh, plan, destroy) all get the same error.
The Situation
I have a list of Fargate services, and a set of maps which correlate different features of the fargate services such as the "Target Group" for the load balancer (I've provided some code below). The problem appears to be that Terraform is not picking up that these resources have been manually deleted or is somehow getting confused because they don't exist. At this point if I run a refresh, plan or destroy I get an error stating that a specific list is empty, even though it isn't (or should not be).
In the failed run I added a new service to the list below along with a new url (see code below)
Objective
At this point I would settle for destroying the entire environment (its my dev environment), however; ideally I want to just get the system working such that Terraform will detect the changes and work properly.
Terraform Script is Valid
I have reverted my Terraform scripts back to the last known good version. I have run the good version against our staging environment and it works fine.
Configuration Info
MacOS Mojave 10.14.6 (18G103)
Terraform v0.12.24.
provider.archive v1.3.0
provider.aws v2.57.0
provider.random v2.2.1
provider.template v2.1.2
The Terraform state file is being stored in a S3 bucket, and terraform init --reconfigure has been called.
What I've done
I was originally getting a similar error but it was in a different location, after many hours Googling and trying stuff (which I didn't write down) I decided to manually remove the AWS resources associated with the problematic code (the ALB, Target Groups, security groups)
Example Terraform Script
Unfortunately I can't post the actual script as it is private, but I've posted what I believe is the pertinent parts but have redacted some info. The reason I mention this is that any syntax type error you might see would be caused by this redaction, as I stated above the script works fine when run in our staging environment.
globalvars.tf
In the root directory. In the case of the failed Terraform run I added a new name to the service_names (edd = "edd") list (I added as the first element). In the service_name_map_2_url I added the new entry (edd = "edd") as the last entry. I'm not sure if the fact that I added these elements in different 'order' is the problem, although it really shouldn't since I access the map via the name and not by index
variable "service_names" {
type = list(string)
description = "This is a list/array of the images/services for the cluster"
default = [
"alert",
"alert-config"
]
}
variable service_name_map_2_url {
type = map(string)
description = "This map contains the base URL used for the service"
default = {
alert = "alert"
alert-config = "alert-config"
}
}
alb.tf
In modules/alb. In this module we create an ALB and then a target group for each service, which looks like this. The items from globalvars.tf are passed into this script
locals {
numberOfServices = length(var.service_names)
}
resource "aws_alb" "orchestration_alb" {
name = "orchestration-alb"
subnets = var.public_subnet_ids
security_groups = [var.alb_sg_id]
tags = {
environment = var.environment
group = var.tag_group_name
app = var.tag_app_name
contact = var.tag_contact_email
}
}
resource "aws_alb_target_group" "orchestration_tg" {
count = local.numberOfServices
name = "${var.service_names[count.index]}-tg"
port = 80
protocol = "HTTP"
vpc_id = var.vpc_id
target_type = "ip"
deregistration_delay = 60
tags = {
environment = var.environment
group = var.tag_group_name
app = var.tag_app_name
contact = var.tag_contact_email
}
health_check {
path = "/${var.service_name_map_2_url[var.service_names[count.index]]}/health"
port = var.app_port
protocol = "HTTP"
healthy_threshold = 2
unhealthy_threshold = 5
interval = 30
timeout = 5
matcher = "200-308"
}
}
output.tf
This is the output of the alb.tf, other things are outputted but this is the one that matters for this issue
output "target_group_arn_suffix" {
value = aws_alb_target_group.orchestration_tg.*.arn_suffix
}
cloudwatch.tf
In modules/cloudwatch. I attempt to create a dashboard
data "template_file" "Dashboard" {
template = file("${path.module}/dashboard.json.template")
vars = {
...
alert-tg = var.target_group_arn_suffix[0]
alert-config-tg = var.target_group_arn_suffix[1]
edd-cluster-name = var.ecs_cluster_name
alb-arn-suffix = var.alb-arn-suffix
}
}
Error
When I run terraform refresh (or plan or destroy) I get the following error (I get the same error for alert-config as well)
Error: Invalid index
on modules/cloudwatch/cloudwatch.tf line 146, in data "template_file" "Dashboard":
146: alert-tg = var.target_group_arn_suffix[0]
|----------------
| var.target_group_arn_suffix is empty list of string
The given key does not identify an element in this collection value.
AWS Environment
I have manually deleted the ALB. Dashboard and all Target Groups. I would expect (and this has worked in the past) that Terraform would detect this and update its state file appropriately such that when running a plan it would know it has to create the ALB and target groups.
Thank you
Terraform trusts its state as the single source of truth. Using Terraform in the presence of manual change is possible, but problematic.
If you manually remove infrastructure, you need to run terraform state rm [resource path] on the manually removed resource.
Gruntwork has what they call The Golden Rule of Terraform:
The master branch of the live repository should be a 1:1 representation of what’s actually deployed in production.
I'm trying to configure a list at the top of my file to list all the SQS resources that should subscribe to a SNS topic. It throws a "resource variables must be three parts: TYPE.NAME.ATTR"
I used locals because it seems they support interpolated values while variables did not.
locals {
update-subscribers = [
"${var.prefix}-${terraform.workspace}-contribution-updates"
]
}
Here is a snippet of my sns topic subscription.
resource "aws_sns_topic_subscription" "subscription" {
count = "${length(locals.update-subscribers.*)}"
topic_arn = "${aws-sns-update-topic.topic.arn}"
protocol = "sqs"
endpoint = "arn:aws:sqs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:${element(locals.update-subscribers, count.index)}"
endpoint_auto_confirms = true
}
It would be nice to be able to use my variable list so I can switch around the workspaces without having any issues on the AWS site. All examples I can find point to a static list of CIDR settings, while I want my list to be based on the interpolated strings. I also tried
locals.contribution-update-subscribers[count.index]
Terraform did not like that either. How should my file be setup to support this or can it be supported?
There are two problems with the configuration given here:
The object name for accessing local values is called local, not locals.
You don't need to (and currently, cannot) use the splat syntax to count the number of elements in what is already a list.
Addressing both of these would give the following configuration, which I think should work:
resource "aws_sns_topic_subscription" "subscription" {
count = "${length(local.update-subscribers)}"
topic_arn = "${aws_sns_update_topic.topic.arn}"
protocol = "sqs"
endpoint = "arn:aws:sqs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:${local.update-subscribers[count.index])}"
endpoint_auto_confirms = true
}
Although dashes are allowed in identifiers in the Terraform language to allow the use of different naming schems in other systems, the idiomatic style is to use underscores for names defined within Terraform itself, such as your local value name update-subscribers.
I have two plans, in which I am creating two different servers(just for example otherwise it's really complex). In one plan, I am outputing the value of the security group like this:
output "security_group_id" {
value = "${aws_security_group.security_group.id}"
}
I have second plan, in which I want to use that value, how I can achieve it, I have tried couple of things but nothing work for me.
I know how to use the output value return by module but don't know that how I can use the output of one plan to another.
When an output is used in the top-level module of a configuration (the directory where you run terraform plan) its value is recorded in the Terraform state.
In order to use this value from another configuration, the state must be published to a location where it can be read by the other configuration. The usual way to achieve this is to use Remote State.
With remote state enabled for the first configuration, it becomes possible to read the resulting values from the second configuration using the terraform_remote_state data source.
For example, it's possible to keep the state for the first configuration in Amazon S3 by using a backend configuration like the following:
terraform {
backend "s3" {
bucket = "example-s3-bucket"
key = "example-bucket-key"
region = "us-east-1"
}
}
After adding this to the first configuration, Terraform will prompt you to run terraform init to initialize the new backend, which includes migrating the existing state to be stored on S3.
Then in the second configuration this can be retrieved by providing the same configuration to the terraform_remote_state data source:
data "terraform_remote_state" "example" {
backend = "s3"
config {
bucket = "example-s3-bucket"
key = "example-bucket-key"
region = "us-east-1"
}
}
resource "aws_instance" "foo" {
# ...
vpc_security_group_ids = "${data.terraform_remote_state.example.security_group_id}"
}
Note that since the second configuration is reading the state from the first it is necessary to terraform apply the first configuration so that this value will actually be recorded in the state. The second config must be re-applied any time the outputs are changed in the first.
For the local backend the process is same. In first step, we need to declare the following code snippet to publish the state.
terraform {
backend local {
path = "./terraform.tfstate"
}
}
When you execute terraform init and terraform apply command, please observe that in .terraform directory new terraform.tfsate file would be created which contains backend information and tell terraform to use the following tfstate file.
Now in the second configuration we need to use data source to import the outputs by using this code snippet
data "terraform_remote_state" "test" {
backend = "local"
config {
path = "${path.module}/../regionalvpc/terraform.tfstate"
}
}