Terraform variable pointing to current file name - terraform

Is there some special variable available in Terraform configuration files which would point to current file name?
I'd like to use it for description fields in various resources, so that someone seeing these resources in the systems would know where is the master definition for them.
e.g.
in myinfra.tf
resource "aws_iam_policy" "my_policy" {
name = "something-important"
description = "Managed by Terraform at ${HERE_I_WOULD_LIKE_TO_USE_THE_VARIABLE}"
policy = <<EOF
[...]
EOF
}
And I would hope the description becomes:
description = "Managed by Terraform at myinfra.tf"
I tried ${path.module} but that only gives "filesystem path of the module where the expression is placed", so pragmatically speaking - everything but the file name I want.

Here's what I can share. Use the data external resource to call an external script that would get the directory/file name and then return it back as a string or any other type that your resources require. Obviously it's not exactly what you wanted as you'll get the dir/file name indirectly but hopefully it helps for others or even yourself for use-cases.
We use that only for azurerm and for very complex integrations that are not yet supported with the current provider versions. I have have not tested it specifically for AWS but since it's a core Terraform resource provider, I'm guessing it might work across the board.
data "external" "cwd" {
program = ["./script.sh"]
query = {
cwd = "${path.cwd}"
}
}
resource "aws_iam_policy" "my_policy" {
name = "something-important"
description = "Managed by Terraform at ${data.external.dir_script.result.filename}"
policy = <<EOF
[...]
EOF
This is how my script looks like:
#!/bin/sh
#echo '{"cwd":"for_testing"}' | ./dir_name.sh | xargs
PIPED=`cat`
errPrint "INFO: Got PIPED data:\n$PIPED"
DIR=`jq -r .cwd <<< $PIPED`
cd $DIR
filename=`ls | grep \.tf$ | xargs`
errPrint "INFO: Returning this as STDOUT:${filename}"
echo "{\"name\":\"$filename\"}"
You need to be that the data from the script needs to return a valid JSON object.
The program must then produce a valid JSON object on stdout, which will be used to populate the result attribute exported to the rest of the Terraform configuration. This JSON object must again have all of its values as strings. On successful completion it must exit with status zero.
Unfortunately, like the others mentioned, there's no other way to get the current file name being 'applied'.

I think you might benefit from using something like yor from Bridge Crew.
From the project's README:
Yor is an open-source tool that helps add informative and consistent tags across infrastructure-as-code frameworks such as Terraform, CloudFormation, and Serverless.
Yor is built to run as a GitHub Action automatically adding consistent tagging logics to your IaC. Yor can also run as a pre-commit hook and a standalone CLI.
So basically, it updates your resources tags with things like:
tags = {
env = var.env
yor_trace = "912066a1-31a3-4a08-911b-0b06d9eac64e"
git_repo = "example"
git_org = "bridgecrewio"
git_file = "applyTag.md"
git_commit = "COMMITHASH"
git_modifiers = "bana/gandalf"
git_last_modified_at = "2021-01-08 00:00:00"
git_last_modified_by = "bana#bridgecrew.io"
}
Maybe that would be good enough to provide what you're trying to do?
As far as my testimony, I have not used yor since my tagging uses a different approach. Instead of having "raw" tags, we use a label module that builds the tags for us and then merges in local tags.
Just sharing this info FYI in case it helps.

Related

How to design a Terraform resource with multiple sensitive attributes

Context: I'm developing a new resource for my TF Provider.
This foo resource has a name and associated config: a list of key value pairs (both sensitive and non-sensitive).
There're 3 options I've identified:
resource "foo" "option1" {
name = "option1"
config = {
"name" = "option1"
"errors.length" = 3
"tasks.type" = "FOO"
}
config_sensitive = {
"jira.key" = "..."
"credentials.json" = "..."
}
}
resource "foo" "option2" {
name = "option2"
config = {
"name" = "option1"
"errors.length" = 3
"tasks.type" = "FOO"
"jira.key" = "..."
"credentials.json" = "..."
}
}
resource "foo" "option3" {
name = "option3"
config = file("config.json")
}
The advantage of option #3 is it looks very readable but requires a user to store an extra json file (with secrets) in the same folder (I'm not sure how acceptable that setup is). Option #2 looks tempting but foo should accept updates and if we mark the whole block as sensitive (since it may contain secret key-value pairs), the update functionality will suffer (user won't see the expected change). So Option #1 is the winner in my eyes since it's the most explicit one and allows us to distinguish between sensitive and non-sensitive attributes (while allowing updates for non-sensitive ones). Reading from file the whole config is probably not ideal since it doesn't really allow an engineer to see how the config looks like without opening another file.
There's also this weird duplicated name attribute but let's ignore it for now.
What configuration is the most acceptable and used by other TF Providers?
Option #3 should be struck immediately for three reasons:
You cannot realiably use the sensitive flag in the schema struct like you can with 1 and 2.
It requires a JSON format value which is cumbersome to work with unless you are forced into it (e.g. security policies).
Someone could inline the JSON and not store it in a file, which would completely workaround your attempt to obscure the secrets.
Options 1 and 2 are honestly no different from a secrets management perspective. You could apply the sensitive flag to either in the nested schema struct on a per-attribute basis, and use e.g. Vault to pass in values on a KV basis for either.
I would opt for 1 over 2 simply because it appears to me from your question that the arguments and values in the two blocks have no relationship with each other. Therefore, it makes more sense to organize your schema into two separate blocks for code cleanliness purposes.
I will also mention that if it is possible to refactor the credentials.json into your provider, and leverage the JIRA provider for the jira.key, then that would be best practices by both code architecture and security. It is also how the major providers handle this situation.
Terraform providers should handle the credential/auth implementation and the resource handles the resource configuration.
e.g.
resource "jira_issue" "some_story" {
title = "My story"
type = "story"
labels = ["someexampleonstackoverflow","jakewashere"]
}
Notice there's no config that doesn't relate to the thing I'm creating inside the Terraform resource.
It's very acceptable to have some documented convention in your provider that reads credentials from somewhere, whether that's an OS variable, file on disk etc.
For example: The Google Cloud provider, will read an environment variable if it's populated, if not it'll attempt to read either a configuration file that sits inside a hidden directory within $HOME or attempts to read a localhost http metadata server for the credentials.

How to put Dashboards in the right folder dynamically using the Terraform Grafana provider

I have the following use-case: I'm using a combination of the Azure DevOps pipelines and Terraform to synchronize our TAP for Grafana (v7.4). Intention is that we can tweak and tune our dashboards on Test, and push the changes to Acceptance (and Production) via the pipelines.
I've got one pipeline that pulls in the state of the Test environment and writes it to a set of json files (for the dashboards) and a single json array (for the folders).
The second pipeline should use these resources to synchronize the Acceptance environment.
This works flawlessly for the dashboards, but I'm hitting a snag putting the dashboards in the right folder dynamically. Here's my latest working code:
resource "grafana_folder" "folders" {
for_each = toset(var.grafana_folders)
title = each.key
}
resource "grafana_dashboard" "dashboards" {
for_each = fileset(path.module, "../dashboards/*.json")
config_json = file("${path.module}/${each.key}")
}
The folder resources pushes the folders based on a variable list of names that I pass via variables. This generates the folders correctly.
The dashboard resource pushes the dashboards correctly, based on all dashboard files in the specified folder.
But now I'd like to make sure the dashboards end up in the right folder. The provider specifies that I need to do this based on the folder UID, which is generated when the folder is created. So I'd like to take the output from the grafana_folder resource and use it in the grafana_dashboard resource. I'm trying the following:
resource "grafana_folder" "folders" {
for_each = toset(var.grafana_folders)
title = each.key
}
resource "grafana_dashboard" "dashboards" {
for_each = fileset(path.module, "../dashboards/*.json")
config_json = file("${path.module}/${each.key}")
folder = lookup(transpose(grafana_folder.folders), "Station_Details", "Station_Details")
depends_on = [grafana_folder.folders]
}
If I read the Grafana Provider github correctly, the grafana_folder resource should output a map of [uid, title]. So I figured if I transpose that map, and (by way of test) lookup a folder title that I know exists, I can test the concept.
This gives the following error:
on main.tf line 38, in resource "grafana_dashboard" "dashboards":
38: folder = lookup(transpose(grafana_folder.folders),
"Station_Details", "Station_Details")
Invalid value for "default" parameter: the default value must have the
same type as the map elements.
Both Uid and Title should be strings, so I'm obviously overlooking something.
Does anyone have an inkling where I'm going wrong and/or have suggestions on how I can do this (better)?
I think the problem this error is trying to report is that grafana_folder.folders is a map of objects, and so passing it to transpose doesn't really make sense but seems to be succeeding because Terraform has found some clever way to do automatic type conversions to produce some result, but then that result (due to the signature of transpose) is a map of lists rather than a map of strings, and so "Station_Details" (a string, rather than a list) isn't a valid fallback value for that lookup.
My limited familiarity with folders in Grafana leaves me unsure as to what to suggest instead, but I expect the final expression will look something like the following:
folder = grafana_folder.folders[SOMETHING].id
SOMETHING here will be an expression that allows you to know for a given dashboard which folder key it ought to belong to. I'm not seeing an answer to that from what you shared in your question, but just as a placeholder to make this a complete answer I'll suggest that one option would be to make a local map from dashboard filename to folder name:
locals {
# a local value probably isn't actually the right answer
# here, but I'm just showing it as a placeholder for one
# possible way to map from dashboard filename to folder
# name. These names should all be elements of
# var.grafana_folders in order for this to work.
dashboard_folders = {
"example1.json" = "example-folder"
"example2.json" = "example-folder"
"example3.json" = "another-folder"
}
}
resource "grafana_dashboard" "dashboards" {
for_each = fileset("${path.module}/dashboards", "*.json")
config_json = file("${path.module}/dashboards/${each.key}")
folder = grafana_folder.folders[local.dashboard_folders[each.key]].id
}

How to pass multiple template files to user_Data variable in terraform

assign multiple templates files to user_Data variable.
we dont want to merge this template in single file due to some architectural pattern .
I'm new to terraform so struggling on this.
data "template_file" "userdata_lin1" {
template = <<EOF
#!/bin/bash
crontab cronjobfileremote
EOF
}
data "template_file" "userdata_lin2" {
template = <<EOF
#!/bin/bash
echo "hello"
EOF
}
user_data = "${data.template_file.user_data1.rendered}"
It is a fundamental constraint of most cloud platforms that "user data" or "custom metadata" etc (terminology varies by vendor) is a single opaque string of bytes. The interpretation of those bytes depends on what software you have installed in your virtual machine image that makes use of it.
A common choice of such software is cloud-init. If you are using cloud-init then the "user data" can be provided in a number of different formats.
The main way to provide multiple distinct sections to cloud-init is via a MIME-Multipart archive, which is a concatenation of several values interspersed with headers to allow cloud-init to identify the boundaries and understand how you intend each part to be interpreted.
Because cloud-init is a very common choice of software for interpreting "user data", Terraform has a cloudinit provider which includes a data source for constructing a MIME-Multipart archive.
data "cloudinit_config" "example" {
gzip = false
base64_encode = false
part {
content_type = "text/x-shellscript"
filename = "userdata_lin1"
content = <<-EOF
#!/bin/bash
crontab cronjobfileremote
EOT
}
part {
content_type = "text/x-shellscript"
filename = "userdata_lin2"
content = <<-EOF
#!/bin/bash
echo "hello"
EOT
}
}
You can then set your user_data argument to the output from this data source:
user_data = data.cloudinit_config.example.rendered
It's important to note that from the perspective of Terraform and from your cloud compute provider, the content of user_data is just an arbitrary string. Any issues in processing the string must be debugged within the target operating system itself, by reading the cloud-init logs to see how it interpreted the configuration and what happened when it tried to take those actions.
What I've done in the past is to combine two template files in one.
Like this:
data "template_file" "userdata" {
template = "${format("%s%s", file("${path.module}/../common.sh"), file("${path.module}/fo.sh"))}"
vars {
efs_url = "${var.efs_url}"
hostname = "${data.template_file.hostname.rendered}"
api_key = "${var.api_key}"
}
}
As you can see I'm combining two template files:
../common.sh
fo.sh
In my case, this was part of a project with multiple terraform modules common.sh was in the parent folder and contained everything that was needed for all the child modules, then each module added it's own specific requirements.
You can also see that one of the variables for interpolation is also a template_file:
hostname = "${data.template_file.hostname.rendered}"
That is perfectly fine and helps nesting templates. Just in case if you need it...

Terraform doesn't seem to pick up manual changes

I have a very frustrating Terraform issue, I made some changes to my terraform script which failed when I applied the plan. I've gone through a bunch of machinations and probably made the situation worse as I ended up manually deleting a bunch of AWS resources in trying to resolve this.
So now I am unable to use Terraform at all (refresh, plan, destroy) all get the same error.
The Situation
I have a list of Fargate services, and a set of maps which correlate different features of the fargate services such as the "Target Group" for the load balancer (I've provided some code below). The problem appears to be that Terraform is not picking up that these resources have been manually deleted or is somehow getting confused because they don't exist. At this point if I run a refresh, plan or destroy I get an error stating that a specific list is empty, even though it isn't (or should not be).
In the failed run I added a new service to the list below along with a new url (see code below)
Objective
At this point I would settle for destroying the entire environment (its my dev environment), however; ideally I want to just get the system working such that Terraform will detect the changes and work properly.
Terraform Script is Valid
I have reverted my Terraform scripts back to the last known good version. I have run the good version against our staging environment and it works fine.
Configuration Info
MacOS Mojave 10.14.6 (18G103)
Terraform v0.12.24.
provider.archive v1.3.0
provider.aws v2.57.0
provider.random v2.2.1
provider.template v2.1.2
The Terraform state file is being stored in a S3 bucket, and terraform init --reconfigure has been called.
What I've done
I was originally getting a similar error but it was in a different location, after many hours Googling and trying stuff (which I didn't write down) I decided to manually remove the AWS resources associated with the problematic code (the ALB, Target Groups, security groups)
Example Terraform Script
Unfortunately I can't post the actual script as it is private, but I've posted what I believe is the pertinent parts but have redacted some info. The reason I mention this is that any syntax type error you might see would be caused by this redaction, as I stated above the script works fine when run in our staging environment.
globalvars.tf
In the root directory. In the case of the failed Terraform run I added a new name to the service_names (edd = "edd") list (I added as the first element). In the service_name_map_2_url I added the new entry (edd = "edd") as the last entry. I'm not sure if the fact that I added these elements in different 'order' is the problem, although it really shouldn't since I access the map via the name and not by index
variable "service_names" {
type = list(string)
description = "This is a list/array of the images/services for the cluster"
default = [
"alert",
"alert-config"
]
}
variable service_name_map_2_url {
type = map(string)
description = "This map contains the base URL used for the service"
default = {
alert = "alert"
alert-config = "alert-config"
}
}
alb.tf
In modules/alb. In this module we create an ALB and then a target group for each service, which looks like this. The items from globalvars.tf are passed into this script
locals {
numberOfServices = length(var.service_names)
}
resource "aws_alb" "orchestration_alb" {
name = "orchestration-alb"
subnets = var.public_subnet_ids
security_groups = [var.alb_sg_id]
tags = {
environment = var.environment
group = var.tag_group_name
app = var.tag_app_name
contact = var.tag_contact_email
}
}
resource "aws_alb_target_group" "orchestration_tg" {
count = local.numberOfServices
name = "${var.service_names[count.index]}-tg"
port = 80
protocol = "HTTP"
vpc_id = var.vpc_id
target_type = "ip"
deregistration_delay = 60
tags = {
environment = var.environment
group = var.tag_group_name
app = var.tag_app_name
contact = var.tag_contact_email
}
health_check {
path = "/${var.service_name_map_2_url[var.service_names[count.index]]}/health"
port = var.app_port
protocol = "HTTP"
healthy_threshold = 2
unhealthy_threshold = 5
interval = 30
timeout = 5
matcher = "200-308"
}
}
output.tf
This is the output of the alb.tf, other things are outputted but this is the one that matters for this issue
output "target_group_arn_suffix" {
value = aws_alb_target_group.orchestration_tg.*.arn_suffix
}
cloudwatch.tf
In modules/cloudwatch. I attempt to create a dashboard
data "template_file" "Dashboard" {
template = file("${path.module}/dashboard.json.template")
vars = {
...
alert-tg = var.target_group_arn_suffix[0]
alert-config-tg = var.target_group_arn_suffix[1]
edd-cluster-name = var.ecs_cluster_name
alb-arn-suffix = var.alb-arn-suffix
}
}
Error
When I run terraform refresh (or plan or destroy) I get the following error (I get the same error for alert-config as well)
Error: Invalid index
on modules/cloudwatch/cloudwatch.tf line 146, in data "template_file" "Dashboard":
146: alert-tg = var.target_group_arn_suffix[0]
|----------------
| var.target_group_arn_suffix is empty list of string
The given key does not identify an element in this collection value.
AWS Environment
I have manually deleted the ALB. Dashboard and all Target Groups. I would expect (and this has worked in the past) that Terraform would detect this and update its state file appropriately such that when running a plan it would know it has to create the ALB and target groups.
Thank you
Terraform trusts its state as the single source of truth. Using Terraform in the presence of manual change is possible, but problematic.
If you manually remove infrastructure, you need to run terraform state rm [resource path] on the manually removed resource.
Gruntwork has what they call The Golden Rule of Terraform:
The master branch of the live repository should be a 1:1 representation of what’s actually deployed in production.

Terraform: What's the point using Both Data Source and Resource on the same type?

I'm new to Terraform, and I'm working on a project to use Docker/AWS ECR/ECS infrastructure on AWS. I see in this post where the author specify something like
data "aws_ecs_task_definition" "test" {
task_definition = "${aws_ecs_task_definition.test.family}"
depends_on = ["aws_ecs_task_definition.test"]
}
resource "aws_ecs_task_definition" "test" {
family = "test-family"
# ...
}
why is he using both data source AND resource on aws_ecs_task_definition? I can't find an explanation or similar example after hours of digging into the official doc as well as googling articles.
I see later on when he's setting up the service, he uses the following code to reference both of them: (again, I'm not sure what's going on here)
task_definition = "${aws_ecs_task_definition.test.family}:${max("${aws_ecs_task_definition.test.revision}", "${data.aws_ecs_task_definition.test.revision}")}"
I am now confused of what is the difference between using both data & resource on the same type, versus just using resource. Is there any difference in terms of lifecycle?
I'm now trying to create a AWS ECR for my docker image and I want terraform to manage it (create/update/destroy), should I use both data source & resource for the type aws_ecr_repository as well?
It makes sense. The guy is using the data source to get the latest task definition revision. This is because he might be using some other tool(jenkins/circleci) to push changes to the task definition or revision.
Hence, if he will run that code again then terraform should pick up the latest version and update the ecs service accordingly.
Check the below code:
resource "aws_ecs_service" "test-ecs-service" {
name = "test-vz-service"
cluster = "${aws_ecs_cluster.test-ecs-cluster.id}"
task_definition = "${aws_ecs_task_definition.test.family}:${max("${aws_ecs_task_definition.test.revision}", "${data.aws_ecs_task_definition.test.revision}")}"
desired_count = 1
iam_role = "${aws_iam_role.ecs-service-role.name}"
load_balancer {
target_group_arn = "${aws_alb_target_group.test.id}"
container_name = "nginx"
container_port = "80"
}
He is updating the service with the latest revision. He is using MAX function which is returning the maximum value. You may check terraform interpolation syntax, here.
if the task definition does not exist, will this terraform script create it?
Yes, It will create it with respect to the task definition which it has in it state file. If you have created task definition manually then it will increment the revision number.
if task definition exists and the data source block retrieved it, will the resource block re-create another revised task definition, or will it just do nothing?
If there is a change in any of the configuration of the resource then it will create new task definition and that task definition will be allocated to ecs service resource but if there is no change in the resource then it will do nothing.
I'm also unclear if this terraform script is intended to run only once (initial infra creation) or upon change?
This should be run at the time of infra creation or if you wanted to do any other update to task definition resource.

Resources