Using Terraform to load a single file from git repo - terraform

We want to load a file from a git repo and place it into a parameter store. The file contains configuration data that is custom to each of several organizational-accounts, which are being constructed with Terraform and are otherwise identical. The data will be stored in AWS SM Parameter Store.
For example the Terraform code to store a string as a parameter is:
resource "aws_ssm_parameter" "parameter_config" {
name = "config_object"
type = "String"
value = "long json string with configuration data"
}
I know there is a file() operator (reference) from Terraform and I know that TF can load files from remote git repos, but I'm not sure if I can bring all this together.

There are a few ways that you can do this.
The first would be to use the github provider with the github_repository_file data source:
terraform {
required_providers {
github = {
source = "integrations/github"
version = "5.12.0"
}
}
}
provider "github" {
token = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
owner = "org-name"
}
data "github_repository_file" "config" {
repository = "my-repo"
branch = "master"
file = "config-file"
}
resource "aws_ssm_parameter" "parameter_config" {
name = "config_object"
type = "String"
value = data.github_repository_file.config.content
}
You could also do this with the http provider:
data "http" "config" {
url = "https://raw.githubusercontent.com/my-org/my-repo/master/config-file"
}
resource "aws_ssm_parameter" "parameter_config" {
name = "config_object"
type = "String"
value = data.http.config.response_body
}
Keep in mind that you may get multiline string delimiters when using the http data source method. (e.g.: <<EOT ... EOT)

Related

Terragrunt dependencies for aws_ssm_parameter

I tried to figure out a good way to use AWS Parameter Store in my Terragrunt config.
I created module secrets to create all my secret parameters:
# secrets/main.tf
resource "aws_ssm_parameter" "postgres_password" {
name = "${var.env}-postgres-password"
description = "The parameter description"
type = "SecureString"
value = var.postgres_password
tags = var.tags
}
With outputs:
# secrets/outputs.tf
output "postgres_password" {
description = "Postgres password"
value = try(aws_ssm_parameter.postgres_password.name, "")
}
I then created the dependency for this module:
# dependency-blocks/secrets.hcl
dependency "secrets" {
config_path = "${get_original_terragrunt_dir()}/../../data-stores/secrets"
mock_outputs = {
postgres_password= "staging-postgres-password"
}
mock_outputs_allowed_terraform_commands = ["validate", "plan", "init"]
}
In my database config, I included the secrets dependency and set it as input for my Database module:
# db/terragrunt.hcl
include "secrets" {
path = "../../../../dependency-blocks/secrets.hcl"
expose = true
merge_strategy = "deep"
}
inputs = {
postgres_password = dependency.secrets.outputs.postgres_password
}
Than I tried to refer to this parameter in db module:
# db/main.tf
data "aws_ssm_parameter" "password" {
name = var.postgres_password
}
When I run terragrunt run-all plan it ends with error:
Error describing SSM parameter (staging-postgres-password): ParameterNotFound: Parameter staging-postgres-password not found.
and I know that's because I mocked only the name, but I don't have ssm parameter resource created yet.
My questions:
how to make this plan command work?
It also led me to the question that maybe creating a module for parameters is not a good idea and I should manage them outside Terraform config?
Could you recommend some of your good practices to work with parameters in Terragrunt/Terraform?

Terraform optional provider for optional resource

I have a module where I want to conditionally create an s3 bucket in another region. I tried something like this:
resource "aws_s3_bucket" "backup" {
count = local.has_backup ? 1 : 0
provider = "aws.backup"
bucket = "${var.bucket_name}-backup"
versioning {
enabled = true
}
}
but it appears that I need to provide the aws.backup provider even if count is 0. Is there any way around this?
NOTE: this wouldn't be a problem if I could use a single provider to create buckets in multiple regions, see https://github.com/terraform-providers/terraform-provider-aws/issues/8853
Based on your description, I understand that you want to create resources using the same "profile", but in a different region.
For that case I would take the following approach:
Create a module file for you s3_bucket_backup, in that file you will build your "backup provider" with variables.
# Module file for s3_bucket_backup
provider "aws" {
region = var.region
profile = var.profile
alias = "backup"
}
variable "profile" {
type = string
description = "AWS profile"
}
variable "region" {
type = string
description = "AWS profile"
}
variable "has_backup" {
type = bool
description = "AWS profile"
}
variable "bucket_name" {
type = string
description = "VPC name"
}
resource "aws_s3_bucket" "backup" {
count = var.has_backup ? 1 : 0
provider = aws.backup
bucket = "${var.bucket_name}-backup"
}
In your main tf file, declare your provider profile using local variables, call the module passing the profile and a different region
# Main tf file
provider "aws" {
region = "us-east-1"
profile = local.profile
}
locals {
profile = "default"
has_backup = false
}
module "s3_backup" {
source = "./module"
profile = local.profile
region = "us-east-2"
has_backup = true
bucket_name = "my-bucket-name"
}
And there you have it, you can now build your s3_bucket_backup using the same "profile" with different regions.
In the case above, the region used by the main file is us-east-1 and the bucket is created on us-east-2.
If you set has_backup to false, it won't create anything.
Since the "backup provider" is build inside the module, your code won't look "dirty" for having multiple providers in the main tf file.

Terraform Resource does not have attribute for variable

Running Terraform 0.11.7 and getting the following error:
module.frontend_cfg.var.web_acl: Resource 'data.terraform_remote_state.waf' does not have attribute 'waf_nonprod_id' for variable 'data.terraform_remote_state.waf.waf_nonprod_id'
Below is the terraform file:
module "frontend_cfg"
{
source = "../../../../modules/s3_fe/developers"
region = "us-east-1"
dev_shortname = "cfg"
web_acl = "${data.terraform_remote_state.waf.waf_nonprod_id}"
}
data "terraform_remote_state" "waf" {
backend = "local"
config = {
name = "../../../global/waf/terraform.tfstate"
}
}
The file which creates the tfstate file referenced above is below. This file has had no issues building.
resource "aws_waf_web_acl" "waf_fe_nonprod"
{
name = "fe_nonprod_waf"
metric_name = "fenonprodwaf"
default_action
{
type = "ALLOW"
}
}
output waf_nonprod_id
{
value = "${aws_waf_web_acl.waf_fe_nonprod.id}"
}
I will spare the full output of the cloudfront file, however, the following covers the text:
resource "aws_cloudfront_distribution" "fe_distribution"
{
web_acl_id = "${var.web_acl}"
}
If I put the ID of the waf ID into the web_acl variable, it works just fine, so I suspect the issue is something to do with the way I am calling data. This appears to match documentation though.
Use path instead of name in terraform_remote_state,
https://www.terraform.io/docs/backends/types/local.html
data "terraform_remote_state" "waf" {
backend = "local"
config = {
path = "../../../global/waf/terraform.tfstate"
}
}
or
data "terraform_remote_state" "waf" {
backend = "local"
config = {
path = "${path.module}/../../../global/waf/terraform.tfstate"
}
}
I tested it with terraform version 0.11.7 and 0.11.14
If you upgrade terraform to version 0.12.x, syntax using remote_state ouput has changed.
So change
web_acl = "${data.terraform_remote_state.waf.waf_nonprod_id}"
to
web_acl = data.terraform_remote_state.waf.outputs.waf_nonprod_id
or
web_acl = "${data.terraform_remote_state.waf.outputs.waf_nonprod_id}"

terraform nested interpolation with count

Using terraform I wish to refer to the content of a list of files (ultimately I want to zip them up using the archive_file provider, but in the context of this post that isn't important). These files all live within the same directory so I have two variables:
variable "source_root_dir" {
type = "string"
description = "Directory containing all the files"
}
variable "source_files" {
type = "list"
description = "List of files to be added to the cloud function. Locations are relative to source_root_dir"
}
I want to use the template data provider to refer to the content of the files. Given the number of files in source_files can vary I need to use a count to carry out the same operation on all of them.
Thanks to the information provided at https://stackoverflow.com/a/43195932/201657 I know that I can refer to the content of a single file like so:
provider "template" {
version = "1.0.0"
}
variable "source_root_dir" {
type = "string"
}
variable "source_file" {
type = "string"
}
data "template_file" "t_file" {
template = "${file("${var.source_root_dir}/${var.source_file}")}"
}
output "myoutput" {
value = "${data.template_file.t_file.rendered}"
}
Notice that that contains nested string interpolations. If I run:
terraform init && terraform apply -var source_file="foo" -var source_root_dir="./mydir"
after creating file mydir/foo of course then this is the output:
Success!
Now I want to combine that nested string interpolation syntax with my count. Hence my terraform project now looks like this:
provider "template" {
version = "1.0.0"
}
variable "source_root_dir" {
type = "string"
description = "Directory containing all the files"
}
variable "source_files" {
type = "list"
description = "List of files to be added to the cloud function. Locations are relative to source_root_dir"
}
data "template_file" "t_file" {
count = "${length(var.source_files)}"
template = "${file("${"${var.source_root_dir}"/"${element("${var.source_files}", count.index)}"}")}"
}
output "myoutput" {
value = "${data.template_file.t_file.*.rendered}"
}
yes it looks complicated but syntactically, its correct (at least I think it is). However, if I run init and apply:
terraform init && terraform apply -var source_files='["foo", "bar"]' -var source_root_dir='mydir'
I get errors:
Error: data.template_file.t_file: 2 error(s) occurred:
* data.template_file.t_file[0]: __builtin_StringToInt: strconv.ParseInt: parsing "mydir": invalid syntax in:
${file("${"${var.source_root_dir}"/"${element("${var.source_files}", count.index)}"}")}
* data.template_file.t_file1: __builtin_StringToInt: strconv.ParseInt: parsing "mydir": invalid syntax in:
${file("${"${var.source_root_dir}"/"${element("${var.source_files}", count.index)}"}")}
My best guess is that its interpreting the / as a division operation hence its attempting to parse the value mydir in source_root_dir as an int.
I've played around with this for ages now and can't figure it out. Can someone figure out how to use nested string interpolations together with a count in order to refer to the content of multiple files using the template provider?
OK, I think I figured it out. formatlist to the rescue
provider "template" {
version = "1.0.0"
}
variable "source_root_dir" {
type = "string"
description = "Directory containing all the files"
}
variable "source_files" {
type = "list"
description = "List of files to be added to the cloud function. Locations are relative to source_root_dir"
}
locals {
fully_qualified_source_files = "${formatlist("%s/%s", var.source_root_dir, var.source_files)}"
}
data "template_file" "t_file" {
count = "${length(var.source_files)}"
template = "${file(element("${local.fully_qualified_source_files}", count.index))}"
}
output "myoutput" {
value = "${data.template_file.t_file.*.rendered}"
}
when applied:
terraform init && terraform apply -var source_files='["foo", "bar"]' -var source_root_dir='mydir'
outputs:
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
myoutput = [
This is the content of foo
,
This is the content of bar
]

Configuring remote state in terrform seems duplicated?

I am configuring remote state in terraform like:
provider "aws" {
region = "ap-southeast-1"
}
terraform {
backend "s3" {
bucket = "xxx-artifacts"
key = "terraform_state.tfstate"
region = "ap-southeast-1"
}
}
data "terraform_remote_state" "s3_state" {
backend = "s3"
config {
bucket = "xxx-artifacts"
key = "terraform_state.tfstate"
region = "ap-southeast-1"
}
}
It seems very duplicated tho, why is it like that? I have the same variables in terraform block and the terraform_remote_state data source block. Is this actually required?
The terraform.backend configuration is for configuring where to store remote state for the Terraform context/directory where Terraform is being ran from.
This allows you to share state between different machines, backup your state and also co-ordinate between usages of a Terraform context via state locking.
The terraform_remote_state data source is, like other data sources, for retrieving data from an external source, in this case a Terraform state file.
This allows you to retrieve information stored in a state file from another Terraform context and use that elsewhere.
For example in one location you might create an aws_elasticsearch_domain but then need to lookup the endpoint of the domain in another context (such as for configuring where to ship logs to). Currently there isn't a data source for ES domains so you would need to either hardcode the endpoint elsewhere or you could look it up with the terraform_remote_state data source like this:
elasticsearch/main.tf
resource "aws_elasticsearch_domain" "example" {
domain_name = "example"
elasticsearch_version = "1.5"
cluster_config {
instance_type = "r4.large.elasticsearch"
}
snapshot_options {
automated_snapshot_start_hour = 23
}
tags = {
Domain = "TestDomain"
}
}
output "es_endpoint" {
value = "$aws_elasticsearch_domain.example.endpoint}"
}
logstash/userdata.sh.tpl
#!/bin/bash
sed -i 's/|ES_DOMAIN|/${es_domain}/' >> /etc/logstash.conf
logstash/main.tf
data "terraform_remote_state" "elasticsearch" {
backend = "s3"
config {
bucket = "xxx-artifacts"
key = "elasticsearch.tfstate"
region = "ap-southeast-1"
}
}
data "template_file" "logstash_config" {
template = "${file("${path.module}/userdata.sh.tpl")}"
vars {
es_domain = "${data.terraform_remote_state.elasticsearch.es_endpoint}"
}
}
resource "aws_instance" "foo" {
# ...
user_data = "${data.template_file.logstash_config.rendered}"
}

Resources