I have created the following resources using Terraform:
aws_athena_database: Amazon Athena database
aws_glue_catalog_table: A CSV table for Athena
It is easy to change my my primary workgroup's default location for query results in the AWS console:
How can I achieve this using Terraform?
I have specified a custom bucket argument for the aws_athena_database to store my query results, which works well if I'm querying outside the Athena console (like Tableau), but if I'm working in the Athena console it defaults to a generic Athena-provided S3 bucket.
You want to us something like this in your terraform, however; this will produce an error because the workgroup already exists, so you will need to use terraform import in order to make terraform add this pre-existing resource to your state file:
terraform import aws_athena_workgroup.primary primary
resource "aws_athena_workgroup" "primary" {
name = "primary"
depends_on = [aws_s3_bucket.my-results-bucket]
configuration {
enforce_workgroup_configuration = false
publish_cloudwatch_metrics_enabled = true
result_configuration {
output_location = "s3://${aws_s3_bucket.my-results-bucket.bucket}/"
encryption_configuration {
encryption_option = "SSE_S3"
}
}
}
}
Related
I have a very frustrating Terraform issue, I made some changes to my terraform script which failed when I applied the plan. I've gone through a bunch of machinations and probably made the situation worse as I ended up manually deleting a bunch of AWS resources in trying to resolve this.
So now I am unable to use Terraform at all (refresh, plan, destroy) all get the same error.
The Situation
I have a list of Fargate services, and a set of maps which correlate different features of the fargate services such as the "Target Group" for the load balancer (I've provided some code below). The problem appears to be that Terraform is not picking up that these resources have been manually deleted or is somehow getting confused because they don't exist. At this point if I run a refresh, plan or destroy I get an error stating that a specific list is empty, even though it isn't (or should not be).
In the failed run I added a new service to the list below along with a new url (see code below)
Objective
At this point I would settle for destroying the entire environment (its my dev environment), however; ideally I want to just get the system working such that Terraform will detect the changes and work properly.
Terraform Script is Valid
I have reverted my Terraform scripts back to the last known good version. I have run the good version against our staging environment and it works fine.
Configuration Info
MacOS Mojave 10.14.6 (18G103)
Terraform v0.12.24.
provider.archive v1.3.0
provider.aws v2.57.0
provider.random v2.2.1
provider.template v2.1.2
The Terraform state file is being stored in a S3 bucket, and terraform init --reconfigure has been called.
What I've done
I was originally getting a similar error but it was in a different location, after many hours Googling and trying stuff (which I didn't write down) I decided to manually remove the AWS resources associated with the problematic code (the ALB, Target Groups, security groups)
Example Terraform Script
Unfortunately I can't post the actual script as it is private, but I've posted what I believe is the pertinent parts but have redacted some info. The reason I mention this is that any syntax type error you might see would be caused by this redaction, as I stated above the script works fine when run in our staging environment.
globalvars.tf
In the root directory. In the case of the failed Terraform run I added a new name to the service_names (edd = "edd") list (I added as the first element). In the service_name_map_2_url I added the new entry (edd = "edd") as the last entry. I'm not sure if the fact that I added these elements in different 'order' is the problem, although it really shouldn't since I access the map via the name and not by index
variable "service_names" {
type = list(string)
description = "This is a list/array of the images/services for the cluster"
default = [
"alert",
"alert-config"
]
}
variable service_name_map_2_url {
type = map(string)
description = "This map contains the base URL used for the service"
default = {
alert = "alert"
alert-config = "alert-config"
}
}
alb.tf
In modules/alb. In this module we create an ALB and then a target group for each service, which looks like this. The items from globalvars.tf are passed into this script
locals {
numberOfServices = length(var.service_names)
}
resource "aws_alb" "orchestration_alb" {
name = "orchestration-alb"
subnets = var.public_subnet_ids
security_groups = [var.alb_sg_id]
tags = {
environment = var.environment
group = var.tag_group_name
app = var.tag_app_name
contact = var.tag_contact_email
}
}
resource "aws_alb_target_group" "orchestration_tg" {
count = local.numberOfServices
name = "${var.service_names[count.index]}-tg"
port = 80
protocol = "HTTP"
vpc_id = var.vpc_id
target_type = "ip"
deregistration_delay = 60
tags = {
environment = var.environment
group = var.tag_group_name
app = var.tag_app_name
contact = var.tag_contact_email
}
health_check {
path = "/${var.service_name_map_2_url[var.service_names[count.index]]}/health"
port = var.app_port
protocol = "HTTP"
healthy_threshold = 2
unhealthy_threshold = 5
interval = 30
timeout = 5
matcher = "200-308"
}
}
output.tf
This is the output of the alb.tf, other things are outputted but this is the one that matters for this issue
output "target_group_arn_suffix" {
value = aws_alb_target_group.orchestration_tg.*.arn_suffix
}
cloudwatch.tf
In modules/cloudwatch. I attempt to create a dashboard
data "template_file" "Dashboard" {
template = file("${path.module}/dashboard.json.template")
vars = {
...
alert-tg = var.target_group_arn_suffix[0]
alert-config-tg = var.target_group_arn_suffix[1]
edd-cluster-name = var.ecs_cluster_name
alb-arn-suffix = var.alb-arn-suffix
}
}
Error
When I run terraform refresh (or plan or destroy) I get the following error (I get the same error for alert-config as well)
Error: Invalid index
on modules/cloudwatch/cloudwatch.tf line 146, in data "template_file" "Dashboard":
146: alert-tg = var.target_group_arn_suffix[0]
|----------------
| var.target_group_arn_suffix is empty list of string
The given key does not identify an element in this collection value.
AWS Environment
I have manually deleted the ALB. Dashboard and all Target Groups. I would expect (and this has worked in the past) that Terraform would detect this and update its state file appropriately such that when running a plan it would know it has to create the ALB and target groups.
Thank you
Terraform trusts its state as the single source of truth. Using Terraform in the presence of manual change is possible, but problematic.
If you manually remove infrastructure, you need to run terraform state rm [resource path] on the manually removed resource.
Gruntwork has what they call The Golden Rule of Terraform:
The master branch of the live repository should be a 1:1 representation of what’s actually deployed in production.
I am using terraform to create a parameter in the AWS Parameter Store.
resource "aws_ssm_parameter" "username" {
name = "username"
type = "SecureString"
value = "to_be_defined"
overwrite = false
}
provider "aws" {
version = "~> 1.53"
}
When I run terraform apply for the first time, if the parameter does not exist terraform creates the parameter. However, if I run it again (usually with a different value) I get the error
ParameterAlreadyExists: The parameter already exists. To overwrite
this value, set the overwrite option in the request to true
If I understand correctly, this is due to the behaviour of AWS Cli (not specific to the provider).
The current behavior for overwrite = false is
If the parameter does not exist, create it
If the parameter exists, throw exception
What I want to achieve is
If the parameter does not exist, create it
If the parameter exists, do nothing
I did not find a way in AWS CLI documentation to achieve the desired behavior.
I would like to know if there is any way to achieve the desired behaviour using terraform (or directly via AWS CLI)
I agree with #ydaetskcoR that you should maintain the value with terraform state as well.
But if you insist to ignore the value to be updated if the SSM key is exist, you can use lifecycle ignore_changes(https://www.terraform.io/docs/configuration/resources.html#ignore_changes)
So in your case, you can update the code to
resource "aws_ssm_parameter" "username" {
name = "username"
type = "SecureString"
value = "to_be_defined"
overwrite = false
lifecycle {
ignore_changes = [
value,
]
}
}
overwrite - (Optional) Overwrite an existing parameter. If not specified, will default to false if the resource has not been created by terraform to avoid overwrite of existing resource and will default to true otherwise (terraform lifecycle rules should then be used to manage the update behavior).
By the way, it is not good design to manage SecureString SSM key/value with terraform, because its tfstate file is not encrypted.
I'm going to use an external data source to create an AWS resource that is not currently supported (an AWS Organizations OU). I'm doing some testing with creating an S3 bucket, because the workflow is similar. I'm using the external data source to call a bash script, and pass the parameters (e,g, bucket name, region).
The script is checking if the S3 bucket exists, and if not, creating it. The script is written to return success etc. if it does already exist.
What I'm seeing is that the script is executed even in the "terraform plan" phase, and there does not appear to be either a stdin argument or environment variable passed to the script in order to indicate the plan/apply/destroy phase. Is this possible to do/deduce ? Ideally the script would be invoked with "plan" and then return if the resource exists already. Then the "apply" phase could then create if necessary.
Is it possible to get the Terraform phase (plan/apply/destroy) in the script ? Or other ways to avoid the script getting invoked fully during the plan stage ?
As a reminder, ultimately I'm not trying to create an S3 bucket (because I know I can do that in Terraform already), I want to create an Organizations OU, but this is an easier workflow to test with...
Example calling code :
data "external" "create_bucket" {
program = ["bash", "create_bucket.sh"]
query = {
region = "ca-central-1"
bucket_name = "mytestbucket"
}
}
Also tried using local-exec :
When I have the following calling code:
variable "bucket_name_list" {
type = "list"
default = ["neilp-dummylist-1","neilp-dummylist-2",
"neilp-dummylist-3"]
}
module "local" {
source = "../local"
bucket_name_list = ["${var.bucket_name_list}"]
bucket_region = "ca-central-1"
}
With the module :
resource "null_resource" "test" {
count = "${length(var.bucket_name_list)}"
provisioner "local-exec" {
command = "aws s3api create-bucket
--bucket
${element(var.bucket_name_list, count.index)}
--create-bucket-configuration
LocationConstraint=${var.bucket_region}"
}
}
The issue here, is that it just tracks the number of entries in the bucket_name_list, and if I change a value but keep the number of entries the same, it does not notice.
I have two plans, in which I am creating two different servers(just for example otherwise it's really complex). In one plan, I am outputing the value of the security group like this:
output "security_group_id" {
value = "${aws_security_group.security_group.id}"
}
I have second plan, in which I want to use that value, how I can achieve it, I have tried couple of things but nothing work for me.
I know how to use the output value return by module but don't know that how I can use the output of one plan to another.
When an output is used in the top-level module of a configuration (the directory where you run terraform plan) its value is recorded in the Terraform state.
In order to use this value from another configuration, the state must be published to a location where it can be read by the other configuration. The usual way to achieve this is to use Remote State.
With remote state enabled for the first configuration, it becomes possible to read the resulting values from the second configuration using the terraform_remote_state data source.
For example, it's possible to keep the state for the first configuration in Amazon S3 by using a backend configuration like the following:
terraform {
backend "s3" {
bucket = "example-s3-bucket"
key = "example-bucket-key"
region = "us-east-1"
}
}
After adding this to the first configuration, Terraform will prompt you to run terraform init to initialize the new backend, which includes migrating the existing state to be stored on S3.
Then in the second configuration this can be retrieved by providing the same configuration to the terraform_remote_state data source:
data "terraform_remote_state" "example" {
backend = "s3"
config {
bucket = "example-s3-bucket"
key = "example-bucket-key"
region = "us-east-1"
}
}
resource "aws_instance" "foo" {
# ...
vpc_security_group_ids = "${data.terraform_remote_state.example.security_group_id}"
}
Note that since the second configuration is reading the state from the first it is necessary to terraform apply the first configuration so that this value will actually be recorded in the state. The second config must be re-applied any time the outputs are changed in the first.
For the local backend the process is same. In first step, we need to declare the following code snippet to publish the state.
terraform {
backend local {
path = "./terraform.tfstate"
}
}
When you execute terraform init and terraform apply command, please observe that in .terraform directory new terraform.tfsate file would be created which contains backend information and tell terraform to use the following tfstate file.
Now in the second configuration we need to use data source to import the outputs by using this code snippet
data "terraform_remote_state" "test" {
backend = "local"
config {
path = "${path.module}/../regionalvpc/terraform.tfstate"
}
}
As part of a setup, I create TLS certs and store them in S3. Creating the certs is done via external data source that runs the command to generate the certs. I then use those outputs to create S3 bucket object resources.
This works very well the first time I run terraform apply. However, if I change any other (non-cert) variable, resource, etc. and rerun, it reruns the external command, which generates a new key/cert pair, uploads them to S3, and breaks everything that already works.
Is there any way to create the resource conditionally? What pattern could I use to make the certs created only if they don't exist?
I did look at storing the generated keys/certs locally, but this is sensitive key material; I do not want it stored in local disk (and there are keys per environment).
Key/cert generation and storage:
data "external" "ca" {
program = ["sh","-c","jq '.root|fromjson' | cfssl gencert -initca -"]
#
query = {root = "${ data.template_file.etcd-ca-csr.rendered }"}
# the result will be saved in
# data.external.etcd-ca.result.key
# data.external.etcd-ca.result.csr
# data.external.etcd-ca.result.cert
}
resource "aws_s3_bucket_object" "ca_cert" {
bucket = "${aws_s3_bucket.my_bucket.id}"
key = "ca.pem"
content = "${data.external.ca.result.cert}"
}
resource "aws_s3_bucket_object" "ca_key" {
bucket = "${aws_s3_bucket.my_bucket.id}"
key = "ca-key.pem"
content = "${data.external.ca.result.key}"
}
Happy to look at using some form of conditional or entirely different generation pattern.
The reason for this behavior is that external is a data source, and thus Terraform expects that it is is read-only and side-effect-free. It re-runs data sources for every plan.
In order to do this via an external script, it would be necessary to use a resource provisioner to run the script and upload it to S3, since there is currently no external equivalent for resources, which are allowed to have side-effects, and provisioners are side-effect-only (that is, they can't produce results to use elsewhere in config.)
Another approach, though, would be to use Terraform's built-in TLS provider, which allows creation of certificates within Terraform itself. In this case it looks like you're trying to create a new CA cert and key, which could be done with tls_self_signed_cert like this:
resource "tls_private_key" "ca" {
algorithm = "RSA"
rsa_bits = 2048
}
resource "tls_self_signed_cert" "ca" {
key_algorithm = "RSA"
private_key_pem = "${tls_private_key.ca.private_key_pem}"
# ... subject and validity settings, as appropriate
is_ca_certificate = true
allowed_uses = ["cert_signing"]
}
resource "aws_s3_bucket_object" "ca_cert" {
bucket = "${aws_s3_bucket.my_bucket.id}"
key = "ca.pem"
content = "${resource.tls_self_signed_cert.ca.cert_pem}"
}
resource "aws_s3_bucket_object" "ca_key" {
bucket = "${aws_s3_bucket.my_bucket.id}"
key = "ca-key.pem"
content = "${resource.tls_self_signed_cert.ca.private_key_pem}"
}
The generated private key will be included in the state for use on future runs, so it's important to ensure that the state is stored securely. Note that this would also be true using the external data source, since data source results are also stored in state. Thus this approach is equivalent from the standpoint of where the secrets get stored.
I wrote more details about using Terraform for TLS certificate management in an article on my website. Its scope is broader than your requirements here, but may be of some interest.