Terraform Import Resources and Looping Over Those Resources - terraform

I am new to Terraform and looking to utilize it for management of Snowflake environment using the provider of "chanzuckerberg/snowflake". I am specifically looking to leverage it for managing an RBAC model for roles within Snowflake.
The scenario is that I have about 60 databases in Snowflake which would equate to a resource for each in Terraform. We will then create 3 roles (reader, writer, all privileges) for each database. We will expand our roles from there.
The first question is, can I leverage map or object variables to define all database names and their attributes and import them using a for_each within a single resource or do I need to create a resource for each database and then import them individually?
The second question is, what would be the best approach for creating the 3 roles per database? Is there a way to iterate over all the resources of type snowflake_database and create the 3 roles? I was imagining the use of modules, variables, and resources based on the research I have done.
Any help in understanding how this can be accomplished would be super helpful. I understand the basics of Terraform but this is a bit of a complex situation for a newbie like myself to visualize enough to implement it. Thanks all!
Update:
This is what my project looks like and the error I am receiving is below it.
variables.tf:
variable "databases" {
type = list(object(
{
name = string
comment = string
retention_days = number
}))
}
databases.auto.tfvars:
databases = [
{
name = "TEST_DB1"
comment = "Testing state."
retention_days = 90
},
{
name = "TEST_DB2"
comment = ""
retention_days = 1
}
]
main.tf:
terraform {
required_providers {
snowflake = {
source = "chanzuckerberg/snowflake"
version = "0.25.25"
}
}
}
provider "snowflake" {
username = "user"
account = "my_account"
region = "my_region"
password = "pwd"
role = "some_role"
}
resource "snowflake_database" "sf_database" {
for_each = { for idx, db in var.databases: idx => db }
name = each.value.name
comment = each.value.comment
data_retention_time_in_days = each.value.retention_days
}
To Import the resource I run:
terraform import snowflake_database.sf_databases["TEST_DB1"]
db_test_db1
I am left with this error:
Error: resource address
"snowflake_database.sf_databases["TEST_DB1"]" does not exist in the
configuration.
Before importing this resource, please create its configuration in the
root module. For example:
resource "snowflake_database" "sf_databases" { # (resource
arguments) }

You should be able to define the databases using for_each and referring to the actual resources with brackets in the import command. Something like:
terraform import snowflake_database.resource_id_using_for_each[foreachkey]
You could then create three snowflake_role and three snowflake_database_grant definitions using for_each over the same map of databases used for the database definitions.

had this exact same problem and in the end the solution was quite simple.
you just need to wrap your import statement within single brackets.
so instead of
terraform import snowflake_database.sf_databases["TEST_DB1"] db_test_db1
do
terraform import 'snowflake_database.sf_databases["TEST_DB1"]' db_test_db1
this took way to long to figure out!

Related

Validating through terraform that a vault policy exists before using it in a group

I have the following structure
module "policies" {
source = "../../../../path/to/my/custom/modules/groups"
for_each = var.config.policies
name = each.key
policy = each.value
}
module "groups" {
source = "../../../../path/to/my/custom/modules/groups"
for_each = var.config.groups
name = each.key
type = each.value.type
policies = each.value.policies
depends_on = [
module.policies
]
}
Policies and groups are declared in a yaml file from which through yamldecode the corresponding variables to for_each are created.
Is there any way to make sure that the policies passed to policies = each.value.policies of the groups module DO exist?
I mean, OK I have the depends_on clause, but I want to also provision for typos in the yaml file and other similar situations.
The usual way to declare a dependency on an external object (managed elsewhere) in Terraform is to use a data block using a data source defined by the provider responsible for that object. If the goal is only to verify that the object exists then it's enough to declare the data source and then have your downstream object's configuration refer to anything about its result, just so Terraform can see that the data source is a dependency and so should be resolved first.
Unfortunately it seems like the hashicorp/vault provider doesn't currently have a data source for declaring a dependency on a policy, although there is a feature request for it.
Assuming that it did exist then the pattern might look something like this:
data "vault_policy" "needed" {
for_each = var.config.policies
name = each.value
}
module "policies" {
source = "../../../../path/to/my/custom/modules/groups"
for_each = var.config.policies
name = each.key
# Accessing this indirectly via the data resource tells
# Terraform that it must complete the data lookup before
# planning anything which depends on this "policy" argument.
policy = data.vault_policy.needed[each.key].name
}
Without a data source for this particular object type I don't think there will be an elegant way to solve this, but you may be able to work around it by using a more general data source like hashicorp/external's external data source for collecting data by running an external program that prints JSON.
Again because you don't actually seem to need any specific data from the policy and only want to check whether it exists, it would be sufficient to write an external program which queries vault and then exists with an unsuccessful status if the request fails, or prints an empty JSON object {} if the request succeeds.
data "external" "vault_policy" {
for_each = var.config.policies
program = ["${path.module}/query-vault"]
query = {
policy_name = each.value
}
}
module "policies" {
source = "../../../../path/to/my/custom/modules/groups"
for_each = var.config.policies
name = each.key
policy = data.external.vault_policy.query.policy_name
}
I'm not familiar enough with Vault to suggest a specific implementation of this query-vault program, but you may be able to use a shell script wrapping the vault CLI program if you follow the advice in Processing JSON in shell scripts. You only need to do the input parsing part of that, because your result would be communicated either by exit 1 to signal failure or echo '{}' followed by exiting successfully to signal success.

How to create multiple resources in a for loop with terraform?

I have looked at several bits of documentation as well as a udemy course on terraform and I do not understand how to do the thing that I want to do. I want to create a for loop and in it I want to create an S3 event notification, create an Sns topic that listens to that notification, create an Sqs queue and then subscribe the queue to the sns topic. It seems like for loops in terraform are not advanced enough to do this. Am I wrong, is there any documentation or examples that explain how to use for loops for this use case?
Thanks in advance.
An example to create AWS VPC subnets then give them to AWS EC2 instances.
resource "aws_subnet" "public" {
count = length(var.public_subnet_cidr_blocks)
vpc_id = var.vpc_id
cidr_block = var.public_subnet_cidr_blocks[count.index]
}
resource "aws_instance" "public_ec2" {
count = length(var.public_subnet_ids)
subnet_id = var.public_subnet_ids[count.index]
ami = var.ami_id
instance_type = "t2.micro"
tags = {
Name = "PublicEC2${count.index}}"
}
provisioner "local-exec" {
command = <<EOF
echo "Public EC2 ${count.index} ID is ${self.id}"
EOF
}
}
There is no syntax like below to create resources.
[ for name in var.names:
aws_s3_bucket {...}
aws_sns_topic {...}
]
For expression is basically for values, not for creating resources.
for Expressions
A for expression creates a complex type value by transforming another complex type value.
To create multiple resources, as below in the document use for_each or count.
for_each: Multiple Resource Instances Defined By a Map, or Set of Strings
By default, a resource block configures one real infrastructure object. However, sometimes you want to manage several similar objects, such as a fixed pool of compute instances. Terraform has two ways to do this: count and for_each.
The for_each example looks like the following:
variable names {
type = list(string)
description = "List of names"
}
resoure "aws_foo" "bar" {
for_each = toset(var.names)
name = each.key
}
This will create N resources (length of names), using the each.key to specify the value to assign to name each time

Terraform updating a one of many ECS service/task

Happy Friday! hoping someone can help me with this issue or point out the flaws in my thinking.
$ terraform --version
Terraform v0.12.7
+ provider.aws v2.25.0
+ provider.template v2.1.2
Preface
This is my first time using Terraform. We have an existing AWS ECS/Fargate environment up and running, this is in a 'test' environment. We recently (e.g. after setting up the test env) started to use Terraform for IaC purposes.
Current Config
The environment has a single ECS cluster, we're using FARGATE but I'm not sure that matters for this question. The cluster has several services, each service has a single task (docker image) associated with it - so they can be individually scaled. Each docker image has its own Repo.
What I'm trying to do
So with Terraform I was hoping to be able to create, update and destroy the environment. Creating/destroying seems fairly straight forward, however; I'm hitting a road-block for updating.
As I said each task has its own repo, when a pull request is made against the repo our CI platform (CircleCI if that matters) builds the new docker image, tags it and pushes it. Then we use an API call to trigger a build of the Terraform Repo passing the name of the service/task that was updated.
Problem
The problem we're facing is that when going through the services (described below) I can't figure out how to get Terraform to either ignore the services that are not being updated, or how I can provide the correct container_definitions in the aws_ecs_task_definition, specifically the current image tag (we don't use the latest tag). So I'm trying to figure out how I can get the latest container information (tag) or just tell Terraform to skip the unmodified task.
Terraform Script
Here is a stripped down version of what I have tried, this is in a module called ecs.tf, the var.ecs_svc_names is a list of the service names. I have removed some elements as I don't think they pertain to this issue and having them makes this very large.
CAVEATS
I have not run the Terraform 'script' as shown below due to the issues I am asking about, so my syntax maybe a off. Sorry if that is the case, hopefully this will show you what I'm trying to do....
ecs.tf
/* ecs_service_names is passed in, but here is its definition:
variable "ecs_service_names" {
type = list(string)
description = "This is a list/array of the images/services that are contained in the cluster"
default = [
"main",
"sub-service1",
"sub-service2"]
}
*/
locals {
numberOfServices = length(var.ecs_svc_names)
}
resource "aws_ecs_cluster" "ecs_cluster" {
name = "${var.env_type}-ecs-cluster"
}
// Create the service objects (1 for each service)
resource "aws_ecs_service" "ecs-service" {
// How many services are being created
count = local.numberOfServices
name = var.ecs_svc_names[count.index]
cluster = aws_ecs_cluster.ecs_cluster.id
definition[count.index].family}:${max(aws_ecs_task_definition.ecs-task-definition[count.index].revision, data.aws_ecs_task_definition.ecs-task-def.revision)}"
desired_count = 1
launch_type = "FARGATE"
// stuff removed
}
resource "aws_ecs_task_definition" "ecs-task-definition" {
// How many tasks. There is a 1-1 relationship between tasks and services
count = local.numberOfServices
family = var.ecs_svc_names[count.index]
network_mode = "awsvpc"
requires_compatibilities = ["FARGATE"]
// cpu/memory stuff removed
task_role_arn = var.ecs_task_role_arn
container_definitions = data.template_file.ecs_containers_json[count.index].rendered
}
data.tf
locals {
numberOfServices = length(var.ecs_svc_names)
}
data "aws_ecs_task_definition" "ecs-task-def" {
// How many services are being created, 1-1 relationship between tasks and services
count = local.numberOfServices
task_definition = aws_ecs_task_definition.ecs-task-definition[count.index].family
depends_on = [
"aws_ecs_task_definition.ecs-task-definition",
]
}
data "template_file" "ecs_containers_json" {
// How many tasks. There is a 1-1 relationship between tasks and services
count = local.numberOfServices
template = file("${path.module}/container.json.template")
vars = {
// vars removed
image = aws_ecs_task_definition.ecs-task-definition[count.index].family
// This is where I hit the road-block, how do I get the current docker tag from Terraform?
tag = var.ecs_svc_name_2_update == var.ecs_svc_names[count.index]
? var.ecs_svc_image_tag
: data.aws_ecs_task_definition.ecs-task-def[count.index].
}
I didn't post the JSON document, if you need that I can provide it...
Thank you
It is necessary to pass the updated image attribute in the container definition of the task definition revision.
You can data source the container definition of the current task revision which is used by the service and pass it to the terraform. You may follow the code below.
data "template_file" "example" {
template = "${file("${path.module}/example.json")}"
vars {
image = "${data.aws_ecs_container_definition.example.image}"
}
}
resource "aws_ecs_task_definition" "example" {
family = "${var.project_name}-${var.environment_name}-example"
container_definitions = "${data.template_file.example.rendered}"
cpu = 192
memory = 512
}
data "aws_ecs_container_definition" "example" {
task_definition = "${var.project_name}-${var.environment_name}-example"
container_name = "example"
}

Iterate over map with lists in terraform 0.12

I am using terraform 0.12.8 and I am trying to write a resource which would iterate over the following variable structure:
variable "applications" {
type = map(string)
default = {
"app1" = "test,dev,prod"
"app2" = "dev,prod"
}
}
My resource:
resource "aws_iam_user" "custom" {
for_each = var.applications
name = "circleci-${var.tags["ServiceType"]}-user-${var.tags["Environment"]}-${each.key}"
path = "/"
}
So, I can iterate over my map. However, I can't figure out how to verify that var.tags["Environment"] is enabled for specific app e.g. app1.
Basically, I want to ensure that the resource is created for each application as long as the Environment variable is in the list referencing app name in the applications map.
Could someone help me out here?
Please note that I am happy to go with a different variable structure if you have something to propose that would accomplish my goal.

Does Terraform execute parallely in multi-cloud deployments?

I want to create a VMs in different cloud provider from a single Terraform script e.g. GCP, AWS, Azure using Terraform.
So, I wanted to know that, will Terraform make the VM instances in parallel in all public clouds?
Terraform builds a directed, acyclical graphic (also referred to as a DAG) to understand the dependencies between things. If something isn't dependent on something else then it will execute it in parallel up to the number specified by the -parallelism flag which defaults to 10.
If things are completely separate across multiple providers (you're just creating the same stack in n cloud providers) then it will be comfortably parallel across those stacks.
However, I'd recommend against applying multiple environments/cloud providers at the same time like this because of blast radius issues and in general erring towards minimising how much changes in one operation.
If you have cross provider dependencies then Terraform is great for handling this but it still relies on building that DAG so it can understand your dependencies.
For example you might want to create an instance in GCP and use DNS to resolve the IP address but use AWS' Route53 for all your DNS. For this you could use something like this:
resource "google_compute_instance" "test" {
name = "test"
machine_type = "n1-standard-1"
zone = "us-central1-a"
tags = ["foo", "bar"]
boot_disk {
initialize_params {
image = "debian-cloud/debian-9"
}
}
// Local SSD disk
scratch_disk {
}
network_interface {
network = "default"
access_config {
// Ephemeral IP
}
}
metadata = {
foo = "bar"
}
metadata_startup_script = "echo hi > /test.txt"
service_account {
scopes = ["userinfo-email", "compute-ro", "storage-ro"]
}
}
data "aws_route53_zone" "example" {
name = "example.com."
}
resource "aws_route53_record" "www" {
zone_id = "${data.aws_route53_zone.example.zone_id}"
name = "www.${data.aws_route53_zone.example.name}"
type = "A"
ttl = "300"
records = ["${google_compute_instance.test.network_interface.0.access_config.0.nat_ip}"]
}
This would build a graph that has the aws_route53_record.www depending on both the data.aws_route53_zone.example data source and also the google_compute_instance.test resource so Terraform knows that both of these need to complete before it can start work on the Route53 record.

Resources