I am using terraform to upload a file with contents to s3.However, when the content changes, I need to update the s3 file as well. But since the state file stores that the s3 upload was completed, it doesn't upload a new file.
resource "local_file" "timestamp" {
filename = "timestamp"
content = "${timestamp()}"
}
resource "aws_s3_bucket_object" "upload" {
bucket = "bucket"
key = "date"
source = "timestamp"
}
expected:
aws_s3_bucket_object change detected
aws_s3_bucket_object.timestamp Creating...
result:
aws_s3_bucket_object Refreshing state...
When you give Terraform the path to a file rather than the direct content to upload, it is the name of the file that decides whether the resource needs to be updated, rather than the file's contents.
For a short piece of data as shown in your example, the easiest solution is to specify the data directly in the resource configuration:
resource "aws_s3_bucket_object" "upload" {
bucket = "bucket"
key = "date"
content = "${timestamp()}"
}
If your file is actually too large to reasonably load into a string variable, or if it contains raw binary data that cannot be loaded into a string, you can set the etag of the object to an MD5 hash of the content so that the provider can see when the content has changed:
resource "aws_s3_bucket_object" "upload" {
bucket = "bucket"
key = "date"
source = "${path.module}/timestamp"
etag = "${filemd5("${path.module}/timestamp")}"
}
By setting the etag, any change to the content of the file will cause this hash result to change and thus allow the provider to detect that the object needs to be updated.
Related
I have a list of users whom I want to provide read access to an object stored in my GCS Bucket.
I am able to do this task manually by adding one one user, but I want to do this programmatically.
Please guide me if there is any such way to do it.
If you are comfortable with Terraform and it's possible for you to use it, you can use the dedicated resource :
You can configure the users access as a variable in a map :
variables.tf file
variable "users_object_access" {
default = {
user1 = {
entity = "user-user1#gmail.com"
role = "READER"
}
user2 = {
entity = "user-user2#gmail.com"
role = "OWNER"
}
}
}
Then in the Terraform resource, you can use a foreach in your users access list configured previously.
main.tf file :
resource "google_storage_object_access_control" "public_rule" {
for_each = var.users_object_access
object = google_storage_bucket_object.object.output_name
bucket = google_storage_bucket.bucket.name
role = each.value["role"]
entity = each.value["entity"]
}
resource "google_storage_bucket" "bucket" {
name = "static-content-bucket"
location = "US"
}
resource "google_storage_bucket_object" "object" {
name = "public-object"
bucket = google_storage_bucket.bucket.name
source = "../static/img/header-logo.png"
}
If it's to one particular object in a bucket then it sounds like more of an ACL approach.
gsutil will make things easier. You have a couple of options depending on your specific needs. If those users already have authenticated Google accounts then you can use the authenticatedRead predefined ACL:
gsutil acl set authenticatedRead gs://BUCKET_NAME/OBJECT_NAME
This will gives the bucket or object owner OWNER permission, and gives all authenticated Google account holders READER permission.
Or, with ACL enabled, you can retrieve the ACL of that particular object, make some edits to the JSON file, and set the updated ACL back on the object.
Retrieve the ACL of the object:
sutil acl get gs://BUCKET_NAME/OBJECT_NAME > acl.txt
Then make the permission edits by adding the required users/groups, and apply the updated ACL back to the object:
gsutil acl set acl.txt gs://BUCKET_NAME/OBJECT_NAME
You can apply the updated ACL to a particular object, bucket, or pattern (all images, etc).
Summary
The terraform config below creates aws_glue_catalog_database and aws_glue_catalog_table resources, but does not define an s3 bucket output location which is necessary to use these resources in the context of Athena. I can add the s3 output location manually through the AWS console, but need to do it programmatically using terraform.
Detail
Minimal example terraform config which creates an aws glue database and table:
resource "aws_glue_catalog_database" "GlueDB" {
name = "gluedb"
}
resource "aws_glue_catalog_table" "GlueTable" {
name = "gluetable"
database_name = aws_glue_catalog_database.gluedb.name
table_type = "EXTERNAL_TABLE"
parameters = {
EXTERNAL = "TRUE"
}
storage_descriptor {
location = var.GLUE_SOURCE_S3_LOCATION
input_format = "org.apache.hadoop.mapred.TextInputFormat"
output_format = "org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat"
ser_de_info {
name = "jsonserde"
serialization_library = "org.openx.data.jsonserde.JsonSerDe"
parameters = {
"serialization.format" = "1"
}
}
columns {
name = "messageId"
type = "string"
comment = ""
}
}
}
The aim is to be able to access the table either via the Athena query editor (AWS console), or using the python boto3 library (boto3.client('athena')).
However, before Athena access works in either case I need to define an output location for the query. This is easy to do in the AWS console (Amazon Athena -> Query Editor -> Manage settings -> Location of query result), but I need to do this via terraform so that the entire aws infrastructure stack can be setup in one go.
There is a terraform resource called aws_athena_workgroup which has an output_location property, but it is unclear how a separate aws_athena_workgroup resource would be related to the aws_glue_catalog_database already defined (there doesn't seem to be any way to link these two resources).
This answer suggests importing the existing primary workgroup into terraform and modifying it. But what I need is a terraform implementation which sets everything up from scratch in one go.
Any suggestions on how to wire up an s3 output-location in terraform so the above glue resources can be used in the context of Athena would be greatly appreciated!
AWS Glue and Athena are two independent services. Glue doesn't have to know the Athena query output location configuration at all. It just stores the query results run in Athena.
You can simply create a new resources for aws_athena_workgroup next to Glue resources and define the result configuration bucket.
resource "aws_athena_workgroup" "example" {
name = "example"
configuration {
enforce_workgroup_configuration = true
publish_cloudwatch_metrics_enabled = true
result_configuration {
output_location = "s3://${aws_s3_bucket.example.bucket}/output/"
encryption_configuration {
encryption_option = "SSE_KMS"
kms_key_arn = aws_kms_key.example.arn
}
}
}
}
Is there a way in Terraform to check if a resource in Google Cloud exists prior to trying to create it?
I want to check if the following resources below exist in my CircleCI CI/CD pipeline during a job. I have access to terminal commands, bash, and gcloud commands. If the resources do exist, I want to use them. If they do not exist, I want to create them. I am doing this logic in CircleCI's config.yml as steps where I have access to terminal commands and bash. My goal is to create my necessary infrastructure (resources) in GCP when they are needed, otherwise use them if they are created, without getting Terraform errors in my CI/CD builds.
If I try to create a resource that already exists, Terraform apply will result in an error saying something like, "you already own this resource," and now my CI/CD job fails.
Below is pseudo code describing the resources I am trying to get.
resource "google_artifact_registry_repository" "main" {
# this is the repo for hosting my Docker images
# it does not have a data source afaik because it is beta
}
For my google_artifact_registry_repository resource. One approach I have is to do a Terraform apply using a data source block and see if a value is returned. The problem with this is that the google_artifact_registry_repository does not have a data source block. Therefore, I must create this resource once using a resource block and every CI/CD build thereafter can rely on it being there. Is there a work-around to read that it exists?
resource "google_storage_bucket" "bucket" {
# bucket containing the folder below
}
resource "google_storage_bucket_object" "content_folder" {
# folder containing Terraform default.tfstate for my Cloud Run Service
}
For my google_storage_bucket and google_storage_bucket_object resources. If I do a Terraform apply using a data source block to see if these exist, one issue I run into is when the resources are not found, Terraform takes forever to return that status. It would be great if I could determine if a resource exists within like 10-15 seconds or something, and if not assume these resources do not exist.
data "google_storage_bucket" "bucket" {
# bucket containing the folder below
}
output bucket {
value = data.google_storage_bucket.bucket
}
When the resource exists, I can use Terraform output bucket to get that value. If it does not exist, Terraform takes too long to return a response. Any ideas on this?
Thanks to the advice of Marcin, I have a working example of how to solve my problem of checking if a resource exists in GCP using Terraform's external data sources. This is one way that works. I am sure there are other approaches.
I have a CircleCI config.yml where I have a job that uses run commands and bash. From bash, I will init/apply a Terraform script that checks if my resource exists, like so below.
data "external" "get_bucket" {
program = ["bash","gcp.sh"]
query = {
bucket_name = var.bucket_name
}
}
output "bucket" {
value = data.external.get_bucket.result.name
}
Then in my gcp.sh, I use gsutil to get my bucket if it exists.
#!/bin/bash
eval "$(jq -r '#sh "BUCKET_NAME=\(.bucket_name)"')"
bucket=$(gsutil ls gs://$BUCKET_NAME)
if [[ ${#bucket} -gt 0 ]]; then
jq -n --arg name "" '{name:"'$BUCKET_NAME'"}'
else
jq -n --arg name "" '{name:""}'
fi
Then in my CircleCI config.yml, I put it all together.
terraform init
terraform apply -auto-approve -var bucket_name=my-bucket
bucket=$(terraform output bucket)
At this point I check if the bucket name is returned and determine how to proceed based on that.
TF does not have any build in tools for checking if there are pre-existing resources, as this is not what TF is meant to do. However, you can create your own custom data source.
Using the custom data source you can program any logic you want, including checking for pre-existing resources and return that information to TF for future use.
There is a way to check if a resource already exists before creating the resource. But you should be aware of whether it exists. Using this approach, you need to know if the resource exists. If the resource does not exist, it'll give you an error.
I will demonstrate it by create/reading data from an Azure Resource Group. First, create a boolean variable azurerm_create_resource_group. You can set the value to true if you need to create the resource; otherwise, if you just want to read data from an existing resource, you can set it to false.
variable "azurerm_create_resource_group" {
type = bool
}
Next up, get data about the resource using the ternary operator supplying it to count, next do the same for creating the resource:
data "azurerm_resource_group" "rg" {
count = var.azurerm_create_resource_group == false ? 1 : 0
name = var.azurerm_resource_group
}
resource "azurerm_resource_group" "rg" {
count = var.azurerm_create_resource_group ? 1 : 0
name = var.azurerm_resource_group
location = var.azurerm_location
}
The code will create or read data from the resource group based on the value of the var.azurerm_resource_group. Next, combine the data from both the data and resource sections into a locals.
locals {
resource_group_name = element(coalescelist(data.azurerm_resource_group.rg.*.name, azurerm_resource_group.rg.*.name, [""]), 0)
location = element(coalescelist(data.azurerm_resource_group.rg.*.location, azurerm_resource_group.rg.*.location, [""]), 0)
}
Another way of doing it might be using terraformer to import the infra code.
I hope this helps.
This work for me:
Create data
data "gitlab_user" "user" {
for_each = local.users
username = each.value.user_name
}
Create resource
resource "gitlab_user" "user" {
for_each = local.users
name = each.key
username = data.gitlab_user.user[each.key].username != null ? data.gitlab_user.user[each.key].username : split("#", each.value.user_email)[0]
email = each.value.user_email
reset_password = data.gitlab_user.user[each.key].username != null ? false : true
}
P.S.
Variable
variable "users_info" {
type = list(
object(
{
name = string
user_name = string
user_email = string
access_level = string
expires_at = string
group_name = string
}
)
)
description = "List of users and their access to team's groups for newcommers"
}
Locals
locals {
users = { for user in var.users_info : user.name => user }
}
I used terraform import to link an aws_s3_bucket resources with the least parameters.
Since the bucket is in my state, it's allow me to reflect the real resource paramters (the first terraform apply failed, but it's intended).
I have some buckets with acl="private" which gave me errors and invite me to add some grants blocks. When i'm doing it, of course terraform gave me two ConflictWith errors since acl and grants cannont be used together.
But if for example i use a s3 bucket with the proper grants blocks, terraform invite me to add an acl="private" statement.
On the same time, I have a strange behavior with the force_destroy = false block. Which seems to be not detected.
Can somebody help me with me ? maybe i'm doing something wrong.
Thanks.
Code example:
resource "aws_s3_bucket" "s3-bucket-example" {
bucket = "s3-bucket-example"
force_destroy = false
grant {
permissions = [
"READ",
"READ_ACP",
"WRITE",
]
type = "Group"
uri = "http://acs.amazonaws.com/groups/s3/LogDelivery"
}
grant {
id = "xxxxxxxxxxxxxxx"
permissions = [
"FULL_CONTROL",
]
type = "CanonicalUser"
}
}
Result
# aws_s3_bucket.s3-bucket-jolivdi-acces will be updated in-place
~ resource "aws_s3_bucket" "s3-bucket-example" {
+ acl = "private"
+ force_destroy = false
id = "s3-bucket-example"
# (7 unchanged attributes hidden)
# (4 unchanged blocks hidden)
}
Your code is absolutely right and working fine.
I had checked the ACLs for the S3 bucket also. The permissions are perfectly applied as written in the terraform code.
If you still have any problems then please elaborate a bit in a comment.
I'm running Terraform v11.13 with the AWS provider. Is it possible to attach multiple data template files to a single resource?
An example of this is where you have a single aws_iam_policy resouce, but for it to create multiple IAM polices from different data template files.
It works when it is just a single data template file with a count index. It also works when the file is static, as in not a template file.
Here is the code example
variable "policy_list"{
type = "list"
default = ["s3,"emr","lambda"]
}
resource "aws_iam_policy" "many_policies" {
count = "${length(var.policy_list)}"
name = "Policy_${var.policy_list[count.index]}_${var.environment}"
policy = "${file("${path.module}/files/policies/${var.environment}/${var.policy_list[count.index]}.json")}"
}
resource "aws_iam_role_policy_attachment" "many_policies_attachment" {
count = "${length(var.policy_list)}"
role = "${aws_iam_role.iam_roles.*.name[index(var.role_list, "MyRole"))]}"
policy_arn = "${aws_iam_policy.many_policies.*.arn[count.index]}"
}
But what fails is
resource "aws_iam_policy" "many_policies" {
count = "${length(var.policy_list)}"
name = "Policy_${var.policy_list[count.index]}_${var.environment}"
policy = "${data.template_file.${var.policy_list[count.index]}_policy_file.*.rendered[count.index]}"
}
With an error message similar to
parse error expected "}" but found invalid sequence "$"
Any ideas on how this can be achieved?
Based on the errors messages and the suggestion by Matt Schuchard, it's fair to conclude that the data.template_file option does not support interpolation in v11.13