I am trying to use etag when i update my bucket S3, but i get this error:
Error: Error in function call
on config.tf line 48, in resource "aws_s3_bucket_object" "bucket_app":
48: etag = filemd5("${path.module}/${var.env}/app-config.json")
|----------------
| path.module is "."
| var.env is "develop"
Call to function "filemd5" failed: no file exists at develop/app-config.json.
However, this works fine:
resource "aws_s3_bucket_object" "bucket_app" {
bucket = "${var.app}-${var.env}-app-assets"
key = "config.json"
source = "${path.module}/${var.env}/app-config.json"
// etag = filemd5("${path.module}/${var.env}/app-config.json")
depends_on = [
local_file.app_config_json
]
}
I am genereting the file this way:
resource "local_file" "app_config_json" {
content = local.app_config_json
filename = "${path.module}/${var.env}/app-config.json"
}
I really don't get what i am doing wrong...
If you happen to arrive here and are using an archive_file Data Source, there is an exported attribute called output_md5. This seems to provide the same results that you would get from filemd5(data.archive_file.app_config_json.output_path).
Here is a full example:
data archive_file config {
type = "zip"
output_path = "${path.module}/config.zip"
source {
filename = "config/template-configuration.json"
content = "some content"
}
}
resource aws_s3_bucket_object config{
bucket = aws_s3_bucket.stacks.bucket
key = "config.zip"
content_type = "application/zip"
source = data.archive_file.config.output_path
etag = data.archive_file.config.output_md5
}
All functions in Terraform run during the initial configuration processing, not during the graph walk. For all of the functions that read files on disk, that means that the files must be present on disk prior to running Terraform as part of the configuration itself -- usually, checked in to version control -- rather than being generated dynamically during the Terraform operation.
The documentation for [file], which filemd5 builds on, has the following to say about it:
This function can be used only with files that already exist on disk at the beginning of a Terraform run. Functions do not participate in the dependency graph, so this function cannot be used with files that are generated dynamically during a Terraform operation. We do not recommend using dynamic local files in Terraform configurations, but in rare situations where this is necessary you can use the local_file data source to read files while respecting resource dependencies.
As the documentation there suggests, the local_file data source provides a way to read a file into memory as a resource during the graph walk, although its result would still need to be passed to md5 to get the result you needed here.
Because you're creating the file with a local_file resource anyway, you can skip the need for the additional data resource and derive the MD5 hash directly from your local_file.app_config_json resource:
resource "aws_s3_bucket_object" "bucket_app" {
bucket = "${var.app}-${var.env}-app-assets"
key = "config.json"
source = local_file.app_config_json.filename
etag = md5(local_file.app_config_json.content)
}
Note that we don't need to use depends_on if we derive the configuration from attributes of the local_file.app_config_json resource, because Terraform can therefore already see that the dependency relationship exists.
Related
Just an example scenario would be that I have a module, and that has variables. For example an S3 bucket would need a variable about the name of the bucket.
The variables.tf file would look like:
variable "bucket_name" { type = string }
Now if I want to use that module to create the S3 bucket, and then assign it to a cloudfront distribution, I have to assign the bucket name at that level to the bucket module. For example I want to give the distribution a name, I need to create a variables.tf for the distribution with the distribution name defined, but because I use the module of the S3 bucket, I have to include all variables of the S3 bucket as well. So the variables.tf file of the distribution would look like:
variable "bucket_name" { type = string }
variable "distribution_name" { type = string }
If I have a module that is only for us-east-1, and it needs the cloudfront distribution, I need to create a variables.tf file for that as well containing all the above.
If I for example have multiple AWS accounts for staging, and prod, and want the distribution included in both (or for only one, doesn't matter), I have to create yet another variables.tf file containing all the above.
Every time I would add a new variable, I'd have to update at least 4 variables.tf files which is definitely not a way (terraform is supposed to reduce hard coding).
What I would like is as I go up the ladder (from s3 to environment module), Import the variables.tf file of the "child module", and extend it.
Just for an imaginary scenario, I'd like to:
# bucket/variables.tf
variable "bucket_name" { type = string }
# distribution/variables.tf
import_variables {
source = "../bucket"
}
# us-east-1/variables.tf
import_variables {
source = "../distribution"
}
# staging/us-east-1/variables.tf
import_variables {
source = "../../us-east-1"
}
Am I having a completely wrong approach to terraform, or I just missed a method with which this variable definition sharing could be done?
I have the following structure
module "policies" {
source = "../../../../path/to/my/custom/modules/groups"
for_each = var.config.policies
name = each.key
policy = each.value
}
module "groups" {
source = "../../../../path/to/my/custom/modules/groups"
for_each = var.config.groups
name = each.key
type = each.value.type
policies = each.value.policies
depends_on = [
module.policies
]
}
Policies and groups are declared in a yaml file from which through yamldecode the corresponding variables to for_each are created.
Is there any way to make sure that the policies passed to policies = each.value.policies of the groups module DO exist?
I mean, OK I have the depends_on clause, but I want to also provision for typos in the yaml file and other similar situations.
The usual way to declare a dependency on an external object (managed elsewhere) in Terraform is to use a data block using a data source defined by the provider responsible for that object. If the goal is only to verify that the object exists then it's enough to declare the data source and then have your downstream object's configuration refer to anything about its result, just so Terraform can see that the data source is a dependency and so should be resolved first.
Unfortunately it seems like the hashicorp/vault provider doesn't currently have a data source for declaring a dependency on a policy, although there is a feature request for it.
Assuming that it did exist then the pattern might look something like this:
data "vault_policy" "needed" {
for_each = var.config.policies
name = each.value
}
module "policies" {
source = "../../../../path/to/my/custom/modules/groups"
for_each = var.config.policies
name = each.key
# Accessing this indirectly via the data resource tells
# Terraform that it must complete the data lookup before
# planning anything which depends on this "policy" argument.
policy = data.vault_policy.needed[each.key].name
}
Without a data source for this particular object type I don't think there will be an elegant way to solve this, but you may be able to work around it by using a more general data source like hashicorp/external's external data source for collecting data by running an external program that prints JSON.
Again because you don't actually seem to need any specific data from the policy and only want to check whether it exists, it would be sufficient to write an external program which queries vault and then exists with an unsuccessful status if the request fails, or prints an empty JSON object {} if the request succeeds.
data "external" "vault_policy" {
for_each = var.config.policies
program = ["${path.module}/query-vault"]
query = {
policy_name = each.value
}
}
module "policies" {
source = "../../../../path/to/my/custom/modules/groups"
for_each = var.config.policies
name = each.key
policy = data.external.vault_policy.query.policy_name
}
I'm not familiar enough with Vault to suggest a specific implementation of this query-vault program, but you may be able to use a shell script wrapping the vault CLI program if you follow the advice in Processing JSON in shell scripts. You only need to do the input parsing part of that, because your result would be communicated either by exit 1 to signal failure or echo '{}' followed by exiting successfully to signal success.
New to terraform, and have been building out the infrastructure recently.
I am trying to pull secrets from azure key vault and assign the keys to the variables.tf file depending on the environment(dev.tfvars, test.tfvars, etc). However when I execute the plan with the tfvar file as the parameter, I get an error with the following message:
Error: Variables not allowed
Here are the files and the relevant contents of it.
variables.tf:
variable "user_name" {
type = string
sensitive = true
}
data.tf (referencing the azure key vault):
data "azurerm_key_vault" "test" {
name = var.key_vault_name
resource_group_name = var.resource_group
}
data "azurerm_key_vault_secret" "test" {
name = "my-key-vault-key-name"
key_vault_id = data.azurerm_key_vault.test.id
}
test.tfvars:
user_name = "${data.azurerm_key_vault_secret.test.value}" # Where the error occurrs
Can anyone point out what I'm doing wrong here? And if so is there another way to achieve such a thing?
In Terraform a variable can be used for user input only. You can not assign to them anything dynamically computed from your code. They are like read-only arguments, for more info see Input Variables from the doc.
If you want to assign a value to something for later use, you must use locals. For example:
locals {
user_name = data.azurerm_key_vault_secret.test.value
}
Local values can be changed dynamically during execution. For more info, see Local Values.
You can't create dynamic variables. All variables must have known values before execution of your code. The only thing you could do is to use local, instead of variabile:
locals {
user_name = data.azurerm_key_vault_secret.test.value
}
and then refer to it as local.user_name.
I want to install a CRD with terraform, I was hoping it would be easy as doing this:
data "http" "crd" {
url = "https://raw.githubusercontent.com/kubernetes-sigs/application/master/deploy/kube-app-manager-aio.yaml"
request_headers = {
Accept = "text/plain"
}
}
resource "kubernetes_manifest" "install-crd" {
manifest = data.http.crd.body
}
But I get this error:
can't unmarshal tftypes.String into *map[string]tftypes.Value, expected
map[string]tftypes.Value
Trying to convert it to yaml with yamldecode also doesn't work because yamldecode doesn't support multi-doc yaml files.
I could use exec, but I was already doing that while waiting for the kubernetes_manifest resource to be released. Does kubernetes_manifest only support a single resource or can it be used to create several from a raw text manifest file?
kubernetes_manifest (emphasis mine)
Represents one Kubernetes resource by supplying a manifest attribute
That sounds to me like it does not support multiple resources / a multi doc yaml file.
However you can manually split the incoming document and yamldecode the parts of it:
locals {
yamls = [for data in split("---", data.http.crd.body): yamldecode(data)]
}
resource "kubernetes_manifest" "install-crd" {
count = length(local.yamls)
manifest = local.yamls[count.index]
}
Unfortunately on my machine this then complains about
'status' attribute key is not allowed in manifest configuration
for exactly one of the 11 manifests.
And since I have no clue of kubernetes I have no idea what that means or wether or not it needs fixing.
Alternatively you can always use a null_resource with a script that fetches the yaml document and uses bash tools or python or whatever is installed to convert and split and filter the incoming yaml.
I got this to work using kubectl provider. Eventually kubernetes_manifest should work as well, but it is currently (v2.5.0) still beta and has some bugs. This example only uses kind+name, but for full uniqueness, it should also include the API and the namespace params.
resource "kubectl_manifest" "cdr" {
# Create a map { "kind--name" => yaml_doc } from the multi-document yaml text.
# Each element is a separate kubernetes resource.
# Must use \n---\n to avoid splitting on strings and comments containing "---".
# YAML allows "---" to be the first and last line of a file, so make sure
# raw yaml begins and ends with a newline.
# The "---" can be followed by spaces, so need to remove those too.
# Skip blocks that are empty or comments-only in case yaml began with a comment before "---".
for_each = {
for pair in [
for yaml in split(
"\n---\n",
"\n${replace(data.http.crd.body, "/(?m)^---[[:blank:]]*(#.*)?$/", "---")}\n"
) :
[yamldecode(yaml), yaml]
if trimspace(replace(yaml, "/(?m)(^[[:blank:]]*(#.*)?$)+/", "")) != ""
] : "${pair.0["kind"]}--${pair.0["metadata"]["name"]}" => pair.1
}
yaml_body = each.value
}
Once Hashicorp fixes kubernetes_manifest, I would recommend using the same approach. Do not use count+element() because if the ordering of the elements change, Terraform will delete/recreate many resources without needed it.
resource "kubernetes_manifest" "crd" {
for_each = {
for value in [
for yaml in split(
"\n---\n",
"\n${replace(data.http.crd.body, "/(?m)^---[[:blank:]]*(#.*)?$/", "---")}\n"
) :
yamldecode(yaml)
if trimspace(replace(yaml, "/(?m)(^[[:blank:]]*(#.*)?$)+/", "")) != ""
] : "${value["kind"]}--${value["metadata"]["name"]}" => value
}
manifest = each.value
}
P.S. Please support Terraform feature request for multi-document yamldecode. Will make things far easier than the above regex.
Terraform can split a multi-resource yaml (---) for you (docs):
# fetch a raw multi-resource yaml
data "http" "knative_serving_crds" {
url = "https://github.com/knative/serving/releases/download/knative-v1.7.1/serving-crds.yaml"
}
# split raw yaml into individual resources
data "kubectl_file_documents" "knative_serving_crds" {
content = data.http.knative_serving_crds.body
}
# apply each resource from the yaml one by one
resource "kubectl_manifest" "knative_serving_crds" {
depends_on = [kops_cluster_updater.updater]
for_each = data.kubectl_file_documents.knative_serving_crds.manifests
yaml_body = each.value
}
I have a tf.json file that declares a bunch of local variables. One of the variables is a array of complex objects like so:
{
"locals": [
{
"ordered_cache_behaviors": [
{
"path_pattern": "/poc-app-angular*",
"s3_target": "dev-ui-static",
"ingress": "external"
}
]
}
]
}
Here is what I want to do... instead of declaring the variable ordered_cache_behaviors statically in my file I want this to be a computed value. I will get this configuration from a S3 bucket and set the value here. So the value statically will only be an empty array [] that I will append to with a script after getting the data from S3.
This logic needs to execute each time before a terraform plan or terraform apply. What is the best way to do this? I am assuming I need to use a Provisioner to fire off a script? If so how do I then set the local variable here?
If the cache configuration data can be JSON-formatted, you may be able to use the s3_bucket_object datasource plus the jsondecode function as an alternative approach:
Upload your cache configuration data to the poc-app-cache-config bucket as cache-config.json, and then use the following to have Terraform download that file from S3 and parse it into your local ordered_cache_behaviors variable:
data "aws_s3_bucket_object" "cache_configuration" {
bucket = "poc-app-cache-config"
key = "cache-config.json" # JSON-formatted cache configuration map
}
...
locals {
ordered_cache_behaviors = jsondecode(aws_s3_bucket_object.cache_configuration.body)
}