Terraform : depends_on argument not creating the specified resource first - terraform

I want to push the terraform state file to a github repo. The file function in Terraform fails to read .tfstate files, so I need to change their extension to .txt first. Now to automate it, I created a null resource which has a provisioner to run the command to copy the tfstate file as a txt file in the same directory. I came across this 'depends_on' argument which lets you specify if a particular resource needs to be made first before running the current. However, it is not working and I am straight away getting the error that 'terraform.txt' file doesn't exit when the file function demands it.
provider "github" {
token = "TOKEN"
owner = "USERNAME"
}
resource "null_resource" "tfstate_to_txt" {
provisioner "local-exec" {
command = "copy terraform.tfstate terraform.txt"
}
}
resource "github_repository_file" "state_push" {
repository = "TerraformStates"
file = "terraform.tfstate"
content = file("terraform.txt")
depends_on = [null_resource.tfstate_to_txt]
}

The documentation for the file function explains this behavior:
This function can be used only with files that already exist on disk at the beginning of a Terraform run. Functions do not participate in the dependency graph, so this function cannot be used with files that are generated dynamically during a Terraform operation. We do not recommend using dynamic local files in Terraform configurations, but in rare situations where this is necessary you can use the local_file data source to read files while respecting resource dependencies.
This paragraph also includes a suggestion for how to get the result you wanted: use the local_file data source, from the hashicorp/local provider, to read the file as a resource operation (during the apply phase) rather than as part of configuration loading:
resource "null_resource" "tfstate_to_txt" {
triggers = {
source_file = "terraform.tfstate"
dest_file = "terraform.txt"
}
provisioner "local-exec" {
command = "copy ${self.triggers.source_file} ${self.triggers.dest_file}"
}
}
data "local_file" "state" {
filename = null_resource.tfstate_to_txt.triggers.dest_file
}
resource "github_repository_file" "state_push" {
repository = "TerraformStates"
file = "terraform.tfstate"
content = data.local_file.state.content
}
Please note that although the above should get the order of operations you were asking about, reading the terraform.tfstate file while Terraform running is a very unusual thing to do, and is likely to result in undefined behavior because Terraform can repeatedly update that file at unpredictable moments throughout terraform apply.
If your intent is to have Terraform keep the state in a remote system rather than on local disk, the usual way to achieve that is to configure remote state, which will then cause Terraform to keep the state only remotely, and not use the local terraform.tfstate file at all.

depends_on does not really work with null_resource.provisioner.
here's a workaround that can help you :
resource "null_resource" "tfstate_to_txt" {
provisioner "local-exec" {
command = "copy terraform.tfstate terraform.txt"
}
}
resource "null_resource" "delay" {
provisioner "local-exec" {
command = "sleep 20"
}
triggers = {
"before" = null_resource.tfstate_to_txt.id
}
}
resource "github_repository_file" "state_push" {
repository = "TerraformStates"
file = "terraform.tfstate"
content = file("terraform.txt")
depends_on = ["null_resource.delay"]
}
the delay null resource will make sure the resource 2 runs after the first if the copy command takes more time just change the sleep to higher number

Related

Uploading a folder to azure storage using terraform (azurerm_storage_blob) -> Nothing appear when execute plan

I have the following resource defined:
resource "azurerm_storage_blob" "widget_upload" {
for_each = fileset("${path.root}/../packages/widget/dist", "**/*")
name = each.key
storage_account_name = azurerm_storage_account.main_storage.name
storage_container_name = "$web"
type = "Block"
source = "${path.root}/../packages/widget/dist/${each.key}"
content_md5 = filemd5(each.key)
}
When I execute
terraform plan
Nothing appear for this resource.
If I do (with terraform CLI in console mode)
❯ terraform console
> fileset("${path.root}/../packages/widget/dist", "**/*")
toset([
"index.html",
"v0.0.0/209-88ecfac5fde48cb32584.js",
"v0.0.0/209-88ecfac5fde48cb32584.js.map",
"v0.0.0/index-9158507e7eab071ec137.js",
"v0.0.0/index-9158507e7eab071ec137.js.map",
])
We can see the files I am trying to upload ...
What I am missing? Why terraform is not adding the changes to the plan?
OK I found why .... console mode can list file from an upper directory.
But "plan" cannot use ".." to access an upper dir.
If you want to do something similar, move the directory you try to upload inside the "terraform workspace".

Conditionally triggering of Terraform local_exec provisioner based on local_file changes

I'm using terraform 0.14 and have 2 resources, one is a local_file that creates a file on the local machine based on a variable and the other is a null_resource with a local_exec provisioner.
This all works as intended but I can only get it to either always run the provisioner (using an always-changing trigger, like timestamp()) or only run it once. Now I'd like to get it to run every time (and only when) the local_file actually changes.
Does anybody know how I can set a trigger that changes when the local_file content has changed? e.g. a last-updated-timestamp or maybe a checksum value?
resource "local_file" "foo" {
content = var.foobar
filename = "/tmp/foobar.txt"
}
resource "null_resource" "null" {
triggers = {
always_run = timestamp() # this will always run
}
provisioner "local-exec" {
command = "/tmp/somescript.py"
}
}
You can try using file hash to indicate its change:
resource "null_resource" "null" {
triggers = {
file_changed = md5(local_file.foo.content)
}
provisioner "local-exec" {
command = "/tmp/somescript.py"
}
}

Terraform - Resource dependency on module

I have a Terraform module, which we'll call parent and a child module used inside of it that we'll refer to as child. The goal is to have the child module run the provisioner before the kubernetes_deployment resource is created. Basically, the child module builds and pushes a Docker image. If the image is not already present, the kubernetes_deployment will wait and eventually timeout because there's no image for the Deployment to use for creation of pods. I've tried everything I've been able to find online, output variables in the child module, using depends_on in the kubernetes_deployment resource, etc and have hit a wall. I would greatly appreciate any help!
parent.tf
module "child" {
source = ".\\child-module-path"
...
}
resource "kubernetes_deployment" "kub_deployment" {
...
}
child-module-path\child.tf
data "external" "hash_folder" {
program = ["powershell.exe", "${path.module}\\bin\\hash_folder.ps1"]
}
resource "null_resource" "build" {
triggers = {
md5 = data.external.hash_folder.result.md5
}
provisioner "local-exec" {
command = "${path.module}\\bin\\build.ps1 ${var.argument_example}"
interpreter = ["powershell.exe"]
}
}
Example Terraform error output:
module.parent.kubernetes_deployment.kub_deployment: Still creating... [10m0s elapsed]
Error output:
Error: Waiting for rollout to finish: 0 of 1 updated replicas are available...
In your child module, declare an output value that depends on the null resource that has the provisioner associated with it:
output "build_complete" {
# The actual value here doesn't really matter,
# as long as this output refers to the null_resource.
value = null_resource.build.triggers.md5
}
Then in your "parent" module, you can either make use of module.child.build_complete in an expression (if including the MD5 string in the deployment somewhere is useful), or you can just declare that the resource depends on the output.
resource "kubernetes_deployment" "example" {
depends_on = [module.child.build_complete]
...
}
Because the output depends on the null_resource and the kubernetes_deployment depends on the output, transitively the kubernetes_deployment now effectively depends on the null_resource, creating the ordering you wanted.

Dependency on local file creation

I am setting up an EKS cluster with Terraform following the example https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/aws_auth.tf and I now have two Terraform files:
kubeconfig.tf
resource "local_file" "kubeconfig" {
content = "${data.template_file.kubeconfig.rendered}"
filename = "tmp/kubeconfig"
}
data "template_file" "kubeconfig" {
template = "${file("template/kubeconfig.tpl")}"
...
}
aws-auth.tf
resource "null_resource" "update_config_map_aws_auth" {
provisioner "local-exec" {
command = "kubectl apply -f tmp/config-map-aws-auth_${var.cluster-name}.yaml --kubeconfig /tmp/kubeconfig"
}
...
}
When I run this the local-exec command fails with
Output: error: stat tmp/kubeconfig: no such file or directory
On a second run it succeeds. I think that the file is created after local-exec tries to use it and local-exec should depend on the file resource. So I try to express the dependency by using interpolation (implicit dependency) like this:
resource "null_resource" "update_config_map_aws_auth" {
provisioner "local-exec" {
command = "kubectl apply -f tmp/config-map-aws-auth_${var.cluster-name}.yaml --kubeconfig ${resource.local_file.kubeconfig.filename}"
}
But this always gives me
Error: resource 'null_resource.update_config_map_aws_auth' provisioner
local-exec (#1): unknown resource 'resource.local_file' referenced in
variable resource.local_file.kubeconfig.filename
You don't need the resource. part when using the interpolation in the last code block.
When Terraform first started it just had resources so you don't need to say that something's a resource as that was the only case. They then added modules and data sources which required some differentiation in the naming so these get module. and data. so Terraform can tell resources and data sources etc apart.
So you probably want something like this:
resource "local_file" "kubeconfig" {
content = "${data.template_file.kubeconfig.rendered}"
filename = "tmp/kubeconfig"
}
data "template_file" "kubeconfig" {
template = "${file("template/kubeconfig.tpl")}"
...
}
resource "null_resource" "update_config_map_aws_auth" {
provisioner "local-exec" {
command = "kubectl apply -f tmp/config-map-aws-auth_${var.cluster-name}.yaml --kubeconfig ${local_file.kubeconfig.filename}"
}
}

How to run command before data.archive_file zips folder in Terraform?

I try to implement aws lambda function using terraform.
I simply have null_resource that have local provisioner and resource.archive_file that zips source code after all preparation is done.
resource "null_resource" "deps" {
triggers = {
package_json = "${base64sha256(file("${path.module}/src/package.json"))}"
}
provisioner "local-exec" {
command = "cd ${path.module}/src && npm install"
}
}
resource "archive_file" "function" {
type = "zip"
source_dir = "${path.module}/src"
output_path = "${path.module}/function.zip"
depends_on = [ "null_resource.deps" ]
}
Recent changes to Terraform deprecated resource.archive_file, so data.archive_file should be used instead. Unfortunately, data executes before resources, and so local provisioner from dependent resource is called way after zip is created. So code bellow does not produce warning any more, however not working at all.
resource "null_resource" "deps" {
triggers = {
package_json = "${base64sha256(file("${path.module}/src/package.json"))}"
}
provisioner "local-exec" {
command = "cd ${path.module}/src && npm install"
}
}
data "archive_file" "function" {
type = "zip"
source_dir = "${path.module}/src"
output_path = "${path.module}/function.zip"
depends_on = [ "null_resource.deps" ]
}
Am I missing something? What is correct way to do this with recent versions.
Terraform: v0.7.11
OS: Win10
Turns out there is an issue with the way Terraform core handles depends_on for data resources. There are a couple of issues reported, one in the archive provider and another in the core.
The following workaround is listed in the archive provider issue. Note that it uses a data.null_data_source to sit between the null_resource and data.archive_file which makes it an explicit dependency as opposed to an implicit dependency with depends_on.
resource "null_resource" "lambda_exporter" {
# (some local-exec provisioner blocks, presumably...)
triggers = {
index = "${base64sha256(file("${path.module}/lambda-files/index.js"))}"
}
}
data "null_data_source" "wait_for_lambda_exporter" {
inputs = {
# This ensures that this data resource will not be evaluated until
# after the null_resource has been created.
lambda_exporter_id = "${null_resource.lambda_exporter.id}"
# This value gives us something to implicitly depend on
# in the archive_file below.
source_dir = "${path.module}/lambda-files/"
}
}
data "archive_file" "lambda_exporter" {
output_path = "${path.module}/lambda-files.zip"
source_dir = "${data.null_data_source.wait_for_lambda_exporter.outputs["source_dir"]}"
type = "zip"
}
There is a new data source in Terraform 0.8, external that allows you to run external commands and extract output. See data.external
The data source should only be used for the retrieval of some depedency value, not the execution of the npm install, you should still do that via the null_resource. Since this is a Terraform data source, it should not have any side effects (although you may need some in this case, not sure).
So basically, null_resource does the dependencies, data.external grabs some value that you can depend on for the archive (directory path for example), then data.archive_file performs the archiving.
This would probably work best with a pseudo random directory name potentially to make dirty checks work a little cleaner.
Here is another working example that doesn't use the 'null_data_source' solution. This work around combines everything into the null_resource block as I was still facing issues with the provided solution.
resource "null_resource" "dependancies" {
provisioner "local-exec" {
command = <<EOT
mkdir script-files
cp import.py script-files
pip3 install requests --target script-files
cd script-files
chmod -R 644 $(find . -type f)
chmod -R 755 $(find . -type d)
zip -r ../lambda_function.zip *
EOT
working_dir = path.module
}
triggers = {
always_run = "${timestamp()}"
}
}

Resources