Terraform long term lock - terraform

Using Terraform 0.12 with the remote state in an S3 bucket with DynamoDB locking.
It seems that a common pattern for Terraforming automation goes more or less like this:
terraform plan -out=plan
[review plan]
terraform apply plan
But then, maybe I'm overlooking something obvious, there's no guarantee other terraform apply invocations haven't updated the infrastructure between 1 and 3 above.
I know locking will prevent a concurrent run of terraform apply while another one is running (and locking is enabled) but can I programmatically grab a "long term locking" so the effective workflow looks like this?
[something to the effect of...] "terraform lock"
terraform plan -out=plan
[review plan]
terraform apply plan
[something to the effect of...] "terraform release lock"
Are there any other means to "protect" infrastructure from concurrent/interdependant updates that I'm overlooking?

You don't need this as long as you are only worrying about the state file changing.
If you provide a plan output file to apply and the state has changed since then Terraform will error before making any changes, complaining that the saved plan is stale.
As an example:
$ cat <<'EOF' >> main.tf
> resource "random_pet" "bucket_suffix" {}
>
> resource "aws_s3_bucket" "example" {
> bucket = "example-${random_pet.bucket_suffix.id}"
> acl = "private"
>
> tags = {
> ThingToChange = "foo"
> }
> }
> EOF
$ terraform init
# ...
$ terraform apply
# ...
$ $ sed -i 's/foo/bar/' main.tf
$ terraform plan -out=plan
# ...
$ sed -i 's/bar/baz/' main.tf
$ terraform apply
# ...
$ terraform apply plan
Error: Saved plan is stale
The given plan file can no longer be applied because the state was changed by
another operation after the plan was created.
What it won't do is fail if something outside of Terraform has changed anything. So if instead of applying Terraform again with baz as the tag for the bucket I had changed the tag on the bucket via the AWS CLI or the AWS console then Terraform would have happily changed it back to bar on the apply with the stale plan.

Related

how does terraform state and it's refresh work

I'm new to terraform and want to understand how the terraform state and the refresh work together. As I understand, you write the desired resources in a .tf file and run 'terraform apply' and the resources are created, if they did not exist. Also, the terraform.tfstate file is created, containing all the metadata from terraform's viewpoint about the resource. This way, terraform can compare the desired state (in the .tf file) to the current state (in the terraform.state file) . So far so good.
But what happens, if there is any change on the 'real world' resource?
Shouldn't terraform recognize this? This should be done by the 'refresh' of terraform, or?
However, I've experimented a bit and created this simple resource:
resource "local_file" "mypet" {
filename = "/tmp/mypet.txt"
content = "I love pets"
file_permission = "0755"
}
Running 'terraform init && terraform apply' created the file /tmp/mypet.txt with the content "I love pets". Cool.
Then I changed the file permissions:
chmod 0444 /tmp/mypet.txt"
ls -l /tmp/mypet.txt
-r--r--r--. 1 devel devel 11 Nov 25 09:28 /tmp/mypet.txt
And another 'terraform apply':
local_file.mypet: Refreshing state... [id=4246687863ee17bf4daf5fc376ab01222e989aca]
No changes. Your infrastructure matches the configuration.
Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Actually, terraform does not recognize this real world change and that the target configuration has drifted from the state known to terraform!
Another interesting example: I did a vi /tmp/mypet.txt but did not change the content at all. However, I left the vi using :wq . And this changes the checksum of the file:
sha256sum /tmp/mypet.txt
595e30750adb4244cfcfc31de9b37d1807c20e7ac2aef049755b6c0e5f580170 /tmp/mypet.txt
[devel#fedora test]$ !vi
vi /tmp/mypet.txt
[devel#fedora test]$ sha256sum /tmp/mypet.txt
4fcc76cae3c8950e9633ca8220e36f4313a97d0f436ab050fd1d586b40d682f0 /tmp/mypet.txt
This is recognized by terraform and thus it wants to recreate the file:
terraform apply
local_file.mypet: Refreshing state... [id=4246687863ee17bf4daf5fc376ab01222e989aca]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
+ create
Terraform will perform the following actions:
# local_file.mypet will be created
+ resource "local_file" "mypet" {
+ content = "I love pets"
+ directory_permission = "0777"
+ file_permission = "0755"
+ filename = "/tmp/mypet.txt"
+ id = (known after apply)
}
Plan: 1 to add, 0 to change, 0 to destroy.
Means, terraform this time recognized the real-world change.
When I call 'terraform apply' without refreshing, it does not see the change that happened in the real world, as expected:
terraform apply --refresh=false
No changes. Your infrastructure matches the configuration.
Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
It seems, that only some attributes of the file resource-type of the local provider are able to recognize a drift from the terraform state.
Is that true? Is this documented somewhere, if an attribute of a resource-type is "drift-aware" or not?

Terraform rename a resource without deleting

I created AzDo repositories using the Terraform. But what is the best way to rename the repository without deleting it. I found a command "terraform state mv" for the purpose. Please let me know if there is a better way.
Currently when I use the Terraform state mv command I'm getting below error
terraform state mv "module.azdo-module.azuredevops_git_repository[\"repo1\"]" "module.azdo-module.azuredevops_project.azuredevops_git_repository[\"repo2\"]"
Getting below error
Error: Invalid address
│
│ on line 1:
│ (source code not available)
│
│ A resource name is required.
maybe
terraform state mv "module.azdo-module.azuredevops_git_repository.repo1" "module.azdo-module.azuredevops_project.azuredevops_git_repository.repo2"
list resources to check the name
terraform state list
Terraform has introduced a new declarative way to refactor resources with the move block syntax in 1.1 version.
Steps:
rename the resource to the new name
moved {
from = module.azdo-module.azuredevops_git_repository.repo1
to = module.azdo-module.azuredevops_project.azuredevops_git_repository.repo2
}
update references to a new resource name
plan and apply
remove the moved block (given no one else is using your modules)
More info:
https://developer.hashicorp.com/terraform/language/modules/develop/refactoring#moved-block-syntax

"Invalid legacy provider address" error on Terraform

I'm trying to deploy a bitbucket pipeline using terraform v0.14.3 to create resources in google cloud. after running terraform command, the pipeline fails with this error:
Error: Invalid legacy provider address
This configuration or its associated state refers to the unqualified provider
"google".
You must complete the Terraform 0.13 upgrade process before upgrading to later
versions.
We updated our local version of terraform to v.0.13.0 and then ran: terraform 0.13upgrade as referenced in this guide: https://www.terraform.io/upgrade-guides/0-13.html. A versions.tf file was generated requiring terraform version >=0.13 and our required provider block now looks like this:
terraform {
backend "gcs" {
bucket = "some-bucket"
prefix = "terraform/state"
credentials = "key.json" #this is just a bitbucket pipeline variable
}
required_providers {
google = {
source = "hashicorp/google"
version = "~> 2.20.0"
}
}
}
provider "google" {
project = var.project_ID
credentials = "key.json"
region = var.project_region
}
We still get the same error when initiating the bitbucket pipeline. Does anyone know how to get past this error? Thanks in advance.
Solution
If you are using a newer version of Terraform, such as v0.14.x, you should:
use the replace-provider subcommand
terraform state replace-provider \
-auto-approve \
"registry.terraform.io/-/google" \
"hashicorp/google"
#=>
Terraform will perform the following actions:
~ Updating provider:
- registry.terraform.io/-/google
+ registry.terraform.io/hashicorp/google
Changing x resources:
. . .
Successfully replaced provider for x resources.
initialize Terraform again:
terraform init
#=>
Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/google from the dependency lock file
- Using previously-installed hashicorp/google vx.xx.x
Terraform has been successfully initialized!
You may now begin working with Terraform. Try . . .
This should take care of installing the provider.
Explanation
Terraform only supports upgrades from one major feature upgrade at a time. Your older state file was, more than likely, created using a version earlier than v0.13.x.
If you did not run the apply command before you upgraded your Terraform version, you can expect this error: the upgrade from v0.13.x to v0.14.x was not complete.
You can find more information here.
in our case, we were on aws and had similar error
...
Error: Invalid legacy provider address
This configuration or its associated state refers to the unqualified provider
"aws".
the steps to resolve were :
ensure syntax was upgraded by running terraform init again
check the warnings and resolve them
and finally updating the statefile with following method.
# update provider in state file
terraform state replace-provider -- -/aws hashicorp/aws
# reinit
terraform init
specific to ops problem, if issue still occurs, verify access to the bucket location from local and from pipeline. also verify the version of terraform running in pipeline. depending on configuration it may be the remote statefile is/can not be updated.
Same issue for me. I ran:
terraform providers
That gave me:
Providers required by configuration:
registry.terraform.io/hashicorp/google
Providers required by state:
registry.terraform.io/-/google
So I ran:
terraform state replace-provider registry.terraform.io/-/google registry.terraform.io/hashicorp/google
That did the trick.
To add on, I had installed terraform 0.14.6 but the state seemed to be stuck in 0.12. In my case I had 3 references that were off, this article helped me pinpoint which ones (all the entries in "Providers required by state" which had a - in the link. https://github.com/hashicorp/terraform/issues/27615
I corrected it by running the replace-provider command for each entry which was off, then running terraform init. I note doing this and running a git diff, the tfstate has been updated and now uses 0.14.x terraform instead of my previous 0.12.x. i.e.
terraform providers
terraform state replace-provider registry.terraform.io/-/azurerm registry.terraform.io/hashicorp/azurerm
Explanation: Your terraform project contains tf.state file that is outdated and refereeing to old provider address. The Error message will present this error:
Error: Invalid legacy provider address
This configuration or its associated state refers to the unqualified provider
<some-provider>.
You must complete the Terraform <some-version> upgrade process before upgrading to later
versions.
Solution: In order to solve this issue you should change the tf.state references to link to the newer required providers, update the tf.state file and initialize the project again. The steps are:
Create / Edit the required providers block with the relevant package name and version, I'd rather doing it on versions.tf file.
example:
terraform {
required_version = ">= 0.14"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.35.0"
}
}
}
Run terraform providers command to present the required providers from configuration against the required providers that saved on state.
example:
Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/aws] >= 3.35.0
Providers required by state:
provider[registry.terraform.io/-/aws]
Switch and reassign the required provider source address in the terraform state ( using terraform state replace-provider command) so we can tell terraform how to interpret the legacy provider.
The terraform state replace-provider subcommand allows re-assigning
provider source addresses recorded in the Terraform state, and so we
can use this command to tell Terraform how to reinterpret the "legacy"
provider addresses as properly-namespaced providers that match with
the provider source addresses in the configuration.
Warning: The terraform state replace-provider subcommand, like all of
the terraform state subcommands, will create a new state snapshot and
write it to the configured backend. After the command succeeds the
latest state snapshot will use syntax that Terraform v0.12 cannot
understand, so you should perform this step only when you are ready to
permanently upgrade to Terraform v0.13.
example:
terraform state replace-provider registry.terraform.io/-/aws registry.terraform.io/hashicorp/aws
output:
~ Updating provider:
- registry.terraform.io/-/aws
+ registry.terraform.io/hashicorp/aws
run terraform init to update references.
While you were under TF13 did you apply state at least once for the running project?
According to TF docs: https://www.terraform.io/upgrade-guides/0-14.html
There is no automatic update command (separately) in 0.14 (like there was in 0.13). The only way to upgrade is to force state on a project at least once, while under command when moving TF13 to 14.
You can also try terraform init in the project directory.
my case was like this
Error: Invalid legacy provider address
This configuration or its associated state refers to the unqualified provider
"openstack".
You must complete the Terraform 0.13 upgrade process before upgrading to later
versions.
for resolving the issue
remove the .terraform folder
the execute the following command
terraform state replace-provider -- -/openstack terraform-provider-openstack/openstack
after this command, you will see the below print, enter yes
Terraform will perform the following actions:
~ Updating provider:
- registry.terraform.io/-/openstack
+ registry.terraform.io/terraform-provider-openstack/openstack
Changing 11 resources:
openstack_compute_servergroup_v2.kubernetes_master
openstack_networking_network_v2.kube_router
openstack_compute_instance_v2.kubernetes_worker
openstack_networking_subnet_v2.internal
openstack_networking_subnet_v2.kube_router
data.openstack_networking_network_v2.external_network
openstack_compute_instance_v2.kubernetes_etcd
openstack_networking_router_interface_v2.internal
openstack_networking_router_v2.internal
openstack_compute_instance_v2.kubernetes_master
openstack_networking_network_v2.internal
Do you want to make these changes?
Only 'yes' will be accepted to continue.
Enter a value: yes
Successfully replaced provider for 11 resources.
I recently ran into this using Terraform Cloud for the remote backend. We had some older AWS-related workspaces set to version 0.12.4 (in the cloud) that errored out with "Invalid legacy provider address" and refused to run with the latest Terraform client 1.1.8.
I am adding my answer because it is much simpler than the other answers. We did not do any of the following:
terraform providers
terraform 0.13upgrade
remove the .terraform folder
terraform state replace-provider
Instead we simply:
In a clean folder (no local state, using local terraform.exe version 0.13.7) ran 'terraform init'
Made a small insignificant change (to ensure apply would write state) to a .tf file in the workspace
In Terraform Cloud set the workspace version to 0.13.7
Using local 0.13.7 terraform.exe ran apply - that saved new state.
Now we can use cloud and local terraform.exe version 1.1.8 and no more problems.
Note that we did also need to update a few AWS S3-related resources to the newer AWS provider syntax to get all our workspaces working with the latest provider.
We encountered a similar problem in our operational environments today. We successfully completed the terraform 0.13upgrade command. This indeed introduced a versions.tf file.
However, performing a terraform init with this setup was still not possible, and the following error popped up:
Error: Invalid legacy provider address
Further investigation in the state file revealed that, for some resources, the provider block was not updated. We hence had to run the following command to finalize the upgrade process.
terraform state replace-provider "registry.terraform.io/-/google" "hashicorp/google"
EDIT Deployment to the next environment revealed that this was caused by conditional resources. To easily enable/disable some resources we leverage the count attribute and use either 0 or 1. For the resources with count = 0, that were unaltered with Terraform 0.13, the provider was not updated.
I was using terragrunt with remote s3 state and dynamo db and sadly this does not work for me. So posting it here might help someone else.
A long way to make this work, as terragrunt state replace-provider does work for me
download the state file from s3
aws s3 cp s3://bucket-name/path/terraform.tfstate terraform.tfstate --profile profile
replace the provider using terraform
terraform state replace-provider "registry.terraform.io/-/random" "hashicorp/random"
terraform state replace-provider "registry.terraform.io/-/aws" "hashicorp/aws"
upload the state file back to s3 as even terragrunt state push terraform.tfstate does not work for me
aws s3 cp terraform.tfstate s3://bucket-name/path/terraform.tfstate --profile profile
terragrunt apply
the command will throw error with digest value,
update the dynamo db table digest value that received in previous command
Initializing the backend...
Error refreshing state: state data in S3 does not have the expected content.
This may be caused by unusually long delays in S3 processing a previous state
update. Please wait for a minute or two and try again. If this problem
persists, and neither S3 nor DynamoDB are experiencing an outage, you may need
to manually verify the remote state and update the Digest value stored in the
DynamoDB table to the following value: fe2840edf8064d9225eea6c3ef2e5d1d
finally, run terragrunt apply
The other way that this can be strange is if you are using terraform workspaces - especially with the remote state files.
Using a terraform workspace - the order of operations is important.
terraform init - connecting to the default workspace
terraform workspace switch <env> - Even if you have specified the workspace here, the init will happen using the default workspace.
This is an assumption that terraform makes - sometimes erroneously
To fix this - you can run your init using:
TF_WORKSPACE=<your_env> terraform init
Or remove the default workspace.

Terraform Cloud (i.e. remote backend) TF_VAR_ environment substitutions not working?

Terraform version: 0.12.24
This is really weird because I have used the TF_VAR_ substitution syntax before and it has worked fine.
provider.tf
# Configure the AWS Provider
provider "aws" {
version = "~> 2.0"
region = "ap-southeast-2"
access_key = var.aws_access_key_id
secret_key = var.aws_secret_access_key
}
vars.tf
variable "aws_access_key_id" {
description = "Access Key for AWS IAM User"
}
variable "aws_secret_access_key" {
description = "Secret Access Key for AWS IAM User"
}
variable "terraform_cloud_token" {
description = "Token used to log into Terraform Cloud via the CLI"
}
backend.tf for terraform cloud
terraform {
backend "remote" {
organization = "xx"
workspaces {
name = "xx"
}
}
}
Build logs
---------------
TF_VAR_aws_secret_access_key=***
TF_VAR_aws_access_key_id=***
TF_VAR_terraform_cloud_token=***
---------------
It also fails locally when I try to run this in a local Docker Container
Dockerfile
FROM hashicorp/terraform:0.12.24
COPY . /app
COPY .terraformrc $HOME
ENV TF_VAR_aws_secret_access_key 'XX'
ENV TF_VAR_aws_access_key_id 'XX'
ENV TF_VAR_terraform_cloud_token 'XX'
WORKDIR /app
ENTRYPOINT ["/app/.github/actions/terraform-plan/entrypoint.sh"]
entrypoint.sh
#!/bin/sh -l
# move terraform cloud configuration file to user root as expected
# by the backend resource
mv ./.terraformrc ~/
terraform init
terraform plan
output from docker container run
$ docker run -it tf-test
---------------
TF_VAR_aws_secret_access_key=XX
TF_VAR_aws_access_key_id=XX
TF_VAR_terraform_cloud_token=XX
---------------
Initializing the backend...
Successfully configured the backend "remote"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.56.0...
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Running plan in the remote backend. Output will stream here. Pressing Ctrl-C
will stop streaming the logs, but will not stop the plan running remotely.
Preparing the remote plan...
To view this run in a browser, visit:
https://app.terraform.io/app/XX/XX/runs/run-XX
Waiting for the plan to start...
Terraform v0.12.24
Configuring remote state backend...
Initializing Terraform configuration...
2020/04/03 01:43:04 [DEBUG] Using modified User-Agent: Terraform/0.12.24 TFC/05d5abc3eb
Error: No value for required variable
on vars.tf line 1:
1: variable "aws_access_key_id" {
The root module input variable "aws_access_key_id" is not set, and has no
default value. Use a -var or -var-file command line argument to provide a
value for this variable.
Error: No value for required variable
on vars.tf line 5:
5: variable "aws_secret_access_key" {
The root module input variable "aws_secret_access_key" is not set, and has no
default value. Use a -var or -var-file command line argument to provide a
value for this variable.
Error: No value for required variable
on vars.tf line 9:
9: variable "terraform_cloud_token" {
The root module input variable "terraform_cloud_token" is not set, and has no
default value. Use a -var or -var-file command line argument to provide a
value for this variable.
Okay... it is confusing because the logs generated in Terraform's VMs are streamed to your own terminal/run logs.
But this is what I found out. There are two options available to you when you use Terraform Cloud.
Use Terraform's VMs to run your terraform commands
Use your own (or your CI/CD platform's) infrastructure to run those terraform commands.
If you choose the first option (which is annoyingly the default)... you must set your environment variables within the Terraform Cloud Dashboard. This is because all terraform commands for this execution type are run in their VMs and the environment variables in your local environment, for good security reasons, aren't passed through to Terraform.
If you have the remote option selected, once you do this, it will work as expected.

terraform: How to destroy oldest instance when lowering aws_instance count

Given a pair of aws instances deployed with
provider "aws" {
region = "us-east-1"
}
resource "aws_instance" "example" {
count = 2
ami = "ami-2757f631"
instance_type = "t2.micro"
tags = {
Name = "Test${count.index}"
}
}
Lowering count = 1 will destroy the last instance deployed:
Terraform will perform the following actions:
- aws_instance.example[1]
Is it possible to get terraform to destroy the first instance. ie.
Terraform will perform the following actions:
- aws_instance.example[0]
Terraform is tracking which instance is which via its state. When you reduce your count on the aws_instance resource Terraform will simply remove the later instances. While this shouldn't really be much of an issue because I would only really recommend that you are deploying groups of homogenous instances that can handle the load being interrupted (and would sit behind some form of load balancer mechanism) if you really needed to you could edit the state file to reorder the instances before reducing the number of instances.
The state file is serialised as JSON so you can just edit it directly (making sure it's uploaded to whatever you're using for remote state if you are using remote state) or better yet you can use the first class tools for editing remote state that the Terraform CLI provides with terraform state mv.
As an example you can do this:
# Example from question has been applied already
# `count` is edited from 2 to 1
$ terraform plan
...
aws_instance.example[1]: Refreshing state... (ID: i-0c227dfbfc72fb0cd)
aws_instance.example: Refreshing state... (ID: i-095fd3fdf86ce8254)
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
- destroy
Terraform will perform the following actions:
- aws_instance.example[1]
Plan: 0 to add, 0 to change, 1 to destroy.
...
$
$
$
$ terraform state list
aws_instance.example[0]
aws_instance.example[1]
$
$
$
$ terraform state mv aws_instance.example[1] aws_instance.example[2]
Moved aws_instance.example[1] to aws_instance.example[2]
$ terraform state mv aws_instance.example[0] aws_instance.example[1]
Moved aws_instance.example[0] to aws_instance.example[1]
$ terraform state mv aws_instance.example[2] aws_instance.example[0]
Moved aws_instance.example[2] to aws_instance.example[0]
$
$
$
$ terraform plan
...
aws_instance.example[1]: Refreshing state... (ID: i-095fd3fdf86ce8254)
aws_instance.example: Refreshing state... (ID: i-0c227dfbfc72fb0cd)
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-place
- destroy
Terraform will perform the following actions:
~ aws_instance.example
tags.Name: "Test1" => "Test0"
- aws_instance.example[1]
Plan: 0 to add, 1 to change, 1 to destroy.
...

Resources