As some of you might already know, when you execute Terraform CLI it really is finicky about what Terraform version that was used to execute the last plan.
For example I am on Terraform v0.11.8. I haven't updated my Terraform in awhile. A new employee comes onboard and installs the latest Terraform without knowing that our infrastructure is deployed using v0.11.8. The new hire mistakenly executes Terraform with their new version and now I won't be able to execute with my v0.11.8.
Is there a feature in Terraform CLI that reads in the state file, sees that it was executed on v0.11.8, and tells the caller that they're not on the correct version?
You can use the required_version parameter in the terraform configuration block:
terraform {
required_version = "=0.11.8"
}
Related
I just recently created on my terraform base module a composer environment instance (resource google_composer_environment) with the following attribute : image_version = "composer-2.0.30-airflow-2.3.3".
When you try to upgrade via the console, the cluster is not destroyed (I think?), but when I try to do it via terraform using image_version = "composer-2.0.32-airflow-2.3.4" the environment will apparently be scrapped and recreated :
# module.dev-omni-orchestrator-instance.google_composer_environment.omni-orchestrator-instance must be replaced
...
Plan: 1 to add, 0 to change, 1 to destroy.
Is there a way to achieve these version upgrade within terraform without destroying the existing environment? Perhaps upgrade via the console or gcloud and then update+import somehow?
I had the same issue previously, to solve it :
Upgrade the version from the Cloud Composer console, in this case the cluster will not be destroyed
Upgrade the version in your Terraform code
When you will plan your Terraform Composer resource, Terraform will detect a change in the real infrastructure and will no plan change and nothing to update (because the version from the real infra is the same than in the Terraform resource)
I think it's an issue of the resource in the Google provider but it was not blocking for me with this technique.
We use terraform for Azure PAAS resources creation and it runs as a separate pipeline steps for each component. For instance - first step data component plan and apply, second step web component plan and apply and so on. So the code is arranged into multiple components and each of those will have it's own definition for provider azurerm block. Inside the block we want to pin the provider version and we want to control it in centralized manner. So currently we came up with the following approach.
provider "azurerm" {
version = "=${ps.AzureRmVersion}"
skip_provider_registration = "true"
features {}
}
When the release process runs there is a powershell functionality that replaces the ps.AzureRmVerison marker with the version. My question is if there is another way to control the provider version without involving third party such as powerhsell to control it.
The version argument in provider blocks is a legacy pattern from older versions of Terraform for specifying version constraints (a set of versions that this module is compatible with) rather than version selections (a single selected version that you want to use).
Since you want to centrally control which exact version is selected I think the best approach would be to have your automation script generate a Dependency Lock File containing the versions you want to prescribe.
Normally Terraform manages this lock file itself as you install and upgrade providers, but in that case each configuration will have its own set of locks and may therefore differ from one another. Since you want to impose central policy, you can instead use Terraform CLI with a simple configuration that only contains provider requirements declarations for the providers you want to use:
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "1.0.0"
}
}
}
In that directory you can run terraform providers lock to cause Terraform to select that particular version from the registry and generate a .terraform.lock.hcl file recording the checksums for all of the platforms you specified:
terraform providers lock -platform=windows_amd64 -platform=linux_amd64
You can then save that .terraform.lock.hcl file to your central location and configure your automation to copy that file into the working directory each time (overwriting any file that might already be there) before running terraform init. Terraform will then select whatever package the lock file recorded, and make sure that it matches the checksums previously recorded.
Your individual Terraform configurations may optionally contain their own non-exact version constraints specifying which Terraform versions they are compatible with, which will then cause Terraform to report an error if the centrally-selected version recorded in your shared lock file is not compatible with one of your configurations.
Note that the lock file only constrains providers that are already recorded in it. If one of your configurations requires a different provider that's not already in the lock file then by default terraform init will select the newest compatible version of that provider and overwrite the lock file to include it.
If you want to prevent that and require all new providers to be added to the centrally-maintained lock file, you can add an additional option to terraform init to tell Terraform to fail if the action it's taking would require changes to the locked providers:
terraform init -lockfile=readonly
To add a new provider with this usage pattern, you'd need to return to the requirements-only configuration I described earlier, add the new provider to it, re-run the same terraform providers lock command to regenerate it, and then update your "master" lock file to that new version of the file.
My Apologies if this is a repeat question. I want to lock down the terraform required version at a global level so that no one in my team can create another resource using a different terraform version.
terraform {
required_version = "1.0.2". ---> I want to lock this.
My repo is the conventional structure - TF_ROOT_DIR/env, env/stage, env/prod TF_ROOT_DIR/global, F_ROOT_DIR/modules
Is it only possible to do so in modules using a versions.tf?
Since 1995, we have used an update mechanism which
cleanly updates and removes software
centrally stores all software meta-data internally to manage needs and artifacts from a single source of truth
NEVER triggers itself arbitrarily.
While we understand terraform has begun reaching out to a registry in a brave reinvention of that wheel without any of those features, we wish to disable it completely. Our current kit includes only one plugin:
terraform-0.13.0-1.el7.harbottle.x86_64
golang-github-terraform-provider-vsphere-1.13.0-0.1.x86_64
The goal is
never check the registry
return an error if the given module is not installed
and I'd be very grateful for good suggestions toward that end. Is there a setting I've overlooked, or can we fake it by telling it to look somewhere empty? Is there a -stay-in-your-lane switch?
Clarification:
the add-on package is a go-build package which delivers a single artifact /usr/bin/terraform-provider-vsphere and nothing else. This has worked wonderfully for all previous incarnations and may have only begun to act up in v13.
Update: These things failed:
terraform init -plugin-dir=/dev/shm
terraform init -get-plugins=false
terraform init -get=false
setting terraform::required_providers::vsphere::source=""
echo "disable_checkpoint = true" > ~/.terraformrc
$ terraform init -get-plugins=false
Initializing the backend...
Initializing provider plugins...
- Finding latest version of -/vsphere...
- Finding latest version of hashicorp/vsphere...
Update: I'm still a bit off:
rpm -qlp golang-github-terraform-provider-vsphere
/usr/share/terraform/plugins/registry.terraform.io/hashicorp/vsphere/1.14.0/linux_amd64/terraform-provider-vsphere
I feel I'm really close. /usr/share/ is in the XDG default search path, and it DOES seem to find the location, but it seems to check the registry first/at-all, which is unexpected.
Initializing provider plugins...
- Finding latest version of hashicorp/vsphere...
- Finding latest version of -/vsphere...
- Installing hashicorp/vsphere v1.14.0...
- Installed hashicorp/vsphere v1.14.0 (unauthenticated)
Error: Failed to query available provider packages
Are we sure it stops checking if it has something local, and that it does that by default? Did I read that right?
What you are describing here sounds like the intention of the Provider Installation settings in Terraform's CLI configuration file.
Specifically, you can put your provider files in a local filesystem directory of your choice -- for the sake of this example, I'm going to arbitrarily choose /usr/local/lib/terraform, and then write the following in the CLI configuration file:
provider_installation {
filesystem_mirror {
path = "/usr/local/lib/terraform"
}
}
If you don't already have a CLI configuration file, you can put this in the file ~/.terraformrc.
With the above configuration, your golang-github-terraform-provider-vsphere-1.13.0-0.1.x86_64 package would need to place the provider's executable at the following path (assuming that you're working with a Linux system):
/usr/local/lib/terraform/registry.terraform.io/hashicorp/vsphere/1.30.0/linux_amd64/terraform-provider-vsphere_v1.13.0_x4
(The filename above is the one in the official vSphere provider release, but if you're building this yourself from source then it doesn't matter what exactly it's called as long as it starts with terraform-provider-vsphere.)
It looks like you are in the process of completing an upgrade from Terraform v0.12, and so Terraform is also trying to install the legacy (un-namespaced) version of this provider, -/vsphere. Since you won't have that in your local directory the installation of that would fail, but with the knowledge that this provider is now published at hashicorp/vsphere we can avoid that by manually migrating it in the state, thus avoiding the need for Terraform to infer this automatically on the next terraform apply:
terraform state replace-provider 'registry.terraform.io/-/vsphere' 'registry.terraform.io/hashicorp/vsphere'
After you run this command your latest state snapshot will not be compatible with Terraform 0.12 anymore, so if you elect to abort your upgrade and return to 0.12 you will need to restore the previous version from a backup. If your state is not stored in a location that naturally retains historical versions, one way to get such a backup is to run terraform state pull with a Terraform 0.12 executable and save the result to a file. (By default, Terraform defers taking this action until terraform apply to avoid upgrading the state format until it would've been making other changes anyway.)
The provider_installation configuraton above is an answer if you want to make this true for all future use of Terraform, which seems to be your goal here, but for completeness I also want to note that the following command should behave in an equivalent way to the result of the above configuration if you want to force a local directory only for one particular invocation of terraform init:
terraform init -plugin-dir=/usr/local/lib/terraform
Since you seem to be upgrading from Terraform 0.12, it might also interest you to know that Terraform 0.13's default installation behavior (without any special configuration) is the same as Terraform 0.12 with the exception of now expecting a different local directory structure than before, to represent the hierarchical provider namespace. (That is, to distinguish hashicorp/vsphere from a hypothetical othernamespace/vsphere.)
Specifically, Terraform 0.13 (as with Terraform 0.12) will skip contacting the remote registry for any provider for which it can discover at least one version available in the local filesystem.
It sounds like your package representing the provider was previously placing a terraform-provider-vsphere executable somewhere that Terraform 0.12 could find and use it. You can adapt that strategy to Terraform 0.13 by placing the executable at the following location:
/usr/local/share/terraform/plugins/registry.terraform.io/hashicorp/vsphere/1.30.0/linux_amd64/terraform-provider-vsphere_v1.13.0_x4
(Again, the exact filename here isn't important as long as it starts with terraform-provider-vsphere.)
/usr/local/share here is assuming one of the default data directories from the XDG Base Directory specification, but if you have XDG_DATA_HOME/XDG_DATA_DIRS overridden on your system then Terraform should respect that and look in the other locations you've listed.
The presence of such a file, assuming you haven't overridden the default behavior with an explicit provider_installation block, will cause Terraform to behave as if you had written the following in the CLI configuration:
provider_installation {
filesystem_mirror {
path = "/usr/local/share/terraform/plugins"
include = ["hashicorp/vsphere"]
}
direct {
exclude = ["hashicorp/vsphere"]
}
}
This form of the configuration forces local installation only for the hashicorp/vsphere provider, thus mimicking what Terraform 0.12 would've done with a local plugin file terraform-provider-vsphere. You can get the more thorough behavior of never contacting remote registries with a configuration like the one I opened this answer with, which doesn't include a direct {} block at all.
After upgrading I was getting messages like the following when running a terraform plan:
Error: Invalid resource instance data in state
on iam_server_backup.tf line 4:
4: resource "aws_iam_access_key" "backup" {
Instance aws_iam_access_key.backup data could not be decoded from
the state: unsupported attribute "ses_smtp_password".
The way I fixed it was by removing the state (terraform state rm aws_iam_access_key.backup). However, that created new access keys when I ran terraform apply, which was time-consuming because I had to change all my access keys in all my apps. Is there a better way to fix this issue?
I encountered this same unsupported attribute "ses_smtp_password" issue when I updated my aws terraform provider, and was able to fix it by manually downloading and modifying the state.
terraform state pull > state.json
now edit state.json and
remove any line with ses_smtp_password
increment the serial attribute (ex "serial": 21, -> "serial": 22,)
save
terraform state push state.json
terraform plan # should work again
optional, but makes it so you can't accidentally commit your state file
rm state.json
I too just faced this issue and wanted to provide a solution that does not require mucking with states.
The solution to this is to upgrade the AWS provider to ~> 3.0 before upgrading to terraform 0.13. This caused the ses_smtp_password field to be removed from the state which then allowed upgrading to terraform 0.13 possible without issue.
Unfortunately I do not understand how this worked and am guessing there is a change in TF 0.13 which causes an exception over removing the deprecated property since TF 0.12 was using the same version of the provider that 0.13 was using.
This error is not related to your upgrade to Terraform 0.13, it is actually due to upgrading from version 2.x of the AWS Terraform provider to version 3.x. As stated in the AWS Terraform provider 3.0 upgrade guide you need to switch from using ses_smtp_password to ses_smtp_password_v4.
The reason for this change is that SES will stop accepting the older type of password in October 2020, so you have to upgrade to the password that uses version 4 signatures before then.
As you've seen, you need to delete the old passwords from your Terraform state, and let Terraform generate new ses_smtp_password_v4 passwords.
I had the same problem with azurerm provider. It was pinned to version ~> 1.44 and suddenly after update to terraform 0.13, unsupported attribute errors started to appear.
What solved it for me (without the need to upgrade the provider):
Run terraform 0.13upgrade - this will create versions.tf with provider version contraints in the new format, i.e.:
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "~> 1.44"
}
}
required_version = ">= 0.13"
}
Now, change source = "hashicorp/azurerm" to source = "-/azurerm". This should make terraform use the old provider version.
For AWS provider, you should have something very similar, with different provider name and version.