What specific commands must we use in order to figure out why terraform hangs indefinitely when it is supposed to be installing registry.terraform.io/hashicorp/azurerm v2.61.0 from a local registry?
CURRENT ERROR MESSAGE:
Here is the error message that terraform writes to the console when it stalls indefinitely/permanently:
2021/08/12 13:29:38 [TRACE] providercache.Dir.InstallPackage: installing registry.terraform.io/hashicorp/azurerm v2.61.0 from C:\path\to\registry.terraform.io\hashicorp\azurerm\2.61.0\windows_amd64
I looked in the C:\path\to\registry.terraform.io\hashicorp\azurerm\2.61.0\windows_amd64 directory, and there is indeed an executable named terraform-provider-azurerm_v2.61.0_x5.
Also, terraform is set to TRACE so that I think we are getting more verbose logs than we would get with DEBUG.
We note the extra characters in terraform-provider-azurerm_v2.61.0_x5 compared to terraform-provider-azurerm_v2.61.0 , but terraform-provider-azurerm_v2.61.0_x5 is what terraform installed from the official hashicorp registry when we ran the same module without specifying any local registry using a local terraform.rc. We simply moved the terraform-provider-azurerm_v2.61.0_x5 that terraform installed into our private local registry at the same path and ran terraform apply for the module with a terraform.rc pointing to the same executable in the private registry like we do with all our providers, and we got this error result.
What can we do to identify the root cause of this problem?
Related
We have been given a remote Terraform Private registry to utilise. Along with that came a credentials name and token.
Once we configured the general details on the terraform script, we created a .terraformrc file in the same dir (mac) and created the following
credentials "my remote registy"
token = "tokenvaluegoeshere"
When we run a terraform init we get the following error (for all modules)
Error: Error Accessing remote module registry
failed to retrieve available versions for module "x" from external.location - failed to request discovery document: 401 Unauthorised
It feels like i haven't got something correctly setup in terraform (even though it looks fine)
I have tried running terraform from different locations on my mac also created new .terraformrc files but still doesn't work.
I am stuck in a path where when I run terraform init. the provider is not getting downloaded and it gives me no error. I am using the main.tf file and in it, I have provider "azurerm" syntax only. So when I run the terraform init I get the below output only and I see nowhere the terraform provider file getting initialized or getting downloaded. Logged in and authenticated to Azure login page too.
Terraform Code> terraform init
Initializing the backend...
Initializing provider plugins...
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Terraform creates a hidden folder to store the provider. Make sure you set your OS permissions to see hidden files and folders.
From the documentation:
A hidden .terraform directory, which Terraform uses to manage cached provider plugins and modules, record which workspace is currently active, and record the last known backend configuration in case it needs to migrate state on the next run. This directory is automatically managed by Terraform, and is created during initialization.
If I run terraform -version from the directory that I ran terraform init, terraform correctly finds the plugins.
But if I run terraform -version from any other directory, terraform does NOT find any provider plugins.
My ~/.terraformrc file looks like this:
provider_installation {
filesystem_mirror {
path = "/.terraform/providers"
include = ["registry.terraform.io/hashicorp/*"]
}
}
Inside that directory, I have the aws provider binary (it was placed there by terraform init, so I know the directory structure is correct:
/.terraform/providers/registry.terraform.io/hashicorp/aws/3.55.0/linux_amd64/terraform-provider-aws_v3.55.0_x5
When I cd to /, terraform correctly finds the provider:
# terraform -version
Terraform v1.0.6
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.55.0
But if I cd to /tmp, terraform does NOT find the provider:
/tmp # terraform -version
Terraform v1.0.6
on linux_amd64
So that tells me there is something not right with the .terraformrc file.
If I run that with TRACE, it doesn't say much:
2021-09-08T03:01:30.336Z [DEBUG] Attempting to open CLI config file: /root/.terraformrc
2021-09-08T03:01:30.336Z [INFO] Loading CLI configuration from /root/.terraformrc
2021-09-08T03:01:30.336Z [DEBUG] Explicit provider installation configuration is set
2021-09-08T03:01:30.336Z [TRACE] Selected provider installation method cliconfig.ProviderInstallationFilesystemMirror("/.terraform/providers") with includes [registry.terraform.io/hashicorp/*] and excludes []
2021-09-08T03:01:30.337Z [INFO] CLI command args: []string{"version", "-version"}
Terraform v1.0.6
on linux_amd64
How can I tell terraform to look in `/.terraform/providers" for all of it's providers?
I think in your investigations here you are confusing two different concepts for Terraform provider installation: a local filesystem mirror (which you've configured in your CLI configuration), and your current working directory's provider cache (which is where terraform init installs the providers for the current working directory).
A local filesystem mirror happens to support a similar directory structure to a provider cache directory, and so what you've achieved by running Terraform in the root directory is to trick Terraform into thinking that your local mirror is the local cache directory for that working directory. It doesn't work anywhere else because in that case these two separate directories are truly separate, which is the expected way to use Terraform.
The other relevant thing to know in what you tried here is that terraform version will only show providers that are already installed and activated by terraform init. Again, because you tricked Terraform into treating your filesystem mirror as the local cache directory it happened to find the required hashicorp/aws plugin there, but when you ran it elsewhere there was no local cache directory and so it returned nothing.
You mentioned that your goal is to have Terraform look for providers only in your local filesystem directory, in /.terraform. I'll note that a dot file in the root of your filesystem is a rather unusual place to keep a local filesystem mirror, but I'll use that path here anyway since you asked about it in your question.
First, configure the filesystem mirror as you already did, and also configure the "direct" installation method (installing from the origin registry) to exclude those same providers:
provider_installation {
filesystem_mirror {
path = "/.terraform/providers"
include = ["registry.terraform.io/hashicorp/*"]
}
direct {
exclude = ["registry.terraform.io/hashicorp/*"]
}
}
The above means that for any provider in the hashicorp namespace on the public registry, Terraform will only look in /.terraform. For all other providers, Terraform will contact the origin registry in the usual way and try to download the provider over the network.
With that configuration in place, change your shell's current working directory to be the root directory of your configuration, which is the directory containing your root set of .tf files. To initialize that as a working directory, which will install all of the needed providers and external modules, run:
terraform init
Terraform will then read the configuration to learn which providers it requires, and will automatically install each of them in turn. Because of the CLI configuration above, Terraform will not contact the origin registry for any of the hashicorp namespace providers, and will instead expect them to already be present in the mirror directory. Regardless of whether each provider was found in the local filesystem mirror or directly in the origin registry, they'll all then be cached in the .terraform/providers subdirectory of your working directory, which is the local cache directory.
You specifically asked about preventing Terraform from accessing the origin registry at all here and so I've answered with that in mind. However, a more common request is to have Terraform contact the registry the first time it installs each provider and then cache the result locally for future installation. If that is what you want to achieve here then you should enable the global provider plugin cache instead.
In that case, the global cache directory becomes a secondary read-through cache for the local cache directory I was describing above. When installing new providers to the local cache directory, Terraform will first check the global cache directory and either copy or symlink (depending on whether your system can support symlinks) the global cache entry into the local cache directory, and thus avoid re-downloading the same package from the registry again.
I'm trying to deploy a bitbucket pipeline using terraform v0.14.3 to create resources in google cloud. after running terraform command, the pipeline fails with this error:
Error: Invalid legacy provider address
This configuration or its associated state refers to the unqualified provider
"google".
You must complete the Terraform 0.13 upgrade process before upgrading to later
versions.
We updated our local version of terraform to v.0.13.0 and then ran: terraform 0.13upgrade as referenced in this guide: https://www.terraform.io/upgrade-guides/0-13.html. A versions.tf file was generated requiring terraform version >=0.13 and our required provider block now looks like this:
terraform {
backend "gcs" {
bucket = "some-bucket"
prefix = "terraform/state"
credentials = "key.json" #this is just a bitbucket pipeline variable
}
required_providers {
google = {
source = "hashicorp/google"
version = "~> 2.20.0"
}
}
}
provider "google" {
project = var.project_ID
credentials = "key.json"
region = var.project_region
}
We still get the same error when initiating the bitbucket pipeline. Does anyone know how to get past this error? Thanks in advance.
Solution
If you are using a newer version of Terraform, such as v0.14.x, you should:
use the replace-provider subcommand
terraform state replace-provider \
-auto-approve \
"registry.terraform.io/-/google" \
"hashicorp/google"
#=>
Terraform will perform the following actions:
~ Updating provider:
- registry.terraform.io/-/google
+ registry.terraform.io/hashicorp/google
Changing x resources:
. . .
Successfully replaced provider for x resources.
initialize Terraform again:
terraform init
#=>
Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/google from the dependency lock file
- Using previously-installed hashicorp/google vx.xx.x
Terraform has been successfully initialized!
You may now begin working with Terraform. Try . . .
This should take care of installing the provider.
Explanation
Terraform only supports upgrades from one major feature upgrade at a time. Your older state file was, more than likely, created using a version earlier than v0.13.x.
If you did not run the apply command before you upgraded your Terraform version, you can expect this error: the upgrade from v0.13.x to v0.14.x was not complete.
You can find more information here.
in our case, we were on aws and had similar error
...
Error: Invalid legacy provider address
This configuration or its associated state refers to the unqualified provider
"aws".
the steps to resolve were :
ensure syntax was upgraded by running terraform init again
check the warnings and resolve them
and finally updating the statefile with following method.
# update provider in state file
terraform state replace-provider -- -/aws hashicorp/aws
# reinit
terraform init
specific to ops problem, if issue still occurs, verify access to the bucket location from local and from pipeline. also verify the version of terraform running in pipeline. depending on configuration it may be the remote statefile is/can not be updated.
Same issue for me. I ran:
terraform providers
That gave me:
Providers required by configuration:
registry.terraform.io/hashicorp/google
Providers required by state:
registry.terraform.io/-/google
So I ran:
terraform state replace-provider registry.terraform.io/-/google registry.terraform.io/hashicorp/google
That did the trick.
To add on, I had installed terraform 0.14.6 but the state seemed to be stuck in 0.12. In my case I had 3 references that were off, this article helped me pinpoint which ones (all the entries in "Providers required by state" which had a - in the link. https://github.com/hashicorp/terraform/issues/27615
I corrected it by running the replace-provider command for each entry which was off, then running terraform init. I note doing this and running a git diff, the tfstate has been updated and now uses 0.14.x terraform instead of my previous 0.12.x. i.e.
terraform providers
terraform state replace-provider registry.terraform.io/-/azurerm registry.terraform.io/hashicorp/azurerm
Explanation: Your terraform project contains tf.state file that is outdated and refereeing to old provider address. The Error message will present this error:
Error: Invalid legacy provider address
This configuration or its associated state refers to the unqualified provider
<some-provider>.
You must complete the Terraform <some-version> upgrade process before upgrading to later
versions.
Solution: In order to solve this issue you should change the tf.state references to link to the newer required providers, update the tf.state file and initialize the project again. The steps are:
Create / Edit the required providers block with the relevant package name and version, I'd rather doing it on versions.tf file.
example:
terraform {
required_version = ">= 0.14"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.35.0"
}
}
}
Run terraform providers command to present the required providers from configuration against the required providers that saved on state.
example:
Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/aws] >= 3.35.0
Providers required by state:
provider[registry.terraform.io/-/aws]
Switch and reassign the required provider source address in the terraform state ( using terraform state replace-provider command) so we can tell terraform how to interpret the legacy provider.
The terraform state replace-provider subcommand allows re-assigning
provider source addresses recorded in the Terraform state, and so we
can use this command to tell Terraform how to reinterpret the "legacy"
provider addresses as properly-namespaced providers that match with
the provider source addresses in the configuration.
Warning: The terraform state replace-provider subcommand, like all of
the terraform state subcommands, will create a new state snapshot and
write it to the configured backend. After the command succeeds the
latest state snapshot will use syntax that Terraform v0.12 cannot
understand, so you should perform this step only when you are ready to
permanently upgrade to Terraform v0.13.
example:
terraform state replace-provider registry.terraform.io/-/aws registry.terraform.io/hashicorp/aws
output:
~ Updating provider:
- registry.terraform.io/-/aws
+ registry.terraform.io/hashicorp/aws
run terraform init to update references.
While you were under TF13 did you apply state at least once for the running project?
According to TF docs: https://www.terraform.io/upgrade-guides/0-14.html
There is no automatic update command (separately) in 0.14 (like there was in 0.13). The only way to upgrade is to force state on a project at least once, while under command when moving TF13 to 14.
You can also try terraform init in the project directory.
my case was like this
Error: Invalid legacy provider address
This configuration or its associated state refers to the unqualified provider
"openstack".
You must complete the Terraform 0.13 upgrade process before upgrading to later
versions.
for resolving the issue
remove the .terraform folder
the execute the following command
terraform state replace-provider -- -/openstack terraform-provider-openstack/openstack
after this command, you will see the below print, enter yes
Terraform will perform the following actions:
~ Updating provider:
- registry.terraform.io/-/openstack
+ registry.terraform.io/terraform-provider-openstack/openstack
Changing 11 resources:
openstack_compute_servergroup_v2.kubernetes_master
openstack_networking_network_v2.kube_router
openstack_compute_instance_v2.kubernetes_worker
openstack_networking_subnet_v2.internal
openstack_networking_subnet_v2.kube_router
data.openstack_networking_network_v2.external_network
openstack_compute_instance_v2.kubernetes_etcd
openstack_networking_router_interface_v2.internal
openstack_networking_router_v2.internal
openstack_compute_instance_v2.kubernetes_master
openstack_networking_network_v2.internal
Do you want to make these changes?
Only 'yes' will be accepted to continue.
Enter a value: yes
Successfully replaced provider for 11 resources.
I recently ran into this using Terraform Cloud for the remote backend. We had some older AWS-related workspaces set to version 0.12.4 (in the cloud) that errored out with "Invalid legacy provider address" and refused to run with the latest Terraform client 1.1.8.
I am adding my answer because it is much simpler than the other answers. We did not do any of the following:
terraform providers
terraform 0.13upgrade
remove the .terraform folder
terraform state replace-provider
Instead we simply:
In a clean folder (no local state, using local terraform.exe version 0.13.7) ran 'terraform init'
Made a small insignificant change (to ensure apply would write state) to a .tf file in the workspace
In Terraform Cloud set the workspace version to 0.13.7
Using local 0.13.7 terraform.exe ran apply - that saved new state.
Now we can use cloud and local terraform.exe version 1.1.8 and no more problems.
Note that we did also need to update a few AWS S3-related resources to the newer AWS provider syntax to get all our workspaces working with the latest provider.
We encountered a similar problem in our operational environments today. We successfully completed the terraform 0.13upgrade command. This indeed introduced a versions.tf file.
However, performing a terraform init with this setup was still not possible, and the following error popped up:
Error: Invalid legacy provider address
Further investigation in the state file revealed that, for some resources, the provider block was not updated. We hence had to run the following command to finalize the upgrade process.
terraform state replace-provider "registry.terraform.io/-/google" "hashicorp/google"
EDIT Deployment to the next environment revealed that this was caused by conditional resources. To easily enable/disable some resources we leverage the count attribute and use either 0 or 1. For the resources with count = 0, that were unaltered with Terraform 0.13, the provider was not updated.
I was using terragrunt with remote s3 state and dynamo db and sadly this does not work for me. So posting it here might help someone else.
A long way to make this work, as terragrunt state replace-provider does work for me
download the state file from s3
aws s3 cp s3://bucket-name/path/terraform.tfstate terraform.tfstate --profile profile
replace the provider using terraform
terraform state replace-provider "registry.terraform.io/-/random" "hashicorp/random"
terraform state replace-provider "registry.terraform.io/-/aws" "hashicorp/aws"
upload the state file back to s3 as even terragrunt state push terraform.tfstate does not work for me
aws s3 cp terraform.tfstate s3://bucket-name/path/terraform.tfstate --profile profile
terragrunt apply
the command will throw error with digest value,
update the dynamo db table digest value that received in previous command
Initializing the backend...
Error refreshing state: state data in S3 does not have the expected content.
This may be caused by unusually long delays in S3 processing a previous state
update. Please wait for a minute or two and try again. If this problem
persists, and neither S3 nor DynamoDB are experiencing an outage, you may need
to manually verify the remote state and update the Digest value stored in the
DynamoDB table to the following value: fe2840edf8064d9225eea6c3ef2e5d1d
finally, run terragrunt apply
The other way that this can be strange is if you are using terraform workspaces - especially with the remote state files.
Using a terraform workspace - the order of operations is important.
terraform init - connecting to the default workspace
terraform workspace switch <env> - Even if you have specified the workspace here, the init will happen using the default workspace.
This is an assumption that terraform makes - sometimes erroneously
To fix this - you can run your init using:
TF_WORKSPACE=<your_env> terraform init
Or remove the default workspace.
I have a repository with several separate configs which share some modules, and reference those modules using relative paths that look like ../../modules/rabbitmq. The directories are setup like this:
tf/
configs/
thing-1/
thing-2/
modules/
rabbitmq/
cluster/
...
The configs are setup with a remote backend to use TF Cloud for runs and state:
terraform {
backend "remote" {
hostname = "app.terraform.io"
organization = "my-org"
workspaces {
prefix = "config-1-"
}
}
}
Running terraform init works fine. When I try to run terrform plan locally, it gives me an error saying:
Initializing modules...
- rabbitmq in
Error: Unreadable module directory
Unable to evaluate directory symlink: lstat ../../modules: no such file or
directory
...as if the modules directory isn't being uploaded to TF Cloud or something. What gives?
It turns out the problem was (surprise, surprise!) it was not uploading the modules directory to TF Cloud. This is because neither the config nor the TF Cloud workspace settings contained any indication that this config folder was part of a larger filesystem. The default is to upload just the directory from which you are running terraform (and all of its contents).
To fix this, I had to visit the "Settings > General" page for the given workspace in Terraform Cloud, and change the Terraform Working Directory setting to specify the path of the config, relative to the relevant root directory - in this case: tf/configs/config-1
After that, running terraform plan displays a message indicating which parent directory it will upload in order to convey the entire context relevant to the workspace. 🎉
Update #mlsy answer with a screenshot. Using Terraform Cloud with free account. Resolving module source to using local file system.
terraform version
Terraform v1.1.7
on linux_amd64
Here is the thing I worked for me. I used required_version = ">= 0.11"
and then put all those tf files which have provider and module in a subfolder. Kept the version.tf which has required providers at root level. Somehow I have used the same folder path where terraform.exe is present. Then Built the project instead of executing at main.tf level or doing execution without building. It downloaded all providers and modules for me. I am yet to run on GCP.
enter image description here - Folder path on Windows
enter image description here - InteliJ structure
enter image description hereenter image description here
Use this source = "mhmdio/rabbitmq/aws
I faced this problem when I started. Go to hashicorp/terraform site and search module/provider block. They have full path. The code snippets are written this way. Once you get path run Terraform get -update Terraform init - upgrade
Which will download modules and provider locally.
Note: on cloud the modules are in repo but still you need to give path if by default repo path not mapped
I have similar issue, which I think someone might encounter.
I have issue where in my project the application is hosted inside folder1/folder2. However when I run terraform plan inside the folder2 there was an issue because it tried to load every folder from the root repository.
% terraform plan
Running plan in the remote backend. Output will stream here. Pressing Ctrl-C
will stop streaming the logs, but will not stop the plan running remotely.
Preparing the remote plan...
The remote workspace is configured to work with configuration at
infrastructure/prod relative to the target repository.
Terraform will upload the contents of the following directory,
excluding files or directories as defined by a .terraformignore file
at /Users/yokulguy/Development/arepository/.terraformignore (if it is present),
in order to capture the filesystem context the remote workspace expects:
/Users/yokulguy/Development/arepository
╷
│ Error: Failed to upload configuration files: Failed to get symbolic link destination for "/Users/yokulguy/Development/arepository/docker/mysql/mysql.sock": lstat /private/var/run/mysqld: no such file or directory
│
│ The configured "remote" backend encountered an unexpected error. Sometimes this is caused by network connection problems, in which case you could retry the command. If the issue persists please open a support
│ ticket to get help resolving the problem.
╵
The solution is sometime I just need to remove the "bad" folder whic is the docker/mysql and then rerun the terraform plan and it works.