I have been developing Terraform v0.14.7 to run via GitLab Pipeline. Everything's been going well until I tried upgrading Terraform to v0.15.5 (Upgrading above that is beyond my control at the moment). I followed the advice here and a how-to guide here.
When I run terraform version on my Terraform server, it shows the proper upgraded version, and I can successfully run terraform init each time just fine. However, when I run my pipeline, the version seems to flip between the pre, validate, and plan stages with no consistent pattern. I have it print terraform version at the start of each stage, and sometimes pre outputs 0.14.7, validate outputs 0.15.5, and then plan flips back to 0.14.7. Sometimes the order is different as well, such as pre outputting 0.15.5 and the other two stages 0.14.7. I can't figure out why it's doing this since I can't recreate the problem manually.
It's been very confusing trying to solve - I tried clearing the runner cache, removing the .terraform cache and .terraform.lock.hcl files, and switching up how I run the init (adding backend=false or lockfile=readonly flags). Clearing the cache and files allows for one single clean pipeline run, but the next run is always back to being mixed versions. Does anyone know what I'm doing wrong?
Edit: The closest documented issue I can find is this issue here, but a solution was never posted. I have also tried terraform refresh, init -migrate-state, and init -reconfigure with no luck.
Edit 2: I am running Terraform offline with no internet access, so upgrades need to be done manually in case that changes any answer.
Edit 3: I'm unable to paste the exact code but here's as much as I can give, not that the output at the beginning of each stage is Terraform has been successfully initialized!:
.gitlab-ci.yml:
before_script:
- |
export TF_VAR_pipeline=$CI_PIPELINE_URL
terraform -chdir="$WORKING_DIR" init -backend=false
pre:
stage: pre
script:
- which terraform
- terraform version
validate:
stage: validate
script:
- which terraform
- terraform version
- terraform -chdir="$WORKING_DIR" validate
plan:
stage: plan
script:
- which terraform
- terraform version
- terraform -chdir="$WORKING_DIR" plan -out=plan_file.txt
artifacts:
paths:
- $WORKING_DIR/plan_file.txt
when: on success
example pre output:
$ which terraform
/usr/bin/terraform
$ terraform version
Terraform v0.15.5
on linux_amd64
Saving cache
Creating cache default-3...
...
No URL provided, cache will be not up-loaded to shared cache server. Cache will be stored only locally.
Created cache
Cleaning up file based variables
Job succeeded
example validate output:
$ which terraform
/usr/bin/terraform
$ terraform version
Terraform v0.15.5
on linux_amd64
$ terraform -chdir="$WORKING_DIR" validate
Success! The configuration is valid.
Saving cache
Creating cache default-3...
...
No URL provided, cache will be not up-loaded to shared cache server. Cache will be stored only locally.
Created cache
Cleaning up file based variables
Job succeeded
example plan output:
$ which terraform
/usr/bin/terraform
$ terraform version
Terraform v0.14.7
$ terraform -chdir="$WORKING_DIR" plan -out=plan_file
An execution plan has been generated and is shown below.
...
Unfortunately this issue was never solved. I believe it had something to do with the Gitlab cache but I couldn't figure it out. I ended up spinning up a new Terraform server instance and installing a fresh gitlab-runner, all seems to be working well now.
I just installed aws cli on my pc, and I want to run a terraform script to build an instance.
Here is the error message:
Error refreshing state: AccessDenied: Access Denied
status code: 403, request id:
I think it's a problem with the SSL certificate because when I do an aws s3 ls s3://MyBucketName --no-verify-ssl
How can I import the certificate?
I didn't see any documentation about how we can configure aws cli on windows.
Thanks,
John.
I doubt whether it's a ssl cert. problem. Try running
aws s3 ls
If it gives the following error -
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
Then it's a ssl cert problem. Otherwise try these steps -
Delete the .terraform directory
Place the access_key and secret_key under the backend block. like below given code
Run terraform init
backend "s3" {
bucket = "great-name-terraform-state-2"
key = "global/s3/terraform.tfstate"
region = "eu-central-1"
access_key = "<access-key>"
secret_key = "<secret-key>"
}
}
Should work.
In case it's an SLS cert problem,
If you want to use SSL and not have to specify the --no-verify-ssl option, then you need to set the AWS_CA_BUNDLE environment variable. e.g from PowerShell:
setx AWS_CA_BUNDLE "C:\Users\UserX\Documents\RootCert.pem"
The PEM file is a saved copy of the root certificate for the AWS endpoint you are trying to connect to. To generate it, first export the certificate in DER format (For details on how to do this, see here. Then run the following command to convert to the PEM format:
openssl x509 -inform der -in "C:\Users\UserX\Documents\RootCert.der" -out RootCert.pem
If you are using Powershell and not bash, then you will need to first install openssl.
For a full list of environment variables supported by the AWS CLI, see here
I'm trying to deploy a bitbucket pipeline using terraform v0.14.3 to create resources in google cloud. after running terraform command, the pipeline fails with this error:
Error: Invalid legacy provider address
This configuration or its associated state refers to the unqualified provider
"google".
You must complete the Terraform 0.13 upgrade process before upgrading to later
versions.
We updated our local version of terraform to v.0.13.0 and then ran: terraform 0.13upgrade as referenced in this guide: https://www.terraform.io/upgrade-guides/0-13.html. A versions.tf file was generated requiring terraform version >=0.13 and our required provider block now looks like this:
terraform {
backend "gcs" {
bucket = "some-bucket"
prefix = "terraform/state"
credentials = "key.json" #this is just a bitbucket pipeline variable
}
required_providers {
google = {
source = "hashicorp/google"
version = "~> 2.20.0"
}
}
}
provider "google" {
project = var.project_ID
credentials = "key.json"
region = var.project_region
}
We still get the same error when initiating the bitbucket pipeline. Does anyone know how to get past this error? Thanks in advance.
Solution
If you are using a newer version of Terraform, such as v0.14.x, you should:
use the replace-provider subcommand
terraform state replace-provider \
-auto-approve \
"registry.terraform.io/-/google" \
"hashicorp/google"
#=>
Terraform will perform the following actions:
~ Updating provider:
- registry.terraform.io/-/google
+ registry.terraform.io/hashicorp/google
Changing x resources:
. . .
Successfully replaced provider for x resources.
initialize Terraform again:
terraform init
#=>
Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/google from the dependency lock file
- Using previously-installed hashicorp/google vx.xx.x
Terraform has been successfully initialized!
You may now begin working with Terraform. Try . . .
This should take care of installing the provider.
Explanation
Terraform only supports upgrades from one major feature upgrade at a time. Your older state file was, more than likely, created using a version earlier than v0.13.x.
If you did not run the apply command before you upgraded your Terraform version, you can expect this error: the upgrade from v0.13.x to v0.14.x was not complete.
You can find more information here.
in our case, we were on aws and had similar error
...
Error: Invalid legacy provider address
This configuration or its associated state refers to the unqualified provider
"aws".
the steps to resolve were :
ensure syntax was upgraded by running terraform init again
check the warnings and resolve them
and finally updating the statefile with following method.
# update provider in state file
terraform state replace-provider -- -/aws hashicorp/aws
# reinit
terraform init
specific to ops problem, if issue still occurs, verify access to the bucket location from local and from pipeline. also verify the version of terraform running in pipeline. depending on configuration it may be the remote statefile is/can not be updated.
Same issue for me. I ran:
terraform providers
That gave me:
Providers required by configuration:
registry.terraform.io/hashicorp/google
Providers required by state:
registry.terraform.io/-/google
So I ran:
terraform state replace-provider registry.terraform.io/-/google registry.terraform.io/hashicorp/google
That did the trick.
To add on, I had installed terraform 0.14.6 but the state seemed to be stuck in 0.12. In my case I had 3 references that were off, this article helped me pinpoint which ones (all the entries in "Providers required by state" which had a - in the link. https://github.com/hashicorp/terraform/issues/27615
I corrected it by running the replace-provider command for each entry which was off, then running terraform init. I note doing this and running a git diff, the tfstate has been updated and now uses 0.14.x terraform instead of my previous 0.12.x. i.e.
terraform providers
terraform state replace-provider registry.terraform.io/-/azurerm registry.terraform.io/hashicorp/azurerm
Explanation: Your terraform project contains tf.state file that is outdated and refereeing to old provider address. The Error message will present this error:
Error: Invalid legacy provider address
This configuration or its associated state refers to the unqualified provider
<some-provider>.
You must complete the Terraform <some-version> upgrade process before upgrading to later
versions.
Solution: In order to solve this issue you should change the tf.state references to link to the newer required providers, update the tf.state file and initialize the project again. The steps are:
Create / Edit the required providers block with the relevant package name and version, I'd rather doing it on versions.tf file.
example:
terraform {
required_version = ">= 0.14"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.35.0"
}
}
}
Run terraform providers command to present the required providers from configuration against the required providers that saved on state.
example:
Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/aws] >= 3.35.0
Providers required by state:
provider[registry.terraform.io/-/aws]
Switch and reassign the required provider source address in the terraform state ( using terraform state replace-provider command) so we can tell terraform how to interpret the legacy provider.
The terraform state replace-provider subcommand allows re-assigning
provider source addresses recorded in the Terraform state, and so we
can use this command to tell Terraform how to reinterpret the "legacy"
provider addresses as properly-namespaced providers that match with
the provider source addresses in the configuration.
Warning: The terraform state replace-provider subcommand, like all of
the terraform state subcommands, will create a new state snapshot and
write it to the configured backend. After the command succeeds the
latest state snapshot will use syntax that Terraform v0.12 cannot
understand, so you should perform this step only when you are ready to
permanently upgrade to Terraform v0.13.
example:
terraform state replace-provider registry.terraform.io/-/aws registry.terraform.io/hashicorp/aws
output:
~ Updating provider:
- registry.terraform.io/-/aws
+ registry.terraform.io/hashicorp/aws
run terraform init to update references.
While you were under TF13 did you apply state at least once for the running project?
According to TF docs: https://www.terraform.io/upgrade-guides/0-14.html
There is no automatic update command (separately) in 0.14 (like there was in 0.13). The only way to upgrade is to force state on a project at least once, while under command when moving TF13 to 14.
You can also try terraform init in the project directory.
my case was like this
Error: Invalid legacy provider address
This configuration or its associated state refers to the unqualified provider
"openstack".
You must complete the Terraform 0.13 upgrade process before upgrading to later
versions.
for resolving the issue
remove the .terraform folder
the execute the following command
terraform state replace-provider -- -/openstack terraform-provider-openstack/openstack
after this command, you will see the below print, enter yes
Terraform will perform the following actions:
~ Updating provider:
- registry.terraform.io/-/openstack
+ registry.terraform.io/terraform-provider-openstack/openstack
Changing 11 resources:
openstack_compute_servergroup_v2.kubernetes_master
openstack_networking_network_v2.kube_router
openstack_compute_instance_v2.kubernetes_worker
openstack_networking_subnet_v2.internal
openstack_networking_subnet_v2.kube_router
data.openstack_networking_network_v2.external_network
openstack_compute_instance_v2.kubernetes_etcd
openstack_networking_router_interface_v2.internal
openstack_networking_router_v2.internal
openstack_compute_instance_v2.kubernetes_master
openstack_networking_network_v2.internal
Do you want to make these changes?
Only 'yes' will be accepted to continue.
Enter a value: yes
Successfully replaced provider for 11 resources.
I recently ran into this using Terraform Cloud for the remote backend. We had some older AWS-related workspaces set to version 0.12.4 (in the cloud) that errored out with "Invalid legacy provider address" and refused to run with the latest Terraform client 1.1.8.
I am adding my answer because it is much simpler than the other answers. We did not do any of the following:
terraform providers
terraform 0.13upgrade
remove the .terraform folder
terraform state replace-provider
Instead we simply:
In a clean folder (no local state, using local terraform.exe version 0.13.7) ran 'terraform init'
Made a small insignificant change (to ensure apply would write state) to a .tf file in the workspace
In Terraform Cloud set the workspace version to 0.13.7
Using local 0.13.7 terraform.exe ran apply - that saved new state.
Now we can use cloud and local terraform.exe version 1.1.8 and no more problems.
Note that we did also need to update a few AWS S3-related resources to the newer AWS provider syntax to get all our workspaces working with the latest provider.
We encountered a similar problem in our operational environments today. We successfully completed the terraform 0.13upgrade command. This indeed introduced a versions.tf file.
However, performing a terraform init with this setup was still not possible, and the following error popped up:
Error: Invalid legacy provider address
Further investigation in the state file revealed that, for some resources, the provider block was not updated. We hence had to run the following command to finalize the upgrade process.
terraform state replace-provider "registry.terraform.io/-/google" "hashicorp/google"
EDIT Deployment to the next environment revealed that this was caused by conditional resources. To easily enable/disable some resources we leverage the count attribute and use either 0 or 1. For the resources with count = 0, that were unaltered with Terraform 0.13, the provider was not updated.
I was using terragrunt with remote s3 state and dynamo db and sadly this does not work for me. So posting it here might help someone else.
A long way to make this work, as terragrunt state replace-provider does work for me
download the state file from s3
aws s3 cp s3://bucket-name/path/terraform.tfstate terraform.tfstate --profile profile
replace the provider using terraform
terraform state replace-provider "registry.terraform.io/-/random" "hashicorp/random"
terraform state replace-provider "registry.terraform.io/-/aws" "hashicorp/aws"
upload the state file back to s3 as even terragrunt state push terraform.tfstate does not work for me
aws s3 cp terraform.tfstate s3://bucket-name/path/terraform.tfstate --profile profile
terragrunt apply
the command will throw error with digest value,
update the dynamo db table digest value that received in previous command
Initializing the backend...
Error refreshing state: state data in S3 does not have the expected content.
This may be caused by unusually long delays in S3 processing a previous state
update. Please wait for a minute or two and try again. If this problem
persists, and neither S3 nor DynamoDB are experiencing an outage, you may need
to manually verify the remote state and update the Digest value stored in the
DynamoDB table to the following value: fe2840edf8064d9225eea6c3ef2e5d1d
finally, run terragrunt apply
The other way that this can be strange is if you are using terraform workspaces - especially with the remote state files.
Using a terraform workspace - the order of operations is important.
terraform init - connecting to the default workspace
terraform workspace switch <env> - Even if you have specified the workspace here, the init will happen using the default workspace.
This is an assumption that terraform makes - sometimes erroneously
To fix this - you can run your init using:
TF_WORKSPACE=<your_env> terraform init
Or remove the default workspace.
I have integrated my self hosted Gitlab with Hashicorp vault. I have followed the steps here https://docs.gitlab.com/ee/ci/examples/authenticating-with-hashicorp-vault/ and tried to run the pipeline.
I am receiving the certificate error while running the pipeline.
Error writing data to auth/jwt/login: Put "https://vault.systems:8200/v1/auth/jwt/login": x509: certificate signed by unknown authority
My .gitlab yml file -
Vault Client:
image:
name: vault:latest
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
before_script:
script:
- export VAULT_ADDR=https:/vault.systems:8200/
- export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=staging jwt=$CI_JOB_JWT)"
- export PASSWORD="$(vault kv get -field=password kv/project/staging/db)"
- echo $PASSWORD
If i use -tls-skip-verify flag then it works fine.
Do i need to place the self signed server certificate somewhere on the vault server or gitlab server?
Please let me know if anyone has any ideas on this one?
The containers that are managed by the docker/kube executor must be configured to trust the self-signed cert(s). You can edit the config.toml for your runner to mount in the trusted certs/CA roots to GitLab CI job containers
For example, on Linux-based docker executors:
[[runners]]
name = "docker"
url = "https://example.com/"
token = "TOKEN"
executor = "docker"
[runners.docker]
image = "ubuntu:latest"
# Add path to your ca.crt file in the volumes list
volumes = ["/cache", "/path/to-ca-cert-dir/ca.crt:/etc/gitlab-runner/certs/ca.crt:ro"]
See the docs for more info.
I was able to solve this by using this variable VAULT_CACERT in my gitlab.yml file :
- export VAULT_CACERT=/etc/gitlab-runner/certs/ca.crt. The certificate path here is the path of the mounted container which we specify during the start of container.
Posting this so if anyone is looking for it, this is the solution. :)
Error writing data to auth/jwt/login: Put "https://vault.systems:8200/v1/auth/jwt/login": x509: certificate signed by unknown authority
The error you're receiving is being returned from Vault, so it's Vault that you need to get to accept that certificate. There's a decent note on how to do it in the Deployment Guide. (I used to work for HashiCorp Vault so I knew where to dig it up.)
You can use -tls-skip-verify in your vault command vault kv get -tls-skip-verify -field=password kv/project/staging/db , or if you have vault's ca-cert you have to export CA CERT path by setting VAULT_CACERT to the right path .
Within Octopus Deploy I've setup a Terraform Apply Step using their Apply a Terraform template
In my Terraform main.tf file I want to use a connection to run an remote-exec on a Amazon Linux EC2 instance in AWS
resource "aws_instance" "nginx" {
ami = "${var.aws_ami}"
instance_type = "t2.nano"
key_name = "${var.key_name}"
connection {
type = "ssh"
user = "ec2-user"
private_key = "${var.aws_key_path}"
}
provisioner "remote-exec" {
inline = [
"sudo amazon-linux-extras install epel -y",
"sudo yum update -y",
"sudo amazon-linux-extras install nginx1.12 -y",
"sudo systemctl enable nginx.service",
"sudo systemctl start nginx.service",
"sudo systemctl status nginx.service"
]
}
}
As part of the connection block we need to connect using an SSH key pair using a Private Key PEM to auth with the Public Key stored on AWS
My Private Key is stored as a variable in my Project in Octopus deploy
For my private key to be interpreted correctly in Terraform as a multi-line string I had to use 'here doc' syntax using a starting EOF and an ending EOF
This syntax explanation can be found on the Terraform official documentation at
https://www.terraform.io/docs/configuration-0-11/syntax.html
This was my original problem that my variable syntax was falling over as I wasn't handling the multi-line PEM file correctly and I raised the ticket below with Octopus Deploy Support
https://help.octopus.com/t/terraform-apply-step-pem-variable-set-to-unix-lf-ucs-2-le-bom/23659
Where they kindly were able to point me in the direction of the EOF syntax
This all worked great on Terraform v0.11 but we've a lot of code here on this side that's been written in the latest HCL2 in v0.12
So I wanted to force Octopus Deploy to use a v0.12 binary rather than the prepackaged v0.11 that Octopus Deploy comes with. And they offer a built in Special var so you can use a different binary
But when I run it with this binary the script blows up with the error below
Error: Unterminated template string
No closing marker was found for the string.
August 6th 2019 14:54:07 Error
Calamari.Integration.Processes.CommandLineException: The following command: "C:\Program Files\Octopus Deploy\Octopus\bin\terraform.exe" apply -no-color -auto-approve -var-file="octopus_vars.tfvars"
August 6th 2019 14:54:07 Error
With the working directory of: C:\Octopus\Work\20190806135350-47862-353\staging
August 6th 2019 14:54:07 Error
Failed with exit code: 1
August 6th 2019 14:54:07 Error
Error: Unterminated template string
August 6th 2019 14:54:07 Error
on octopus_vars.tfvars line 34:
I've had a look at the official documentation for v0.12
https://www.terraform.io/docs/configuration/syntax.html#terraform-syntax
And I'm not sure if there is anything that helps in relation to how to manage multi-line that they had in v0.11
Here is the code block that worked in v0.11 successfully from my tfvars file
aws_ami = "#{ami}"
key_name = "#{awsPublicKey}"
aws_private_key = <<-EOF
#{testPrivateKey}
-EOF
The expected result when I ran this with the latest version of Terraform v0.12.6 was that it would function normally and run my Terraform Apply within Octopus Deploy
My hope here is that someone from Hashicorp has a workaround for this as I see this was supposed to be fixed with https://github.com/hashicorp/terraform/pull/20281
But I'm using the latest binary at the time of writing this v0.12.6 downloaded today
Any suggestions anyone on how to get this working in v0.12? Cheers
The correct syntax for a "flush heredoc" does not include a dash on the final marker:
aws_key_path = <<-EOF
#{martinTestPrivateKey}
EOF
If prior versions were accepting -EOF to end the heredoc then that unfortunately was a bug, which has now been fixed in Terraform 0.12 and so moving forward you must use the syntax as documented, with the marker alone on the final line.