I want to use a GitLab Runner to deploy to AWS with Terraform. I have setup AWS credentials in GitLab "Variables" (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY). I must be misunderstanding how .gitlab-cy.yml performs string interpolation because I cannot get the credentials to populate.
The stage in question looks like this:
validate:
stage: validate
dependencies:
- lint
- unit
image:
name: hashicorp/terraform:light
entrypoint:
- "/usr/bin/env"
- "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
- "AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}"
- "AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}"
before_script:
- rm -rf terraform/test/.terraform
- terraform --version
- terraform init -input=false -backend-config="access_key=${AWS_ACCESS_KEY_ID}" -backend-config="secret_key=${AWS_SECRET_ACCESS_KEY}" terraform/test
script:
- terraform validate
The pipeline fails without fail on the terraform init command. However, just to confirm I'm not crazy, I did try a pipeline run with the credentials hardcoded and it worked (I also immediately learned about how to permanently delete commits and pipelines).
From the relevant GitLab documentation on variable usage, I don't see anything obviously wrong.
Error message:
Initializing the backend...
Error: error using credentials to get account ID: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
Related
I am using Gitlab terraform & the yaml file is as below for deploy stage,
build:
extends: .terraform:build
script:
- cd "${TF_ROOT}"
- gitlab-terraform plan --var-file=local.tfvars --var-file=common.tfvars
- gitlab-terraform plan-json --var-file=local.tfvars --var-file=common.tfvars
deploy:
extends: .terraform:deploy
script:
- cd "${TF_ROOT}"
- gitlab-terraform apply --var-file=local.tfvars --var-file=common.tfvars
environment:
name: $TF_STATE_NAME
First time it deployed perfectly, However my Gitlab log shows the following error now for the second time,
│ Error: Can't set variables when applying a saved plan
The -var and -var-file options cannot be used when applying a saved plan
file, because a saved plan includes the variable values that were set when
it was created.
ERROR: Job failed: exit code 1
I am looking through the internet but no help till now.
Can someone please suggest me the terraform command I should use to avoid this error ?
I'm trying to set up Terrafom validation on Gitlab CI.
However a build fails with an error: "Terraform has no command named "sh". Did you mean "show"?"
Why does it happen? How could it be fixed?
My .gitlab-ci.yml
image: hashicorp/terraform:light
before_script:
- terraform init
validate:
script:
- terraform validate
You need to override the entrypoint in the terraform image so you have access to the shell.
image:
name: hashicorp/terraform:light
entrypoint: [""]
before_script:
- terraform init
validate:
script:
- terraform validate
You can also take a look at the official gitlab documentation how to integrate terraform with gitlab, as the have a template for that.
I have defined the following, simple pipeline:
image:
name: hashicorp/terraform:light
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
variables:
PLAN: dbrest.tfplan
STATE: dbrest.tfstate
cache:
paths:
- .terraform
before_script:
- terraform --version
- terraform init
stages:
- validate
- build
- deploy
- destroy
validate:
stage: validate
script:
- terraform validate
plan:
stage: build
script:
- terraform plan -state=$STATE -out=$PLAN
artifacts:
name: plan
paths:
- $PLAN
- $STATE
apply:
stage: deploy
environment:
name: production
script:
- terraform apply -state=$STATE -input=false $PLAN
- terraform state show aws_instance.bastion
dependencies:
- plan
when: manual
only:
- master
destroy:
stage: destroy
environment:
name: production
script:
- terraform destroy -state=$STATE -auto-approve
dependencies:
- apply
when: manual
only:
- master
When I run it, everything succeeds wonderfully - but the destroy stage doesn't in fact destroy the environment I've created in the apply stage. This is what I see:
Running with gitlab-runner 10.5.0 (80b03db9)
on ip-10-74-163-110 5cf66672
Using Docker executor with image hashicorp/terraform:light ...
Pulling docker image hashicorp/terraform:light ...
Using docker image sha256:5d5c9faad78b96bb84555a584fe729260d7ff7d3fb973e105690ddc0dab48fb5 for hashicorp/terraform:light ...
Running on runner-5cf66672-project-1136-concurrent-0 via ip-10-197-79-116...
Fetching changes...
Removing .terraform/
Removing dbrest.tfplan
Removing dbrest.tfstate
HEAD is now at f798b05 Update .gitlab-ci.yml
Checking out f798b05a as master...
Skipping Git submodules setup
Checking cache for default-1...
Successfully extracted cache
$ terraform --version
Terraform v0.12.13
+ provider.aws v2.34.0
$ terraform init
Initializing the backend...
Initializing provider plugins...
The following providers do not have any version constraints in configuration,
so the latest version was installed.
To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.
* provider.aws: version = "~> 2.34"
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
$ terraform destroy -state=$STATE -auto-approve
Destroy complete! Resources: 0 destroyed.
Creating cache default-1...
.terraform: found 5 matching files
Created cache
Job succeeded
It seems obvious that something is missing in the way I call terraform destroy, but I don't know what - can somebody shed some light on this, please?
You aren't correctly passing the state from the apply job because you haven't set the artifacts up like you did for plan -> apply. Your apply job should look like this:
apply:
stage: deploy
environment:
name: production
script:
- terraform apply -state=$STATE -input=false $PLAN
- terraform state show aws_instance.bastion
artifacts:
name: apply
paths:
- $STATE
dependencies:
- plan
when: manual
only:
- master
A better solution, however, would be to not use file based state here and instead use proper remote state (eg S3 if you're using AWS) or you're going to have a ton of problems later on when multiple users (including CI as a potentially self concurrent user) are running Terraform. This allows you to take advantage of state locking and also allow for versioning the state file in case things go wrong during a Terraform operation such as moving state as part of a refactor.
This question already has an answer here:
Terraform destroy fails in CircleCI
(1 answer)
Closed 3 years ago.
I'm quite new with Terraform and I'm trying to replicate in my terraform configuration the stack I have already built for production (basically: Api gateway - Lambda - DynamoDB).
If I run terraform init, terraform plan and then terraform apply from my local host, everything is created as I want.
The problem arises when it comes to my Gitlab CI/CD pipeline, as Terraform complains about the existing resources (the first time runs properly, the second time complains and throws an error).
My Terraform steps in my .gitlab-ci.yml file:
plan:
stage: plan
image:
name: hashicorp/terraform:light
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
script:
- cd terraform
- rm -rf .terraform
- terraform --version
- terraform init
- terraform plan
deploy:
stage: deploy
image:
name: hashicorp/terraform:light
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
script:
- cd terraform
- terraform init
- terraform apply -auto-approve
dependencies:
- plan
when: manual
I see in my pipeline console the following error:
So after some Googling I saw that maybe the terraform import command could help.
Then added this import command to my .gitlab-ci.yml:
script:
- cd terraform
- terraform init
- terraform import aws_dynamodb_table.demo-dynamodb-table demo-dynamodb-table
- terraform apply -auto-approve
And the error in the Gitlab console was:
In the meantime I tried also this last change locally, and the error was:
So to summarize: I would need to know how to use Terraform in the right way to be able to run the apply command in my Gitlab CI/CD pipeline without conflicts with the resource that was created in the previous run of this same pipeline.
As others have stated, you need to store the Terraform state.
In my GitLab projects, I use a S3 bucket to store the Terraform state. But, have the CI pipeline fill in the key based on the GitLab project's path by setting the TF_CLI_ARGS_init environment variable.
terraform {
backend "s3" {
bucket = "bucket-name-here"
region = "us-west-2"
# key = $CI_PROJECT_PATH_SLUG
}
}
I also set the Terraform workspace based on the project. This can be modified to support branches. I also set the name variable to the project name, for use in the Terraform configuration. And, set input to false so that the CI job doesn't get hung up on user prompts.
variables:
TF_INPUT: "false"
TF_WORKSPACE: "$CI_PROJECT_NAME"
TF_VAR_name: "$CI_PROJECT_NAME"
TF_CLI_ARGS_init: "-upgrade=true"
For destroys, I also make sure to delete the workspace, so that there isn't stuff left over in the bucket.
.destroy:
extends: .terraform
stage: Cleanup
script:
- terraform init
- terraform destroy
-auto-approve
- export WORKSPACE=$TF_WORKSPACE
- export TF_WORKSPACE=default
- terraform workspace delete "$WORKSPACE"
I am currently using CircleCI as my CI tool to build AWS infrastructure using Terraform
My flow is,
Create an AWS instance using Terraform
Install Docker and run Nginx image on it
Destroy the infrastructure
My CircleCI config is as follows,
version: 2
jobs:
terraform_apply:
working_directory: ~/tmp
docker:
- image: hashicorp/terraform:light
- image: ubuntu:16.04
steps:
- checkout
- run:
name: terraform apply
command: |
terraform init
terraform apply -auto-approve
- store_artifacts:
path: terraform.tfstate
terraform_destroy:
working_directory: ~/tmp
docker:
- image: hashicorp/terraform:light
- image: ubuntu:16.04
steps:
- checkout
- run:
name: terraform destroy
command: |
terraform init
terraform destroy -auto-approve
workflows:
version: 2
terraform:
jobs:
- terraform_apply
- click_here_to_delete:
type: approval
requires:
- terraform_apply
- terraform_destroy:
requires:
- click_here_to_delete
Here I am using 2 jobs, One for the creation and one for Deletion in CircleCI workflow.
My first job is running successfully but when I started second it start from scratch so I could not get previous terraform apply state hence terraform could not destroy my already created infrastructure.
I am looking for some solution where I can somehow save state file and copy it to next job where terraform can destroy my previous architecture
You should be using remote state.
Local state is only ever useful if you are always running from the same machine and don't care about loss of your state file if you accidentally delete something etc.
You can mix and match any of the available state backends but as you're using AWS already it probably makes most sense to use the S3 backend.
You will need to define the state configuration for each location which can be done entirely hardcoded in config, entirely by command line flags or partially with both.
As an example you should have something like this block in each of the directories you would run Terraform in:
terraform {
backend "s3" {}
}
You could then finish configuring this during terraform init:
terraform init -backend-config="bucket=uniquely-named-terraform-state-bucket" \
-backend-config="key=state-key/terraform.tfstate"
Once you have ran terraform init, Terraform will fetch the state from S3 for any plans. Then on a terraform apply or terraform destroy it will update the state file as necessary.
This will then allow you to share the state easily among colleagues and also CI/CD machines. You should also consider looking into state locking using DynamoDB to prevent state from being corrupted by multiple people modifying state at the same time. Equally you should also consider enabling versioning on the S3 bucket used for storing your state so you can always get back to an earlier version of the state in the event of any issues.