Gitlab CI: terraform destroy doesn't destroy? - terraform

I have defined the following, simple pipeline:
image:
name: hashicorp/terraform:light
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
variables:
PLAN: dbrest.tfplan
STATE: dbrest.tfstate
cache:
paths:
- .terraform
before_script:
- terraform --version
- terraform init
stages:
- validate
- build
- deploy
- destroy
validate:
stage: validate
script:
- terraform validate
plan:
stage: build
script:
- terraform plan -state=$STATE -out=$PLAN
artifacts:
name: plan
paths:
- $PLAN
- $STATE
apply:
stage: deploy
environment:
name: production
script:
- terraform apply -state=$STATE -input=false $PLAN
- terraform state show aws_instance.bastion
dependencies:
- plan
when: manual
only:
- master
destroy:
stage: destroy
environment:
name: production
script:
- terraform destroy -state=$STATE -auto-approve
dependencies:
- apply
when: manual
only:
- master
When I run it, everything succeeds wonderfully - but the destroy stage doesn't in fact destroy the environment I've created in the apply stage. This is what I see:
Running with gitlab-runner 10.5.0 (80b03db9)
on ip-10-74-163-110 5cf66672
Using Docker executor with image hashicorp/terraform:light ...
Pulling docker image hashicorp/terraform:light ...
Using docker image sha256:5d5c9faad78b96bb84555a584fe729260d7ff7d3fb973e105690ddc0dab48fb5 for hashicorp/terraform:light ...
Running on runner-5cf66672-project-1136-concurrent-0 via ip-10-197-79-116...
Fetching changes...
Removing .terraform/
Removing dbrest.tfplan
Removing dbrest.tfstate
HEAD is now at f798b05 Update .gitlab-ci.yml
Checking out f798b05a as master...
Skipping Git submodules setup
Checking cache for default-1...
Successfully extracted cache
$ terraform --version
Terraform v0.12.13
+ provider.aws v2.34.0
$ terraform init
Initializing the backend...
Initializing provider plugins...
The following providers do not have any version constraints in configuration,
so the latest version was installed.
To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.
* provider.aws: version = "~> 2.34"
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
$ terraform destroy -state=$STATE -auto-approve
Destroy complete! Resources: 0 destroyed.
Creating cache default-1...
.terraform: found 5 matching files
Created cache
Job succeeded
It seems obvious that something is missing in the way I call terraform destroy, but I don't know what - can somebody shed some light on this, please?

You aren't correctly passing the state from the apply job because you haven't set the artifacts up like you did for plan -> apply. Your apply job should look like this:
apply:
stage: deploy
environment:
name: production
script:
- terraform apply -state=$STATE -input=false $PLAN
- terraform state show aws_instance.bastion
artifacts:
name: apply
paths:
- $STATE
dependencies:
- plan
when: manual
only:
- master
A better solution, however, would be to not use file based state here and instead use proper remote state (eg S3 if you're using AWS) or you're going to have a ton of problems later on when multiple users (including CI as a potentially self concurrent user) are running Terraform. This allows you to take advantage of state locking and also allow for versioning the state file in case things go wrong during a Terraform operation such as moving state as part of a refactor.

Related

Github Actions Terraform Init initialized empty directory

I am new to using Github actions and coding into YAML file.
Currently, I setup Terraform Cloud - Github actions for my Datadog POC.
I arrived on the issue:
terraform init
/home/runner/work/_temp/85297372-6fed-4b1d-88f8-3c6b5527569f/terraform-bin init
Terraform initialized in an empty directory!
The directory has no Terraform configuration files. You may begin working
with Terraform immediately by creating Terraform configuration files.
and the current github actions yaml file is:
I use the terraform github actions yaml file
name: 'Terraform'
on:
push:
branches:
- "main"
pull_request:
permissions:
contents: read
jobs:
terraform:
name: 'Terraform'
runs-on: ubuntu-latest
environment: production
# Use the Bash shell regardless whether the GitHub Actions runner is ubuntu-latest, macos-latest, or windows-latest
defaults:
run:
shell: bash
steps:
# Checkout the repository to the GitHub Actions runner
- name: Checkout
uses: actions/checkout#v3
# Install the latest version of Terraform CLI and configure the Terraform CLI configuration file with a Terraform Cloud user API token
- name: Setup Terraform
uses: hashicorp/setup-terraform#v1
with:
cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}
# Initialize a new or existing Terraform working directory by creating initial files, loading any remote state, downloading modules, etc.
- name: Terraform Init
run: terraform init
# Checks that all Terraform configuration files adhere to a canonical format
- name: Terraform Format
run: terraform fmt -check
# Generates an execution plan for Terraform
- name: Terraform Plan
run: terraform plan -input=false
# On push to "main", build or change infrastructure according to Terraform configuration files
# Note: It is recommended to set up a required "strict" status check in your repository for "Terraform Cloud". See the documentation on "strict" required status checks for more information: https://help.github.com/en/github/administering-a-repository/types-of-required-status-checks
- name: Terraform Apply
if: github.ref == 'refs/heads/"main"' && github.event_name == 'push'
run: terraform apply -auto-approve -input=false
What remedy should I do here?
I was trying to change the directory included in the terraform init to
run:
working-directory: ./DataDog-Demo/terraform
but I received also error.
Thank you
You will have to cd into the directory which has all the terraform files. Something like this
- name: Build Docker image
run: |
cd dir
Alternatively, you can also set working directory like this
steps:
- uses: actions/checkout#v1
- name: Setup and run tests
working-directory: ./app
run: |
cp .env .env
OR
jobs:
unit:
runs-on: ubuntu-latest
defaults:
run:
working-directory: ./app
steps:
- uses: actions/checkout#v1
- name: Do stuff
https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepsrun

Terraform apply being run twice in gitlab CI - how to prevent?

We have a pipeline that includes "terraform plan" and "terraform apply" as separate CI steps, so that in production we can manually review changes before applying (however in review apps / staging we're happy for them to run automatically). The plan is passed as an artifact between the jobs.
We've had a couple of issues where developers have re-run the "terraform apply" job without re-running "terraform plan". I'm trying to work out how to identify this and prevent it.
I'm surprised that the terraform plan doesn't e.g. include a hash of the terraform state, and so apply could identify that the state has changed and refuse to continue.
Is there a suggested way to fix this? We've tried:
Searching for options in terraform to avoid this (nothing so far)
Searching for options in gitlab to avoid this (nothing so far)
We're currently looking into taking our own checksum of the tfstate file in the plan stage, and then checking that at the start of the apply stage - but I can't help feeling this ought to be out there already.
(State is stored in an S3 bucket. We also use dynamodb for locking)
Cut down .gitlab-ci.yml for illustration:
stages:
- plan
- apply
terraform plan:
image: hashicorp/terraform:0.12.26
stage: plan
script:
- terraform init
- terraform plan -out terraform.plan
artifacts:
paths:
- terraform.plan
terraform apply:
image: hashicorp/terraform:0.12.26
stage: apply
script:
- terraform apply -auto-approve terraform.plan
rules:
- if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"
when: manual
- when: on_success

How can I get a gitlab pipeline to stop and ask me to input variables?

I have created a pipeline in gitlab, with
image:
name: hashicorp/terraform:light
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
variables:
PLAN: dbrest.tfplan
STATE: dbrest.tfstate
cache:
paths:
- .terraform
before_script:
- terraform --version
- terraform init
stages:
- validate
- build
- deploy
- destroy
validate:
stage: validate
script:
- terraform validate
plan:
stage: build
script:
- terraform plan -state=$STATE -out=$PLAN
artifacts:
name: plan
paths:
- $PLAN
- $STATE
apply:
stage: deploy
environment:
name: production
script:
- terraform apply -state=$STATE -input=false $PLAN
- terraform state show aws_instance.bastion
dependencies:
- plan
when: manual
only:
- master
destroy:
stage: destroy
environment:
name: production
script:
- terraform destroy -state=$STATE -auto-approve
dependencies:
- apply
when: manual
only:
- master
I have also created a variable under 'Settings. -> 'CI/CD' -> 'Variables' - I was under the impression that when I came to the manual stage deploy, gitlab should pause and ask me to input a value for that variable, but this does not happen - what is missing?
You have mixed a job with when: manual to when you trigger a pipeline manually. This is the one you want:
https://docs.gitlab.com/ee/ci/pipelines/#run-a-pipeline-manually
You could use this together with an only for some variable. Something like:
...
apply:
stage: deploy
environment:
name: production
script:
- terraform apply -state=$STATE -input=false $PLAN
- terraform state show aws_instance.bastion
dependencies:
- plan
only:
refs:
- master
variables:
- $RELEASE == "yes"
destroy:
stage: destroy
environment:
name: production
script:
- terraform destroy -state=$STATE -auto-approve
dependencies:
- apply
only:
refs:
- master
variables:
- $RELEASE == "yes"
With something like this, you can have jobs that are never run normally, but only if you manually start a new pipeline on the master branch and set the variable $RELEASE to yes. I haven't tested this, so my apologies if it doesn't work!
I saw your comment and I still think there's more needed for a complete answer, but I do believe this will work for what you're asking for. I included what I've used in the past as well, "MODE" but then you'd need to update the Rules section on your jobs to reflect when you wanted the job to run based on MODE vs RELEASE
variables:
PLAN: dbrest.tfplan
STATE: dbrest.tfstate
RELEASE:
description: "Provide YES or NO to trigger jobs"
default: "NO"
MODE:
description: "Actions for terraform to perform, ie: PLAN, APPLY, DESTROY, PLAN_APPLY, ALL"
default: "PLAN"

Terraform apply command fails when runs twice in Gitlab pipeline and resources were already created [duplicate]

This question already has an answer here:
Terraform destroy fails in CircleCI
(1 answer)
Closed 3 years ago.
I'm quite new with Terraform and I'm trying to replicate in my terraform configuration the stack I have already built for production (basically: Api gateway - Lambda - DynamoDB).
If I run terraform init, terraform plan and then terraform apply from my local host, everything is created as I want.
The problem arises when it comes to my Gitlab CI/CD pipeline, as Terraform complains about the existing resources (the first time runs properly, the second time complains and throws an error).
My Terraform steps in my .gitlab-ci.yml file:
plan:
stage: plan
image:
name: hashicorp/terraform:light
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
script:
- cd terraform
- rm -rf .terraform
- terraform --version
- terraform init
- terraform plan
deploy:
stage: deploy
image:
name: hashicorp/terraform:light
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
script:
- cd terraform
- terraform init
- terraform apply -auto-approve
dependencies:
- plan
when: manual
I see in my pipeline console the following error:
So after some Googling I saw that maybe the terraform import command could help.
Then added this import command to my .gitlab-ci.yml:
script:
- cd terraform
- terraform init
- terraform import aws_dynamodb_table.demo-dynamodb-table demo-dynamodb-table
- terraform apply -auto-approve
And the error in the Gitlab console was:
In the meantime I tried also this last change locally, and the error was:
So to summarize: I would need to know how to use Terraform in the right way to be able to run the apply command in my Gitlab CI/CD pipeline without conflicts with the resource that was created in the previous run of this same pipeline.
As others have stated, you need to store the Terraform state.
In my GitLab projects, I use a S3 bucket to store the Terraform state. But, have the CI pipeline fill in the key based on the GitLab project's path by setting the TF_CLI_ARGS_init environment variable.
terraform {
backend "s3" {
bucket = "bucket-name-here"
region = "us-west-2"
# key = $CI_PROJECT_PATH_SLUG
}
}
I also set the Terraform workspace based on the project. This can be modified to support branches. I also set the name variable to the project name, for use in the Terraform configuration. And, set input to false so that the CI job doesn't get hung up on user prompts.
variables:
TF_INPUT: "false"
TF_WORKSPACE: "$CI_PROJECT_NAME"
TF_VAR_name: "$CI_PROJECT_NAME"
TF_CLI_ARGS_init: "-upgrade=true"
For destroys, I also make sure to delete the workspace, so that there isn't stuff left over in the bucket.
.destroy:
extends: .terraform
stage: Cleanup
script:
- terraform init
- terraform destroy
-auto-approve
- export WORKSPACE=$TF_WORKSPACE
- export TF_WORKSPACE=default
- terraform workspace delete "$WORKSPACE"

Terraform destroy fails in CircleCI

I am currently using CircleCI as my CI tool to build AWS infrastructure using Terraform
My flow is,
Create an AWS instance using Terraform
Install Docker and run Nginx image on it
Destroy the infrastructure
My CircleCI config is as follows,
version: 2
jobs:
terraform_apply:
working_directory: ~/tmp
docker:
- image: hashicorp/terraform:light
- image: ubuntu:16.04
steps:
- checkout
- run:
name: terraform apply
command: |
terraform init
terraform apply -auto-approve
- store_artifacts:
path: terraform.tfstate
terraform_destroy:
working_directory: ~/tmp
docker:
- image: hashicorp/terraform:light
- image: ubuntu:16.04
steps:
- checkout
- run:
name: terraform destroy
command: |
terraform init
terraform destroy -auto-approve
workflows:
version: 2
terraform:
jobs:
- terraform_apply
- click_here_to_delete:
type: approval
requires:
- terraform_apply
- terraform_destroy:
requires:
- click_here_to_delete
Here I am using 2 jobs, One for the creation and one for Deletion in CircleCI workflow.
My first job is running successfully but when I started second it start from scratch so I could not get previous terraform apply state hence terraform could not destroy my already created infrastructure.
I am looking for some solution where I can somehow save state file and copy it to next job where terraform can destroy my previous architecture
You should be using remote state.
Local state is only ever useful if you are always running from the same machine and don't care about loss of your state file if you accidentally delete something etc.
You can mix and match any of the available state backends but as you're using AWS already it probably makes most sense to use the S3 backend.
You will need to define the state configuration for each location which can be done entirely hardcoded in config, entirely by command line flags or partially with both.
As an example you should have something like this block in each of the directories you would run Terraform in:
terraform {
backend "s3" {}
}
You could then finish configuring this during terraform init:
terraform init -backend-config="bucket=uniquely-named-terraform-state-bucket" \
-backend-config="key=state-key/terraform.tfstate"
Once you have ran terraform init, Terraform will fetch the state from S3 for any plans. Then on a terraform apply or terraform destroy it will update the state file as necessary.
This will then allow you to share the state easily among colleagues and also CI/CD machines. You should also consider looking into state locking using DynamoDB to prevent state from being corrupted by multiple people modifying state at the same time. Equally you should also consider enabling versioning on the S3 bucket used for storing your state so you can always get back to an earlier version of the state in the event of any issues.

Resources