Gitlab CI: terraform destroy doesn't destroy? - terraform

Question

I have defined the following, simple pipeline:
image:
name: hashicorp/terraform:light
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
variables:
PLAN: dbrest.tfplan
STATE: dbrest.tfstate
cache:
paths:
- .terraform
before_script:
- terraform --version
- terraform init
stages:
- validate
- build
- deploy
- destroy
validate:
stage: validate
script:
- terraform validate
plan:
stage: build
script:
- terraform plan -state=$STATE -out=$PLAN
artifacts:
name: plan
paths:
- $PLAN
- $STATE
apply:
stage: deploy
environment:
name: production
script:
- terraform apply -state=$STATE -input=false $PLAN
- terraform state show aws_instance.bastion
dependencies:
- plan
when: manual
only:
- master
destroy:
stage: destroy
environment:
name: production
script:
- terraform destroy -state=$STATE -auto-approve
dependencies:
- apply
when: manual
only:
- master
When I run it, everything succeeds wonderfully - but the destroy stage doesn't in fact destroy the environment I've created in the apply stage. This is what I see:
Running with gitlab-runner 10.5.0 (80b03db9)
on ip-10-74-163-110 5cf66672
Using Docker executor with image hashicorp/terraform:light ...
Pulling docker image hashicorp/terraform:light ...
Using docker image sha256:5d5c9faad78b96bb84555a584fe729260d7ff7d3fb973e105690ddc0dab48fb5 for hashicorp/terraform:light ...
Running on runner-5cf66672-project-1136-concurrent-0 via ip-10-197-79-116...
Fetching changes...
Removing .terraform/
Removing dbrest.tfplan
Removing dbrest.tfstate
HEAD is now at f798b05 Update .gitlab-ci.yml
Checking out f798b05a as master...
Skipping Git submodules setup
Checking cache for default-1...
Successfully extracted cache
$ terraform --version
Terraform v0.12.13
+ provider.aws v2.34.0
$ terraform init
Initializing the backend...
Initializing provider plugins...
The following providers do not have any version constraints in configuration,
so the latest version was installed.
To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.
* provider.aws: version = "~> 2.34"
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
$ terraform destroy -state=$STATE -auto-approve
Destroy complete! Resources: 0 destroyed.
Creating cache default-1...
.terraform: found 5 matching files
Created cache
Job succeeded
It seems obvious that something is missing in the way I call terraform destroy, but I don't know what - can somebody shed some light on this, please?

Answer 1

You aren't correctly passing the state from the apply job because you haven't set the artifacts up like you did for plan -> apply. Your apply job should look like this:
apply:
stage: deploy
environment:
name: production
script:
- terraform apply -state=$STATE -input=false $PLAN
- terraform state show aws_instance.bastion
artifacts:
name: apply
paths:
- $STATE
dependencies:
- plan
when: manual
only:
- master
A better solution, however, would be to not use file based state here and instead use proper remote state (eg S3 if you're using AWS) or you're going to have a ton of problems later on when multiple users (including CI as a potentially self concurrent user) are running Terraform. This allows you to take advantage of state locking and also allow for versioning the state file in case things go wrong during a Terraform operation such as moving state as part of a refactor.