Managing multiple terraform statefiles with gitlab - gitlab

Question

Just as an example, my git repository would look like so:
└── git_repo_example
├── modules
│   ├── module_1
│   │   ├── main.tf
│   │   └── var.tf
│   └── module_2
│   ├── main.tf
│   └── var.tf
└── projects
├── project_1
│   ├── main.tf
│   ├── terraform.tfstate
│   └── vars.tf
└── project_2
├── main.tf
├── terraform.tfstate
└── vars.tf
7 directories, 10 files
My team wants to make our terraform state files gitlab-managed, so that the statefiles would be locked in case multiple people want to run or modify a single project at the same time.
All of the examples I can find for managing terraform via gitlab only seem to assume 1 tfstate file and project, but my repository has multiple. Breaking this up into multiple repositories would be difficult to manage since they all reference the same modules, and it seems that placing everything into one folder is against terraform best-practices.
How would one best go about managing one repository with multiple terraform projects / statefiles via gitlab?

Answer 1

I have a similar-ish directory structure that works well with GitLab managed state per project directory.
I'd recommend replacing local TF development with GitLab CI/CD pipelines, using the provided GitLab container image as it supports the GitLab backend by default.
I use environments (representing each project directory) to mange the pipeline (CI/CD variables are managed by environment). The TF statefile is named according to the TF_ADDRESS variable:
image: registry.gitlab.com/gitlab-org/terraform-images/stable:latest
variables:
TF_ROOT: ${CI_PROJECT_DIR}/${ENVIRONMENT}
TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${ENVIRONMENT}
Here a build job is defined to create the TF plan and run only when the development directory is modified and merged to the default branch. An identical job for the production directory is also defined. Each environment references a unique TF state file managed by GitLab:
.plan:
stage: build
environment:
name: $ENVIRONMENT
script:
- gitlab-terraform plan
- gitlab-terraform plan-json
cache:
policy: pull # don't update the cache
artifacts:
name: plan
paths:
- ${TF_ROOT}/plan.cache
reports:
terraform: ${TF_ROOT}/plan.json
Development Build Terraform Plan:
extends: .plan
variables:
ENVIRONMENT: development
rules:
- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
changes:
- "*"
- "development/**/*"
Production Build Terraform Plan:
extends: .plan
variables:
ENVIRONMENT: production
rules:
- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
changes:
- "*"
- "production/**/*"

Answer 2

I have exactly the same kind of terraform scripts repository, with a "run" script at the top, which will do, for each application, a
cd modules/modulesx
terraform init -backend-config=backend.tf -reconfigure
With backend.tf (here for Azure):
container-name = "tfbackend"
key = "aStorageAccount/aFileShare/path/to/modulex.tfstate"
resource_group_name = "xxx"
That does create a modules/modulex/.terraform/terraform.tfstate
However, this file is local and not versioned nor "locked".