Why is TF build failing with "Error refreshing state: HTTP remote state endpoint requires auth"? - gitlab

Question

My pipeline builds, plans and applies just fine for my dev branch. When I push to my master branch, I get "Error refreshing state: HTTP remote state endpoint requires auth". See pipeline logs: (and since someone will ask, the token "$onbaord_cloud_account_into_monitoring" has complete read/write access to the scoped project API.)...
Running with gitlab-runner 13.4.1 (REDACTED)
on runner-gitlab-runner-REDACTED-REDACTED REDACTED
Resolving secrets
00:00
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-managed-apps
Using Kubernetes executor with image my-gitlab.io:5005/team-cloud-platform-team/terraform-modules/root-module-deployment/python-terraform14 ...
Preparing environment
00:03
Waiting for pod gitlab-managed-apps/runner-REDACTED-project-REDACTED-concurrent-REDACTED to be running, status is Pending
Running on runner-REDACTED-project-REDACTED-concurrent-REDACTED via runner-gitlab-runner-REDACTED-REDACTED...
Getting source from Git repository
00:02
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/team-cloud-platform-team/teamcloudv2/workers/onboard-cloud-account-into-monitoring/.git/
Created fresh repository.
Checking out REDACTED as master...
Skipping Git submodules setup
Restoring cache
00:00
Checking cache for REDACTEDREDACTEDREDACTEDREDACTED...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
Executing "step_script" stage of the job script
00:05
$ git config --global credential.helper cache
$ git fetch
From https://my-gitlab.io/team-cloud-platform-team/teamcloudv2/workers/onboard-cloud-account-into-monitoring
* [new branch] dev -> origin/dev
$ if [ "$CI_COMMIT_REF_NAME" == "master" ]; then TF_ACCOUNT="REDACTED";fi
$ if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then TF_ACCOUNT="REDACTED";fi
$ if [ "$CI_COMMIT_REF_NAME" == "master" ]; then ORG_ACCOUNT="REDACTED";fi
$ if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then ORG_ACCOUNT="REDACTED";fi
$ echo ${TF_ACCOUNT}
REDACTED
$ echo ${TF_ADDRESS}
https://my-gitlab.io/api/v4/projects/10849/terraform/state/onboard-cloud-account-into-monitoring-master
$ echo TF_HTTP_ADDRESS=${TF_ADDRESS}
TF_HTTP_ADDRESS=https://my-gitlab.io/api/v4/projects/10849/terraform/state/onboard-cloud-account-into-monitoring-master
$ echo TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
TF_HTTP_LOCK_ADDRESS=https://my-gitlab.io/api/v4/projects/10849/terraform/state/onboard-cloud-account-into-monitoring-master/lock
$ echo TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
TF_HTTP_UNLOCK_ADDRESS=https://my-gitlab.io/api/v4/projects/10849/terraform/state/onboard-cloud-account-into-monitoring-master/lock
$ echo TF_ADDRESS=${TF_ADDRESS}
TF_ADDRESS=https://my-gitlab.io/api/v4/projects/10849/terraform/state/onboard-cloud-account-into-monitoring-master
$ export TF_HTTP_ADDRESS=${TF_ADDRESS}
$ export TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
$ export TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
$ export TF_HTTP_LOCK_METHOD="POST"
$ export TF_HTTP_UNLOCK_METHOD="DELETE"
$ export TF_HTTP_RETRY_WAIT_MIN='5'
$ export TF_HTTP_USERNAME='JOHN.DOE'
$ export TF_HTTP_PASSWORD=${onbaord_cloud_account_into_monitoring}
$ export TF_VAR_ACCOUNT=${TF_ACCOUNT}
$ export TF_VAR_ORG_ACCOUNT=${ORG_ACCOUNT}
$ export AWS_DEFAULT_REGION='us-east-1'
$ terraform init
Initializing modules...
Downloading git::https://my-gitlab.io/team-cloud-platform-team/teamcloudv2/terraform/lambda-modules.git?ref=dev for lambda...
- lambda in .terraform/modules/lambda
Downloading git::https://my-gitlab.io/team-cloud-platform-team/teamcloudv2/terraform/iam-modules/lambda-iam.git?ref=dev for lambda_iam...
- lambda_iam in .terraform/modules/lambda_iam
Initializing the backend...
Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
Error refreshing state: HTTP remote state endpoint requires auth
Cleaning up file based variables
00:00
ERROR: Job failed: command terminated with exit code 1
A peek at my gitlab-ci.yml:
image:
name: my-gitlab.io:5005/team-cloud-platform-team/terraform-modules/root-module-deployment/python-terraform14
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
stages:
- build
- plan
- apply
variables:
PLAN: plan.tfplan
JSON_PLAN_FILE: tfplan.json
WORKSPACE: "dev"
TF_IN_AUTOMATION: "true"
ZIP_FILE: "lambda_package.zip"
STATE_FILE: ${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}
TF_ACCOUNT: ""
ORG_ACCOUNT: ""
TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}
TF_LOCK: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}/lock
TF_UNLOCK: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}/lock
cache:
key: "$CI_COMMIT_SHA"
paths:
- .terraform
before_script:
- git config --global credential.helper cache
- git fetch
- if [ "$CI_COMMIT_REF_NAME" == "master" ]; then TF_ACCOUNT="REDACTED";fi
- if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then TF_ACCOUNT="REDACTED";fi
- if [ "$CI_COMMIT_REF_NAME" == "master" ]; then ORG_ACCOUNT="REDACTED";fi
- if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then ORG_ACCOUNT="REDACTED";fi
- echo ${TF_ACCOUNT}
- echo ${TF_ADDRESS}
- echo TF_HTTP_ADDRESS=${TF_ADDRESS}
- echo TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
- echo TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
- echo TF_ADDRESS=${TF_ADDRESS}
- export TF_HTTP_ADDRESS=${TF_ADDRESS}
- export TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
- export TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
- export TF_HTTP_LOCK_METHOD="POST"
- export TF_HTTP_UNLOCK_METHOD="DELETE"
- export TF_HTTP_RETRY_WAIT_MIN='5'
- export TF_HTTP_USERNAME='MATTHEW.FETHEROLF'
- export TF_HTTP_PASSWORD=${onbaord_cloud_account_into_monitoring}
- export TF_VAR_ACCOUNT=${TF_ACCOUNT}
- export TF_VAR_ORG_ACCOUNT=${ORG_ACCOUNT}
- export AWS_DEFAULT_REGION='us-east-1'
- terraform init
# -----BUILD-----
# 1 build job per lambda function
buildLambda:
stage: build
tags:
- cluster
script:
- echo "Beginning Build"
- cd lambda_code/
- echo "Switched into Lambda dir"
- pip3 install -r requirements.txt --target .
- echo "Requirements installed"
- echo $ZIP_FILE
- zip -r $ZIP_FILE *
- echo "Zip file packaged up"
artifacts:
paths:
- lambda_code/
only:
- dev
- master
- merge_requests
# -----PLAN-----
planMerge:
stage: plan
tags:
- cluster
script:
- terraform plan
dependencies:
- buildLambda
only:
- merge_requests
planCommit:
stage: plan
tags:
- cluster
script:
- terraform plan
dependencies:
- buildLambda
# -----APPLY-----
# dev branch will auto deploy
applyDev:
stage: apply
tags:
- cluster
script:
- echo "Running Terraform Apply"
- terraform apply -auto-approve
dependencies:
- buildLambda
only:
- dev
#when: manual
# prod branch will require manual deployment approval
applyProd:
stage: apply
tags:
- cluster
script:
- echo "Running Terraform Apply"
- terraform apply -auto-approve
when: manual
dependencies:
- buildLambda
only:
- master
backend.tf:
terraform {
backend "http" {
}
}
main.tf:
locals {
common_tags = {
SVC_ACCOUNT_ID = "REDACTED",
CLOUD_PLATFORM_PROD_ID = "REDACTED"
}
}
module "lambda" {
source = "git::https://my-gitlab.io/team-cloud-platform-team/teamcloudv2/terraform/lambda-modules.git?ref=dev"
lambda_name = var.name
lambda_role = "arn:aws:iam::${var.ACCOUNT}:role/${var.lambda_role}"
lambda_handler = var.handler
lambda_runtime = var.runtime
default_lambda_timeout = var.timeout
env = local.common_tags
ACCOUNT = var.ACCOUNT
}
module "lambda_iam" {
source = "git::https://my-gitlab.io/team-cloud-platform-team/teamcloudv2/terraform/iam-modules/lambda-iam.git?ref=dev"
lambda_policy = var.lambda_policy
ACCOUNT = var.ACCOUNT
lambda_role = var.lambda_role
}
inputs.tf:
variable "handler" {
type = string
default = "handler.lambda_handler"
}
variable "runtime" {
type = string
default = "python3.8"
}
variable "name" {
type = string
default = "onboard-cloud-account-into-monitoring"
}
variable "timeout"{
type = string
default = "120"
}
variable "lambda_role" {
type = string
default = "cloud-platform-onboard-to-monitoring"
}
variable "ACCOUNT" {
type = string
}
variable "ORG_ACCOUNT" {
type = string
}
variable "lambda_policy" {
default = "{\"Version\": \"2012-10-17\",\"Statement\": [{\"Sid\": \"VisualEditor0\",\"Effect\": \"Allow\",\"Action\": [\"logs:CreateLogStream\",\"logs:CreateLogGroup\"],\"Resource\": \"*\"},{\"Sid\": \"VisualEditor1\",\"Effect\": \"Allow\",\"Action\": \"logs:PutLogEvents\",\"Resource\": \"*\"},{\"Sid\": \"VisualEditor2\",\"Effect\": \"Allow\",\"Action\": \"sts:AssumeRole\",\"Resource\": \"*\"},{\"Sid\": \"VisualEditor3\",\"Effect\": \"Allow\",\"Action\": \"secretsmanager:GetSecretValue\",\"Resource\": \"*\"}]}"
}
provider.tf:
provider "aws" {
region = "us-east-1"
assume_role {
role_arn ="arn:aws:iam::${var.ACCOUNT}:role/team-platform-onboard-to-monitoring"
}
default_tags {
tags = {
owner = "REDACTED#REDACTED.com"
altowner = "REDACTED#REDACTED.com"
blc = "REDACTED"
costcenter = "REDACTED"
itemid = "PlatformAutomation"
}
}
}
Perhaps I also need to share the terraform modules being invoked?

Answer 1

We found the problem:
The pipeline referenced a protected environment variable. The master branch was not a protected branch. The solution was to unprotect the environment variable or protect the branch. Instantly solved the problem.