GitHub Action for reading JSON file - terraform

I would like to write a GH action that uses JSON data as a parameter.
The plan is to save the Terraform output as a JSON file in one step and access it in the sequential step.
name: Test and Terraform
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
Unit-Tests:
...
Terraform:
runs-on: ubuntu-18.04
needs: [ Unit-Tests, S3-Sync ]
steps:
...
- name: Terraform Apply
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
run: |
cd terraform
terraform apply -auto-approve
terraform output --json > output.json
- name: Set Terraform Output
id: output
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
run: |
content=`cat ./terraform/output.json`
# the following lines are only required for multi line json
content="${content//'%'/'%25'}"
content="${content//$'\n'/'%0A'}"
content="${content//$'\r'/'%0D'}"
# end of optional handling for multi line json
echo "::set-output name=terraform::$content"
- name: Gatsby Cloud Sync
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
run: |
echo "${{fromJson(steps.output.outputs.terraform)}}"
The current error I get with this is that the final step is formatted improperly.
The template is not valid. .github/workflows/main.yml (Line: 154, Col: 14): Unexpected character encountered while parsing value: c. Path '', line 1, position 1.
How is the last step, Gatsby Cloud Sync, formatted incorrectly?

Related

Gitlabci: How to pass dynamic variable to downstream pipeline to use it for remote include line?

I can pass the variable successfully from upstream to downstream pipeline. But I can't get the dynamic variable value for ref: $branch line. I'm getting the following error in the downstream pipeline: Project main/project reference does not exist! I think that somehow I can't give the value to ref section.
Upstream pipeline:
---
stages:
- test
test:
stage: test
variables:
branch: "dev"
trigger:
project: child/test
strategy: depend
branch: master
allow_failure: false
rules:
- if: $CI_COMMIT_BRANCH == 'master' || $CI_COMMIT_BRANCH == 'dev'
when: never
Child pipeline:
---
stages:
- hellotest
hellotest_1:
stage: hellotest
trigger:
include:
- project: 'main/project'
file: 'samplejob/example.gitlab-ci.yml'
ref: $branch

$CI_ENVIRONMENT_NAME remains empty after set in previous stage

I have the following GitLab pipeline. For brevity I have removed some code. In deploy-dev stage I am creating environment dev. Then in the next stage test-postdeploy I am using $CI_ENVIRONMENT_NAME to set the base_url variable value. However, in test-postdeploy stage the $CI_ENVIRONMENT_NAME always empty
stages:
deploy:dev
deploy:qa
deploy:prod
test:postdeploy
deploy-dev:
image: node
stage: deploy:dev
environment:
name: dev
script:
- . deploy/scripts/deploy-dev.sh
rules:
- if: $CI_COMMIT_REF_NAME == "dev"
deploy-qa:
image: node
stage: deploy:qa
environment:
name: qa
script:
- . deploy/scripts/deploy-qa.sh
rules:
- if: $CI_COMMIT_REF_NAME == "qa"
deploy-prod:
image: node
stage: deploy:prod
environment:
name: prod
script:
- . deploy/scripts/deploy-prod.sh
rules:
- if: $CI_COMMIT_REF_NAME == "release"
test-postdeploy:
tags:
- xlarge
image: node
stage: test:postdeploy
variables:
base_url: ""
script:
- echo "$CI_ENVIRONMENT_NAME" // this is empty
- script/postdeploy-test.sh
rules:
- if: $CI_ENVIRONMENT_NAME == "dev"
variables:
base_url: "https://dev.example.com"
- if: $CI_ENVIRONMENT_NAME == "qa"
variables:
base_url: "https://qa.example.com"
- if: $CI_ENVIRONMENT_NAME == "prod"
variables:
base_url: "https://prod.example.com"
UPDATE 1
I can probably pass environment name to another stage using https://docs.gitlab.com/ee/ci/variables/#pass-an-environment-variable-to-another-job
However, this does not make sense. Environment Name is most basic value that any ci/cd pipeline would need to know. The CI_ENVIRONMENT_NAME
variable should be available in all stages once its set without doing some extra work.

Gitlab CI automatical run pipeline on merge request

I have below pipeline.
After creating merge request, created Detached merge request pipeline with failed status (app1 - no stages/jobs).
In scope of below pipeline need to run pipeline when merge request is created and after merging changes main. Flow described in here Gitlab CI Child pipeline
Below pipeline does not work.
workflow:
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule"'
- if: '$CI_PIPELINE_SOURCE == "web"'
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
- if: '$CI_PIPELINE_SOURCE == "push"'
stages:
- child-pipelines
app1:
stage: child-pipelines
variables:
COMPONENT NAME: 'app1'
trigger:
include:
- local: .ci/.gitlab-ci.yml
strategy: depend
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- test1/**/*
variables:
DEPLOY_RELEASE: '11111'
- if : '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "main"'
changes:
- test1/**/*
variables:
DEPLOY_RELEASE: '222222'
app2:
stage: child-pipelines
variables:
COMPONENT NAME: 'app1'
trigger:
include:
- local: .ci/.gitlab-ci.yml
strategy: depend
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- test2/**/*
variables:
DEPLOY_RELEASE: '11111'
- if : '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "main"'
changes:
- test2/**/*
variables:
DEPLOY_RELEASE: '222222'
If you want to run a pipeline when merge request was created and after merge to main branch, take a look this example:
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
job1:
script:
- echo "This job runs in merge request and also after merge to main branch"
According to Gitlab documentation.
If you want to run a rule for the entire pipeline:
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
job1:
script:
- echo "This job runs in merge request pipelines"
job2:
script:
- echo "This job also runs in merge request pipelines"
If you want to run a rule in certain job:
job1:
script:
- echo "This job runs in merge request pipelines"
only:
- merge_requests

Why is TF build failing with "Error refreshing state: HTTP remote state endpoint requires auth"?

My pipeline builds, plans and applies just fine for my dev branch. When I push to my master branch, I get "Error refreshing state: HTTP remote state endpoint requires auth". See pipeline logs: (and since someone will ask, the token "$onbaord_cloud_account_into_monitoring" has complete read/write access to the scoped project API.)...
Running with gitlab-runner 13.4.1 (REDACTED)
on runner-gitlab-runner-REDACTED-REDACTED REDACTED
Resolving secrets
00:00
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-managed-apps
Using Kubernetes executor with image my-gitlab.io:5005/team-cloud-platform-team/terraform-modules/root-module-deployment/python-terraform14 ...
Preparing environment
00:03
Waiting for pod gitlab-managed-apps/runner-REDACTED-project-REDACTED-concurrent-REDACTED to be running, status is Pending
Running on runner-REDACTED-project-REDACTED-concurrent-REDACTED via runner-gitlab-runner-REDACTED-REDACTED...
Getting source from Git repository
00:02
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/team-cloud-platform-team/teamcloudv2/workers/onboard-cloud-account-into-monitoring/.git/
Created fresh repository.
Checking out REDACTED as master...
Skipping Git submodules setup
Restoring cache
00:00
Checking cache for REDACTEDREDACTEDREDACTEDREDACTED...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
Executing "step_script" stage of the job script
00:05
$ git config --global credential.helper cache
$ git fetch
From https://my-gitlab.io/team-cloud-platform-team/teamcloudv2/workers/onboard-cloud-account-into-monitoring
* [new branch] dev -> origin/dev
$ if [ "$CI_COMMIT_REF_NAME" == "master" ]; then TF_ACCOUNT="REDACTED";fi
$ if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then TF_ACCOUNT="REDACTED";fi
$ if [ "$CI_COMMIT_REF_NAME" == "master" ]; then ORG_ACCOUNT="REDACTED";fi
$ if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then ORG_ACCOUNT="REDACTED";fi
$ echo ${TF_ACCOUNT}
REDACTED
$ echo ${TF_ADDRESS}
https://my-gitlab.io/api/v4/projects/10849/terraform/state/onboard-cloud-account-into-monitoring-master
$ echo TF_HTTP_ADDRESS=${TF_ADDRESS}
TF_HTTP_ADDRESS=https://my-gitlab.io/api/v4/projects/10849/terraform/state/onboard-cloud-account-into-monitoring-master
$ echo TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
TF_HTTP_LOCK_ADDRESS=https://my-gitlab.io/api/v4/projects/10849/terraform/state/onboard-cloud-account-into-monitoring-master/lock
$ echo TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
TF_HTTP_UNLOCK_ADDRESS=https://my-gitlab.io/api/v4/projects/10849/terraform/state/onboard-cloud-account-into-monitoring-master/lock
$ echo TF_ADDRESS=${TF_ADDRESS}
TF_ADDRESS=https://my-gitlab.io/api/v4/projects/10849/terraform/state/onboard-cloud-account-into-monitoring-master
$ export TF_HTTP_ADDRESS=${TF_ADDRESS}
$ export TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
$ export TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
$ export TF_HTTP_LOCK_METHOD="POST"
$ export TF_HTTP_UNLOCK_METHOD="DELETE"
$ export TF_HTTP_RETRY_WAIT_MIN='5'
$ export TF_HTTP_USERNAME='JOHN.DOE'
$ export TF_HTTP_PASSWORD=${onbaord_cloud_account_into_monitoring}
$ export TF_VAR_ACCOUNT=${TF_ACCOUNT}
$ export TF_VAR_ORG_ACCOUNT=${ORG_ACCOUNT}
$ export AWS_DEFAULT_REGION='us-east-1'
$ terraform init
Initializing modules...
Downloading git::https://my-gitlab.io/team-cloud-platform-team/teamcloudv2/terraform/lambda-modules.git?ref=dev for lambda...
- lambda in .terraform/modules/lambda
Downloading git::https://my-gitlab.io/team-cloud-platform-team/teamcloudv2/terraform/iam-modules/lambda-iam.git?ref=dev for lambda_iam...
- lambda_iam in .terraform/modules/lambda_iam
Initializing the backend...
Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
Error refreshing state: HTTP remote state endpoint requires auth
Cleaning up file based variables
00:00
ERROR: Job failed: command terminated with exit code 1
A peek at my gitlab-ci.yml:
image:
name: my-gitlab.io:5005/team-cloud-platform-team/terraform-modules/root-module-deployment/python-terraform14
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
stages:
- build
- plan
- apply
variables:
PLAN: plan.tfplan
JSON_PLAN_FILE: tfplan.json
WORKSPACE: "dev"
TF_IN_AUTOMATION: "true"
ZIP_FILE: "lambda_package.zip"
STATE_FILE: ${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}
TF_ACCOUNT: ""
ORG_ACCOUNT: ""
TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}
TF_LOCK: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}/lock
TF_UNLOCK: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}/lock
cache:
key: "$CI_COMMIT_SHA"
paths:
- .terraform
before_script:
- git config --global credential.helper cache
- git fetch
- if [ "$CI_COMMIT_REF_NAME" == "master" ]; then TF_ACCOUNT="REDACTED";fi
- if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then TF_ACCOUNT="REDACTED";fi
- if [ "$CI_COMMIT_REF_NAME" == "master" ]; then ORG_ACCOUNT="REDACTED";fi
- if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then ORG_ACCOUNT="REDACTED";fi
- echo ${TF_ACCOUNT}
- echo ${TF_ADDRESS}
- echo TF_HTTP_ADDRESS=${TF_ADDRESS}
- echo TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
- echo TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
- echo TF_ADDRESS=${TF_ADDRESS}
- export TF_HTTP_ADDRESS=${TF_ADDRESS}
- export TF_HTTP_LOCK_ADDRESS=${TF_LOCK}
- export TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK}
- export TF_HTTP_LOCK_METHOD="POST"
- export TF_HTTP_UNLOCK_METHOD="DELETE"
- export TF_HTTP_RETRY_WAIT_MIN='5'
- export TF_HTTP_USERNAME='MATTHEW.FETHEROLF'
- export TF_HTTP_PASSWORD=${onbaord_cloud_account_into_monitoring}
- export TF_VAR_ACCOUNT=${TF_ACCOUNT}
- export TF_VAR_ORG_ACCOUNT=${ORG_ACCOUNT}
- export AWS_DEFAULT_REGION='us-east-1'
- terraform init
# -----BUILD-----
# 1 build job per lambda function
buildLambda:
stage: build
tags:
- cluster
script:
- echo "Beginning Build"
- cd lambda_code/
- echo "Switched into Lambda dir"
- pip3 install -r requirements.txt --target .
- echo "Requirements installed"
- echo $ZIP_FILE
- zip -r $ZIP_FILE *
- echo "Zip file packaged up"
artifacts:
paths:
- lambda_code/
only:
- dev
- master
- merge_requests
# -----PLAN-----
planMerge:
stage: plan
tags:
- cluster
script:
- terraform plan
dependencies:
- buildLambda
only:
- merge_requests
planCommit:
stage: plan
tags:
- cluster
script:
- terraform plan
dependencies:
- buildLambda
# -----APPLY-----
# dev branch will auto deploy
applyDev:
stage: apply
tags:
- cluster
script:
- echo "Running Terraform Apply"
- terraform apply -auto-approve
dependencies:
- buildLambda
only:
- dev
#when: manual
# prod branch will require manual deployment approval
applyProd:
stage: apply
tags:
- cluster
script:
- echo "Running Terraform Apply"
- terraform apply -auto-approve
when: manual
dependencies:
- buildLambda
only:
- master
backend.tf:
terraform {
backend "http" {
}
}
main.tf:
locals {
common_tags = {
SVC_ACCOUNT_ID = "REDACTED",
CLOUD_PLATFORM_PROD_ID = "REDACTED"
}
}
module "lambda" {
source = "git::https://my-gitlab.io/team-cloud-platform-team/teamcloudv2/terraform/lambda-modules.git?ref=dev"
lambda_name = var.name
lambda_role = "arn:aws:iam::${var.ACCOUNT}:role/${var.lambda_role}"
lambda_handler = var.handler
lambda_runtime = var.runtime
default_lambda_timeout = var.timeout
env = local.common_tags
ACCOUNT = var.ACCOUNT
}
module "lambda_iam" {
source = "git::https://my-gitlab.io/team-cloud-platform-team/teamcloudv2/terraform/iam-modules/lambda-iam.git?ref=dev"
lambda_policy = var.lambda_policy
ACCOUNT = var.ACCOUNT
lambda_role = var.lambda_role
}
inputs.tf:
variable "handler" {
type = string
default = "handler.lambda_handler"
}
variable "runtime" {
type = string
default = "python3.8"
}
variable "name" {
type = string
default = "onboard-cloud-account-into-monitoring"
}
variable "timeout"{
type = string
default = "120"
}
variable "lambda_role" {
type = string
default = "cloud-platform-onboard-to-monitoring"
}
variable "ACCOUNT" {
type = string
}
variable "ORG_ACCOUNT" {
type = string
}
variable "lambda_policy" {
default = "{\"Version\": \"2012-10-17\",\"Statement\": [{\"Sid\": \"VisualEditor0\",\"Effect\": \"Allow\",\"Action\": [\"logs:CreateLogStream\",\"logs:CreateLogGroup\"],\"Resource\": \"*\"},{\"Sid\": \"VisualEditor1\",\"Effect\": \"Allow\",\"Action\": \"logs:PutLogEvents\",\"Resource\": \"*\"},{\"Sid\": \"VisualEditor2\",\"Effect\": \"Allow\",\"Action\": \"sts:AssumeRole\",\"Resource\": \"*\"},{\"Sid\": \"VisualEditor3\",\"Effect\": \"Allow\",\"Action\": \"secretsmanager:GetSecretValue\",\"Resource\": \"*\"}]}"
}
provider.tf:
provider "aws" {
region = "us-east-1"
assume_role {
role_arn ="arn:aws:iam::${var.ACCOUNT}:role/team-platform-onboard-to-monitoring"
}
default_tags {
tags = {
owner = "REDACTED#REDACTED.com"
altowner = "REDACTED#REDACTED.com"
blc = "REDACTED"
costcenter = "REDACTED"
itemid = "PlatformAutomation"
}
}
}
Perhaps I also need to share the terraform modules being invoked?
We found the problem:
The pipeline referenced a protected environment variable. The master branch was not a protected branch. The solution was to unprotect the environment variable or protect the branch. Instantly solved the problem.

Azure Pipelines YAML: Unexpected value 'variables'

I'm using Azure Pipelines as a part of Azure DevOps.
I'm trying to define variables in my template file, because I need to use the same value multiple times.
This is my stage-template.yml:
parameters:
- name: param1
type: string
- name: param2
type: string
variables:
var1: path/${{ parameters.param2 }}/to-my-file.yml
stages:
- stage: Deploy${{ parameters.param2 }}
displayName: Deploy ${{ parameters.param1 }}
jobs:
...
When trying to use this pipeline, I get an error message:
/stage-template.yml (Line: 7, Col: 1): Unexpected value 'variables'
Why is this not working? What did I do wrong?
You can't have variables in a template that is included as a stage, job or step template (i.e. included below a stages, jobs or steps element in a pipeline). You can only use variables in a variable template.
The documentation sadly is not really clear about that.
Including a stage template
# pipeline-using-stage-template.yml
stages:
- stage: stage1
[...]
# stage template reference, no 'variables' element allowed in stage-template.yml
- template: stage-template.yml
Including a variable template
# pipeline-using-var-template.yml
variables:
# variable template reference, only variables allowed inside the template
- template: variables.yml
steps:
- script: echo A step.
If you are using a template to include variables in a pipeline, the included template can only be used to define variables.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops#variable-reuse
you cant have parameters in the pipeline, only in the templateReferences:
name: string # build numbering format
resources:
pipelines: [ pipelineResource ]
containers: [ containerResource ]
repositories: [ repositoryResource ]
variables: # several syntaxes, see specific section
trigger: trigger
pr: pr
stages: [ stage | templateReference ]
if you want to use variables in templates you have to use proper syntax:
parameters:
- name: param1
type: string
- name: param2
type: string
stages:
- stage: Deploy${{ parameters.param2 }}
displayName: Deploy ${{ parameters.param1 }}
variables:
var1: path/${{ parameters.param2 }}/to-my-file.yml
jobs:
...
This works for me:
In your parent yaml:
stages:
- stage: stage1
displayName: 'stage from parent'
jobs:
- template: template1.yml
parameters:
param1: 'somevalueforparam1'
inside template1:
parameters:
param1: ''
param2: ''
jobs:
- job: job1
workspace:
clean: all
displayName: 'Install job'
pool:
name: 'yourpool'
variables:
- name: var1
value: 'value1'

Resources