how to dump gitlab ci environment variables to file - gitlab

Question

the question
How to dump all Gitlab CI environment variables (with variables set in the project or group CI/CD settings) to a file, but only them, without environment variables of the host on which gitlab runner is executed?
Background
We are using gitlab CI/CD to deploy our projects to a docker server. Each project contains a docker-compose.yml file which uses various environment variables, eg db passwords. We are using .env file to store this variables, so one can start/restart the containers after deployment from command line, without accessing gitlab.
Our deployments script looks something like this:
deploy:
script:
#...
- cp docker-compose.development.yml {$DEPLOY_TO_PATH}/docker-compose.yml
- env > variables.env
- docker-compose up -d
#...
And the docker-compose.yml file looks like this:
version: "3"
services:
project:
image: some/image
env_file:
- variables.env
...
The problem is now the .env file contains both gitlab variables and hosts system environment variables and in the result the PATH variable is overwritten.
I have developed a workaround with grep:
env | grep -Pv "^PATH" > variables.env
It allowed us to keep this working for now, but I think that the problem might hit us again with another variables which would be set to different values inside a container and on the host system.
I know I can list all the variables in docker-compose and similar files, but we already have quite a few of them in a few projects so it is not a solution.

Answer 1

You need to add to script next command
script:
...
# Read certificate stored in $KUBE_CA_PEM variable and save it in a new file
- echo "$KUBE_CA_PEM" > variables.env
...

Answer 2

This might be late, but I did something like this:
script:
- env |grep -v "CI"|grep -v "FF"|grep -v "GITLAB"|grep -v "PWD"|grep -v "PATH"|grep -v "HOME"|grep -v "HOST"|grep -v "SH" > application.properties
- cat application.properties
It's not the best, but it works.
The one problem with this is you can have variables with a string containing one of the exclusions, ie. "CI","FF","GITLAB","PWD","PATH","HOME","HOME","SH"

Answer 3

My reusable solution /tools/gitlab/script-gitlab-variables.yml:
variables:
# Default values
GITLAB_EXPORT_ENV_FILENAME: '.env.gitlab.cicd'
.script-gitlab-variables:
debug:
# section_start
- echo -e "\e[0Ksection_start:`date +%s`:gitlab_variables_debug[collapsed=true]\r\e[0K[GITLAB VARIABLES DEBUG]"
# command
- env
# section_end
- echo -e "\e[0Ksection_end:`date +%s`:gitlab_variables_debug\r\e[0K"
export-to-env:
# section_start
- echo -e "\e[0Ksection_start:`date +%s`:gitlab_variables_export_to_env[collapsed=true]\r\e[0K[GITLAB VARIABLES EXPORT]"
# verify mandatory variables
- test ! -z "$GITLAB_EXPORT_VARS" && echo "$GITLAB_EXPORT_VARS" || exit $?
# display variables
- echo "$GITLAB_EXPORT_ENV_FILENAME"
# command
- env | grep -E "^($GITLAB_EXPORT_VARS)=" > $GITLAB_EXPORT_ENV_FILENAME
# section_end
- echo -e "\e[0Ksection_end:`date +%s`:gitlab_variables_export_to_env\r\e[0K"
cat-env:
# section_start
- echo -e "\e[0Ksection_start:`date +%s`:gitlab_variables_cat-env[collapsed=true]\r\e[0K[GITLAB VARIABLES CAT ENV]"
# command
- cat $GITLAB_EXPORT_ENV_FILENAME
# section_end
- echo -e "\e[0Ksection_end:`date +%s`:gitlab_variables_cat-env\r\e[0K"
How to use .gitlab-ci.yml:
include:
- local: '/tools/gitlab/script-gitlab-variables.yml'
Your Job:
variables:
GITLAB_EXPORT_VARS: 'CI_BUILD_NAME|GITLAB_USER_NAME'
script:
- !reference [.script-gitlab-variables, debug]
- !reference [.script-gitlab-variables, export-to-env]
- !reference [.script-gitlab-variables, cat-env]
Result cat .env.gitlab.cicd:
CI_BUILD_NAME=Demo
GITLAB_USER_NAME=Benjamin
What you need dump all:
# /tools/gitlab/script-gitlab-variables.yml
dump-all:
- env > $GITLAB_EXPORT_ENV_FILENAME
# .gitlab-ci.yml
script:
- !reference [.script-gitlab-variables, dump-all]
I hope I could help