How to add env vars into Azure Devops pipeline - node.js

I am setting up an Azure pipeline for a Node app with Jest being used to test APIs and integration. The source code lives on Azure DevOps and the code is deployed in Azure Portal.
When I run the tests, it fails in the pipeline as the .env is never checked in the remote repository. The environment variables are living in the Azure Portal in runtime though configuration so the pipeline cannot really access it.
What is some ways to have access or create new location for the environment variables in order for my tests to run the the virtual machine?
My current solution (which I don't know if its right) is to create a variable group and redefine all my environment variables so the pipeline can read the variables also described here: https://damienaicheh.github.io/azure/devops/2019/09/04/how-to-use-variables-inside-your-azure-devops-builds-en.html
My questions are:
Is this correct? Any of the stored variables here have nothing to do with the build neither they are inputs to run commands, rather all my environment variables are required inside the source code so I can test in a virtual machine (Ex: base_url, apiKeys, etc).
If this is right, how can I possible avoid re-writting and re-assigning all the value in the pipeline? Can I source the entire variable group and the source code can interpret? I want to avoid like this
- env
- API_KEY: $(apiKey)
- MAPS_KEY: $(mapsKey)
- CLIENT_KEY: $(clientKey)
- CLIENT_SECRET: $(clientSecret)
-
-
- and so on...
// looking for something like this
-env: myVariableGroup
Any leads to a post, articles to a better solution? I was thinking of using key vault but I think it will essentially the same that I have to import one-by-one.

Pipeline variables are mapped to env variables automatically so no need to extra work. There is only one exception - secrets. You must mapped them explicitly
steps:
- script: echo $MYSECRET
env:
MYSECRET: $(Foo)
So all values from declaration, group or template are mapped to env vars
vars.yaml
variables:
variableFromTemplate: 'valueFromTemplate'
build.yaml
variables:
- group: PROD
- name: variableFromDeclaration
value: 'valueFromDeclaration'
- template: vars.yaml
pool:
vmImage: 'ubuntu-latest'
steps:
- script: env | sort
- script: |
echo $VARIABLEFROMDECLARATION
echo $VARIABLEFROMGROUP
echo $VARIABLEFROMTEMPLATE
- pwsh: |
$url = "https://dev.azure.com/thecodemanual/$(System.TeamProject)/_apis/build/builds/$(Build.BuildId)?api-version=5.1"
$build = Invoke-RestMethod -Uri $url -Headers #{Authorization = "Bearer $env:MY_SECRET"}
Write-Host "Pipeline = $($build | ConvertTo-Json -Depth 100)"
$status = $build.status
Write-Host $status
name: initial
env:
MY_SECRET: $(System.AccessToken)
So for each step you need to define secrets in env section. As a workaround you my try use container jobs and define env mapping on container level.
resources:
containers:
- container: string # identifier (A-Z, a-z, 0-9, and underscore)
image: string # container image name
options: string # arguments to pass to container at startup
endpoint: string # reference to a service connection for the private registry
env: { string: string } # list of environment variables to add
ports: [ string ] # ports to expose on the container
volumes: [ string ] # volumes to mount on the container
mapDockerSocket: bool # whether to map in the Docker daemon socket; defaults to true
mountReadOnly: # volumes to mount read-only - all default to false
externals: boolean # components required to talk to the agent
tasks: boolean # tasks required by the job
tools: boolean # installable tools like Python and Ruby
work: boolean # the work directory

Related

Accessing Variables in Azure pipeline

I am trying to access variables from Pipeline > Library like this
$(AndroidKeyAlias) but I can't see any output. I have tried a couple of other ways as well.
azure-pipelines.yml
trigger:
branches:
include:
- azure-develop
variables:
- group: Example Mobile App
pool:
vmImage: 'macos-latest'
stages:
- stage: initialise_environment
displayName: Initialise Environment
jobs:
- template: steps/azure-initialise-pipeline.yml
azure-initialise-pipeline.yml
jobs:
- job: initialise_node
displayName: initialise node
steps:
- checkout: self
persistCredentials: true
clean: true
- bash: |
FOO="some value"
echo "##vso[task.setvariable variable=FOO]$AndroidKeyAlias"
- bash: |
set -ex
mkdir -p packages
echo $AndroidKeyAlias > packages.json
cat packages.json
env:
CACHIX_AUTH_TOKEN: $(AndroidKeyAlias)
- bash: |
echo "$(FOO)"
As you can see I have tried getting variables values but I am not getting any output.
How do I get this value? I have tied fetching the values using the Azure CLI tool and I am getting following output
authorized: true
description: ''
id: 123
name: Example Mobile App
providerData: null
type: Vsts
variables:
AndroidKeyAlias:
isSecret: true
value: null
AndroidKeyAliasPassword:
isSecret: true
value: null
AndroidKeyStorePassword:
isSecret: true
value: null
What am I doing wrong?
Thanks in advance.
You're completely messing up the notation for bash environment variable substitution and the Azure Pirelines variable substitution. On top of that there are rules for when variables are available to a template and when not and also specific notation that might be needed there...
In general, when you want Azure pipelines to inline the value of a variable for you, use:
$(VariableName)
These variables will be substituted at the time that the task is invoked and will hold whatever value the variable has at that step in the job
Alternatively, use
${{ variables.Variablename }}
For 'compiletime' substitution. These will be expanded before the job even starts to run and will use the value that is available during template expansion and workflow initialisation.
If you are inside a bash script you can use any of the above mentioned syntaxis. You can also use an environment variable to pass the contents in
- bash: |
$VAR
env:
VAR: $(Variablename)
- pwsh: |
$env:VAR
env:
VAR: $(Variablename)
By passing in variables through the environment you don't have to mix the different variable substitution notations. It makes it easier to see what's going on.
When setting an environment variable in a bash step, the value won't be available in a subsequent step. The environment is cleaned up between steps for security and reasons.
For this reason, this won't work:
# Doesn't work
- bash: |
FOO="some value"
- bash: |
echo $FOO
There are special strings you can stream to the log that will instruct the agent to set a variable for the subsequent steps in the same job.
With templates things can become even harder, as you can also pass variables to a template where it's being called. You can do this with template parameters. The advantage of this syntax is that's it's very evident when you're passing in data:
jobs:
- template: templates/npm-with-params.yml # Template reference
parameters:
name: Linux
vmImage: 'ubuntu-latest'
With a template defined as:
parameters:
- name: name # defaults for any parameters that aren't specified
default: ''
- name: vmImage
default: ''
jobs:
- job: ${{ parameters.name }}
pool:
vmImage: ${{ parameters.vmImage }}
steps:
- script: npm install
- script: npm test
See:
templates
variables
stetting variables in scripts

Azure Pipeline: Update variables according to parameter

I have some very simple variables, which I would like to change according to the environment.
I have written the code below in very different ways (including indentation) but none was fruitful. Alternatives I see are
Use variable groups (trying to avoid to have too many of them)
Write a bash script which updates the variables (will work but I
think its not a super neat solution)
variables:
- group : secrets
- name: hello
value: world
${{ if eq(parameters.environment, 'dev') }}:
- name: RabbitMQ_replicaCount
value: 3
${{ if eq(parameters.environment, 'test') }}:
RabbitMQ_replicaCount: '1'
Any other ideas will be appriciated :)
I would rather go by a PS script/Bash script for this task. Why ? The logic part of build where manipulation is needed like setting or overriding var based on branch or env can be done in a better way in script rather than the build yaml itself. Also this part un-necessary elongates the yaml.
Step 1 : Define a var in the build pipe with default env name
and may be another var whose value you want to set based on condition
Step 2 : Add a yml file(lets name it BuildEnv.yml) in your repo which actually contains your PowerShell/Bash code:
steps:
- powershell: |
if($BuildEnv -ne "Test"){
Write-Host "##vso[task.setvariable variable=BuildEnv]Dev"
Write-Host "##vso[task.setvariable variable=RabbitMQ_replicaCount]11"
}
displayName: 'Override Build Env'
# MORE CODE HERE
Step 3: Plug your yml in the build pipe as a template-
trigger:
branches:
include:
- master
name: $(date:yyyy-MM-dd_HH.mm)_$(rev:.r)
stages:
- stage: Build_Stage
displayName: Build_Stage
jobs:
- job: Build_Job
pool:
name: ABC
steps:
- template: ..\BuildEnv.yml
#REST CODE
That's it. You are done.
Reference : Template usage in Azure DevOps build - https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops

Setup Gitlab CI/CD environment variables via UI

I am new at gitlab CI/CD Settings. I don't want to put sensitive credentials (like API-Keys, passwords...) into my branch. For this, GitLab (and other CI/CD-Services) are able to set environment variables.
What I have done so far:
Via UI (Project ⇒ Settings ⇒ CI/CD ⇒ Variables)
First go to Project ⇒ Settings ⇒ CI/CD ⇒ Variables and add them like this:
enter image description here
Now here trying to get the File with all your config-values(e.g. with dotenv).
require("dotenv");
module.exports = process.env.NODE_ENV.trim() === "production" ? _config.production : _config.development;
Current .gitlab-ci.yaml file is:
image: node:8.9.0
cache:
paths:
- node_modules/
stages:
- ver
- init
- test
- build
- deploy
ver:
stage: ver
script:
- node -v
init:
stage: init-dev
script:
- npm install
tags:
- dev_server
only:
- dev
variables:
ENV_PRODUCTION: "/builds/AkdiD/8/abcde/projectName/ENV_PRODUCTION"
test:
stage: test
script:
- npm test
build:
stage: build
script:
- echo "BUILD_VERSION=production" >> build.env
artifacts:
reports:
dotenv: build.env
deploy:
stage: deploy-dev
script:
- npm run killcurrent
- echo $ENV_PRODUCTION
- echo $BUILD_VERSION
- npm run staging
tags:
- dev_server
only:
- dev
Question: where do I need to keep this ENV_PRODUCTION file name (yaml file or other place) so that server take that value ??
Edited variable like this- still server it not fetching these variables. Should I change/put something in .gitlab-ci.yml file?
Settings up a Custom environment variables of type File (GItLab 11.11+) does not seem the way to reference/set a list of variables, including ones with sensitive information.
A variable of type file is generally there to represent, for instance, a certificate.
You should define variables, possibly Group-level environment variables
You can define per-project or per-group variables that are set in the pipeline environment.
Group-level variables are stored out of the repository (not in .gitlab-ci.yml) and are securely passed to GitLab Runner, which makes them available during a pipeline run.
For Premium users who do not use an external key store or who use GitLab’s integration with HashiCorp Vault, we recommend using group environment variables to store secrets like passwords, SSH keys, and credentials.

How to set environment variable for node js build job in azure devops pipeline

I am importing some secrets from Azure Key Vault to Variable Group to CI / CD pipeline.
I am able to map the required secrets in VariableGroup from KeyVault using Azure Devops UI.
In my pipeline YAML i am able to read and print those VariableGroup variables which are AzureKeyVault secrets.
trigger:
- dev
# define the VM image
pool:
vmImage: "Ubuntu 16.04"
# define variables to use during the build
variables:
- group: SecretVarGroup # it has keyvault variable 'KV_API_KEY'
- group: PublicVarGroup # it has a variable 'API_CLIENTID'
# define the step to export key to env varaiable
steps:
- script: echo $MYSECRETAPIKEY
env:
MYSECRETAPIKEY: $(KV_API_KEY)
## Run the npm build
- script: |
npm run build
displayName: "npm build"
I am able to see value for 'KV_API_KEY' secret printed as *** value in the build output log which i assume its able to consume. I also see value for API_CLIENTID printed in build log as well as node js process.env object.
I was assuming the variable "MYSECRETAPIKEY" will be available in my node js process.env object. But it's not avaialble.
The way i tested it is in my node js project build config i have a print statement which prints process.env object. It printed all the environment variables of pipeline build agent including my PUBLICVARGROUP variable 'API_CLIENTID'. But i don't see my secret variable 'MYSECRETAPIKEY' in the process.env object.
env:
MYSECRETAPIKEY: $(KV_API_KEY)
I thought above line would export variable to specific language process environment. But it is not. How can i fix this?
# define the step to export key to env varaiable
steps:
## Run the npm build
- script: |
npm run build
displayName: "npm build"
env:
MYSECRETAPIKEY: $(KV_API_KEY)
Looks like secrets are scoped on the agent for individual tasks and scripts to use. The issue was i had env: declaraion in a separate adhoc task.Moving it to the same place of my script declaration in the above code has fixed the issue.

How to set and read user environment variable in Azure DevOps Pipeline?

I have some test automation code that reads some values from an environment variable stored on my local machine, like this:
Environment.GetEnvironmentVariable("SAUCE_USERNAME", EnvironmentVariableTarget.User);
I'm trying to use Azure Pipelines to create this variable during pipeline execution and then read it in my test automation code. Using a YAML file.
Im reading this variable in the VS Test step of the Azure Pipeline. So if I set the variable, it has to be for the life of the Azure Pipeline.
I've tried to use the documentation here but have been unsuccessful.
Tried this code below as well but it fails with this error:
azure-pipelines.yml (Line: 39, Col: 1, Idx: 1252) - (Line: 39, Col: 1,
Idx: 1252): While scanning a simple key, could not find expected ':'.
# Create a secret variable
- powershell: |
Write-Host '##vso[task.setvariable variable=sauce.userName;issecret=true]abc'
# Attempt to output the value in various ways
- powershell: |
# Using an input-macro:
Write-Host "This works: $(sauce.userName)"
# Using the env var directly:
Write-Host "This does not work: $env:SAUCE_USERNAME"
# Using the mapped env var:
Write-Host "This works: $env:SAUCE_USERNAME"
env:
SAUCE_USERNAME: $(sauce.userName)
The easiest method is to pass the Azure DevOps(ADO) Env Variable values into your keys like this:
- task: DotNetCoreCLI#2
displayName: 'Run tests'
env:
SAUCE_USERNAME: $(sauceUsername) #this will store the value from 'sauceUsername' into SAUCE_USERNAME
SAUCE_ACCESS_KEY: $(sauceKey)
Displaying or using the value will work if you try
- bash: echo $(SAUCE_USERNAME) # will output our username stored in SAUCE_USERNAME env variable
And if you are referencing SAUCE_USERNAME in your code, the code will pick up the value from the Azure server.
This article has a good explanation
Previously, I also used Powershell, but this method is more involved and convoluted:
Create your variables in your Azure DevOps pipeline and provide those variables the values.
Create a Powershell script that you will run in the beginning to set your Env Variables. This is what my Posh looks like.
Run this Posh in the beginning as a separate step in your CI pipeline and this will set the environment variables for the VM that's being used to run your pipeline.
This is another detailed article that could help you with this.
As per request, I'm also attaching the PowerShell code that makes this possible.
Param(
[string]$sauceUserName,
[string]$sauceAccessKey,
[string]$sauceHeadlessUserName,
[string]$sauceHeadlessAccessKey
)
Write-Output "sauce.userName that was passed in from Azure DevOps=>$sauceUserName"
Write-Output "sauce.accessKey that was passed in from Azure DevOps=>$sauceAccessKey"
Write-Output "sauce.headless.userName that was passed in from Azure DevOps=>$sauceHeadlessUserName"
Write-Output "sauce.headless.access.key that was passed in from Azure DevOps=>$sauceHeadlessAccessKey"
[Environment]::SetEnvironmentVariable("SAUCE_USERNAME", "$sauceUserName", "User")
[Environment]::SetEnvironmentVariable("SAUCE_ACCESS_KEY", "$sauceAccessKey", "User")
[Environment]::SetEnvironmentVariable("SAUCE_HEADLESS_USERNAME", "$sauceUserName", "User")
[Environment]::SetEnvironmentVariable("SAUCE_HEADLESS_ACCESS_KEY", "$sauceAccessKey", "User")
I tried using both of the following syntax as suggested in answers above, but the environment variable was always blank when trying to use it in tasks further down in the pipeline (like during the build or while running tests).
[Environment]::SetEnvironmentVariable("SAUCE_USERNAME", "$(sauceUserName)", "User")
variables:
sauceUserName: '$(sauceUserName)'
What worked for me was to use the syntax to write Azure DevOps variables in an inline PowerShell script task:
- task: PowerShell#2
displayName: Add the username as an environment variable so the tests can find it.
inputs:
targetType: 'inline'
script: |
Write-Host "Making the sauceUsername available as an environment variable."
Write-Host "##vso[task.setvariable variable=SAUCE_USERNAME;]$(sauceUserName)"
My build task was then able to find the environment variable, and I could also access it in PowerShell script tasks further down in the pipeline with code like:
- task: PowerShell#2
displayName: Display the environment variable value for debugging purposes.
inputs:
targetType: 'inline'
script: |
[string] $username= $Env:SAUCE_USERNAME
Write-Host "The SAUCE_USERNAME environment variable value is '$username'."
set up pipeline variables and then try this mapping in your yaml file:
# ASP.NET Core (.NET Framework)
# Build and test ASP.NET Core projects targeting the full .NET Framework.
# Add steps that publish symbols, save build artifacts, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/dotnet-core
pool:
vmImage: 'VS2017-Win2016'
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
yourEnvVar: '$(yourPipelineVariable)'
yourOtherEnvVar: '$(yourOtherPipelineVariable)'
We fought with this for hours thinking it was due to not being able to set environment variables but in the end it had nothing to do with that.
Here's some of our notes from trying to figure this out for Azure DevOps:
Pipeline Variables are Environment Variables and are injected into the pipeline in the first step of the process. You can confirm this by looking at the logs for the Initialize job task which lists all the environment variables.
In order to prove that the pipeline variables are in the system you can add a powershell task and inline:
Write-Host "Pipeline Variable: $(FASTLANE_APPLE_APPLICATION_SPECIFIC_PASSWORD)"
Write-Host "Environment Variable: $Env:FASTLANE_APPLE_APPLICATION_SPECIFIC_PASSWORD"
The () around the pipeline variable are critical if you want to use the pipeline variable somewhere - e.g. $(myVar)
Unrelated to the users question but perhaps helpful to those who are trying to upload to AppStore using two-factor auth (2FA). This is the documentation we were following:
We had to set FASTLANE_APPLE_APPLICATION_SPECIFIC_PASSWORD and FASTLANE_SESSION in order to get it to bypass the 2FA. We think the session one allowed us to log into AppStoreConnect but the App specific password was used for uploading. The logs sorta hint that's what's happening although you never see the session variable used.
As defined here, Secret variables should be set explicitly in step/task env variable section
To pass variables that are defined outside of the YAML it is important that you do NOT define the variable in the YAML as the one in YAML will take precedence over the exterior one.
Once the variable is defined, it can be referenced in the inline powershell script using the syntax $(variablename) however the variablename must be capitalized
per the devops hint:
To use a variable in a script, use environment variable syntax.
Replace . and space with _, capitalize the letters, and then use your platform's syntax for referencing an environment variable.
If you need to map it using an environment variable (e.g. secrets) then this is the correct syntax:
script: |
write-host "env $env:myvariable"
env:
#not sure if case matters here
myvariable: $(myvariable)
full example:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
#log the mapped variable, case doesn't matter
write-host "env $env:myvariable"
#directly access pipeline variable, CASE MATTERS!
write-host "pipeline $(MYVARIABLE)"
#show all environment variables
get-childitem env:
env:
#not sure if case matters here
myvariable: $(myvariable)

Resources