I am looking for a solution to run my cypress end to end tests in the pipeline. The tests use credentials to log in. For security purposes I want to fetch the secret credentials from the Azure Key vault. Fetching the secret works through the 'Azure key vault' task. But then theres the problem, when I want to use
- task: AzureKeyVault#2
inputs:
azureSubscription: 'mysubscription'
KeyVaultName: 'mykeyvaultname'
SecretsFilter: 'mysecret'
RunAsPreJob: false
Then I can access the secret through $(mysecret) but the value is still kept secret in the pipeline.
npx cypress run #--ci-build-id $BUILD_BUILDNUMBER --record --parallel --env mypw=$(mysecret)
This shows an output when running the pipeline
Generating script.
Script contents:
npx cypress run #--ci-build-id $BUILD_BUILDNUMBER --record --parallel --env mypw=***
This makes the tests fail.
This instruction (bottom of page) by microsoft suggests to write the password to a TXT file: https://learn.microsoft.com/en-us/azure/devops/pipelines/release/azure-key-vault?view=azure-devops&tabs=yaml
This works, but then when I read from the txt file in the pipeline again, the value is once again encrypted to ***
is there a way around this to use the keyvault secret in my tests?
For security reason, Azure DevOps will redact the secret value as star(*) on the log console UI, this is by designed behavior. But you should still be able to use the secret value with syntax $(secretname).
To validate if the failure is caused by the secret value, you can temporarily disable AzureKeyVault#2 task, use real value for secret in npx cypress run command, check it will work.
in addition, it appears there's an extra # in the npx cypress run command, which commmentted the latter parameters, please remove the # for a check.
I found how to pass secrets as environment variables from vault, i need to write annotations and a string in deployment manifest:
args: ["sh", "-c", "source /vault/secrets/config && <entry-point script>"]
But i don’t have entry-point script in my micro services, so what should i write there instead of script?
I tried to launch deployment without a script, after that i don't have secrets in my environment variables.
I am trying to create a pipeline in which I'll run terraform configs against an Azure subscription from Azure DevOps pipelines. All works fine, but when I am trying to log in as user with az cli it fails with:
ERROR: Authentication failed due to error of 'Unsupported wstrust endpoint version. Current support version is wstrust2005 or wstrust13.' This typically happens when attempting a Microsoft account, which requires interactive login. Please invoke 'az login' to cross check. More details are available at https://github.com/AzureAD/microsoft-authentication-library-for-python/wiki/Username-Password-Authentication
ERROR: Please run 'az login' to setup account.
Although from cli on my local it works to do az log in -u user -p pass
Command was executed from a script, because after log in I'll move to terraform commands which requires those creds:
- script: |
az login -u $(u) -p $(p)
terraform init
terraform plan
I know it's not a best practice to use an user instead of a service principal, but for now I have to stick with this method. So is there a way to automate az login from Azure DevOps pipelines?
The Azure CLI task can be used instead of the Script task
It works like the normal script tasks and you select what scripting language you want to run with the scriptTypeproperty:
Type of script: PowerShell/PowerShell Core/Bat/Shell script. Select
bash/pscore script when running on Linux agent or batch/ps/pscore
script when running on Windows agent. PowerShell Core script can run
on cross-platform agents (Linux, macOS, or Windows)
It also takes a service connection reference in the azureSubscription input. The service connection should be of type Azure Resource Manager and can be created either automatically or by using an existing service principal.
The azure connection details are safely stored in the service connection and when your script starts executing Azure CLI has already been logged in using the service connection
Below is an example of how your pipeline task would look
- task: AzureCLI#2
displayName: Azure CLI
inputs:
azureSubscription: <Name of the Azure Resource Manager service connection>
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
terraform init
terraform plan
az login -u $(secretUser) -p $(secretPassword)
Put the user ID and password into Azure Key Vault, named secretUser and secretPassword, and then use the AzureKeyVault#1 task to populate it
- task: AzureKeyVault#1
inputs:
ConnectedServiceName: Your Service Connection Name
KeyVaultName: Your Key Vault Name
SecretsFilter: 'secretUser,secretPassword'
RunAsPreJob: true
- script: |
az login -u $(secretUser) -p $(secretPassword)
terraform init
terraform plan
I am importing some secrets from Azure Key Vault to Variable Group to CI / CD pipeline.
I am able to map the required secrets in VariableGroup from KeyVault using Azure Devops UI.
In my pipeline YAML i am able to read and print those VariableGroup variables which are AzureKeyVault secrets.
trigger:
- dev
# define the VM image
pool:
vmImage: "Ubuntu 16.04"
# define variables to use during the build
variables:
- group: SecretVarGroup # it has keyvault variable 'KV_API_KEY'
- group: PublicVarGroup # it has a variable 'API_CLIENTID'
# define the step to export key to env varaiable
steps:
- script: echo $MYSECRETAPIKEY
env:
MYSECRETAPIKEY: $(KV_API_KEY)
## Run the npm build
- script: |
npm run build
displayName: "npm build"
I am able to see value for 'KV_API_KEY' secret printed as *** value in the build output log which i assume its able to consume. I also see value for API_CLIENTID printed in build log as well as node js process.env object.
I was assuming the variable "MYSECRETAPIKEY" will be available in my node js process.env object. But it's not avaialble.
The way i tested it is in my node js project build config i have a print statement which prints process.env object. It printed all the environment variables of pipeline build agent including my PUBLICVARGROUP variable 'API_CLIENTID'. But i don't see my secret variable 'MYSECRETAPIKEY' in the process.env object.
env:
MYSECRETAPIKEY: $(KV_API_KEY)
I thought above line would export variable to specific language process environment. But it is not. How can i fix this?
# define the step to export key to env varaiable
steps:
## Run the npm build
- script: |
npm run build
displayName: "npm build"
env:
MYSECRETAPIKEY: $(KV_API_KEY)
Looks like secrets are scoped on the agent for individual tasks and scripts to use. The issue was i had env: declaraion in a separate adhoc task.Moving it to the same place of my script declaration in the above code has fixed the issue.
I have some test automation code that reads some values from an environment variable stored on my local machine, like this:
Environment.GetEnvironmentVariable("SAUCE_USERNAME", EnvironmentVariableTarget.User);
I'm trying to use Azure Pipelines to create this variable during pipeline execution and then read it in my test automation code. Using a YAML file.
Im reading this variable in the VS Test step of the Azure Pipeline. So if I set the variable, it has to be for the life of the Azure Pipeline.
I've tried to use the documentation here but have been unsuccessful.
Tried this code below as well but it fails with this error:
azure-pipelines.yml (Line: 39, Col: 1, Idx: 1252) - (Line: 39, Col: 1,
Idx: 1252): While scanning a simple key, could not find expected ':'.
# Create a secret variable
- powershell: |
Write-Host '##vso[task.setvariable variable=sauce.userName;issecret=true]abc'
# Attempt to output the value in various ways
- powershell: |
# Using an input-macro:
Write-Host "This works: $(sauce.userName)"
# Using the env var directly:
Write-Host "This does not work: $env:SAUCE_USERNAME"
# Using the mapped env var:
Write-Host "This works: $env:SAUCE_USERNAME"
env:
SAUCE_USERNAME: $(sauce.userName)
The easiest method is to pass the Azure DevOps(ADO) Env Variable values into your keys like this:
- task: DotNetCoreCLI#2
displayName: 'Run tests'
env:
SAUCE_USERNAME: $(sauceUsername) #this will store the value from 'sauceUsername' into SAUCE_USERNAME
SAUCE_ACCESS_KEY: $(sauceKey)
Displaying or using the value will work if you try
- bash: echo $(SAUCE_USERNAME) # will output our username stored in SAUCE_USERNAME env variable
And if you are referencing SAUCE_USERNAME in your code, the code will pick up the value from the Azure server.
This article has a good explanation
Previously, I also used Powershell, but this method is more involved and convoluted:
Create your variables in your Azure DevOps pipeline and provide those variables the values.
Create a Powershell script that you will run in the beginning to set your Env Variables. This is what my Posh looks like.
Run this Posh in the beginning as a separate step in your CI pipeline and this will set the environment variables for the VM that's being used to run your pipeline.
This is another detailed article that could help you with this.
As per request, I'm also attaching the PowerShell code that makes this possible.
Param(
[string]$sauceUserName,
[string]$sauceAccessKey,
[string]$sauceHeadlessUserName,
[string]$sauceHeadlessAccessKey
)
Write-Output "sauce.userName that was passed in from Azure DevOps=>$sauceUserName"
Write-Output "sauce.accessKey that was passed in from Azure DevOps=>$sauceAccessKey"
Write-Output "sauce.headless.userName that was passed in from Azure DevOps=>$sauceHeadlessUserName"
Write-Output "sauce.headless.access.key that was passed in from Azure DevOps=>$sauceHeadlessAccessKey"
[Environment]::SetEnvironmentVariable("SAUCE_USERNAME", "$sauceUserName", "User")
[Environment]::SetEnvironmentVariable("SAUCE_ACCESS_KEY", "$sauceAccessKey", "User")
[Environment]::SetEnvironmentVariable("SAUCE_HEADLESS_USERNAME", "$sauceUserName", "User")
[Environment]::SetEnvironmentVariable("SAUCE_HEADLESS_ACCESS_KEY", "$sauceAccessKey", "User")
I tried using both of the following syntax as suggested in answers above, but the environment variable was always blank when trying to use it in tasks further down in the pipeline (like during the build or while running tests).
[Environment]::SetEnvironmentVariable("SAUCE_USERNAME", "$(sauceUserName)", "User")
variables:
sauceUserName: '$(sauceUserName)'
What worked for me was to use the syntax to write Azure DevOps variables in an inline PowerShell script task:
- task: PowerShell#2
displayName: Add the username as an environment variable so the tests can find it.
inputs:
targetType: 'inline'
script: |
Write-Host "Making the sauceUsername available as an environment variable."
Write-Host "##vso[task.setvariable variable=SAUCE_USERNAME;]$(sauceUserName)"
My build task was then able to find the environment variable, and I could also access it in PowerShell script tasks further down in the pipeline with code like:
- task: PowerShell#2
displayName: Display the environment variable value for debugging purposes.
inputs:
targetType: 'inline'
script: |
[string] $username= $Env:SAUCE_USERNAME
Write-Host "The SAUCE_USERNAME environment variable value is '$username'."
set up pipeline variables and then try this mapping in your yaml file:
# ASP.NET Core (.NET Framework)
# Build and test ASP.NET Core projects targeting the full .NET Framework.
# Add steps that publish symbols, save build artifacts, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/dotnet-core
pool:
vmImage: 'VS2017-Win2016'
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
yourEnvVar: '$(yourPipelineVariable)'
yourOtherEnvVar: '$(yourOtherPipelineVariable)'
We fought with this for hours thinking it was due to not being able to set environment variables but in the end it had nothing to do with that.
Here's some of our notes from trying to figure this out for Azure DevOps:
Pipeline Variables are Environment Variables and are injected into the pipeline in the first step of the process. You can confirm this by looking at the logs for the Initialize job task which lists all the environment variables.
In order to prove that the pipeline variables are in the system you can add a powershell task and inline:
Write-Host "Pipeline Variable: $(FASTLANE_APPLE_APPLICATION_SPECIFIC_PASSWORD)"
Write-Host "Environment Variable: $Env:FASTLANE_APPLE_APPLICATION_SPECIFIC_PASSWORD"
The () around the pipeline variable are critical if you want to use the pipeline variable somewhere - e.g. $(myVar)
Unrelated to the users question but perhaps helpful to those who are trying to upload to AppStore using two-factor auth (2FA). This is the documentation we were following:
We had to set FASTLANE_APPLE_APPLICATION_SPECIFIC_PASSWORD and FASTLANE_SESSION in order to get it to bypass the 2FA. We think the session one allowed us to log into AppStoreConnect but the App specific password was used for uploading. The logs sorta hint that's what's happening although you never see the session variable used.
As defined here, Secret variables should be set explicitly in step/task env variable section
To pass variables that are defined outside of the YAML it is important that you do NOT define the variable in the YAML as the one in YAML will take precedence over the exterior one.
Once the variable is defined, it can be referenced in the inline powershell script using the syntax $(variablename) however the variablename must be capitalized
per the devops hint:
To use a variable in a script, use environment variable syntax.
Replace . and space with _, capitalize the letters, and then use your platform's syntax for referencing an environment variable.
If you need to map it using an environment variable (e.g. secrets) then this is the correct syntax:
script: |
write-host "env $env:myvariable"
env:
#not sure if case matters here
myvariable: $(myvariable)
full example:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
#log the mapped variable, case doesn't matter
write-host "env $env:myvariable"
#directly access pipeline variable, CASE MATTERS!
write-host "pipeline $(MYVARIABLE)"
#show all environment variables
get-childitem env:
env:
#not sure if case matters here
myvariable: $(myvariable)