I'm creating a Continuous Integration pipeline that uses Bash script tasks in order to create the initial variables for runtime.
I have a variable that I call: datebuild, which is formatted accordingly : $(date +%Y%m%d_%H%M%S).
Currently I'm using the pipeline variable that's how I'm declaring it
When using the datebuild variable under Bash#3 task, it successfully formatting it.
Afterwards I want to take the formatted output in order to use it on different tasks inside one agent job.
On the second task I need to copy file to the Artifact Staging Directory:
20200423_141808 is the file and Artifact Staging Directory is the Destination Directory, for example.
I've been reading that it can be used with feature called Output Variables.
Created the reference variable named: ref1, and on the task I want to take the output variable I'm using the ref1.datebuild in order to access the variable
Used the following documentation in order to use the output variable it doesn't seem to work.
here's the task inside the pipeline:
Trying to understand What I'm missing.
You can take the formatted date and set it as a variable for the next steps in the job.
For example, in YAML pipeline:
variables:
datebuild: '$(date +%Y%m%d_%H%M%S)'
steps:
- bash: |
formated="$(datebuild)"
echo "##vso[task.setvariable variable=formatedDate]$formated"
- bash: |
echo $(formatedDate)
In the editor:
The second bash task output is:
Related
In Azure Pipelines, I see that you can access the environment variables from scripts in node.js during a pipeline run. However, I want to actually return a value and then capture/use it.
Does anyone know how to do this? I can't find any references on how to do this in documentation.
For consistency's sake it'd be nice to use node scripts for everything and not go back and forth between node and bash.
Thanks
Okay I finally figured this out. Azure documentation is a bit confusing on the topic, but my approach was what follows. In this example, I'm going to make a rather pointless simple script that sets a variable whose value is the name of the source branch, but all lower case.
1) Define your variable
Defining a variable can be done simply (though there is a lot of depth to how variables are used and I suggest consulting Azure documentation on variable creation for more). However, at the top of your pipeline yaml file you can define it as such:
variables
lowerCaseBranchName: ''
This creates an empty variable for use across your jobs. We'll use this variable as our example.
2) Create your script
"Returning a value" from your script simply means outputting it via node's stdout, the output of which will be consumed by the task to set it as a pipeline variable.
An important thing to remember is that any environment variables from the pipeline can be used within node, they are just reformatted and moved under node's process.env global. For instance, the commonly used Build.SourceBranchName environment variable in azure pipelines is accessible in your node script via its alias process.env.BUILD_SOURCEBRANCHNAME. This uppercase name transformation should be uniform across all environment variables.
Here's an example node.js script:
const lowerCaseBranchName = process.env.BUILD_SOURCEBRANCHNAME.toLowerCase();
process.stdout.write(lowerCaseBranchName);
3) Consume the output in the relevant step in azure pipelines
To employ that script in a job step, call it with a script task. Remember that a script task is, in this case, a bash script (though you can use others) that runs node as a command as it sets the value of our variable:
- script: |
echo "##vso[task.setvariable variable=lowerCaseBranchName]$(node path/to/your/script)"
displayName: 'Get lower case branch name'
Breaking down the syntax
Using variable definition syntax is, in my opinion extremely ugly, but pretty easy to use once you understand it. The basic syntax for setting a variable in a script is the following:
##vso[task.setvariable variable=SOME_VARIABLE_NAME]SOME_VARIABLE_VALUE
Above, SOME_VARIABLE_NAME is the name of our variable (lowerCaseBranchName) as defined in our azure pipeline configuration at the beginning. Likewise, SOME_VARIABLE_VALUE is the value we want to set that variable to.
You could do an additional line above this line to create a variable independently that you can then use to set the env variable with, however I chose to just inline the script call as you can see in the example above usign the $() syntax.
That's it. In following tasks, the environment variable lowerCaseBranchName can be utilized using any of the variable syntaxes such as $(lowerCaseBranchName),
Final result
Defining our variable in our yaml file:
variables
lowerCaseBranchName: ''
Our nodejs script:
const lowerCaseBranchName = process.env.BUILD_SOURCEBRANCHNAME.toLowerCase();
process.stdout.write(lowerCaseBranchName);
Our pipeline task implementation/execution of said script:
- script: |
echo "##vso[task.setvariable variable=lowerCaseBranchName]$(node path/to/your/script)"
displayName: 'Get lower case branch name'
A following task using its output:
- script: |
echo "$(lowerCaseBranchName)"
displayName: 'Output lower case branch name'
This will print the lower-cased branch name to the pipline console when it runs.
Hope this helps somebody! Happy devops-ing!
For example the file that I have is test.env
test.env has the content
export SAMPLE="true"
I want the variable SAMPLE to be set as a pipeline variable before running the pipeline
I am trying the below mentioned solution but it is not really helping
before_script:
- git clone test.env
- source test.env
stages:
- publish
test:
stage: publish
trigger:
project: test_pipeline
branch: master
strategy: depend
only:
variables:
- $SAMPLE == 'True'
Is there any way to source the variables in the before hand and then set the pipeline variables so that execution can happen based on those pipeline variables
Currently with Gitlab CI there's no way to provide a file to use as environment variables, at least not in the way you stated. There are a couple of other options however.
First is take all the individual variables you would have in your test.env file and store them as separate Secret Variables. You can set these by going to your project's settings, -> CI/CD, -> Variables. Environment Variables defined here will automatically be available in every pipeline job for this project (although you can select the Protect Variable checkbox, which will only make the variable available for pipelines on Protected Branches).
The next option is to copy the entire test.env file contents, go back to your project’s Secret Variables (as described above), but this time change the Variable Type to "File", and paste the file contents as the value. When you use a "File" type variable, Gitlab will create a temporary file in each of your pipeline jobs (again, unless you check the Protect Variable option). Then the path to that file will be stored as the env variable with the key you selected. This would allow you to do things like cat $my_file_variable, which would evaluate as cat /path/to/temporary/file, then cat the contents.
A final option which is closest to your original request, is to add a job before all your other jobs that would require the test.env file that looks like this:
stage: env_setup # or whatever
script:
- : # this is the bash Null Command that does nothing and always succeeds
artifacts:
reports:
dotenv: test.env
For this job, the only purpose is to turn your test.env file into environment variables. We don't need to do anything else with it, so we use the Null Command for the script section (since a job without at least the script section will fail). The artifacts part is the important stuff here. Gitlab supports a special Report type called dotenv that takes a single argument: a path to a file. The file will get uploaded as an artifact like any other, but for subsequent jobs (or those that use the dependencies keyword with this job name) instead of pulling down the artifact as a file, each item in test.env will be turned into an environment variable, so you can use it like $SAMPLE, etc.
Personally I prefer the first two options over the third, and of the first 2, the 2nd is the easiest as you just have to copy and paste the file you have now into a variable. The reason the third option isn't ideal is that it still allows you to have sensitive variables (like passwords) in your git repository, which isn't ideal from a security standpoint. Either of the first two options eliminate that problem.
I'm trying to get my build repository name as an uppercase string combining predefine variables and expressions on Azure Devops as follows:
variables:
repoNameUpper: $[upper(variables['Build.Repository.Name'])]
- script: |
echo $(repoNameUpper)
Yet I get no output from it, what am I doing wrong here?
Yes, I know I could set a variable to achieve what I need using a bash script, yet I think it would not be so cool.
It because the Build.Repository.Name is agent-scoped, and can be used as an environment variable in a script and as a parameter in a build task. in another words - is not known at plan compile time, only at job execution time.
You can find more info in this GitHub issue.
With Azure Release Pipeline, in a task using the PowerShell Script, I am able to set values of variables and pass to next task using the command
Write-Host '##vso[task.setvariable variable=varResourceExists;isOutput=true;something'
However, when I put this similar command in a task that uses Azure PowerShell, this command is no longer allowed, the task produces a warning:
2019-10-22T00:23:14.3080614Z ##[warning]'##vso[task.setvariable
variable=varResourceExists;isOutput=true;something' contains logging
command keyword '##vso', but it's not a legal command. Please see the
list of accepted commands:
https://go.microsoft.com/fwlink/?LinkId=817296
As a result, the variable varResourceExists cannot be set by my task. I have also tried a conventional PowerShell set value by doing
$varResourceExists = 'something'; # this also does not work
Is there a way I can set this value in Azure Powershell script so that the next task can reference it?
##vso[task.setvariable variable=varResourceExists;isOutput=true;something is not correct syntax. You're missing a closing ].
It should be ##vso[task.setvariable variable=varResourceExists;isOutput=true;]something
Here is how I solved my topic. In the pipeline Azure PowerShell task, I can have code such as Write-Host '##vso[task.setvariable variable=varResourceExists;isOutput=true;]False';
In the Output Variable option, I set a Reference Name "step1":
Output Variable
Then in the next step, I can do a conditional check using a Custom Condition:
Custom Condition
I can also reference the variable in my code such as Write-Host "The step1.varResourceExists says: $(step1.varResourceExists)";
Is it possible that I can create a variable via a batch file and pass it to a Gitlab CI variable?
The background is that I want to declare the link of an environment:
environment:
name: staging
url: https://staging.example.com
a part of the URL results dynamically from the current build date. How can I pass the variable declared in a batch file to the gitlab-ci "url" variable?
The Url would look like this in the batch file:
https://testme.com/Tool_%date:~-2%%date:~-7,2%%date:~-10,2%.zip
Outcome is:
https://testme.com/Tool_180410.zip
and that variable i want to write in the environment URL variable
I don't think creating a variable in a batch file and passing it to a GitLab CI variable is possible, but from what I've gathered from your scenario you could:
set the current build date before you call the batch script
pass in the variable to the batch script
use the variable for your URL generation
For example (running on a Windows runner):
$ set testDate=%date:~-2%%date:~-7,2%%date:~-10,2%
$ echo %testDate%
180410
# Use %testDate% wherever else you need it now for the rest of your build.
With regards to the Environment URL, I don't have any experience using it - but this open issue could be of interest.
I usually set any variable needed in Gitlab's .gitlab-ci.yml in another separate yml file. Let's assume that the name of this second file is "parameters.yml", just to give an example.
You can create the parameters.yml file from your batch and define all your variables there.
Example of what parameters.yml contains:
variables:
name: staging
url: https://staging.example.com
[other variables]
Then, all you have to do is to include this yml into the "main" one (.gitlab-ci.yml), for example, something like this:
stages:
- build
- test
- release
- deploy
include: parameters.yml
And that's it, now you will "see" all the variables defined in "parameters.yml"