I have certain expressions used by multiple steps in Azure Data Factory.
For instance, I get the start time of a pipeline often:
#formatDateTime(pipeline().TriggerTime, 'yyyy-MM-dd HH:mm:ss.fff')
If I need to change it for some reason, I'd need to update it everywhere.
I know about global parameters; however, is there a way to create global expressions?
Based on my knowledge, currently there is no way to have a global expression similar to Global parameters.
For you have to have a common source of truth or 1 common expression, I would suggest you to create a common pipeline in the ADF and call that pipeline via execute pipeline activity to avoid re writing the same code every where.
You could add a variable to the pipeline and set it to the appropriate value at the start of your pipeline run: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-set-variable-activity
You can then refer to this variable as you do with parameters or change its value on the go.
Related
I have a few use cases in my Gitlab setup I would like to be able to support:
If a certain label (let's call it “skip_build”) is set, the deployment steps should not be run when I merge an MR to a main branch. This would be useful when we have multiple MRs being merged right after another and only need the last one built.
If another label (we'll call it “skip_tests”) is set, I should be able to read it as an env var from within the script and alter the flow within the script accordingly (using normal bash syntax), e.g. to alter the package command parameters used a bit. This is useful for small changes where it might not make sense to run a lengthy test suite.
Is this possible with Gitlab, and if so, how?
I’ve tried experimenting with CI_MERGE_REQUEST_LABELS, but it doesn’t seem to be able to read that as an env var from within the script.
You have to use merge request pipelines for the CI_MERGE_REQUEST_LABELS variable (and other MR-related variables) to be present as documented in predefined variables.
You could use a rules: clause to skip jobs. Something like
build:
rules: # only run this job if the regex pattern does not match
- if: $CI_MERGE_REQUEST_LABELS !~ /skip_build/
You can also do this on any other kind of predefined (or user-defined) variable, like branch name, commit messages, MR titles, etc. Whatever works for you.
For example, a built in feature of GitLab is that if your commit message contains [ci skip] it will prevent the pipeline from running. You could implement similar functionality for your jobs and/or pipelines through rules: or workflow:rules:.
I have created a pipeline variable dynamically using powershell script with variable name
$ReleaseVariableName = "[$(Release.EnvironmentName)]_should_run_regression"
Now in next Job, I want to add a customCondition to check if value of above variable is "True".
I have tried couple of steps like using join,
eq(join('_', #(variables['Release.EnvironmentName'], 'should_run_regression')), 'True')
but that doesn't work since I cannot declare array using # in custom condition. So is there a way to use dynamic variable names in azure custom conditions?
PS:
I am working with Classic UI so YAML solutions don't work.
The step where I am using this custom condition is agentless job, so I can't use powershell scripts to read dynamic variable and set a Job level variable
I have an alternative of creating different variables for each stage but that looks awful
In Azure Pipelines, I see that you can access the environment variables from scripts in node.js during a pipeline run. However, I want to actually return a value and then capture/use it.
Does anyone know how to do this? I can't find any references on how to do this in documentation.
For consistency's sake it'd be nice to use node scripts for everything and not go back and forth between node and bash.
Thanks
Okay I finally figured this out. Azure documentation is a bit confusing on the topic, but my approach was what follows. In this example, I'm going to make a rather pointless simple script that sets a variable whose value is the name of the source branch, but all lower case.
1) Define your variable
Defining a variable can be done simply (though there is a lot of depth to how variables are used and I suggest consulting Azure documentation on variable creation for more). However, at the top of your pipeline yaml file you can define it as such:
variables
lowerCaseBranchName: ''
This creates an empty variable for use across your jobs. We'll use this variable as our example.
2) Create your script
"Returning a value" from your script simply means outputting it via node's stdout, the output of which will be consumed by the task to set it as a pipeline variable.
An important thing to remember is that any environment variables from the pipeline can be used within node, they are just reformatted and moved under node's process.env global. For instance, the commonly used Build.SourceBranchName environment variable in azure pipelines is accessible in your node script via its alias process.env.BUILD_SOURCEBRANCHNAME. This uppercase name transformation should be uniform across all environment variables.
Here's an example node.js script:
const lowerCaseBranchName = process.env.BUILD_SOURCEBRANCHNAME.toLowerCase();
process.stdout.write(lowerCaseBranchName);
3) Consume the output in the relevant step in azure pipelines
To employ that script in a job step, call it with a script task. Remember that a script task is, in this case, a bash script (though you can use others) that runs node as a command as it sets the value of our variable:
- script: |
echo "##vso[task.setvariable variable=lowerCaseBranchName]$(node path/to/your/script)"
displayName: 'Get lower case branch name'
Breaking down the syntax
Using variable definition syntax is, in my opinion extremely ugly, but pretty easy to use once you understand it. The basic syntax for setting a variable in a script is the following:
##vso[task.setvariable variable=SOME_VARIABLE_NAME]SOME_VARIABLE_VALUE
Above, SOME_VARIABLE_NAME is the name of our variable (lowerCaseBranchName) as defined in our azure pipeline configuration at the beginning. Likewise, SOME_VARIABLE_VALUE is the value we want to set that variable to.
You could do an additional line above this line to create a variable independently that you can then use to set the env variable with, however I chose to just inline the script call as you can see in the example above usign the $() syntax.
That's it. In following tasks, the environment variable lowerCaseBranchName can be utilized using any of the variable syntaxes such as $(lowerCaseBranchName),
Final result
Defining our variable in our yaml file:
variables
lowerCaseBranchName: ''
Our nodejs script:
const lowerCaseBranchName = process.env.BUILD_SOURCEBRANCHNAME.toLowerCase();
process.stdout.write(lowerCaseBranchName);
Our pipeline task implementation/execution of said script:
- script: |
echo "##vso[task.setvariable variable=lowerCaseBranchName]$(node path/to/your/script)"
displayName: 'Get lower case branch name'
A following task using its output:
- script: |
echo "$(lowerCaseBranchName)"
displayName: 'Output lower case branch name'
This will print the lower-cased branch name to the pipline console when it runs.
Hope this helps somebody! Happy devops-ing!
I'm trying to get my build repository name as an uppercase string combining predefine variables and expressions on Azure Devops as follows:
variables:
repoNameUpper: $[upper(variables['Build.Repository.Name'])]
- script: |
echo $(repoNameUpper)
Yet I get no output from it, what am I doing wrong here?
Yes, I know I could set a variable to achieve what I need using a bash script, yet I think it would not be so cool.
It because the Build.Repository.Name is agent-scoped, and can be used as an environment variable in a script and as a parameter in a build task. in another words - is not known at plan compile time, only at job execution time.
You can find more info in this GitHub issue.
I've created a powershell script that updates a Pipeline Variable during a Release Pipeline. It takes the custom variable and updates it with a new version using semantic versioning with every run.
I've tried to add this custom variable as the Release Pipeline but keeps on giving me an error "Names of releases from this pipeline will not be unique. Use pre-defined variables to generate unique release names."
I've tried setting the variable to "Settable at Release" and putting the scope to "Release"
Does anybody perhaps know if there is a way to let the release pipeline know this is a dynamic variable that changes?
The only other option is to add the revision number to it $(versionnumber)$(rev:.r)
use Custom Pipeline Variable for Release Name
For this issue ,I think it not feasible to achieve it. Release name must be a unique name,
the $(rev:r) token can ensure that every completed build/release has a unique name because it's adding a incremental number for each release. When a build/release is completed, if nothing else in the number has changed, the Rev integer value is incremented by one. So, basically we cannot achieve that without using $(rev:r), unless you can defined a token which has the same function with $(rev:r).
In addition,you can also use $(Build.BuildNumber) or $(Release.ReleaseId) which are also unique.
For the similar issue,please refer to this case .