I want to get some output from the same pipeline that was run previously.
Suppose in run my-pipeline which outputs some hash. Next time I run this pipeline, I want to get the hash from the previous run of my-pipeline.
The reason for that is essentially conditional cache invalidation, So sometimes I need to reuse the same generated hash, while other times I want to generate a new hash that from then on is passed down each pipeline run, until it's changed again.
To transfer output between different pipeline runs, typically we will consider artifacts or store output somewhere which will be not deleted.
According to your description, you can put the hash value to a file and push to your repo in the pipeline, when you check it out in the next pipeline run, you can read from the file to get the value.You can also add tag value to hint it's changed or not.
Related
I need to exclude a job from pipeline in case my project version is pre-release.
How I know it's a pre-release?
I set the following in the version info file, that all project files and tools use:
version = "1.2.3-pre"
From CI script, I parse the file, extract the version value, and know whether it's a pre-release or not, and can set the result in an environment variable.
The only way I know to exclude a job from pipeline is to use rules, while, I know also from gitlab docs that:
rules are evaluated before any jobs run
before_script also is claimed to be called with the script, i.e. after applying the rules.
I can stop the job, only after it starts from the script itself, based on the version value, but what I need is to extract the job from the pipeline in the first place, so it's not displayed in the pipeline history. Any idea?
Thanks
How do you run (start) your pipeline, and is the information whether "it's a pre-release" already known at this point?
If yes, then you could add a flag like IS_PRERELEASE as a variable to the pipeline, and use that in the rules: section of your job. The drawback is that this will not work with automatic pipelines (triggered by a commit or MR); but you can use this approach with manually triggered pipelines (https://docs.gitlab.com/ee/ci/variables/#override-a-variable-when-running-a-pipeline-manually) or via the API (https://docs.gitlab.com/ee/ci/triggers/#pass-cicd-variables-in-the-api-call).
I have a few use cases in my Gitlab setup I would like to be able to support:
If a certain label (let's call it “skip_build”) is set, the deployment steps should not be run when I merge an MR to a main branch. This would be useful when we have multiple MRs being merged right after another and only need the last one built.
If another label (we'll call it “skip_tests”) is set, I should be able to read it as an env var from within the script and alter the flow within the script accordingly (using normal bash syntax), e.g. to alter the package command parameters used a bit. This is useful for small changes where it might not make sense to run a lengthy test suite.
Is this possible with Gitlab, and if so, how?
I’ve tried experimenting with CI_MERGE_REQUEST_LABELS, but it doesn’t seem to be able to read that as an env var from within the script.
You have to use merge request pipelines for the CI_MERGE_REQUEST_LABELS variable (and other MR-related variables) to be present as documented in predefined variables.
You could use a rules: clause to skip jobs. Something like
build:
rules: # only run this job if the regex pattern does not match
- if: $CI_MERGE_REQUEST_LABELS !~ /skip_build/
You can also do this on any other kind of predefined (or user-defined) variable, like branch name, commit messages, MR titles, etc. Whatever works for you.
For example, a built in feature of GitLab is that if your commit message contains [ci skip] it will prevent the pipeline from running. You could implement similar functionality for your jobs and/or pipelines through rules: or workflow:rules:.
I want to set the value of a variable based on whether loading from a cache has succeeded.
I plan to set the variable using an if statement, the same way they do in this example: https://docs.gitlab.com/ee/ci/yaml/#workflowrulesvariables
(The link goes to the wrong part of the page: search for Example of workflow:rules:variables )
If my yaml looks like this:
cache:
key: $CI_COMMIT_REF_SLUG
paths:
- pathtocache
How can I check if pathtocache exists?
This isn't possible due to the lifecycle of when rules are evaluated, which is before the cache/artifacts are restored. Keep in mind the rules can be used to define whether or not the job runs, so they are evaluated at pipeline start before any jobs have been run and thus before you would have generated any cache files.
If you want to test whether the cache has been populated then branch your job's logic based on that, you will have to do so within the script block for your job.
I want to write a task in VSTS that persists a value between phases. The purpose is so I could store a value that points to a record in an external system, and then retrieve and update that record in a later phase that always executes during a release failure. I tried writing an environment variable, but that does not persist:
write-output("##vso[task.setvariable variable=CRQID;]$changeid")
I see you can write an attachment (below), but I can't find any reference to a "get-attachment" cmdlet in the SDK:
write-output "##vso[task.addattachment type=Distributedtask.Core.Summary;name=Change Request;]$fileName"
I was referencing this document.
I thought I might be able to write the file to the file system, but then if the agents were pooled and my second phase executed on another agent the path would be worthless.
For VSTS itself, it can not persist values between phases. But you can archieve it by developing your own task.
And as you found, if you stored the value in the build agent of first phase, the value can not be found if you use another agent in the second phase.
Actually you just need to store the value to a place where both phases (different agents) can get the value. Such as you can store the value in your github repo by below commands:
git clone https://github.com/username/reponame
#copy filename under reponame folder (overwrite of the filename already exist under reponame folder)
cd repo name
git add .
git commit -m 'store values in the filename'
git push https://username:password#github.com/ master
If you want to use the value in another phase, then clone the github repo and get the value from the filename.
We are using Jenkins to automate several of our build and test processes. For some of our process, the engineer starting the build needs to specify a parameter. But the range of possible and optimal values for that parameter change throughout the course of the day.
What I would like to do is let the engineer specify a value - if they know an optimal value - or leave it blank and have a value be calculated by an early build step. If the value is calculated, I would like the calculating build step to update the parameter value of the job. That way, all subsequent build steps don't have to worry about using the parameter or calculating it, they just use the parameter regardless.
It looks like the Groovy Script Plugin might be able to do this, but I can't see how I can SET the build parameters, just GET them.
Found the answer: use the EnvInject Plugin. One of the features is a build step that allows you to "inject" parameters into the build job from a settings file. I used one build step to create the settings file, then another build step to inject the new values. Then, all subsequent build steps and post-build operations used the new value.
Update with an example:
To add a new parameter (REPORT_FILE), based on existing one (JOB_NAME), inject a map with new or modified parameters in the Groovy Script box:
// Setting a map for new build parameters
def paramsMap = [:]
// Set REPORT_FILE based on JOB_NAME
def filename = JOB_NAME.replace(' ','_') + ".html"
paramsMap.put("REPORT_FILE", filename)
// Add or modify other parameters...
return paramsMap
Jenkins does have the ability to parameterize builds. For a string parameter, the developer can leave the field blank and then your build scripts can check to see if the env. variable for the parameter is set. If the env. var. is not set, the script can perform whatever calculation is needed (I don't think Jenkins has "pre-build steps") and pass it along. For a choice parameter the first line can be something like (Default), and again the build script can test its value and act accordingly.
Note on (Default)
I tried leaving the first line of the choice box blank, and Jenkins saved it correctly the first time; but when I came back to reconfigure the build Jenkins ran some kind of trim on options and the leading blank line was removed so I settled on (Default).
I hope this helps,
Zachary