I have a simple gitlab-yaml file that I thought would run a job when only scheduled. However, it is getting fire on a push event as well.
Can anyone please tell me the correct way in which to specify that a job is only run when scheduled.
This is my gitlab-yaml file
job:on-schedule:
only:
- schedules
- branches
script:
- /usr/local/bin/phpunit -c phpunit_config.xml
Thanks
According to the GitLab documentation, branches means "When a branch is pushed".
https://docs.gitlab.com/ce/ci/yaml/README.html#only-and-except-simplified
So including branches in your only: section causes the pipeline job to also run on pushes to any branch.
You can either remove the branches entry, or if you wanted to restrict to pushes for a specific branch you could extend the branch entry to include project and branch name (branches#<project>/<branch>).
My suggestion is to reduce your YML to:
job:on-schedule:
only:
- schedules
script:
- /usr/local/bin/phpunit -c phpunit_config.xml
Related
I would like to check if a branch is a tag or not in a single stage in gitlab-ci.yml. Then based on whether a branch is a tag or not, a specific set of commands are executed:
IF tag:
A SET OF COMMANDS
ELSE:
ANOTHER SET OF COMMAND
I am aware that only: -tags can be used to check if a branch is a tag, but how can I check the branch and then apply different sets of commands based on the branch type in a single gitlab-ci stage? In this case, if a branch is not a tag, I would to apply a different set of commands.
some_stage:
only:
- tags
script:
- | SET OF COMMANDS
You can use Gitlab-CI predefined variables:
CI_COMMIT_BRANCH (The commit branch name. Available in branch pipelines, including pipelines for the default branch. Not available in merge request pipelines or tag pipelines.)
CI_COMMIT_TAG (The commit tag name. Available only in pipelines for tags.)
The "CI_COMMIT_BRANCH" variable will hold the branch name only if pipeline is triggered from branch, otherwise it will not hold any value.
The same applies on the "CI_COMMIT_TAG" variable that will hold values only in case of pipeline triggered from tag.
You can use rules to check these variables and decide if you want to execute a job or not.
for example, if you want to build docker images from tags only:
build_docker:
rules:
- if: '$CI_COMMIT_TAG != null && $CI_BUILD_TAG =~ *SOME_REGEX*'
when: on_success
Update: Or you can use variable inside script to do you check but i recommend that you split the logic into to jobs. one for tags work and another for branch job. and specify with rules when a job is executed.
last thing: there is a deprecated tag called CI_BUILD_TAG, so if you are using an old version of gitlab.
I have a GitLab CI pipeline with a 'migration' job which I want to be added only if certain files changed between current commit and master branch, but in my current project I'm forced to use GitLab CI pipelines for push event which complicates things.
Docs on rules:changes clearly states that it will glitch and will not work properly without MR (my case of push event), so that's out of question.
Docs on rules:if states that it only works with env variables. But docs on passing CI/CD variables to another job clearly states that
These variables cannot be used as CI/CD variables to configure a
pipeline, but they can be used in job scripts.
So, now I'm stuck. I can just skip running the job in question overriding the script and checking for file changes, but what I want is not adding the job in question to pipeline in first place.
While you can't add a job alone to a pipeline based on the output of a script, you can add a child pipeline dynamically based on the script output. The method of using rules: with dynamic variables won't work because rules: are evaluated at the time the pipeline is created, as you found in the docs.
However, you can achieve the same effect using dynamic child-pipelines feature. The idea is you dynamically create the YAML for the desired pipeline in a job. That YAML created by your job will be used to create a child pipeline, which your pipeline can depend on.
Sadly, to add/remove a Gitlab job based on variables created from a previous job is not possible for a given pipeline
A way to achieve this is if your break your current pipeline to an upstream and downstream
The upstream will have 2 jobs
The first one will use your script to define a variable
This job will trigger the downstream, passing this variable
Upstream
check_val:
...
script:
... Script imposes the logic with the needed checks
... If true
- echo "MY_CONDITIONAL_VAR=true" >> var.env
... If false
- echo "MY_CONDITIONAL_VAR=false" >> var.env
artifacts:
reports:
dotenv: var.env
trigger_your_original_pipeline:
...
variables:
MY_CONDITIONAL_VAR: "$MY_CONDITIONAL_VAR"
trigger:
project: "project_namespance/project"
The downstream would be your original pipeline
Downstream
...
migration:
...
rules:
- if: '$MY_CONDITIONAL_VAR == "true"'
Now the MY_CONDITIONAL_VAR will be available at the start of the pipeline, so you can impose rules to add or not the migration job
I have some gitlab jobs in my pipleline which are slow and I'd like to prevent them from running when the changes will not affect the job's outcome.
This is what I have tried:
run_tests:
stage: checks
script:
- cargo test
except:
- master
- tags
only:
changes:
- "**/*.rs"
- "**/Cargo.toml"
- "**/Cargo.lock"
This sort of works. If a merge request has multiple commits, this job will not run on any of the commits after the first one, unless a Rust source file has changed.
But the job will still always run on the first commit of the branch, even if there are no changes to Rust source files between this branch and master. Even worse, if the tests fail on the first commit, subsequent commits might skip the tests so broken code could get merged.
How can I change this filter so that the diff is done against the target branch of the merge request?
Hidden quite deep in the documentation for only: changes, there is this snippet:
Without pipelines for merge requests, pipelines run on branches or tags that don’t have an explicit association with a merge request. In
this case, a previous SHA is used to calculate the diff, which
equivalent to git diff HEAD~. This could result in some unexpected
behavior, including:
When pushing a new branch or a new tag to GitLab, the policy always evaluates to true.
When pushing a new commit, the changed files are calculated using the previous commit as the base SHA
Which is the scenario you are currently experiencing, where the job always runs on the first commit of the new branch.
For this to work as you want, I believe you need to add only: merge_requests, however this would require changing your entire pipeline workflow to integrate only: merge_requests to your pipeline.
From the documentation:
With pipelines for merge requests, it’s possible to define a job to be
created based on files modified in a merge request.
In order to deduce the correct base SHA of the source branch, we
recommend combining this keyword with only: [merge_requests]. This
way, file differences are correctly calculated from any further
commits, thus all changes in the merge requests are properly tested in
pipelines.
Various issues from GitLab:
https://gitlab.com/gitlab-org/gitlab/-/issues/11427
https://gitlab.com/gitlab-org/gitlab/-/issues/27875
I'm trying to implement a job that would be invoked only when a developer tries to merge the MR into master. How do I do that?
I tried using
only:
- merge_requests
and it is almost what I need, except I don't want to run this job on every MR update - I want to run it only before merge into master.
I thought that maybe there is a way to use a new GitLab feature - Pipelines for Merged Results but as far as I understand I'd still need to run this job on every MR update.
Is there a way to do this that I'm missing? For example, can I only call this job for Merge trains - that would also be a solution?
Try the following:
only:
- master
You would probably need to use the advanced version of only and have something like:
only:
refs:
- merge_requests
variables:
- $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"
In this case, it'll only run if both of these conditions are true since
individual keys are logically joined by an AND
In my pipeline, I'd like to have a job run only if the Merge Requests target branch is a certain branch, say master or release.
Is this possible?
I've read through https://docs.gitlab.com/ee/ci/variables/ and unless I missed something, I'm not seeing anything that can help.
Update: 2019-03-21
GitLab has variables for merge request info since version 11.6 (https://docs.gitlab.com/ce/ci/variables/ see the variables start with CI_MERGE_REQUEST_). But, these variables are only available in merge request pipelines.(https://docs.gitlab.com/ce/ci/merge_request_pipelines/index.html)
To configure a CI job for merge requests, we have to set:
only:
- merge_requests
And then we can use CI_MERGE_REQUEST_* variables in those jobs.
The biggest pitfall here is only: merge_request has complete different behavior from normal only/except parameters.
usual only/except parameters:
(https://docs.gitlab.com/ce/ci/yaml/README.html#onlyexcept-basic)
only defines the names of branches and tags for which the job will run.
except defines the names of branches and tags for which the job will not run.
only: merge_request: (https://docs.gitlab.com/ce/ci/merge_request_pipelines/index.html#excluding-certain-jobs)
The behavior of the only: merge_requests parameter is such that only jobs with that parameter are run in the context of a merge request; no other jobs will be run.
I felt hard to reorganize jobs to make them work like before with only: merge_request exists on any job. Thus I'm still using the one-liner in my original answer to get MR info in a CI job.
Original answer:
No.
But GitLab have a plan for this feature in 2019 Q2: https://gitlab.com/gitlab-org/gitlab-ce/issues/23902#final-assumptions
Currently, we can use a workaround to achieve this. The method is as Rekovni's answer described, and it actually works.
There's a simple one-liner, get the target branch of an MR from the current branch:
script: # in any script section of gitlab-ci.yml
- 'CI_TARGET_BRANCH_NAME=$(curl -LsS -H "PRIVATE-TOKEN: $AWESOME_GITLAB_API_TOKEN" "https://my.gitlab-instance.com/api/v4/projects/$CI_PROJECT_ID/merge_requests?source_branch=$CI_COMMIT_REF_NAME" | jq --raw-output ".[0].target_branch")'
Explanation:
CI_TARGET_BRANCH_NAME is a newly defined variable which stores resolved target branch name. Defining a variable is not necessary for various usage.
AWESOME_GITLAB_API_TOKEN is the variable configured in repository's CI/CD variable config. It is a GitLab personal access token(created in User Settings) with api scope.
About curl options: -L makes curl aware of HTTP redirections. -sS makes curl silent(-s) but show(-S) errors. -H specifies authority info accessing GitLab API.
The used API could be founded in https://docs.gitlab.com/ce/api/merge_requests.html#list-project-merge-requests. We use the source_branch attribute to figure out which MR current pipeline is running on. Thus, if a source branch has multiple MR to different target branch, you may want to change the part after | and do your own logic.
About jq(https://stedolan.github.io/jq/), it's a simple CLI util to deal with JSON stuff(what GitLab API returns). You could use node -p or any method you want.
Because of the new env variables in 11.6 $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME and $CI_MERGE_REQUEST_TARGET_BRANCH_NAME jobs can be included or excluded based on the source or target branch.
Using the only and except (complex) expressions, we can build a rule to filter merge requests. For a couple examples:
Merge request where the target branch is master:
only:
refs:
- merge_requests
variables:
- $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"
Merge request except if the source branch is master or release:
only:
- merge_requests
except:
variables:
- $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME == "master"
- $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME == "release"
If you want to use multiple refs (let's say merge_requests and tags) and multiple variables, the refs will be OR'd, the variables will be OR'd, and the result will be AND'd:
If any of the conditions in variables evaluates to truth when using only, a new job is going to be created. If any of the expressions evaluates to truth when except is being used, a job is not going to be created.
If you use multiple keys under only or except, they act as an AND. The logic is:
(any of refs) AND (any of variables) AND (any of changes) AND (if kubernetes is active)
Variable expressions are also quite primitive, only supporting equality and (basic) regex. Because the variables will be OR'd you cannot specify both a source and target branch as of gitlab 11.6, just one or the other.
As of GitLab 11.6, there is CI_MERGE_REQUEST_TARGET_BRANCH_NAME.
If this is what you're really after, there could be an extremely convoluted way (untested) you could achieve this using the merge request API and CI variables.
With a workflow / build step something like:
Create merge request from feature/test to master
Start a build
Using the API (in a script), grab all open merge requests from the current project using CI_PROJECT_ID variable, and filter by source_branch and target_branch.
If there is a merge request open with the source_branch and target_branch being feature/test and master respectively, continue with the build, otherwise just skip the rest of the build.
For using the API, I don't believe you can use the CI_JOB_TOKEN variable to authenticate, so you'll probably need to create your own personal access token and store it as a CI variable to use in the build job.
Hope this helps!
Another example, but using rules:
rules:
# pipeline should run on merge request to master branch
- if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'master'
when: always
# pipeline should run on merge request to release branch
- if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'release'
when: always
- when: never
Gitlab CI is agnostic of Merge Requests (for now). Since the pipeline runs on the origin branch you will not be able to retrieve the destination.