I am using Gitlab and added a .gitlab-ci.yml file to trigger my pipeline. What I don't understand is the purpose of refs when used in combination with the only keyword. Here is an example I use:
only:
- /^newlib.*$/
But in another example I found this:
only:
refs:
- master
The documentation only states:
The refs strategy can take the same values as the simplified only/except configuration.
Can anyone share some light on this? What is the difference between the two?
There is no difference between your two examples. As you quoted the docs yourself:
The refs strategy can take the same values as the simplified only/except configuration.
That said, you only need to use refs if you also want to use the other options: changes, kubernetes and variables.
The docs has some examples using multiple options such as:
test:
script: npm run test
only:
refs:
- master
- schedules
variables:
- $CI_COMMIT_MESSAGE =~ /run-end-to-end-tests/
kubernetes: active
If you remove the refs you will get an invalid yml file
Related
I would like to achieve this Gitlab CI/CD behaviour:
Two jobs defined in the pipeline:
job_beta triggered when a tag is created with a semver number containing a "-rc*" suffix. For example: v1.0.0-rc1.
job_production triggered when a tag is created with a semver number, without any suffix. For example: v1.0.0.
Is there a way to parse the $CI_COMMIT_TAG variable in the .gitlab-ci.yml file of the project? Or any other ideas about how to achieve this?
To achieve this kind of behaviour, you can rely on the only keyword to control which job will be created. Simply specify a regular expression that will either match a semver with the rc suffix or one without any suffix.
You could do it the following way if your semantic versions are prefixed with v:
build-beta:
image: your-build-image:latest
stage: build
only:
- /^v[0-9]+\.[0-9]+\.[0-9]+-rc[0-9]+$/
script:
- ./build-beta.sh # Do something...
build-prod:
image: your-build-image:latest
stage: build
only:
- /^v[0-9]+\.[0-9]+\.[0-9]+$/
script:
- ./build-prod.sh # Do something...
You can also achieve something similar with the rules keyword by matching $CI_COMMIT_REF_NAME with the appropriate regex in an if condition.
I'm wondering if using extends: in global variables: is supposed to work.
The documentation here states, that:
Keyword type: Job keyword. You can use it ONLY as part of a job.
(my highlighting).
However, if I use something like
stages:
- test
.common_variables:
__FOO: "foo"
variables:
extends: .common_variables
__BAR: "bar"
test:
stage: test
script:
- env
rules:
- when: always
then I find both __FOO as well as __BAR variables defined on the runner. However, I also find the extends variable (set to .common_variables defined on the runner).
So I wonder if this is the expected behaviour on which one can rely and the presence of extends variable is just a minor side effect or just a bug in the GitLab version that we use and this keyword is supported only inside jobs as the docs state?
P.S. note, that i specifically ask about extends because i plan to off-load the shared variables into a separate file in a separate repo to be able to reuse it in pipelines of multiple repositories
Yes it works like that.
But you can also read in addition how it can be used correctly if you want to run several scripts in one YAML file.
You can read more details here
I'm absolutely new to all this ci/cd thing and its documentation is too extensive. So i'd like to apologize in advance if there is too easy question.
I have gitlab repo with many configs for different services i.e:
project_folder
--global_conf.json
--service1
----config_folder
------....
--service2
----config_folder
------....
--service3
----config_folder
------....
I'd like to know what should I do (just general plan and some key words to search documentation more precisely already would be fine) to send to hosts [host1:port, host2:port, host3:port] different strings with commands "check {service_name} for changes" or "check {global_config} for changes" e.t.c, depending on the file that was committed.
I already have my service on each host that could perform different operations for different task's strings, so I need just send them that task.
You'll propably end up using "only: changes" as explained here.
Something like this should fit:
stages:
- trigger
service1:
stage: trigger
script:
- curl http://host1:port1
only:
changes:
- service1/*
service2:
stage: trigger
script:
- curl http://host2:port2
only:
changes:
- service2/*
Lets say I have a test block for all my microservices (15-20+). Tests take a long time since there are so many disparate modules in this monorepo.
Lets say I only want to run 1 or maybe 2 at a time if and only if specific code changes have been made underneath a path. How can I best do this? For assembling I do something like this (not sure if this is terrible or not)
Ultimately, I'm trying to only build and test relevant things if they're relevant (based on if they or a related module I can define change)
Module-specific assembles
x:
stage: build
image: gradle:6.0-jdk11
script:
- gradle :x:assemble
artifacts:
paths:
- x/build/libs
only:
changes:
- x
- x/*
- x/*/**
y_build:
stage: build
image: gradle:6.0-jdk11
script:
- gradle :y:assemble
artifacts:
paths:
- y/build/libs
only:
changes:
- y
- y/*
- y/*/**
Current block for testing
test:
stage: test
image: gradle:6.0-jdk11
services:
- name: gitlab-registry.company.com/nap/dynamodb-local:1252954
command: [ "-inMemory", "-sharedDb" ]
alias: dynamodb
script: gradle check
There are many ways that a Gitlab CI Pipeline can be triggered, but the basic way is when a commit is pushed, no matter what the change is. Currently, there isn't a way to inspect which parts of the code are changed, and only run some steps or others based on the result, but you can control which steps run based on the branch or tag name.
So for example, you could have steps that only run when the branch name starts with something like "microservice_3_", and then make sure that when editing Microservice 3, you always start the branch name with "microserver_3_", though this could get complicated the more microservices you have to support.
The easier option (in terms of the pipeline definition) is to maintain the microservices in separate repositories, each with their own pipelines and tests. This way each individual pipeline only runs against a specific component and doesn't care about changes to the others. Then if you need to, you can combine the microservices in another repository (included as git submodules) and have a pipeline that only cares about the services as a whole.
This would add extra overhead while developing the project(s), but it makes the pipelines easier to manage.
In my pipeline, I'd like to have a job run only if the Merge Requests target branch is a certain branch, say master or release.
Is this possible?
I've read through https://docs.gitlab.com/ee/ci/variables/ and unless I missed something, I'm not seeing anything that can help.
Update: 2019-03-21
GitLab has variables for merge request info since version 11.6 (https://docs.gitlab.com/ce/ci/variables/ see the variables start with CI_MERGE_REQUEST_). But, these variables are only available in merge request pipelines.(https://docs.gitlab.com/ce/ci/merge_request_pipelines/index.html)
To configure a CI job for merge requests, we have to set:
only:
- merge_requests
And then we can use CI_MERGE_REQUEST_* variables in those jobs.
The biggest pitfall here is only: merge_request has complete different behavior from normal only/except parameters.
usual only/except parameters:
(https://docs.gitlab.com/ce/ci/yaml/README.html#onlyexcept-basic)
only defines the names of branches and tags for which the job will run.
except defines the names of branches and tags for which the job will not run.
only: merge_request: (https://docs.gitlab.com/ce/ci/merge_request_pipelines/index.html#excluding-certain-jobs)
The behavior of the only: merge_requests parameter is such that only jobs with that parameter are run in the context of a merge request; no other jobs will be run.
I felt hard to reorganize jobs to make them work like before with only: merge_request exists on any job. Thus I'm still using the one-liner in my original answer to get MR info in a CI job.
Original answer:
No.
But GitLab have a plan for this feature in 2019 Q2: https://gitlab.com/gitlab-org/gitlab-ce/issues/23902#final-assumptions
Currently, we can use a workaround to achieve this. The method is as Rekovni's answer described, and it actually works.
There's a simple one-liner, get the target branch of an MR from the current branch:
script: # in any script section of gitlab-ci.yml
- 'CI_TARGET_BRANCH_NAME=$(curl -LsS -H "PRIVATE-TOKEN: $AWESOME_GITLAB_API_TOKEN" "https://my.gitlab-instance.com/api/v4/projects/$CI_PROJECT_ID/merge_requests?source_branch=$CI_COMMIT_REF_NAME" | jq --raw-output ".[0].target_branch")'
Explanation:
CI_TARGET_BRANCH_NAME is a newly defined variable which stores resolved target branch name. Defining a variable is not necessary for various usage.
AWESOME_GITLAB_API_TOKEN is the variable configured in repository's CI/CD variable config. It is a GitLab personal access token(created in User Settings) with api scope.
About curl options: -L makes curl aware of HTTP redirections. -sS makes curl silent(-s) but show(-S) errors. -H specifies authority info accessing GitLab API.
The used API could be founded in https://docs.gitlab.com/ce/api/merge_requests.html#list-project-merge-requests. We use the source_branch attribute to figure out which MR current pipeline is running on. Thus, if a source branch has multiple MR to different target branch, you may want to change the part after | and do your own logic.
About jq(https://stedolan.github.io/jq/), it's a simple CLI util to deal with JSON stuff(what GitLab API returns). You could use node -p or any method you want.
Because of the new env variables in 11.6 $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME and $CI_MERGE_REQUEST_TARGET_BRANCH_NAME jobs can be included or excluded based on the source or target branch.
Using the only and except (complex) expressions, we can build a rule to filter merge requests. For a couple examples:
Merge request where the target branch is master:
only:
refs:
- merge_requests
variables:
- $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"
Merge request except if the source branch is master or release:
only:
- merge_requests
except:
variables:
- $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME == "master"
- $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME == "release"
If you want to use multiple refs (let's say merge_requests and tags) and multiple variables, the refs will be OR'd, the variables will be OR'd, and the result will be AND'd:
If any of the conditions in variables evaluates to truth when using only, a new job is going to be created. If any of the expressions evaluates to truth when except is being used, a job is not going to be created.
If you use multiple keys under only or except, they act as an AND. The logic is:
(any of refs) AND (any of variables) AND (any of changes) AND (if kubernetes is active)
Variable expressions are also quite primitive, only supporting equality and (basic) regex. Because the variables will be OR'd you cannot specify both a source and target branch as of gitlab 11.6, just one or the other.
As of GitLab 11.6, there is CI_MERGE_REQUEST_TARGET_BRANCH_NAME.
If this is what you're really after, there could be an extremely convoluted way (untested) you could achieve this using the merge request API and CI variables.
With a workflow / build step something like:
Create merge request from feature/test to master
Start a build
Using the API (in a script), grab all open merge requests from the current project using CI_PROJECT_ID variable, and filter by source_branch and target_branch.
If there is a merge request open with the source_branch and target_branch being feature/test and master respectively, continue with the build, otherwise just skip the rest of the build.
For using the API, I don't believe you can use the CI_JOB_TOKEN variable to authenticate, so you'll probably need to create your own personal access token and store it as a CI variable to use in the build job.
Hope this helps!
Another example, but using rules:
rules:
# pipeline should run on merge request to master branch
- if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'master'
when: always
# pipeline should run on merge request to release branch
- if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'release'
when: always
- when: never
Gitlab CI is agnostic of Merge Requests (for now). Since the pipeline runs on the origin branch you will not be able to retrieve the destination.