When I take a mergeRequest in GitLab, there is a compile error with description:
CI build failed for 4a0b9b43 , view Details：.gitlab-ci.yml not
found in this commit.
But I have sync with the destination branch and compile succeed on native, how can I fix this issue?
You must place the .gitlab-ci.yml file to the root directory of a project first. If you want to cross-reference other CI configuration locally or from somewhere (GitLab instance required to install on remote locations), you must use the include tag
How-to
Please check your GL instance version first before you are using include. Compare your instance version with the minimum version required from the documentation. If you need to upgrade, back up everything and run an update. It should take hours before you can go back into business.
To cross-reference CI config files locally, use the include:local. Make sure they are on the same branch. If that is on other branch, use ref.
include:
- local: '/templates/.gitlab-ci-template.yml'
In case you want to cross-reference CI config files somewhere within the instance, use
include:
- project: 'my-group/my-project'
file: '/templates/.gitlab-ci-template.yml'
# You can also specify ref, with the default being the head of the object.
- project: 'my-group/my-project'
ref: master # Git branch
file: '/templates/.gitlab-ci-template.yml'
- project: 'my-group/my-project'
ref: v1.0.0 # Git tag
file: '/templates/.gitlab-ci-template.yml'
- project: 'my-group/my-project'
ref: 787123b47f14b552955ca2786bc9542ae66fee5b # Git SHA
file: '/templates/.gitlab-ci-template.yml'
If you prefer to use CI config templates shipped with the instance, use include:template. Check the GitLab's template collection for more details.
include:
- template: Auto-DevOps.gitlab-ci.yml
If deployed outside the instance borders, like GitLab.com, use include:remote.
include:
- remote: 'https://gitlab.com/awesome-project/raw/master/.gitlab-ci-template.yml'
Examples
The first example was derived from GitLab's CI config file for GitLab EE (gitlab-org/gitlab).
image: "registry.gitlab.com/gitlab-org/gitlab-build-images:ruby-2.6.3-golang-1.12-git-2.24-lfs-2.9-chrome-73.0-node-12.x-yarn-1.16-postgresql-9.6-graphicsmagick-1.3.33"
stages:
- sync
- prepare
- quick-test
- test
- post-test
- review-prepare
- review
- qa
- post-qa
- notification
- pages
variables:
RAILS_ENV: "test"
NODE_ENV: "test"
SIMPLECOV: "true"
GIT_DEPTH: "20"
GIT_SUBMODULE_STRATEGY: "none"
GET_SOURCES_ATTEMPTS: "3"
KNAPSACK_RSPEC_SUITE_REPORT_PATH: knapsack/report-master.json
FLAKY_RSPEC_SUITE_REPORT_PATH: rspec_flaky/report-suite.json
BUILD_ASSETS_IMAGE: "false"
ES_JAVA_OPTS: "-Xms256m -Xmx256m"
ELASTIC_URL: "http://elastic:changeme#elasticsearch:9200"
after_script:
- date
include:
- local: .gitlab/ci/cache-repo.gitlab-ci.yml
- local: .gitlab/ci/cng.gitlab-ci.yml
- local: .gitlab/ci/docs.gitlab-ci.yml
- local: .gitlab/ci/frontend.gitlab-ci.yml
- local: .gitlab/ci/global.gitlab-ci.yml
- local: .gitlab/ci/memory.gitlab-ci.yml
- local: .gitlab/ci/notifications.gitlab-ci.yml
- local: .gitlab/ci/pages.gitlab-ci.yml
- local: .gitlab/ci/qa.gitlab-ci.yml
- local: .gitlab/ci/reports.gitlab-ci.yml
- local: .gitlab/ci/rails.gitlab-ci.yml
- local: .gitlab/ci/review.gitlab-ci.yml
- local: .gitlab/ci/setup.gitlab-ci.yml
- local: .gitlab/ci/dev-fixtures.gitlab-ci.yml
- local: .gitlab/ci/test-metadata.gitlab-ci.yml
- local: .gitlab/ci/yaml.gitlab-ci.yml
- local: .gitlab/ci/releases.gitlab-ci.yml
You can see more examples on the .gitlab-ci.yml reference in the GitLab documentation.
Related
I have a gitlab repo which has two folders(folder-a and folder-b) in it, I want them to get triggered based on the commit on their folder. Currently I have a single gitlab.yml file which has below configuration. It triggers folder-a but not folder-b.
folder-a:
trigger:
include:
- local: folder-a/.gitlab-ci.yml
rules:
- changes: [folder-a/*]
folder-b:
trigger:
include:
- local: folder-b/.gitlab-ci.yml
rules:
- changes: [folder-b/*]
Sorry I had the issue with folder-b .gitlab-ci yml file. this works fine, now able to trigger folder based on commits to the branches
I'm using self-hosted gitlab version 14.10 with self-registered gitlab runners using docker executors. In this image I'm using gitlab's secret detection job template and it generates artifact gl-secret-detection-report.json and it seems that it uploads it back to some kind of coordinator (I wonder what is this) but unfortunately I still can't see it in my Job artifacts section at this page.
I've checked this question that seemed to be similar, but its not, I've also checked GitLab's documentation and found nothing similar to my issue.
I even can't see Artifacts column in the Job's list at my Pipeline's page:
Okay, I've found the answer here:
To be able to browse the report output files, include the artifacts:paths keyword.
After I've overrided artifacts section for the job it worked fine. There's still no "Artifacts" column in the Job's row at the Pipeline's job list page.
My .gitlab-ci.yml:
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
include:
- project: 'devops/gitlab-templates'
ref: master
file: '/lib/gitlab/ci/templates/Docker/Docker-Build-Risky.DO.yml'
- project: 'devops/gitlab-templates'
ref: master
file: '/lib/gitlab/ci/templates/Security/Trivy-Scan.yml'
- template: Security/Secret-Detection.gitlab-ci.yml
- project: 'devops/gitlab-templates'
ref: master
file: '/lib/gitlab/ci/templates/Security/Dive-Scan.yml'
- project: 'devops/gitlab-templates'
ref: master
file: '/lib/gitlab/ci/templates/Docker/Docker-Retag-n-Push.DO.yml'
- project: 'devops/gitlab-templates'
ref: master
file: '/lib/gitlab/ci/templates/Docker/Docker-Retag-n-Push.AWS.yml'
- project: 'devops/gitlab-templates'
ref: master
file: '/lib/gitlab/ci/templates/Docker/Docker-Retag-n-Push.GCP.yml'
- project: 'devops/gitlab-templates'
ref: master
file: '/lib/gitlab/ci/templates/AWS/Deploy.yml'
secret_detection:
variables:
SECRET_DETECTION_HISTORIC_SCAN: "true"
allow_failure: false
artifacts:
reports:
secret_detection: gl-secret-detection-report.json
# this is the way to make artifacts appear
paths:
- gl-secret-detection-report.json
expire_in: 1 day
integration-tests:
stage: test
needs:
- job: "docker-build"
artifacts: true
...
deploy-to-aws:
environment: production
variables:
...
I'm trying to set up MobSF SAST within Gitlab-ci and having a few issues.
I've followed the instructions within the Gitlab Docs and within the MobSF Gitlab repo
However, when I add:
To my .gitlab-ci.yml . I get a yml error stating that it could not get access
My .gitlab-ci.yml file looks like:
sast:
stage: Security
tags:
- docker
include:
- project: 'gitlab-org/security-products/analyzers/mobsf'
ref: master
file: '/template/mobsf.gitlab-ci.yml'
I have a docker image on my machine with gitlab-runners as an image. Does anyone have any thoughts about how to implement this so that i can run automated MobSF SAST on both Android and iOS?
So after working through this, It seems that you must have the following included in yoru gitlab-ci.yml file:
variables:
#required for Mobile SAST
SAST_EXPERIMENTAL_FEATURES: "true"
include:
- template: Security/SAST.gitlab-ci.yml
sast:
image: docker:19.03.8
stage: Security
variables:
SEARCH_MAX_DEPTH: 4
artifacts:
reports:
sast: gl-sast-report.json
tags:
- docker
I have a two projects in GitLab and I am trying to integrate SonarQube with my GitLab projects.
Project 1
I have added the 'sonar-scanner.properties' file to Project1 and it's as follows:
sonar-scanner.properties
# SonarQube server
# sonar.host.url & sonar.login are set by the Scanner CLI.
# See https://docs.sonarqube.org/latest/analysis/gitlab-cicd/.
# Project settings.
sonar.projectKey=Trojanwall
sonar.projectName=Trojanwall
sonar.projectDescription=My new interesting project.
sonar.links.ci=https://gitlab.com/rmesi/trojanwallg2-testing/-/pipelines
#sonar.links.issue=https://gitlab.com/rmesi/trojanwallg2-testing/
# Scan settings.
sonar.projectBaseDir=./
#sonar.sources=./
sonar.sources=./
sonar.sourceEncoding=UTF-8
sonar.host.url=http://sonarqube.southeastasia.cloudapp.azure.com:31000
sonar.login=4f4cbabd17914579beb605c3352349229b4fd57b
#sonar.exclusions=,**/coverage/**
# Fail CI pipeline if Sonar fails.
sonar.qualitygate.wait=true
Then I added the sonar scanner job in the gitlab-ci.yml file:
gitlab-ci.yml
sonar-scanner-trojanwall:
stage: sonarqube:scan
image:
name: sonarsource/sonar-scanner-cli:4.5
entrypoint: [""]
variables:
# Defines the location of the analysis task cache
SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"
# Shallow cloning needs to be disabled.
# See https://docs.sonarqube.org/latest/analysis/gitlab-cicd/.
GIT_DEPTH: 0
cache:
key: "${CI_JOB_NAME}"
paths:
- .sonar/cache
script:
- sonar-scanner
only:
- Production
- /^[\d]+\.[\d]+\.1$/
when: on_success
After this, I configured the two variables: 'SONAR_HOST_URL' and 'SONAR_TOKEN' and then, ran the pipeline. It worked perfectly fine for the Project 1.
Project 2
Then, I needed to do the same for the Project 2 as well. I needed the sonar scanner to go into the Project 2, scan and analyze. For that, I created another project in SonarQube with a new token.
I needed to configure in such a way that when the pipeline for Project 1 is triggered, it scans both Project 1 and 2.
For that, I added another job in Project1's pipeline.
It's as follows:
gitlab-ci.yml
sonar-scanner-test-repo:
stage: sonarqube:scan
trigger:
include:
- project: 'rmesi/test-repo'
ref: master
file: 'sonarscanner.gitlab-ci.yml'
only:
- Production
- /^[\d]+\.[\d]+\.1$/
when: on_success
I tried to setup a downstream pipeline to trigger a yaml file in Project 2. So, when The pipeline in Project 1 is triggered and when the job 'sonar-scanner-test-repo' gets triggered, another yaml file in Project 2 is run as a down stream pipeline. That YAML file is as follows:
sonarscanner.gitlab-ci.yml
stages:
- sonarqube:scan
variables:
CI_PROJECT_DIR: /builds/rmesi/test-repo
sonar-scanner:
stage: sonarqube:scan
image:
name: sonarsource/sonar-scanner-cli:4.5
entrypoint: [""]
variables:
# Defines the location of the analysis task cache
SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"
# Shallow cloning needs to be disabled.
# See https://docs.sonarqube.org/latest/analysis/gitlab-cicd/.
GIT_DEPTH: 0
cache:
key: "${CI_JOB_NAME}"
paths:
- .sonar/cache
script:
- cd /builds/rmesi/
- git clone https://gitlab.com/rmesi/test-repo.git test-repo
- sonar-scanner
Then I added the 'sonar-project.properties' file in Project2 which is as follows:
sonar-project.properties
# SonarQube server
# sonar.host.url & sonar.login are set by the Scanner CLI.
# See https://docs.sonarqube.org/latest/analysis/gitlab-cicd/.
# Project settings.
sonar.projectKey=test-repo
sonar.projectName=test-repo
sonar.projectDescription=My new interesting project.
sonar.links.ci=https://gitlab.com/rmesi/test-repo/-/pipelines
#sonar.links.issue=https://gitlab.com/rmesi/test-repo/
# Scan settings.
sonar.projectBaseDir=/builds/rmesi/test-repo/
sonar.sources=/builds/rmesi/test-repo/, ./
sonar.sourceEncoding=UTF-8
sonar.host.url=http://sonarqube.southeastasia.cloudapp.azure.com:31000
sonar.login=b0c40e44fd59155d27ee43ae375b9ad7bf39bbdb
#sonar.exclusions=,**/coverage/**
# Fail CI pipeline if Sonar fails.
sonar.qualitygate.wait=true
The issue is that, when the down stream pipeline is run, I am getting the following error message:
I figured out that the down stream pipeline is not locating the 'sonar-scanner.properties' in Project 2. (Lines 68 and 74)
Where as, on Project 1 while searching for this step, it shows:
INFO: Project root configuration file: /builds/rmesi/trojanwallg2-testing/sonar-project.properties
But in Project 2 it's not working.
Does anyone know how to fix this?
I found the solution to this, myself.
Required to add
"- cd /build/rmesi/test-repo ; sonar-scanner"
in the script section in the job of the 'sonarscanner.gitlab-ci.yml' file.
That way, the runner maps directly to desired directory and execute the 'sonar-scanner' command there.
I am trying to set up a CI with minimal code duplication using .gitlab-ci.yml.
With that, I am separating the configuration in separate files and reusing parts of it that are common.
I have a separate repository with Gitlab CI settings: gitlab-ci and several projects that use it to form their own CI pipelines.
Contents of gitlab-ci repository
template_jobs.yml:
.sample:
rules:
- if: '$CI_PIPELINE_SOURCE == "push"'
when: on_success
- when: never
jobs_architectureA.yml:
include:
- local: '/template_jobs.yml'
.script_core: &script_core
- echo "Running stage"
test_archA:
extends:
- .sample
stage: test
tags:
- architectureA
script:
- *script_core
jobs_architectureB.yml:
include:
- local: '/template_jobs.yml'
.script_core: &script_core
- echo "Running stage"
test_archB:
extends:
- .sample
stage: test
tags:
- architectureB
script:
- *script_core
Project with code contents:
In the actual project (separate repositories per project, and I have a lot of them), I have the following:
.gitlab-ci.yml:
stages:
- test
include:
- project: 'gitlab-ci'
file: '/jobs_architectureA.yml'
- project: 'gitlab-ci'
file: '/jobs_architectureB.yml'
This configuration works fine and allows to include only some architectures for some modules while sharing rules between the job templates.
However, it's easy to notice one code duplication: both jobs_architectureA.yml and jobs_architectureB.yml contain a common section:
.script_core: &script_core
- echo "Running stage"
It would be ideal to move it into a separate file: template_scripts.yml and include from both jobs_architectureA.yml* and jobs_architectureB.yml. However, that results in the invalid YAML (at least from Gitlab's point of view).
With that, I make a conclusion that I can share the rules as the mechanism of their usage is via extends keyword; however, I am not able to do it with the scripts: as it uses &/* anchoring mechanic on the YAML level.
Ideally, I want something along the lines of:
Contents of the ideal (conceptually) gitlab-ci repository
template_jobs.yml:
.sample:
rules:
- if: '$CI_PIPELINE_SOURCE == "push"'
when: on_success
- when: never
template_scripts.yml:
.script_core: &script_core
- echo "Running stage"
jobs_architectureA.yml:
include:
- local: '/template_jobs.yml'
- local: '/template_scripts.yml'
test_archA:
extends:
- .sample
stage: test
tags:
- architectureA
script:
- *script_core # this becomes invalid, as script_core is in the other file, even though it is included at the top
jobs_architectureB.yml:
include:
- local: '/template_jobs.yml'
- local: '/template_scripts.yml'
test_archB:
extends:
- .sample
stage: test
tags:
- architectureB
script:
- *script_core # this becomes invalid, as script_core is in the other file, even though it is included at the top
Am I doing something wrong?
Am I hitting a limitation of the Gitlab mechanic? Is it the implementation of the include directive in this specific YML type, that limits me?
Do I have options to achieve something close to the desired behaviour?
Note, while this might not look like a big deal, in reality, I have many more pieces to the scripts, and the actual script is much larger. Thus, currently, it is duplicated code all over the place which is very prone to mistakes.
my solution is to not include template_jobs.yml and template_scripts.yml directly in jobs_architectureA.yml but only in the "final" .gitlab-ci.yml
taking you exemple, /template_jobs.yml//template_scripts.yml do not change.
jobs_architectureA.yml loses the include:
test_archA:
extends:
- .sample
stage: test
tags:
- architectureB
script:
- *script_core # this becomes invalid, as script_core is in the other file, even though it is included at the top
and .gitlab-ci.yml becomes:
stages:
- test
include:
- local: '/template_jobs.yml'
- local: '/template_scripts.yml'
- project: 'gitlab-ci'
file: '/jobs_architectureA.yml'
- project: 'gitlab-ci'
file: '/jobs_architectureB.yml'
in reality, I have many more pieces to the scripts, and the actual script is much larger
Adding to Cyril's solution, GitLab 13.12 (May 2021) can help scale those includes:
Support wildcards when including YAML CI/CD configuration files
The includes: keyword for CI/CD pipelines lets you break one long .gitlab-ci.yml file into multiple smaller files to increase readability.
It also makes it easier to reuse configuration in multiple places.
Frequently there are multiple files included into a single pipeline, and they all might be stored in the same place.
In this release, we add support to use the * wildcard with the local includes: keyword. You can now make your includes: sections more dynamic, less verbose, and easier to read, check out how we are dogfooding it in GitLab.
See Documentation and Issue.
The extends: keyword can be used with multiple jobs to extend from (on recent versions of GitLab). You can simply do the following:
include:
- local: '/template_jobs.yml'
.script_core:
- echo "Running stage"
test_archA:
extends:
- .sample
- .script_core
stage: test
tags:
- architectureA
And after that move the .script_core to an import.