I use Gitlab on gitlab.com and the issue tracker.
Each time I have an issue, I create a new branch on the button inside the issue, but this triggers a new build (pipeline) in CI.
I don't want this because this branch is coming from master and doesn't need to be built.
How can I achieve that ? Is this a gitlab-ci.yml modification or a repository related configuration ?
You can define, in which branches particular steps of your build will run, via only and except parameters to the builds: https://docs.gitlab.com/ee/ci/yaml/#only-and-except-complex
For example, run java build in all branches except issue branches:
java-build:
stage: build
except:
- /^issue-.*$/
script:
- mvn -U -e install
image: maven:3.5-jdk-8
Restrict build only to master and release branches:
java-build:
stage: build
only:
- master
- /^RELEASE-.*$/
script:
- mvn -U -e install
image: maven:3.5-jdk-8
Based on the Commit message we can avoid building for the newly created branch on GitLab CI/CD pipeline
Build Branch(master):
stage: build
only:
refs:
- master
variables:
- $CI_COMMIT_MESSAGE =~ /^\[master\].*$/
script:
- echo "master branch"
# - sleep 60 | echo "hello master"
# when: delayed
# start_in: 3 minutes
interruptible: true
Build Branch(release):
stage: build
only:
refs:
- /^build_.*$/
variables:
- $CI_COMMIT_MESSAGE =~ /^\[build\].*$/
script:
- echo "release branch"
# - sleep 60 | echo "hello release"
# when: delayed
# start_in: 3 minutes
interruptible: true
The branches will trigger when the commit message has start with [master] or [build]
Related
I have like this gitlab ci cd configuration file:
image: docker:git
stages:
- develop
- production
default:
before_script:
- apk update && apk upgrade && apk add git curl
deploy:
stage: develop
script:
- echo "Hello World"
backup:
stage: develop
when:
- manual
- on_success
remove:
stage: develop
when:
- delayed
- on_success
start_in: 30 minutes
In my case job deploy runs automaticaly and job backup must runs manually only when successfully completed job deploy. But in my case this configuration doesn't works and I get error with message:
Found errors in your .gitlab-ci.yml:
jobs:backup when should be one of:
on_success
on_failure
always
manual
delayed
How I can use multiple when option arguments in my case?
Basically you can't because when does not expect an array. You can work around it though with needs. But this solution does only work if you run your jobs in different stages.
image: docker:git
stages:
- deploy
- backup
- remove
deploy:develop:
stage: deploy
script:
- exit 1
backup:develop:
stage: backup
script:
- echo "backup"
when: manual
needs: ["deploy:develop"]
remove:develop:
stage: remove
script:
- echo "remove"
when: delayed
needs: ["backup:develop"]
start_in: 30 minutes
I have a gitlab ci pipeline in my application repo, A, which calls an end to end testing Repo T to run its tests. The repo A pipeline succesfully triggers the tests from Repo T but if the test job fails in T, the job calling the test job in T from A still passes. How do I get repo A to track the result of Repo T's test job, and pass/fail its pipeline based off of the test jobs in T?
.gitlab-ci.yml for testing Repo T:
stages:
- test
test:
stage: test
image: markhobson/maven-chrome:jdk-11
artifacts:
paths:
- target/surefire-reports
script:
- mvn test
only:
- triggers
.gitlab-ci.yml from application repo A:
job1:
stage: unit-tests ...
job2:
stage: build ...
...
trigger-e2e-repo:
stage: e2e-testing
image: markhobson/maven-chrome
script:
- "curl -X POST -F token=repo-T-token -F ref=repo-T-branch https://repo-A/api/v4/projects/repo-T-id/trigger/pipeline"
only:
- repo-A-branch
Since GitLab 11.8 you can trigger a pipeline via bridge job.
In GitLab 11.8, GitLab provides a new CI/CD configuration syntax to make this task easier, and avoid needing GitLab Runner for triggering cross-project pipelines.
With bridge jobs it is possible to mirror the status of the trigger pipeline to the calling pipeline.
You can mirror the pipeline status from the triggered pipeline to the source bridge job by using strategy: depend.
Example in your case:
trigger-e2e-repo:
stage: e2e-testing
trigger:
project: repo-T
strategy: depend
If the triggered pipeline with the test jobs fails, the calling pipeline also fails.
If you only want to execute a particular job in your repository "Repo T" when executed by a bridge job, then you should use only: pipeline (only) or rules: -if '$CI_PIPELINE_SOURCE == "pipeline"' (rules:if) instead of only: triggers.
I wasn't able to use the bridge job property of mirroring a downstream job result as the version of my gitlab is before 11.8. I did manage to get it to work by creating a trigger for repo A, and making a call from repo T to repo A with this new second trigger. The remaining jobs in repo A will only be activated by triggers and setting of variables (JOB in this case ) as laid out below:
.gitlab-ci.yml for repo T:
test:
stage: test
script:
- mvn test
- "curl -X POST -F token=repo-A-token -F ref=$BRANCH -F variables[JOB]=build https://project.com/api/v4/projects/project_id/trigger/pipeline"
.gitlab-ci.yml in A
job1:
...
except:
- triggers
job2:
...
except:
- triggers
...
trigger-e2e-repo:
stage: e2e-testing
script:
- "curl -X POST -F token=repo-B-token -F ref=repo-B-branch -F variables[BRANCH]=$CI_COMMIT_REF_NAME -F https://project-B/api/v4/projects/project-B-id/trigger/pipeline"
except:
- triggers
build_application_for_prod:
stage: build_prod
script:
- "curl -X POST -F token=repo-A-token -F ref=$CI_COMMIT_REF_NAME -F variables[JOB]=deploy -F variables[SEND]=true https://foo/api/v4/projects/proj_A_id/trigger/pipeline"
only:
variables:
- $JOB == "build"
deploy_production_environment:
stage: deploy_prod
script:
- ...
only:
variables:
- $JOB == "deploy"
Note I also had to add the except statements for the jobs before the end to end tests in repo A so that they won't rerun and loop when repo A's API trigger is called later on.
I am wanting to use the gl-sast-report.json file created during the SAST process in a subsequent stage of my CI but it is not found.
ci.yml
include:
- template: Security/SAST.gitlab-ci.yml
stages:
- test
- .post
sast:
rules:
- if: $CI_COMMIT_TAG
send-reports:
stage: .post
dependencies:
- sast
script:
- ls
- echo "in post stage"
- cat gl-sast-report.json
output:
Running with gitlab-runner 13.2.1 (efa30e33)
on blah blah blah
Preparing the "docker" executor
00:01
.
.
.
Preparing environment
00:01
Running on runner-zqk9bcef-project-4296-concurrent-0 via ff93ba7b6ee2...
Getting source from Git repository
00:01
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in blah blah
Checking out 9c2edf67 as 39-test-dso...
Removing gl-sast-report.json
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:03
$ ls
<stuff in the repo>
$ echo "in .post stage"
in post stage
$ cat gl-sast-report.json
cat: can't open 'gl-sast-report.json': No such file or directory
ERROR: Job failed: exit code 1
You can see the line Removing gl-sast-report.json which I assume is the issue.
I don't see that anywhere in the SAST.gitlab-ci.yml at https://gitlab.com/gitlab-org/gitlab/-/blob/v11.11.0-rc2-ee/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml#L33-45
Any ideas on how to use this artifact in the next stage of my CI pipeline?
UPDATE:
So I tried out k33g_org's suggestion below but to no avail. Seems that this is due to limitations in the sast template specifically. Did the following test.
include:
- template: Security/SAST.gitlab-ci.yml
stages:
- test
- upload
something:
stage: test
script:
- echo "in something"
- echo "this is something" > something.txt
artifacts:
paths: [something.txt]
sast:
before_script:
- echo "hello from before sast"
- echo "this is in the file" > test.txt
artifacts:
reports:
sast: gl-sast-report.json
paths: [gl-sast-report.json, test.txt]
send-reports:
stage: upload
dependencies:
- sast
- something
before_script:
- echo "This is the send-reports before_script"
script:
- echo "in send-reports job"
- ls
artifacts:
reports:
sast: gl-sast-report.json
Three changes:
Updated code with k33g_org's suggestion
Created another artifact in the sast job (to see if it would pass through to send-reports job)
Created a new job (something) where I created a new something.txt artifact (to see if it would pass through to send-reports job)
Output:
Preparing environment
00:01
Running on runner-zqx7qoq-project-4296-concurrent-0 via e3fe672984b4...
Getting source from Git repository
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /<repo>
Checking out 26501c44 as <branch_name>...
Removing something.txt
Skipping Git submodules setup
Downloading artifacts
00:00
Downloading artifacts for something (64950)...
Downloading artifacts from coordinator... ok id=64950
responseStatus=200 OK token=zoJwysdq
Executing "step_script" stage of the job script
00:01
$ echo "This is the send-reports before_script"
This is the send-reports before_script
$ echo "in send-reports job"
in send-reports job
$ ls
...<other stuff in repo>
something.txt
Uploading artifacts for successful job
00:01
Uploading artifacts...
WARNING: gl-sast-report.json: no matching files
ERROR: No files to upload
Cleaning up file based variables
00:01
Job succeeded
Notes:
something.txt made it to this job
all artifacts from the sast job to not make it to subsequent jobs
I can only conclude that there is something internal to the sast template that is not allowing artifacts to propagate to subsequent jobs.
in the first job (sast) add this:
artifacts:
paths: [gl-sast-report.json]
reports:
sast: gl-sast-report.json
and in the next job (send-reports) add this
artifacts:
reports:
sast: gl-sast-report.json
Then you should be able to access the report in the next job (send-reports)
Instead of referencing the gl-sast-report.json artifact as sast report, reference it as a regular artifact.
So what you should do is declare the artifact this way
artifacts:
paths:
- 'gl-sast-report.json'
instead of
reports:
sast: gl-sast-report.json
I spent a full day banging my head against this, trying to access the gl-sast-report.json file generated by the built-in IaC scanner. Here's what ultimately worked for me:
First and foremost, DO NOT use this code suggested by GitLab's documentation:
include:
- template: Security/SAST-IaC.latest.gitlab-ci.yml
The above code works fine if all you want to do is scan for IaC vulnerabilities and download the report from the GitLab UI later. But who wants to do that?! I want to access the report in my next job and fail the pipeline if there are medium+ vulnerabilities in the report!
If that's what you want to do, you'll need to add all of the code from the official GitLab IaC scanner template to your pipeline, and then make some modifications. You can find the latest template code here, or use my example below.
Modified template:
# Read more about this feature here: https://docs.gitlab.com/ee/user/application_security/iac_scanning/
#
# Configure SAST with CI/CD variables (https://docs.gitlab.com/ee/ci/variables/index.html).
# List of available variables: https://docs.gitlab.com/ee/user/application_security/iac_scanning/index.html
variables:
# Setting this variable will affect all Security templates
# (SAST, Dependency Scanning, ...)
TEMPLATE_REGISTRY_HOST: 'registry.gitlab.com'
SECURE_ANALYZERS_PREFIX: "$TEMPLATE_REGISTRY_HOST/security-products"
SAST_IMAGE_SUFFIX: ""
SAST_EXCLUDED_PATHS: "spec, test, tests, tmp"
iac-sast:
stage: test
artifacts:
name: sast
paths:
- gl-sast-report.json
#reports:
# sast: gl-sast-report.json
when: always
rules:
- when: never
# `rules` must be overridden explicitly by each child job
# see https://gitlab.com/gitlab-org/gitlab/-/issues/218444
variables:
SEARCH_MAX_DEPTH: 4
allow_failure: true
script:
- /analyzer run
kics-iac-sast:
extends: iac-sast
image:
name: "$SAST_ANALYZER_IMAGE"
variables:
SAST_ANALYZER_IMAGE_TAG: 3
SAST_ANALYZER_IMAGE: "$SECURE_ANALYZERS_PREFIX/kics:$SAST_ANALYZER_IMAGE_TAG$SAST_IMAGE_SUFFIX"
rules:
- if: $SAST_DISABLED
when: never
- if: $SAST_EXCLUDED_ANALYZERS =~ /kics/
when: never
- if: $CI_COMMIT_BRANCH
Enforce Compliance:
stage: Compliance
before_script:
- apk add jq
script:
- jq -r '.vulnerabilities[] | select(.severity == "Critical") | (.severity, .message, .location, .identifiers[].url)' gl-sast-report.json > results.txt
- jq -r '.vulnerabilities[] | select(.severity == "High") | (.severity, .message, .location, .identifiers[].url)' gl-sast-report.json >> results.txt
- jq -r '.vulnerabilities[] | select(.severity == "Medium") | (.severity, .message, .location, .identifiers[].url)' gl-sast-report.json >> results.txt
- chmod u+x check-sast-results.sh
- ./check-sast-results.sh
You'll also need to make sure to add two stages to your pipeline (if you don't have them already):
stages:
# add these to whatever other stages you already have
- test
- Compliance
Note: it's extremely important that your job which is trying to access gl-sast-report.json ("Compliance" in this case) is not in the same stage as the sast scans themselves ("test" in this case). If they are, then your job will try to access the report before it exists and fail.
I'll include my shell script referenced in the pipeline in case you want to use that too:
#!/bin/sh
if [ -s results.txt ]; then
echo ""
echo ""
cat results.txt
echo ""
echo "ERROR: SAST SCAN FOUND VULNERABILITIES - FIX ALL VULNERABILITIES TO CONTINUE"
echo ""
exit 1
fi
This is a basic script that checks to see if the "results.txt" file has any contents. If it does, it exits with code 1 to break the pipeline and print the vulnerabilities. If there are no contents in the file, the script exits with code 0 and the pipeline continues (allowing you to deploy your infra). Save the file above as "check-sast-results.sh" in the root directory of your GitLab repository (the same level where ".gitlab-ci.yml" resides).
Hope this helps someone out there.
I've found this issue too, also impacts some of the other scanners; I raised an issue with GitLab to fix:
https://gitlab.com/gitlab-org/gitlab/-/issues/345696
I currently have two jobs in my CI file which are nearly identical.
The first is for manually compiling a release build from any git branch.
deploy_internal:
stage: deploy
script: ....<deploy code>
when: manual
The second is to be used by the scheduler to release a daily build from develop branch.
scheduled_deploy_internal:
stage: deploy
script: ....<deploy code from deploy_internal copy/pasted>
only:
variables:
- $MY_DEPLOY_INTERNAL != null
This feels wrong to have all that deploy code repeated in two places. It gets worse. There are also deploy_external, deploy_release, and scheduled variants.
My question:
Is there a way that I can combine deploy_internal and scheduled_deploy_internal such that the manual/scheduled behaviour is retained (DRY basically)?
Alternatively: Is there is a better way that I should structure my jobs?
Edit:
Original title: Deploy job. Execute manually except when scheduled
You can use YAML anchors and aliases to reuse the script.
deploy_internal:
stage: deploy
script:
- &deployment_scripts |
echo "Deployment Started"
bash command 1
bash command 2
when: manual
scheduled_deploy_internal:
stage: deploy
script:
- *deployment_scripts
only:
variables:
- $MY_DEPLOY_INTERNAL != null
Or you can use extends keyword.
.deployment_script:
script:
- echo "Deployment started"
- bash command 1
- bash command 2
deploy_internal:
extends: .deployment_script
stage: deploy
when: manual
scheduled_deploy_internal:
extends: .deployment_script
stage: deploy
only:
variables:
- $MY_DEPLOY_INTERNAL != null
Use GitLab's default section containing a before_script:
default:
before_script:
- ....<deploy code>
job1:
stage: deploy
script: ....<code after than deploy>
job2:
stage: deploy
script: ....<code after than deploy>
Note: the default section fails to function as such if you try to execute a job locally with the gitlab-runner exec command - use YAML anchors instead.
In my repo only source files are checked in — the code is tested and the dist files are generated in a pipeline. I then want to be able to tag a specific version and attach the artifacts generated by this pipeline to it. Ideally this should all happen with as little manual intervention as possible.
What is the best way to reference pipeline artifacts from a release?
You can use the release-cli in a stage after the job you built your app, to upload to a release a file form a previous job you'll need that build job id that you can store in a file in the artifacts:
build:
stage: build
script:
- echo "Build your app"
- echo "${CI_JOB_ID}" > CI_JOB_ID.txt # This way you know the job id in the next stage
artifacts:
paths:
- your_app.exe
- CI_JOB_ID.txt
expire_in: never
rules:
- if: $CI_COMMIT_TAG
release:
stage: release
image: registry.gitlab.com/gitlab-org/release-cli:latest
script:
- |
release-cli create --name "Release $CI_COMMIT_TAG" --tag-name $CI_COMMIT_TAG \
--assets-link "{\"name\":\"Executable file\",\"url\":\"https://gitlab.com/some/repo/-/jobs/`cat CI_JOB_ID.txt`/artifacts/file/your_app.exe\"}"
rules:
- if: $CI_COMMIT_TAG
This way every time you tag your repo, it will create a release if the pipeline succeed.