I have one gitlab CI file like this
stages:
- build
- deploy
build-job:
stage: build
script:
- echo "Compiling the code..."
- echo "Compile complete."
when: manual
deploy-bridge:
stage: deploy
trigger:
project: tests/ci-downstream
What I understand is that the deploy-bridge stage should not be run unless the manual build-job is run successfully. But it is not the case here. Is this normal?
Jobs in the same stage run in parallel. Jobs in the next stage run
after the jobs from the previous stage complete successfully.
You're not defining your deploy-bridge job as a dependent job, or that it needs another job to finish first, so it can run right away as soon as it reaches the stage. Since the previous stage is all manual jobs, GitLab CI/CD sort of interprets it as 'done', at least enough so that other stages can start.
Since it doesn't look like you're uploading the compiled code from build-job as an artifact, we can't use the dependencies keyword here. All that keyword does is control which jobs' dependencies this job needs, but if it needs the artifacts of a prior job, that job will need to run and finish successfully for this job to start. Also, by default all available artifacts from all prior jobs will be downloaded and available for all jobs in the pipeline. The dependencies keyword can also be used to limit which artifacts this job actually needs. However, if there are no artifacts available in the job we "depend" on, it will throw an error. Luckily there's another keyword we can use.
The needs keyword controls the "flow" of the pipeline, so much so that if a job anywhere in the pipeline (even in the last of say 1,000 stages) had needs: [] it will run as soon as the pipeline starts (and as soon there is an available runner). We can use needs here to make the pipeline flow the way you need.
...
deploy-bridge:
stage: deploy
needs: ['build-job']
trigger:
project: tests/ci-downstream
Now, the deploy-bridge job won't run until the build-job has finished successfully. If build-job fails, deploy-bridge will be skipped.
One other use for needs is that it has the same functionality as dependencies, in that it can control what artifacts are downloaded in which jobs, but it won't fail if the "needed" jobs don't have artifacts at all.
Both dependencies and needs accept an empty array which equates to 'don't download any artifacts' and for needs, run as soon as a runner is available.
Related
I have 1 question regarding Gitlab pipeline triggering. We have multiple gitlab projects which trigger 1 common project. They are doing It separately. The idea is to trigger this project only when subprojects are finished. Is there are any way to do It better than create script which checks pipeline status via API? Because didn't find any out-of-the box solution for this
You can use the trigger:strategy. As per the docs:
Use trigger:strategy to force the trigger job to wait for the downstream pipeline to complete before it is marked as success.
So say you have build and test stages, and you want the trigger job in the build stage to succeed before moving on to the test stage, you could do something like this: =
downstream-build:
stage: build
trigger:
include: path/to/child-pipeline.yml
strategy: depend
In my Gitlab project, I'm including multiple .yml files. One of them is remote and the other is a template provided by Gitlab for Code Quality.
The .yml configuration is written like so:
include:
- template: Code-Quality.gitlab-ci.yml
- remote: 'https://raw.githubusercontent.com/checkmarx-ltd/cx-flow/develop/templates/gitlab/v3/Checkmarx.gitlab-ci.yml'
Both of these templates are accessible. The first is located here, and the second Checkmarx one is here.
Both of these .yml configs define jobs that run in the test pipeline stage.
I'm having an issue where only the second include's jobs are running in the test stage, and the Gitlab Code Quality job is completely ignored. If I remove the external Checkmarx include, the Code Quality job runs just fine.
Normally I would just define separate stages, but since these .yml files do not belong to me, I cannot change the stage in which they run.
Is there a way to ensure the jobs all run in the test stage? If not, is there a way to override the stage a job from an external .yml runs in?
Oddly, there seems to be some sort of rules conflict between the two templates, possibly due to the variables that the checkmarx template sets. Even though the CI Lint shows that all 4 jobs should run successfully, I can reproduce your issue with the above code.
Given that it's likely a rules issue, I overrode the rules for running the code_quality job and was able to get both running within the same pipeline:
include:
- template: Code-Quality.gitlab-ci.yml
- remote: 'https://raw.githubusercontent.com/checkmarx-ltd/cx-flow/develop/templates/gitlab/v3/Checkmarx.gitlab-ci.yml'
code_quality:
rules:
- when: on_success
You can lint the above changes to confirm they're successful (though GitLab will warn you that without any workflow:rules, you'll wind up with duplicate pipelines inside MRs, which is true).
You can also see the pipeline running with both jobs here though checkmarx fails because I don't have a subscription to test it with:
I want to add some automation tests (it can be selenium c#/ java tests) to my existing azure pipeline. My pipeline is working correctly i.e. when I push changes to my master branch (GitHub), it triggers the build and it gets deployed to the live. But I want to integrate some automation test which will decide whether to deploy the build or not example:
After committing to the master branch:
run the automation test
if tests passed > deploy
if the test fails > stop deployment
Any suggestions or reference material would be helpful.
Thanks,
Depends on what kind of tests you are going to run - if they are unit tests, it is easy but selenium tests you would need to deploy it to a webserver before running the tests.
When edit the pipeline, and click "Show assistance" if is't not already done, you can see the tasks, including test dashboard etc.
For example if you add a VStests, it will add something like this to the yaml file.
- task: DotNetCoreCLI#2
inputs:
command: test
projects: '**/*Tests/*.csproj'
arguments: '-- no-build --configuration $(buildConfiguration)'
Above just executes 'dotnet test' command which will look for tests in the project and run them.
It is executing a command line, so you can run any tests as far as you can provide a command line using batch/powershell commands.
The Microsoft tutorial at https://learn.microsoft.com/en-us/learn/modules/run-quality-tests-build-pipeline/ explains how to use the test wigits etc.
I would suggest to start with unit tests which is simpler before trying selenium.
Krish
You can specify the conditions under which each stage, job, or step runs. By default, a job or stage runs if it does not depend on any other job or stage, or if all of the jobs or stages that it depends on have completed and succeeded. By default, a step runs if nothing in its job has failed yet and the step immediately preceding it has finished.
In another word, if test step/job/stage passed, the next step/job/stage can precede.
I have a pipeline that contains several jobs triggered by changes on a branch:
deployDev:
only:
- dev
script:
- ...
deployProd:
only:
- master
script:
- ...
If I now hit the "Run pipeline" button in the GitLab UI, will it trigger these jobs or will it only trigger the jobs that specify when:manual ?
And how can I make sure that deployDev and deployProd do not run when I run a manual deploy?
I have checked here: https://docs.gitlab.com/ee/ci/yaml/#onlyexcept-basic, but I'm unsure.
It'll run all jobs corresponding to the branch and will wait for manual action for manual jobs.
So your deploydev will run against your development pipeline and other job will run only in production pipeline
I'm a bit confused what you are trying to achieve from your comment as well, however...
If I now hit the "Run pipeline" button in the GitLab UI, will it trigger these jobs or will it only trigger the jobs that specify when:manual ?
As #Prashanna has said, It'll run all jobs corresponding to the branch and will wait for manual action for manual jobs.
If you do not want deployDev and deployProd to appear in the Pipeline when pressing the Run Pipeline button, you can use:
only:
- master
except:
- web
from the link you showed https://docs.gitlab.com/ee/ci/yaml/#onlyexcept-basic: web -
For pipelines created by using Run pipeline button in the GitLab UI, from the project’s CI/CD > Pipelines section.
The above job would then only appear in the Pipeline, when it is the master branch, and when it has not been triggered by the Run Pipeline button.
enter image description here
Hello - I'm using Gitlab 10.x Enterprise version.
This is in the context of a Production deployment to an environment having eight servers. The requirement is to deploy to one server first, and then after couple of days do a deployment to the remaining servers with the press of just one button.
Pipeline Stages:
Release-Tag-Creation -> Production-OneServer-Deployment -> OneButtonPush -> DeploytoAllServers
Question:
How can I tie the dependency between the stages "OneButtonPush" and "DeploytoAllServers"?. "DeploytoAllServers" stage should be kicked-off only when the Status of the job in the "OneButtonPush" stage is successful. "DeploytoAllServers" stage will have parallel jobs to deploy to each server.
I made few attempts based on the Gitlab CI documentation, but wasn't successful. Also, can the concept of Rolling deployment used in the context of GitLab-CI.
Thanks!
In Gitlab CI, the stages are run one after another. Each stage can have multiple jobs which run in parallel.
As per your use case, you'll need to have different stages each for Release-Tag-Creation, Production-OneServer-Deployment, OneButtonPush and DeploytoAllServers. You can have manual triggers for particular jobs (OneButtonPush in your case) by specifying when: manual in the job definition.
By default, if there is a job awaiting manual trigger, jobs from further stages will start executing considering the job with manual trigger as successful. To change this behavior one will need to use allow_failure: false. Mode details on allow_failure over here
If you want to run the Stage only after then specific one get successful you can use the needs
Example
image: maven:latest
stages:
- deploy_dev_lambda
- test_dev_lambda
deploy_dev_lambda:
stage: deploy_dev_lambda
image: maven:latest
script:
- ./deployLambda
environment: dev
variables:
TARGET_ENV: "dev"
TAG_NAME: main-$CI_COMMIT_SHORT_SHA
test_dev_lambda:
stage: test_dev_lambda
needs:
- deploy_dev_lambda
script:
- ./testLambda
So test_dev_lambda only run after the deploy_dev_lambda stage and if stage deploy_dev_lambda fails test_dev_lambda wont get executed.