We're using Gitlab-CI but we have some troubles to have review and production environments at the same time.
We have several stages in our .gitlab-ci.yml but here I'll focus on the deploy stage:
deploy:
stage: deploy
script:
- some commands
environment:
name: review/$CI_BUILD_REF_NAME
url: http://$CI_BUILD_REF_SLUG.$DEPLOY_SERVER
on_stop: stop_deploy
only:
- /^feature-[cw]\/.*$/
deploy:
stage: deploy
script:
- some other commands
environment:
name: production
only:
- prod
stop_deploy:
stage: deploy
variables:
GIT_STRATEGY: none
script:
- some clean commands
when: manual
environment:
name: review/$CI_BUILD_REF_NAME
action: stop
only:
- /^feature-[cw]\/.*$/
The issue is that the first job is not run on the branches whose name starts with feature-c/. However when removing the second job, the first job is run on those branches.
The job that deploys to production is correctly run when pushed to prod.
So why does the first job is not run when the second job is defined? Where does the conflict comes from?
Thanks!
The answer is quite simple; they can't have the same name :) Name one deploy-review and the other deploy-prod and its fixed.
Related
I'm developing a pipeline on GitLab-ci, in the first job I use gittools/gitversion the obtain the semantic version of my software.
Here a small piece of code of /gitversion-ci-cd-plugin-extension.gitlab-ci.yml (Full documentation here https://gitversion.net/docs/reference/build-servers/gitlab)
.gitversion_function:
image:
name: gittools/gitversion
entrypoint: ['']
stage: .pre
.....
.....
artifacts:
reports:
#propagates variables into the pipeline level
dotenv: thisversion.env
Then a simplified version of my pipeline is as follows
stages:
- .pre
- install_dependencies
- build
- deploy
include:
- local: '/gitversion-ci-cd-plugin-extension.gitlab-ci.yml'
determineversion:
extends: .gitversion_function
install_dependencies:
image: node:16.14
stage: install_dependencies
script:
- echo ${PACKAGE_VERSION}
build:
image: node:16.14
stage: build
script:
- echo $PACKAGE_VERSION
deploy:
image: bitnami/kubectl
stage: deploy
needs: ['build']
script:
- echo $PACKAGE_VERSION
The problem is that the environment variable $PACKAGE_VERSION works in the first two jobs install_dependencies and build.
echo $PACKAGE_NAME; //0.0.1
But when the jobs deploy is executed the environment variable is not expanded by pipeline and I obtain literally this
echo $PACKAGE_NAME; //$PACKAGE_NAME
I found the problem.
In the last job of my pipeline, I use needs (https://docs.gitlab.com/ee/ci/yaml/#needs) to establish dependencies between jobs.
The problem is that artifact is not automatically passed because there is no a dependency between determineversion and deploy, to fix I do this:
...
deploy:
image: bitnami/kubectl
stage: deploy
needs: ['determineversion', 'build'] # <------
script:
- echo $PACKAGE_VERSION
...
I added determineversion as a dependency of deploy, in this way $PACKAGE_VERSION is printed correctly
I have like this gitlab ci cd configuration file:
image: docker:git
stages:
- develop
- production
default:
before_script:
- apk update && apk upgrade && apk add git curl
deploy:
stage: develop
script:
- echo "Hello World"
backup:
stage: develop
when:
- manual
- on_success
remove:
stage: develop
when:
- delayed
- on_success
start_in: 30 minutes
In my case job deploy runs automaticaly and job backup must runs manually only when successfully completed job deploy. But in my case this configuration doesn't works and I get error with message:
Found errors in your .gitlab-ci.yml:
jobs:backup when should be one of:
on_success
on_failure
always
manual
delayed
How I can use multiple when option arguments in my case?
Basically you can't because when does not expect an array. You can work around it though with needs. But this solution does only work if you run your jobs in different stages.
image: docker:git
stages:
- deploy
- backup
- remove
deploy:develop:
stage: deploy
script:
- exit 1
backup:develop:
stage: backup
script:
- echo "backup"
when: manual
needs: ["deploy:develop"]
remove:develop:
stage: remove
script:
- echo "remove"
when: delayed
needs: ["backup:develop"]
start_in: 30 minutes
I am trying to add needs between jobs in the Gitlab CI yaml configuration file.
stages:
- build
- test
- package
- deploy
maven-build:
stage: build
only:
- merge_requests
- master
- branches
...
test:
stage: test
needs: [ "maven-build" ]
only:
- merge_requests
- master
...
docker-build:
stage: package
needs: [ "test" ]
only:
- master
...
deploy-stage:
stage: deploy
needs: [ "docker-build" ]
only:
- master
...
deploy-prod:
stage: deploy
needs: [ "docker-build" ]
only:
- master
when: manual
...
I have used the GitLab CI online lint tools to check my syntax, it is correct.
But when I pushed the codes, it always complains:
'test' job needs 'maven-build' job
but it was not added to the pipeline
You can also test your .gitlab-ci.yml in CI Lint
The GitLab CI did not run at all.
Update: Finally I made it. I think the needs position is sensitive, move all needs under the stage, it works. My original scripts included some other configuration between them.
CI-jobs that depend on each other need to have the same limitations!
In your case that would mean to share the same only targets:
stages:
- build
- test
maven-build:
stage: build
only:
- merge_requests
- master
- branches
test:
stage: test
needs: [ "maven-build" ]
only:
- merge_requests
- master
- branches
that should work from my experience^^
Finally I made it. I think the needs position is sensitive, move all needs under the stage, it works
Actually... that might no longer be the case with GitLab 14.2 (August 2021):
Stageless pipelines
Using the needs keyword in your pipeline configuration helps to reduce cycle times by ignoring stage ordering and running jobs without waiting for others to complete.
Previously, needs could only be used between jobs on different stages.
In this release, we’ve removed this limitation so you can define a needs relationship between any job you want.
As a result, you can now create a complete CI/CD pipeline without using stages by including needs in every job to implicitly configure the execution order.
This lets you define a less verbose pipeline that takes less time to create and can run even faster.
See Documentation and Issue.
The rule in both jobs should be that same or otherwise GitLab cannot create job dependency between the jobs when the trigger rule is different.
I don't know why, but if the jobs are in different stages (as in my case), you have to define the jobs that will be done later with "." at the start.
Another interesting thing is GitLab's own CI/CD Lint online text editor does not complain there is an error. So you have to start the pipeline to see the error.
Below, notice the "." in ".success_notification" and ".failure_notification"
stages:
- prepare
- build_and_test
- deploy
- notification
#SOME CODE
build-StandaloneWindows64:
<<: *build
image: $IMAGE:$UNITY_VERSION-windows-mono-$IMAGE_VERSION
variables:
BUILD_TARGET: StandaloneWindows64
.success_notification:
needs:
- job: "build-StandaloneWindows64"
artifacts: true
stage: notification
script:
- wget https://raw.githubusercontent.com/DiscordHooks/gitlab-ci-discord-webhook/master/send.sh
- chmod +x send.sh
- ./send.sh success $WEBHOOK_URL
when: on_success
.failure_notification:
needs:
- job: "build-StandaloneWindows64"
artifacts: true
stage: notification
script:
- wget https://raw.githubusercontent.com/DiscordHooks/gitlab-ci-discord-webhook/master/send.sh
- chmod +x send.sh
- ./send.sh failure $WEBHOOK_URL
when: on_failure
#SOME CODE
I currently have two jobs in my CI file which are nearly identical.
The first is for manually compiling a release build from any git branch.
deploy_internal:
stage: deploy
script: ....<deploy code>
when: manual
The second is to be used by the scheduler to release a daily build from develop branch.
scheduled_deploy_internal:
stage: deploy
script: ....<deploy code from deploy_internal copy/pasted>
only:
variables:
- $MY_DEPLOY_INTERNAL != null
This feels wrong to have all that deploy code repeated in two places. It gets worse. There are also deploy_external, deploy_release, and scheduled variants.
My question:
Is there a way that I can combine deploy_internal and scheduled_deploy_internal such that the manual/scheduled behaviour is retained (DRY basically)?
Alternatively: Is there is a better way that I should structure my jobs?
Edit:
Original title: Deploy job. Execute manually except when scheduled
You can use YAML anchors and aliases to reuse the script.
deploy_internal:
stage: deploy
script:
- &deployment_scripts |
echo "Deployment Started"
bash command 1
bash command 2
when: manual
scheduled_deploy_internal:
stage: deploy
script:
- *deployment_scripts
only:
variables:
- $MY_DEPLOY_INTERNAL != null
Or you can use extends keyword.
.deployment_script:
script:
- echo "Deployment started"
- bash command 1
- bash command 2
deploy_internal:
extends: .deployment_script
stage: deploy
when: manual
scheduled_deploy_internal:
extends: .deployment_script
stage: deploy
only:
variables:
- $MY_DEPLOY_INTERNAL != null
Use GitLab's default section containing a before_script:
default:
before_script:
- ....<deploy code>
job1:
stage: deploy
script: ....<code after than deploy>
job2:
stage: deploy
script: ....<code after than deploy>
Note: the default section fails to function as such if you try to execute a job locally with the gitlab-runner exec command - use YAML anchors instead.
I need some direction here. I'm reading whatever documentation I can find online but it's not hitting the right synapses or I haven't found the right link yet. On a merge request to a deployable environment, I want to kick off a build on two separate machines. Both machines are IBM Is, running different versions of the OS. I'd like for these builds and subsequent deploys to happen independently of each other.
My .yml file has the entries for the build for the two machines (QQDEV & BNADEV), but the builds occur sequentially, not in parallel. The picture below is what Gitlab draws.
To me, from the above picture, it looks like both build_BNADEV and build_QQDEV are going to run the deploy jobs DEV_BNADEV and DEV_QQDEV. I want build_BNADEV to run DEV_BNADEV, et al, and that is a separate issue aside from the parallel builds.
What do I need here? Another runner? Another pipeline? Just looking for general pointers and direction here.
Here is my YAML.
stages:
- build
- deploy
build_QQDEV:
variables:
THING: "This is a THING for build for QQDEV"
script:
- "bash ./GitLabCI/GitLabCI.Build.sh qqdev"
stage: build
only:
- DEV
- QA
- UAT
- PROD
build_BNADEV:
variables:
THING: "This is a THING for build for BNADEV"
script:
- "bash ./GitLabCI/GitLabCI.Build.sh bnadev"
stage: build
only:
- DEV
- QA
DEV_QQDEV:
variables:
THING: "This is a THING for deploy_DEV_QQDEV"
ASPGRP: "*NONE"
script:
- "bash ./GitLabCI/GitLabCI.Deploy.sh QQDEV EPDEV1_5 /home/quikq/1.5/dev"
stage: deploy
environment:
name: DEV
only:
- DEV
DEV_BNADEV:
variables:
THING: "This is a THING for deploy_DEV_BNADEV"
REBUILD_DEPLOYMENT: "0"
ASPGRP: "DATADEV"
script:
- "bash ./GitLabCI/GitLabCI.Deploy.sh BNADEV EPDEV1_5 /home/quikq/1.5/dev"
stage: deploy
environment:
name: DEV
only:
- DEV
QA_QQDEV:
variables:
THING: "This is a THING for deploy_QA_QQDEV"
ASPGRP: "*NONE"
script:
- "bash ./GitLabCI/GitLabCI.Deploy.sh QQDEV EPQA1_5 /home/quikq/1.5/qa"
stage: deploy
environment:
name: QA
only:
- QA
QA_BNADEV:
variables:
THING: "This is a THING for deploy_QA_BNADEV"
REBUILD_DEPLOYMENT: "0"
ASPGRP: "DATADEV"
script:
- "bash ./GitLabCI/GitLabCI.Deploy.sh BNADEV EPQA1_5 /home/quikq/1.5/qa"
stage: deploy
environment:
name: QA
only:
- QA
UAT_QQ:
variables:
THING: "This is a THING for deploy_UAT_QQ"
ASPGRP: "*NONE"
script:
- "bash ./GitLabCI/GitLabCI.Deploy.sh QQ EPUAT1_5 /home/quikq/1.5/uat"
stage: deploy
environment:
name: UAT
only:
- UAT
UAT_QQBNA:
variables:
THING: "This is a THING for deploy_UAT_QQBNA"
ASPGRP: "*NONE"
script:
- "bash ./GitLabCI/GitLabCI.Deploy.sh QQBNA EPUAT1_5 /home/quikq/1.5/uat"
stage: deploy
environment:
name: UAT
only:
- UAT
PROD_QQ:
variables:
THING: "This is a THING for deploy_PROD_QQ"
ASPGRP: "*NONE"
script:
- "bash ./GitLabCI/GitLabCI.Deploy.sh QQ EPPROD1_5 /home/quikq/1.5/prod"
stage: deploy
environment:
name: PROD
only:
- PROD
PROD_QQBNA:
variables:
THING: "This is a THING for deploy_PROD_QQBNA"
ASPGRP: "*NONE"
script:
- "bash ./GitLabCI/GitLabCI.Deploy.sh QQBNA EPPROD1_5 /home/quikq/1.5/prod"
stage: deploy
environment:
name: PROD
only:
- PROD