In my repo only source files are checked in — the code is tested and the dist files are generated in a pipeline. I then want to be able to tag a specific version and attach the artifacts generated by this pipeline to it. Ideally this should all happen with as little manual intervention as possible.
What is the best way to reference pipeline artifacts from a release?
You can use the release-cli in a stage after the job you built your app, to upload to a release a file form a previous job you'll need that build job id that you can store in a file in the artifacts:
build:
stage: build
script:
- echo "Build your app"
- echo "${CI_JOB_ID}" > CI_JOB_ID.txt # This way you know the job id in the next stage
artifacts:
paths:
- your_app.exe
- CI_JOB_ID.txt
expire_in: never
rules:
- if: $CI_COMMIT_TAG
release:
stage: release
image: registry.gitlab.com/gitlab-org/release-cli:latest
script:
- |
release-cli create --name "Release $CI_COMMIT_TAG" --tag-name $CI_COMMIT_TAG \
--assets-link "{\"name\":\"Executable file\",\"url\":\"https://gitlab.com/some/repo/-/jobs/`cat CI_JOB_ID.txt`/artifacts/file/your_app.exe\"}"
rules:
- if: $CI_COMMIT_TAG
This way every time you tag your repo, it will create a release if the pipeline succeed.
Related
consider the following gitlab-ci.yaml, for a mono repo with multiple microfrontends
stages:
- build
- deploy
build:app1:
stage: build
script:
- sleep 30
- mkdir dist1
- touch dist1/output1.html
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
changes:
- app1/src/*
artifacts:
paths:
- dist1
build:app2:
stage: build
script:
- sleep 30
- mkdir dist2
- touch dist2/output2.html
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
changes:
- app2/src/*
artifacts:
paths:
- dist2
deploy:all:
stage: deploy
script:
- mkdir dist
- cp dist1/* dist
- cp dist2/* dist
- deploy.sh ./dist
artifacts:
paths:
- dist
when ran the order defined in stages is ignored and both the build and deploy jobs run simultaneously
causing a failure for the "deploy:all" job (since its still "building")
if i remove the condition for the changes the stage order is respected and build runs before deploy
how can i both only act on changes and enforce the defined build order?
in the real monorepo there are 10's of micro frontends not just 2 ...
When I run your gitlab_ci.yml in Gitlab CI, it does not run the build and deploy jobs simultaneously. It runs a merge request pipeline (with the build jobs) and a branch pipeline (with the deploy job). Since the artifacts from the build jobs are created in the merge request pipeline, they aren't available to the branch pipeline, and so the deploy job fails.
It's difficult to say how to fix this without knowing what your intention is, but you need to run the build and deploy jobs in the same pipeline, so either have the deploy job run with
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
or have the build jobs run in a branch pipeline.
Additionally it's not clear to me how you're expecting the deploy job to always work, as if you only run the build jobs when there are changes sometimes not every build job will have an artifact and your deploy job will fail.
In your case, you can try to use needs keyword in order to force relationships between jobs: https://docs.gitlab.com/ee/ci/yaml/#needs
You also nee to add optional: true since you have rules conditions in your build jobs.
You can add it in your deploy:all job like this:
deploy:all:
needs:
- job: build:app1
optional: true
- job: build:app2
optional: true
stage: deploy
script:
- mkdir dist
- cp dist1/* dist
- cp dist2/* dist
- deploy.sh ./dist
artifacts:
paths:
- dist
Your deploy stage might need the condition for changes as well. Unless maybe you need the deploy:all stage to always run no matter if there is changes or not.
Also indicating that the deploy stage needs artifacts is always a good idea in your case because your deploy stage is using the artifacts anyways
needs:
- job: build:app1
artifacts: true
- job: build:app2
artifacts: true
I got an error on GitLab Pipeline when I commit the next .gitlab-ci.yml for a repository.
Pipeline executed to build solution, deploy to Artifactory and trigger and API call
Deploy job have to be executed manually, and there are two different job options to execute.
stages:
- build
- deploy
- trigger
variables:
APP_PROJECT_ID: ${CUSTOMER_RELEASED}
build_job:
stage: build
tags:
- dotnet
script:
- echo "build"
only:
- tags
allow_failure: false
.deploy_job_base:
stage: deploy
needs: [build_job]
tags:
- dotnet
script:
- echo "deploy"
dependencies:
- build_job
only:
- tags
deploy_job_sport:
extends: .deploy_job_base
after_script:
- $APP_PROJECT_ID = "2096"
when: manual
allow_failure: false
deploy_job_others:
extends: .deploy_job_base
after_script:
- $APP_PROJECT_ID = "0"
when: manual
allow_failure: false
.trigger_base:
stage: trigger
script:
- echo "Customer Project ID '{$APP_PROJECT_ID}'"
- echo "Call API..."
trigger_sport:
extends: .trigger_base
needs: [deploy_job_sport]
trigger_others:
extends: .trigger_base
needs: [deploy_job_others]
Lint status is correct
but I get that error GitLab Pipeline when I commit changes:
Found errors in your .gitlab-ci.yml: 'trigger_sport' job needs
'deploy_job_sport' job but 'deploy_job_sport' is not in any previous
stage 'trigger_others' job needs 'deploy_job_others' job but
'deploy_job_others' is not in any previous stage
If I remove trigger_sport and trigger_others job and create only one job, it works fine but I don't know how I can target the two manual jobs (deploy_job_sport and deploy_job_others) to a single job.
Do you have any idea?
Thanks in advance.
I think this is related to the fact that you're using only: tags in your template for deploy jobs and the build job also is limited to run when commits contain a tag.
But the trigger template is missing this limitation which most likely causes this error when pushing a commit without a tag because the pipeline creation would add trigger_XY to the pipeline, which has dependencies to the previous deploy_XY jobs.
When updating your job template for trigger jobs to the following this error should be resolved:
.trigger_base:
stage: trigger
script:
- echo "Customer Project ID '{$APP_PROJECT_ID}'"
- echo "Call API..."
only:
- tags
I am trying to add needs between jobs in the Gitlab CI yaml configuration file.
stages:
- build
- test
- package
- deploy
maven-build:
stage: build
only:
- merge_requests
- master
- branches
...
test:
stage: test
needs: [ "maven-build" ]
only:
- merge_requests
- master
...
docker-build:
stage: package
needs: [ "test" ]
only:
- master
...
deploy-stage:
stage: deploy
needs: [ "docker-build" ]
only:
- master
...
deploy-prod:
stage: deploy
needs: [ "docker-build" ]
only:
- master
when: manual
...
I have used the GitLab CI online lint tools to check my syntax, it is correct.
But when I pushed the codes, it always complains:
'test' job needs 'maven-build' job
but it was not added to the pipeline
You can also test your .gitlab-ci.yml in CI Lint
The GitLab CI did not run at all.
Update: Finally I made it. I think the needs position is sensitive, move all needs under the stage, it works. My original scripts included some other configuration between them.
CI-jobs that depend on each other need to have the same limitations!
In your case that would mean to share the same only targets:
stages:
- build
- test
maven-build:
stage: build
only:
- merge_requests
- master
- branches
test:
stage: test
needs: [ "maven-build" ]
only:
- merge_requests
- master
- branches
that should work from my experience^^
Finally I made it. I think the needs position is sensitive, move all needs under the stage, it works
Actually... that might no longer be the case with GitLab 14.2 (August 2021):
Stageless pipelines
Using the needs keyword in your pipeline configuration helps to reduce cycle times by ignoring stage ordering and running jobs without waiting for others to complete.
Previously, needs could only be used between jobs on different stages.
In this release, we’ve removed this limitation so you can define a needs relationship between any job you want.
As a result, you can now create a complete CI/CD pipeline without using stages by including needs in every job to implicitly configure the execution order.
This lets you define a less verbose pipeline that takes less time to create and can run even faster.
See Documentation and Issue.
The rule in both jobs should be that same or otherwise GitLab cannot create job dependency between the jobs when the trigger rule is different.
I don't know why, but if the jobs are in different stages (as in my case), you have to define the jobs that will be done later with "." at the start.
Another interesting thing is GitLab's own CI/CD Lint online text editor does not complain there is an error. So you have to start the pipeline to see the error.
Below, notice the "." in ".success_notification" and ".failure_notification"
stages:
- prepare
- build_and_test
- deploy
- notification
#SOME CODE
build-StandaloneWindows64:
<<: *build
image: $IMAGE:$UNITY_VERSION-windows-mono-$IMAGE_VERSION
variables:
BUILD_TARGET: StandaloneWindows64
.success_notification:
needs:
- job: "build-StandaloneWindows64"
artifacts: true
stage: notification
script:
- wget https://raw.githubusercontent.com/DiscordHooks/gitlab-ci-discord-webhook/master/send.sh
- chmod +x send.sh
- ./send.sh success $WEBHOOK_URL
when: on_success
.failure_notification:
needs:
- job: "build-StandaloneWindows64"
artifacts: true
stage: notification
script:
- wget https://raw.githubusercontent.com/DiscordHooks/gitlab-ci-discord-webhook/master/send.sh
- chmod +x send.sh
- ./send.sh failure $WEBHOOK_URL
when: on_failure
#SOME CODE
I currently have two jobs in my CI file which are nearly identical.
The first is for manually compiling a release build from any git branch.
deploy_internal:
stage: deploy
script: ....<deploy code>
when: manual
The second is to be used by the scheduler to release a daily build from develop branch.
scheduled_deploy_internal:
stage: deploy
script: ....<deploy code from deploy_internal copy/pasted>
only:
variables:
- $MY_DEPLOY_INTERNAL != null
This feels wrong to have all that deploy code repeated in two places. It gets worse. There are also deploy_external, deploy_release, and scheduled variants.
My question:
Is there a way that I can combine deploy_internal and scheduled_deploy_internal such that the manual/scheduled behaviour is retained (DRY basically)?
Alternatively: Is there is a better way that I should structure my jobs?
Edit:
Original title: Deploy job. Execute manually except when scheduled
You can use YAML anchors and aliases to reuse the script.
deploy_internal:
stage: deploy
script:
- &deployment_scripts |
echo "Deployment Started"
bash command 1
bash command 2
when: manual
scheduled_deploy_internal:
stage: deploy
script:
- *deployment_scripts
only:
variables:
- $MY_DEPLOY_INTERNAL != null
Or you can use extends keyword.
.deployment_script:
script:
- echo "Deployment started"
- bash command 1
- bash command 2
deploy_internal:
extends: .deployment_script
stage: deploy
when: manual
scheduled_deploy_internal:
extends: .deployment_script
stage: deploy
only:
variables:
- $MY_DEPLOY_INTERNAL != null
Use GitLab's default section containing a before_script:
default:
before_script:
- ....<deploy code>
job1:
stage: deploy
script: ....<code after than deploy>
job2:
stage: deploy
script: ....<code after than deploy>
Note: the default section fails to function as such if you try to execute a job locally with the gitlab-runner exec command - use YAML anchors instead.
My CI has two main steps. Build and deploy. The result of build is that an artifact is uploaded to maven nexus. And currently manual deploy step just takes the latest artifact from nexus and deploys it.
stages:
- build
- deploy
full:
stage: build
image: ubuntu
script:
- // Build and upload to nexus here
deploy:
stage: deploy
script:
- // Take latest artifact from nexus and deploy
when: manual
But to me this doesn't seem to make that much sense to always deploy latest build from every pipeline. I think ideally deploy step of each pipeline should deploy the artifact that was build by the same pipelines build task. Otherwise deploy step of each pipeline will do exactly the same thing regardless when it is started.
So I have two questions.
1) How can I make my deploy step to deploy the version that was build by this run?
2) If I still want to keep the "deploy latest" functionality, then does gitlab support adding a task separate of each pipeline because as I explained this step doesn't make a lot of seance to be in pipeline? I imagine it being in a separate specific place.
Not too familiar with maven and nexus, but assuming you can name the artifact before you push it, you can add one of the built-in environment variables that dictates which pipeline it's from.
ie:
...
Build:
stage: build
script:
- ./buildAsNormal.sh > build$CI_PIPELINE_ID.extension
- ./pushAsNormal.sh
Deploy:
stage: deploy
script:
- ./deployAsNormal #(but specify the build$CI_PIPELINE_ID.extension file)
There are a lot of CI env variables you can use that are extremely useful. The full list of them is here. The difference with $CI_PIPELINE_ID and $CI_JOB_ID is that the pipeline id is constant for all jobs in the pipeline, no matter when they execute. That means the pipeline id will be the same even if you run a manual step a week after the automated steps. The job id is specific to each job.
Regarding your comment, the usage of artifacts: can solve your problem.
You can put the version number in a file and get the file in the next stage :
stages:
- build
- deploy
full:
stage: build
image: ubuntu
script:
- echo "1.0.0" > version
- // Build and upload to nexus here
artifacts:
paths:
- version
expire_in: 1 week
deploy:
stage: deploy
script:
- VERSION = $(cat version)
- // Take the artifact from nexus using VERSION variable and deploy
when: manual
An alternative is to build, push to nexus and use artifact: to pass the result of the build to the Deploy job :
stages:
- build
- deploy
full:
stage: build
image: ubuntu
script:
- // Build and put the result in out/ directory
- // Upload the result from out/ to nexus
artifacts:
paths:
- out/
expire_in: 1 week
deploy:
stage: deploy
script:
- // Take the artifact from in out/ directory and deploy it
when: manual