I have a CI pipeline in Bitbucket which is used to build and test a shared Node.js library. If we create tag (eg. npm version patch -m "Upgrade to 0.1.2 for bug fix") the new version must be published to the npm repository.
Therefore I have the following pipeline configuration:
pipelines:
default:
- step: *build-test-sonarcloud
tags:
'*':
- step: *build-test-sonarcloud
- step: *build-deploy-npm
However, if I push all changes after 'npm version patch' two pipelines are started. I suppose that this is because the file 'package.json' is also committed and not only a tag.
My idea is that only the 'tags'-pipeline should be started in case of a commit of a tag (with or without any files). Is there a way to only trigger that pipeline and prevent the default pipeline to run?
When you mention '*' after tags step, this means, the pipeline will be running twice every time.
You need to mention tag name, something like below code
pipelines:
default:
- step: *build-test-sonarcloud
tags:
'yourCustomTagName':
- step: *build-test-sonarcloud
- step: *build-deploy-npm
Now whenever you push your code with yourCustomTagName, then both the steps will run in the pipeline, if not only one will run.
Related
I am trying to build GitLab CI/CD for the first time. I have two stages build and deploy The job in the build stage produce artifacts. And then the job in deploy stage wants to upload those artifacts to AWS S3. Both the jobs are using same runner but different docker image.
default:
tags:
- dev-runner
stages:
- build
- deploy
build-job:
image: node:14
stage: build
script:
- npm install
- npm run build:prod
artifacts:
paths:
- deploy/build.zip
deploy-job:
image: docker.xx/xx/gitlab-templates/awscli
stage: deploy
script:
- aws s3 cp deploy/build.zip s3://mys3bucket
The build-job is successfully creating the artifacts. GitLab documentation says artifacts will be automatically downloaded and available in the next stage, however it does not specify where & how these artifacts will be available to consume in the next stage.
Question
In the deploy-job will the artifacts available at the same location? like deploy/build.zip
The artifacts should be available to the second job in the same location, where the first job saved them using the 'artifacts' directive.
I think this question already has an answer on the gitlab forum:
https://forum.gitlab.com/t/access-artifact-in-next-task-to-deploy/9295
Maybe you need to make sure the jobs run in the correct order using the dependencies directive, which is also mentioned in the forum discussion accesible via the link above.
I got a requirement to generate, archive and reuse the artifacts between two different repositories
Repository A: Compile Angular code and create a XLF file
Repository B: Use the 'XLF File' generated above and create a new XLF file
Repository A: Again use the newly generated XLF file to create the final output file
The activities mentioned above should be done using gitlab-ci.yml. I am not sure how to handle this using GitLab CI.
We can push the artifact from Repo A to Repo B. However, CI on Repo A should wait until Repo B pushes a new artifact to Repo A to complete the process
Ideally, you would not push a generated artifact to another Git source repository.
But a GitLab pipeline can retrieve an artifact produced by another one from its URL.
To avoid the back and forth, I would rather have 3 jobs instead of two
the first generates XLF file
the second curls/fetches that file, and use it to generate new XLF file
the third job curls/fetches that file, and complete the process.
How to release built artifacts back-and-forth from one to another repo on GitLab?
Repository A:
Compile Angular code and create a XLF file
Send a hook to repository B that it just compiled
just trigger: https://docs.gitlab.com/ee/ci/yaml/#trigger , works like a charm. It's even nicely visible in the gui.
or API https://docs.gitlab.com/ee/ci/triggers/
pass variables: PARENT_PIPELINE_ID: $CI_PIPELINE_ID to repository B so it can download artifacts from specific pipeline
Repository B:
Use the 'XLF File' generated above
use needs: https://docs.gitlab.com/ee/ci/yaml/#artifact-downloads-to-child-pipelines to download artifacts
or API: have personal access token from repository A https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html added to environment variables and use API to download artifacts https://docs.gitlab.com/ee/api/job_artifacts.html .
create a new XLF file
use trigger: or API to trigger repository A
but this time trigger different .gitlab-ci.yml file like: trigger: - project: repositoryA file: second_stage.gitlab-ci.yml https://docs.gitlab.com/ee/ci/yaml/#trigger-child-pipeline-with-files-from-another-project
or use like variables: SECOND_STAGE: "true" and use a variable to differentiate
Repository A:
run pipeline from the file second_stage.gitlab-ci.yml
download artifacts from repository B - needs: or API
use the newly generated XLF file to create the final output file
Overall, what you need is rules: and needs: documentation. On older gitlab, it was done with API.
CI on Repo A should wait until Repo B pushes a new artifact to Repo A to complete the process
Don't wait. Let the API trigger it.
I tried the following approach and it worked fine or at least I was able to proceed
Due to some reason 'variables' along with CURL did not work as expected but I did not analyze the root cause
Repo A - Pipeline
trigger-repob: (Trigger Project B of Repo B)
stage: repob
trigger:
project: repob-namespace/projectb
branch: devops
test_job:
image: $CI_REGISTRY/$CI_PROJECT_PATH/base-image:latest
stage: test_pipeline
when: delayed
start_in: 2 minutes
needs: (Use artifacts from Repo B/Project B)
-
project: repob-namespace/projectb
job: buildprojectb
ref: devops
artifacts: true
script:
- do something here
Repo B Pipeline
buildprojectb:
image: php:7.4.11
stage: build
script:
- do something here
artifacts:
paths:
- outputs/*.xlf
I am working on a multi-pipeline project, and using trigger keyword to trigger a downstream pipeline, but I'm not able to pass artifacts created in the upstream project. I am using needs to get the artifact like so:
Downstream Pipeline block to get artifacts:
needs:
- project: workspace/build
job: build
ref: master
artifacts: true
Upstream Pipeline block to trigger:
build:
stage: build
artifacts:
paths:
- ./policies
expire_in: 2h
only:
- master
script:
- echo 'Test'
allow_failure: false
triggerUpstream:
stage: deploy
only:
- master
trigger:
project: workspace/deploy
But I am getting the following error:
This job depends on other jobs with expired/erased artifacts:
I'm not sure what's wrong.
Looks like there is a problem sharing artifacts between pipelines as well as between projects. It is known bug and has been reported here:
https://gitlab.com/gitlab-org/gitlab/-/issues/228586
You can find a workaround there but since it needs to add access token to project it is not the best solution.
Your upstream pipeline job "Build" is set to only store its artifacts for 2 hours (from the expire_in: 2h line. Your downstream pipeline must have run at least 2 hours later than the artifacts were created, so the artifact expired and was erased, generating that error.
To solve it you can either update the expire_in field to however long you need them to be active (so for example if you know the downstream pipeline will run up to 5 days later, set it to 5d for 5 days), or rerun the Build job to recreate the artifacts.
You can read more about the expire_in keyword and artifacts in general from the docs
It isn't a problem with expired artifacts, the error is incorrect. In my case I am able to download the artifacts as a zip directly from the UI on the executed job. My expire_in is set to 1 week yet I am still getting this message.
I have a mono-repo with several projects (not my design choice).
Each project has a .gitlab-ci.yml setup to run a pipeline when a "version" file is changed. This is nice because a user can check-in to stage or master (for a hot-fix) and a build is created and deployed to a test environment.
The problem is when a user does a merge from master to stage and commits back to stage (to pull in any hot-fixes). This causes ALL the pipelines to run; even for projects that do not have actual content changes.
How do I allow the pipeline to run from master and/or stage but ONLY when the contents of the "version" file change? Like when a user changes the version number.
Here is an example of the .gitlab-ci.yml (I have 5 of these, 1 for each project in the mono-repo)
#
# BUILD-AND-TEST - initial build
#
my-project-build-and-test:
stage: build-and-test
script:
- cd $MY_PROJECT_DIR
- dotnet restore
- dotnet build
only:
changes:
- "MyProject/.gitlab-ci.VERSION.yml"
# no needs: here because this is the first step
#
# PUBLISH
#
my-project-publish:
stage: publish
script:
- cd $MY_PROJECT_DIR
- dotnet publish --output $MY_PROJECT_OUTPUT_PATH --configuration Release
only:
changes:
- "MyProject/.gitlab-ci.VERSION.yml"
needs:
- my-project-build-and-test
... and so on ...
I am still new to git, GitLab, and CI/pipelines. Any help would be appreciated! (I have little say in changing the mono-repo)
The following .gitlab-ci.yml will run the test_job only if the file version changes.
test_job:
script: echo hello world
rules:
- changes:
- version
See https://docs.gitlab.com/ee/ci/yaml/#ruleschanges
See also
Run jobs only/except for modifications on a path or file
I'm trying GitLab for my first example.
I can't see where's is the error here:
this is for windows running firebase, vue.js, node.js on gitlab
image: node:alpine
cache:
paths:
- node_modules/
deploy_production:
stage: deploy
environment: Production
only:
- master
script:
- npm install
- npm i -g firebase tools
- npm run build
- firebase deploy --non-interactive --token "1/CYHKW-CuYsKOcy2Eo6_oC9akwGjyqtmtRZok93xb5VY"
This GitLab CI configuration is invalid: jobs:deploy_production script
can't be blank
You specify a stage in your deploy_production job but you don't define stages.
Add :
stages:
- deploy
before your job definition.
Late to the party, but one problem here is the indenting of the script tag, which needs to be under the job deploy_production. script isn't allowed at the top level like you've shown it here.
The error is kind of confusing, but does indicate the situation. Because script isn't at the right indent level, it's not part of the job, and a job requires a script.
Should be:
deploy_production:
stage: deploy
environment: Production
only:
- master
script:
- npm install
Another issue is you should screen out your token in the post!
Another way you can get this error message:
Here's what I was trying to do in gitlab-ci.yml:
default:
cache:
paths:
- .gradle
And I was getting this error message:
jobs:default script can't be blank
I was using the documentation here: https://docs.gitlab.com/ee/ci/yaml/
Which clearly indicates how to use default. The message implies gitlab thought default was a job.
ANSWER
You probably know where this is going -- the version I was using was about 3 years behind latest, and the "default" keyword had been added since then.
Check the version of gitlab you're using by going to the Help page (gitlab.domain.com/help), and it's listed at the top of the page.
To find the right documentation, I went to https://gitlab.com/rluna-gitlab/gitlab-ce, then chose my version from the branch drop down. From there went to the docs folder, then clicked on this link in the Popular Documentation table in the README.
https://gitlab.com/rluna-gitlab/gitlab-ce/-/blob/11-6-stable/doc/ci/yaml/README.md