The first job in my GitLab pipeline builds multiple large artifacts of different types:
Applications that should be packaged as Docker images
NuGet packages
Database migrations
The job publishes multiple artifacts, at the moment one for each type. The size of all artifacts are in the realm om hundreds of megabytes.
In the next stage, multiple jobs are run in parallel to process and publish the artifacts from the build step. A significant amount of time is spent to download all artifacts from the build job, even though each job only needs one artifact. How can I configure GitLab CI so that a job depends on a specific artifact from a job?
Related
Gitlab CI/CD
I want to run a job for the Merge Request that needs jar artefacts built during a job run in another pipelines (for commits).
I don't want to build them again because if takes too much time then the job will fail.
Can I do that?
I checked the
needs:
- project:
But that's only with Premium licences.
We use GitLab pipeline and try to implement the cache mechanism to avoid the node packages getting downloaded multiple times in a pipeline through different jobs. Our scenario is below
The pipeline should check for the availability of cache in the local cache directory
if found , then it should download the files and move further with Sonarqube scan
If not found, then the pipeline should execute a specific job which do npm install and once its finished it should move further with sonarqube scan.
As far as I verified the documents , I didn't see any option to run a specific job only when there is no cache available in the local cache location. Any suggestions?
I have a simple Azure Pipeline containing two stages:
Stage 1: Build
Stage 2: Run tests
Stage 1 builds the source code and pushes the binaries to Azure Artifacts. Stage 2 downloads the binaries of Stage 1 and runs multiple tests against them, using different jobs for different tests:
Job 1: Run tests for module A
Job 2: Run tests for module B
Job 3: Run tests for module C
These jobs are completely indepent of each other and run in parallel.
If all stages and jobs succeed, how can I manually trigger Job 3 in Stage 2 without triggering Stage 1 and Job 1 and 2 of Stage 2 again?
One ugly work around might be to use variables and somehow run a new pipeline, skip stage 1 if the variables are set and download the binaries from a previous pipeline. Are there better approaches?
Edit: There is a similar topic ongoing on ms dev community:
https://developercommunity.visualstudio.com/idea/697467/manually-triggered-stages-in-yaml-multi-stage-pipe.html?childToView=903723#comment-903723
You can do what you are after with multi-stage pipelines if you pull the individual Jobs that exist in your current Stage 2 up into their own independent Stages.
Once you do that, you can then manually trigger them independently for reruns.
Or if you had a need, you can also start a new run with only a subset of the stages.
If you define release pipeline in yaml , you can through adding condition: false to disable/skip a job in multiple-job pipeline.
- job: Foo
condition: false
For details , you can refer to this case.
Then you can choose which stages to run when running multi-stages pipeline ,through this, you can skip Stage1.
We have a pipeline with many jobs and the last job failed. I'm trying to debug the issue, but the job requires artifacts from previous jobs.
How can I run this job locally with gitlab-runner so it has access to these artifacts?
That's not possible (yet).
See the limitations of the exec compared to regular CI here (artifacts -> not available).
Consider upvoting the issue to get this fixed.
I have a Gitlab CI pipeline with multiple stages. By default, the same stage between two jobs will be run in parallel. One of the stages I have is an integration test stage that should only have one running instance at a time since it relies on specific resources being setup and torn down.
Is there a simple way to specify a certain stage should only have one concurrent instance?
Gitlab Enterprise Edition v10.0.3
That is possible however it's not straightforward. Define a job with a tag:
job:
tags:
- integration
script:
- run intergation tests
This will make sure the job runs only on a runner that has a specific tag. Now create a runner with the tag integration and concurrency set to 1.
The integration test will run one at a time because there's only one runner who can run them and there's only 1 slot on it.