We have a pipeline with many jobs and the last job failed. I'm trying to debug the issue, but the job requires artifacts from previous jobs.
How can I run this job locally with gitlab-runner so it has access to these artifacts?
That's not possible (yet).
See the limitations of the exec compared to regular CI here (artifacts -> not available).
Consider upvoting the issue to get this fixed.
Related
Gitlab CI/CD
I want to run a job for the Merge Request that needs jar artefacts built during a job run in another pipelines (for commits).
I don't want to build them again because if takes too much time then the job will fail.
Can I do that?
I checked the
needs:
- project:
But that's only with Premium licences.
We use GitLab pipeline and try to implement the cache mechanism to avoid the node packages getting downloaded multiple times in a pipeline through different jobs. Our scenario is below
The pipeline should check for the availability of cache in the local cache directory
if found , then it should download the files and move further with Sonarqube scan
If not found, then the pipeline should execute a specific job which do npm install and once its finished it should move further with sonarqube scan.
As far as I verified the documents , I didn't see any option to run a specific job only when there is no cache available in the local cache location. Any suggestions?
I am designing a dataproc workflow template with multiple spark jobs. These spark jobs would run in sequence one after the other. There could be scenarios where the workflow would run few jobs successfully and might fail for others. Is there a way to just rerun the failed jobs once I have done workaround to fix the issues which failed those jobs in the first place. Please note that I am not looking for job retry mechanism of jobs. I want to re-run the workflow again by avoiding running already successful jobs.
Dataproc Workflows do not support this use case.
Please take at Cloud Composer - Apache Airflow-based orchestration service which is more flexible and should be able to satisfy your use case.
I want to run jobs in the same stage sequentially instead of parallel in GitLab CI. Currently this is what I have:
I want unit-test to run before integration-test and not in parallel. I have looked into the docs and have encountered DAG but it needs the job to be in a prior stage and cannot be on the same stage. Is there a way to achieve this?
Yes its already described in the documentation for stages, jobs are started in parallel in one stage.
It says:
To make a job start earlier and ignore the stage order, use the needs keyword.
As you said, this is not possible in GitLab < 14.2 within a stage (needs):
needs: is similar to dependencies: in that it must use jobs from prior stages, meaning it's impossible to create circular dependencies. Depending on jobs in the current stage is not possible either, but support is planned.
As an alternative, you could define several stages and use the keyword needs between jobs in these stages.
Since GitLab 14.2 (issue) its possible.
We have 4 deploy jobs in the same stage that can be run concurrently. From the Gitlab docs:
The ordering of elements in stages defines the ordering of jobs' execution:
Jobs of the same stage are run in parallel.
Jobs of the next stage are run after the jobs from the previous stage complete successfully.
What happens, however, is that only one of the jobs run at a time and the others stay on pending. Is there perhaps other things that I need to do in order to get it to execute in parallel. I'm using a runner with a shell executor hosted on an Ubuntu 16.04 instance.
Your runner should be configured to enable concurrent jobs( see https://docs.gitlab.com/runner/configuration/advanced-configuration.html)
concurrent = 4
or you may want to setup several runners.
I also ran into this problem. I needed to run several tasks at the same time. I used everything I could find (from needs to parallel). however, my tasks were still performed sequentially. every task I had was on standby. the solution turned out to be very simple. open file /etc/gitlab-runner/config.toml concurent for the required number of parallel tasks for you.