I have one gitlab runner configured for a single project. The issue that I am seeing is that the runner will not wait until the prior job finished, and instead does a checkout in the same directory as the prior job and stomps over everything. I have one job already running, and then another develop commits and thus another job is started. Why can't I configure the pipeline not to run so that it doesn't corrupt the already running workspace?
Here is the log from both of the jobs (only difference is the timestamp)
[0K] Running with gitlab-runner 12.6.0 (ac8e767a)
[0K] on gitlab.xxxx.com rz8RmGp4
[0K] section_start:1578357551:prepare_executor
[0K] Using Docker executor with image my-image-build ...
[0K] Using locally found image version due to if-not-present pull policy
[0K] Using docker image sha256:xxxxxxxxxx for my-image-build ...
[0;msection_end:1578357553:prepare_executor
[0Ksection_start:1578357553:prepare_script
[0K] Running on runner-rz8RmGp4-project-23-concurrent-0 via gitlab.xxxx.com...
section_end:1578357554:prepare_script
[0K] section_start:1578357554:get_sources
[0K[32;1mFetching changes with git depth set to 50...[0;m
Initialized empty Git repository in /builds/my-project/.git/
<proceeds to checkout and stomp over the already running runner>
Main issue I see is that they both checkout to the same directory of Initialized empty Git repository in /builds/my-project/.git/ which causes the problem.
You can use resource_group to keep jobs from running in parallel.
e.g.
Job 1:
stage: My Stage
resource_group: stage-wedge
...
Job 2:
stage: My Stage
resource_group: stage-wedge
...
In the above example Job 2 will run after Job 1 is finished.
Jobs of same stage are executed in parallel.
if you need it to be sequential, you may add a stage for each of those jobs.
see the docs
https://docs.gitlab.com/ee/ci/yaml/#stages
In case of multiple pipelines running you may want to configure your gitlab-runner options: limit / concurrent
https://docs.gitlab.com/runner/configuration/advanced-configuration.html
Related
Have an API in production since May, working on server with Gitlab-CI/CD. Everything worked fine till last update (2021:08:10), when the pipelines started to fail.
I could see different errors, like :
/bin/sh: eval: line 183: docker-compose: not found
or
/bin/sh: eval: line 183: sudo: not found
or other problems like file or directory not found and etc. So fixing one issue was leading to another issue.
I don't show here the .gitlab-ci.yml file, because I assume the problem is in another place. After spending hours, found out that the pipeline is running with Docker executor.
Running with gitlab-runner 14.1.0-rc1 (e9489c8a)
on docker-auto-scale 0277ea0f
Preparing the "docker+machine" executor
Using Docker executor with image docker:latest ...
Very interesting... Because when registering the runner I've chosen the executor shell. And the last successful pipeline 2 months ago was running exactly with executor shell.
I tried to re-register the runner, reset the registration token and register the new one, each time choosing as executor shell, but when checking it appears that the pipelines are running with Docker executor.
Does anyone have idea how it is possible and how to fix this issue?
Actually here is the solution.
Settings were changed somehow in gitlab profile, shared runners became enabled and that caused an impact on my runner and executor.
I am setting up my fork of a Github project with the azure_pipeline.yaml configuration.
This seems to work just fine for everyone else in the community but when I setup the pipeline it gives me the following exception:
A task is missing. The pipeline references a task called 'Cache'. This usually indicates the task isn't installed, and you may be able to install it from the Marketplace: https://marketplace.visualstudio.com. (Task version 2, job 'compile_ci_build', step ''.)
A task is missing. The pipeline references a task called 'Cache'. This usually indicates the task isn't installed, and you may be able to install it from the Marketplace: https://marketplace.visualstudio.com. (Task version 2, job 'test_ci_build', step ''.)
A task is missing. The pipeline references a task called 'Cache'. This usually indicates the task isn't installed, and you may be able to install it from the Marketplace: https://marketplace.visualstudio.com. (Task version 2, job 'e2e_ci_build', step ''.)
Specifically,
Here is my Azure pipeline link
I am creating Flink CI build pipeline according to this instruction.
Which already have an azure-pipeline.yml in the repo
It uses the template to run the job parameterized in tools/azure-pipelines/jobs-template.yml
[UPDATE]
I modified the jobs-templates.yml and commented out all steps with Cache#2 and it runs fine.
Was able to get this working eventually.
Apparently for my Azure account. I am not allow to use Cache#2.
changing all lines with
- task: Cache#2
to
- task: CacheBeta#1
resolves all my problem.
By design, Azure DevOps does not automatically make all tasks available when you run a pipeline.
You have to add them manually as part of the pipeline.
'cache' is one such task.
I'm doing this in classic gui (non-yaml) mode as I find it easier to search for things
But what you do is in your pipeline add a new task and in the task search box type 'cache'.
This will bring up the task.
Click Add to include it in the pipeline.
For more information on this I would recommend reading:
https://learn.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops
I do not see you added cache task in your pipeline from your git repo.
It should look like the example here:
variables:
YARN_CACHE_FOLDER: $(Pipeline.Workspace)/.yarn
steps:
- task: Cache#2
inputs:
key: 'yarn | "$(Agent.OS)" | yarn.lock'
restoreKeys: |
yarn | "$(Agent.OS)"
yarn
path: $(YARN_CACHE_FOLDER)
displayName: Cache Yarn packages
- script: yarn --frozen-lockfile
Source
I was getting the same error A task is missing. The pipeline references a.. on one of azure task, PublishCucumberReport#1 . I resolved it by visiting https://marketplace.visualstudio.com/ and going to the task and then clicking on get free button, which installs it on your pipeline
I install a Specific Runners,and the status is actived.
my .gitlab-ci.ymi file code:
stages:
- build
build_maven:
stage: build
only:
- master
script:
- echo "hello CI/CD"
tags:
- vue-dev-pub
when I push the master branch,the gitlab-runner is running,but it's pending infinitely。
the job page show:
This job has not started yet
This job is in pending state and is waiting to be picked by a runner
if I excute the runner manually,the job can pass.
the command of gitlab-runner verify shows:
Runtime platform arch=amd64 os=linux pid=24616 revision=d0b76032 version=12.0.2
WARNING: Running in user-mode.
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...
Verifying runner... is alive runner=T4iKvsT3
I am waiting for you respond,thanks!
If you run the runner manually in debug mode gitlab-runner --debug run you may see the actual error message, in my case it was:
WARNING: Failed to process runner builds=0 error=failed to update executor: missing Machine options executor=docker+machine runner=pSUsX4yR
That's because on runner creation, I selected option docker+machine rather than docker.
After amending /etc/gitlab-runner/config.toml to docker and running gitlab-runner restart followed by gitlab-runner verify, pipeline started running again.
I had a similar problem with my (shell) runners on linux. It would work fine on runners installed and registered on one of my computers but not another. (Even as tags matched correctly in runner and job)
After
gitlab-runner register I would get:
New runner. Has not connected yet
After
gitlab-runner verify that error would go away. But I would get This job is in pending state and is waiting to be picked by a runner
After
gitlab-runner restart
It would all work.
gitlab-runner status
gitlab-runner: Service is running!
Maybe you have tagged your runner but your job has no tags. Refer :how to run untagged jobs
https://stackoverflow.com/a/53371027/10570524
The tags section in your .gitlab-ci.yml file specifies this job has to be picked by a runner that has the same tags (reference).
tags:
- vue-dev-pub
So unless there is actually a runner available for your project that has the vue-dev-pub tag it will keep waiting for one to become available.
first, remove the old config in sys
rm /etc/systemd/system/gitlab-runner.servicetemd
now, you need install gitlab-runner with gitlab user:
gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
root installations fail
How to configure GitLab CI to store the SSTATE_DIR and the DL_DIR between several jobs? Currently, bitbake rebuilds the complete project every time, which is very time consuming. So i would like to use the sstage again. I tried caching, but building time increases effectively, due to the big zip/unzip overhead.
I don't even need a shared sstate between several projects, just a method to store the output between jobs.
I'm using Gitlab 11.2.3 with a shell executor as runner.
Thanks a lot!
In version 11.10, GIT_CLEAN_FLAGS was added, which could be used to achieve what you want to do with the shell executor.
For completeness: when using the docker executor, this can be achieved by using docker volumes, which are persistent across builds.
If you're only using one runner for this, you could potentially use GIT_STRATEGY: none, which would re-use the project workspace for the following job; relevant documentation. However, this wouldn't be extremely accurate in case you have multiple jobs running which requires the runner, as it could dilute the repository, if jobs are started from across different pipelines.
Another way, if you're still using one runner, is you could just copy the directories out and back into the job you require.
Otherwise, you may potentially be out of luck, and have to wait for the sticky runners issue.
You can reuse a shared-state cache between jobs simply as follows:
Specify the path to the sstate-cache directory in the .yml file of your
gitlab-ci pipeline. An example fragment from one of mine:
myrepo.yml
stages:
...
...
variables:
...
TCM_MACHINE: buzby2
...
SSTATE_CACHE: /sstate-cache/$TCM_MACHINE/PLAT3/
PURGE_SSTATE_CACHE: N
...
In my case /sstate-cache/$TCM_MACHINE/PLAT3/ is a directory in the docker container
that runs the build. This path is mounted in the docker container from a
permanent sstate cache directory on the build-server's filesystem, /var/bitbake/sstate-cache/<machine-id>/PLAT3.
PURGE_SSTATE_CACHE is overridable by a private variable
in the pipeline schedule settings so that I can optionally delete the cache for a squeaky clean
build.
Ensure that the setting of SSTATE_CACHE is appended to the bitbake conf/local.conf
file of the build, e.g.
.build_image: &build_image
stage: build
tags:
...
before_script:
...
script:
- echo "SSTATE_DIR ?= \"$SSTATE_CACHE\"" >> conf/local.conf
...
Apply the same pattern to DL_DIR if you use it.
Variables you use in the .yml file can be overriden by gitlab-ci trigger
or schedule variables. See Priority of variables
When i start a build in gitlab ci (docker executer) its creating container with name "runner-0fa22293-project-174-concurrent-0-postgres"
In this what does "0fa22293-project-174-concurrent-0" represent ?
What is the naming convention used here?
I believe the format is as follows:
runner-<token>-project-<project ID>-concurrent-<concurrent job #>
<token> is the first 8 characters of the Runner Token for the CI runner that is running the build.
<project ID> is the ID of the project. The is the project ID you'd use during API access too.
<concurrent job #> is the number of this job, for this build. GitLab CI lets you run several jobs in a build concurrently, so if you had 3 of 5 jobs running concurrently you would see ...-concurrent-0, ...-concurrent-1, ...-concurrent-2