Is there a way of finding out what tags (Gitlab CI tags, not git tags) a job has from the API?
As background, we are having some issues with contention on our CI pipeline. The runners are selected based on tags, and I want to log what tags are being used/built the most.
For instance if in my .gitlab-ci.yml file I have
build_thing:
stage: build
tags: [some_tag]
script:
- bash build.sh
Is there a way I can use the gitlab jobs API to determine show the tags (some_tag) of that job? I have not been able to find any API endpoints that show this in the documentation.
I am aware that I can look at what tags the active runners have, but unfortunately that is not enough information, as they can have different combinations of tags that overlap the jobs/pipelines.
Related
Only one gitlab runner is available in my project. However, in the runner page I have multiple other runners listed. How do I add these runners to the project such that a runner will be randomly selected to execute the build? The runners are running in Docker containers and I am using a self-managed version of GitLab.
These are the runners I have available
Only runner1 is currently shown in /settings/cicd under Runners:
runner1
Basically, if you go by runners there are actually three types of a runner in Gitlab
Specific: Only for Specific Project
Group: For all projects in the particular group (if need more info on groups https://docs.gitlab.com/ee/user/group)
Shared: These will be available for every project in the GitLab
It will be difficult to specify here that how we can configure the runners but I can point to documentation from where we actually configured our Runners
https://docs.gitlab.com/ee/ci/runners/
Also to make a note of tagging is a very important task that needs to be done so that you can use your runners with your requirement like you said you want to use runners randomly from the list then all the runners in the list should have a common tag that can be further used in your .gitlab-ci.yml file.
eg. We have tag runners as per the AWS ec2 instance size so (like macro, small, medium, large, etc) so that people using can use the runners as per the requirement of there job)
job1:
tags:
- shared-large-runner
job2:
tags:
- shared-micro-runner
Note: you can have multiple tags for a single runner for better control over its usage.
Hopefully, this helps!!
Looking at your screenshot, to be able to use group runners, your project just need to be inside that group
Because the group tag shows that the runner was configured just to accept jobs from one group
I think you have one runner for each group, this is the reason that your project only have the runner1
I'm trying to setup Azure Pipelines for a CI setup and I'm using the YAML syntax to get started. However, I was wondering if it is possible to script the flow at "runtime"? Like you can do in Jenkins script: spawn builds etc.
Depending on the commit I want to have a vastly different flow.
This is because I currently have a mono-repo setup with Conan libraries and I want to rebuild the libraries that are necessary depending on the commit, thus the build-flow is not the same for each commit. I want to spawn jobs so I can take advantage of parallel building on several agents.
For your issue ,do you refer to trigger builds based on specified commits? If so, you can trigger builds by adding tag trigger in yaml. You can create tags on the commits. If the tag created meets the trigger condition of the tag trigger in yaml , then the build will be triggered.
trigger:
tags:
include:
- v2.*
I have a Gitlab project like this (.gitlab-ci.yml):
# Sub-jobs listed as comments
stages:
- check-in-tests
# shellcheck
# pylint
# unit-tests
- several-other-things
# foo
# bar
# baz
- release
# release
# Run some shell code static tests and generate logs/badges
shellcheck:
stage: check-in-tests
script:
- bash run_shellcheck.sh
artifacts:
paths:
- logs/shellcheck.log
- logs/shellcheck.svg
# Run some python code static tests and generate logs/badges
pylint:
stage: check-in-tests
script:
- bash run_pylint.sh
artifacts:
paths:
- logs/pylint.log
- logs/pylint.svg
# <snip>
On my project page I'd like to render the .svg files produced during check-in-tests as badges.
The Gitlab badges tool requires a URL to an image file. It is incapable of loading images from URLs with query strings. Unfortunately, the syntax for accessing specific job artifacts ends in a query string. This effectively means that we can't link to job artifacts as badges.
The most popular workaround is to abuse Gitlab's pages feature to store artifacts as static content. From there we can get clean URLs to our artifacts that don't contain query strings.
My confusion involves the underlying mechanism behind the "pages" job defined in .gitlab-ci.yml. The official documentation here is very sparse. There are a million examples for deploying an actual webpage with various frameworks, but I'm not interested in any of them since I'm just using my project's "page" for file hosting.
The assumption seems to be that I want to deploy my page at the end of the pipeline. However, I want to upload the shellcheck and pylint artifacts near the beginning of the pipeline. Furthermore, I want those artifacts to be uploaded even if the pipeline stages fail.
Syntactically the pages job looks identical to any other job. There's nothing there to describe how it's magically picked up by Gitlab's internals. This leaves me with the following questions:
Can I change the stage from "deploy" to "check-in-tests", or is the deploy stage specifically part of the hidden magic that Gitlab looks for when parsing for a pages job?
If I'm tied to the deploy stage, can I re-arrange the stages to make it come earlier in the pipeline without breaking the magic?
Does the pages job deploy artifacts from the local machine (default behavior for a job), or are the listed paths coming from artifacts which have already been uploade to the Gitlab pipeline by earlier jobs?
If the pages job is only looking for artifacts locally how can I ensure that it runs on the same machine as the earlier jobs so that it finds the artifacts which they produced? Let's assume that the Gitlab executors all come from a pool with the same tag and aren't tagged individually.
Is there any chance of getting the pages job to run within the same Docker container that originally produced the artifacts?
The magic around GitLab pages is in the name of the job. It has to be named "pages", and nothing else. It is possible to move the job to different stages. As soon as the job "pages" has finished successfully, there's a special type of job that is called "pages:deploy". This job is shown in the deploy stage even if you change the stage that the "pages" job is run in.
If you have the pages job in an early stage, jobs in the later stages can fail but the "pages:deploy" job will still run and update GitLab pages.
Other than that, the "pages" job is just like a normal job in GitLab. If you need artifacts from other jobs, you can get these by using artifacts and dependencies:
https://docs.gitlab.com/ee/ci/yaml/#dependencies
The "pages" job should create a folder named "public" and give that folder as an artifact.
I want to use gitlab runners for our builds and our deployment to QA. Can I set up a second runner that ONLY does deployments, or just a specific environment? I don't see a way to do this. (Gitlab.com/private runners)
Found it. This can be achieved using tags. CI runners can be configured to run only jobs that have specific tags, and tags can be applied to jobs.
Gitlab jobs and runners can be tagged - so only runners with the tag can run the job. See documentation here: http://docs.gitlab.com/ce/ci/yaml/README.html#tags
I want to ensure that my runners that have tags only pick up tagged jobs. Currently jobs without any tags are eligible to run on these runners too.
Can this be done?
During the Registration process of the runner: https://docs.gitlab.com/runner/register/index.html
Choose whether the Runner should pick up jobs that do not have tags, you can change this later in GitLab's UI (defaults to false):
In the Admin panel https://<SERVER>/admin/runners
And in the config dialog of your specific runner (URL: https://<PROJECT_URL>/settings/ci_cd)
...you can set the option, to only run tagged jobs.
If you then also tag your jobs accordingly, you have reached your goal.
e.g.:
job:
tags:
- ruby
- postgres