I am trying to get a job to run in parallel within gitlab. I have registered 8 runners on one server and set concurrent=16 in /etc/gitlab-runner/config.toml. I see all 8 runners configured within that file. Below is the beginning of the file:
concurrent = 16
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "gitlab-runner-02-#1"
url = "http://gitlab.example.com"
token = "redacted"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "almalinux:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[[runners]]
name = "gitlab-runner-02-#2"
url = "http://gitlab.example.com"
token = "redacted"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "almalinux:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
.gitlab-ci.yml:
image: almalinux:latest
stages:
- build
foo:
stage: build
script: sleep 60
parallel:
matrix:
- foo: [a, b, c, d, e, f, g, h]
From everything I've read, this should be enough for it to run in parallel. And yet, all the jobs run sequentially. What am I missing here?
Edit:
I've crossposted on gitlab's forum. I will make sure both gitlab's forum and stackoverflow have the answer clearly listed when I find it. https://forum.gitlab.com/t/parallel-matrix-isnt-executing-in-parallel/72095
Update:
I seem to have resolved the issue. I need to do more experimenting, but it appears to be related to another registered runner on a different server not allowing more than one job. I also increased the limit for each individual runner. I need to do more experimenting to understand what actually fixed it still.
Related
My company uses self-managed AWS auto-scaling Docker runners, via Docker Machine. This configuration is documented here
We have a single runner/runner-manager EC2 instance whose config.toml contains several different runner configs, all with different tags so that different groups in our Gitlab org get a dedicated runner, by use of runner tags, all from a single runner which spins up the appropriate executor for the corresponding tag in the job definition.
The runner for my group has been working flawlessly for months. Today I created a job using the parallel:matrix: keyword
Build Images:
image: myimage
stage: build
script:
- docker build -f $DOCKERFILE -t $IMAGE_TAG
- docker push $IMAGE_TAG
parallel:
matrix:
- DOCKERFILE: $CI_PROJECT_DIR/Dockerfile
IMAGE_TAG: myrepo/myimage:standard
- DOCKERFILE: $CI_PROJECT_DIR/super.Dockerfile
IMAGE_TAG: myrepo/myimage:super
rules:
- when: always
When I push a commit neither this job or any others which should run are getting triggered. No error message or anything. The CI/CD->Jobs page does not show any jobs either.
This is the config.toml used on the runner manager. The runner I am attempting to run this job with is the first runner "my-runner"
concurrent = 100
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "my-runner"
limit = 6
url = "https://gitlab.com"
token = "XYZABC"
executor = "docker+machine"
[runners.custom_build_dir]
[runners.cache]
Type = "s3"
Path = "cache"
Shared = true
[runners.cache.s3]
ServerAddress = "s3.amazonaws.com"
BucketName = "mybucket"
BucketLocation = "us-east-1"
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "alpine:latest"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
shm_size = 0
[runners.machine]
IdleCount = 0
IdleTime = 600
MaxBuilds = 10
MachineDriver = "amazonec2"
MachineName = "gitlab-docker-machine-%s"
MachineOptions = ["amazonec2-instance-type=t3.medium", "amazonec2-vpc-id=vpc-xxxxxxxx", "amazonec2-security-group=my-security-group", "amazonec2-iam-instance-profile=xxxxxx", "amazonec2-root-size=32", "amazonec2-ami=ami-218k65t87w8b6posq", "amazonec2-subnet-id=subnet-xxxxxxxxx", "amazonec2-zone=a"]
[[runners.machine.autoscaling]]
Periods = ["* * 13-23 * * mon-fri *"]
Timezone = "UTC"
IdleCount = 1
IdleTime = 600
[[runners.machine.autoscaling]]
Periods = ["* * 2-11 * * * *"]
Timezone = "UTC"
IdleCount = 0
IdleTime = 300
[[runners]]
name = "other-runner"
limit = 6
url = "https://gitlab.com"
token = "LMNOP"
executor = "docker+machine"
...
...
...
There are several more runners defined in this config, but they are all very similar. Each is registered with different tags.
My Question: In the Gitlab CI docs it says
Multiple runners must exist, or a single runner must be configured to run multiple jobs concurrently
and to me it seems like multiple runners do exist, since the runner I am using has a limit of 6. Do the executors need to actually be spun up and sitting idle for this to work? Is there any way that I can get these parallel jobs to run without increasing my runner idle count?
Edit: Some additional information
This is just one job of about a dozen in this file(load-tests.yml)
My gitlab-ci.yml file imports jobs from about 10 other files via
include:
- local: .gitlab/load-test.yml
The pipeline never get created. If I comment out this job then the pipeline runs, including the other jobs in this file.
I can provide the entire file verbatim, but everything works fine if this job is not included. I'm fairly experienced with Gitlab-CI so I'm sure that the issue lies with this job and/or the runner config when using these keywords.
Tag and other keys are set in defaults in the .gitlab-ci.yml file. None of them are of significance, things like default variables, default before_script, cache, etc.
Edit: We are using Gitlab SAAS(premium I believe, but not sure) with the runner manager using Gitlab-Runner v14.1.0
I wish to configure a GitLab runner AutoScale using Docker Machine and Azure Availability sets. I followed this guide https://docs.gitlab.com/runner/executors/docker_machine.html . the docker machine seems to work and new vms are created using the availability set but I get this errorwhen I use docker-machine ls Unable to query docker version: Cannot connect to the docker engine endpoint .
Also I configured the config.toml file and registered with gitlab-runner register and I do not get the runner in the gitlab runners list.
this is the config.toml
concurrent = 30
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "gitlab-runners-bastion"
url = "#############"
token = "#############"
executor = "docker+machine"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
AccountName = "gitlabrunners#############"
AccountKey = "#############"
ContainerName = "runners-cache"
StorageDomain = "blob.core.windows.net"
[runners.docker]
tls_verify = false
image = "alpine:latest"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
shm_size = 0
[runners.machine]
IdleCount = 1
IdleTime = 300
MachineDriver = "azure"
MachineName = "gitlab-runner-%s"
MachineOptions = [
"azure-subscription-id=#############",
"azure-availability-set=",
"azure-client-id=c3f1c633-eb2e-4fb0-9b45-8089ed4d8809",
"azure-client-secret=Cmi8Q~AFbCpPYZV4Uz7FjwrM66Z8Fj3Pd~QQPc2i",
"azure-location=northeurope",
"azure-no-public-ip",
"azure-use-private-ip",
"azure-vnet=BASMACH-NE-PROD-GITLAB-RUNNERS-VNET",
"azure-subnet=default",
"azure-size=Standard_B2s" ,
"azure-resource-group=BASMACH-NE-PROD-GITLAB-RUNNERS"
]
We've set up a Gitlab autoscaling master machine as described in the official docs.
The gitlab-runner instance works, is recognized by gitlab.org, and also successfully spawns runners to execute the jobs.
However, the jobs don't get really started on the spawned runners. They stick at this point.
We have debug-level logging turned on, and the only unhappy-looking messages are these repeated ones:
msg="Failed to request job: runner requestConcurrency meet"
config.toml looks like this:
concurrent = 20
check_interval = 10
log_level = "debug"
log_format = "text"
[session_server]
session_timeout = 1800
[[runners]]
name = "gitlab-runner-master"
url = "https://gitlab.com/"
token = "blabla"
executor = "docker+machine"
limit = 25
[runners.custom_build_dir]
[runners.cache]
Type = "s3"
Shared = true
[runners.cache.s3]
ServerAddress = "s3.amazonaws.com"
AccessKey = "blabla"
SecretKey = "blabla"
BucketName = "gitlab.cache.dyynamo.net"
BucketLocation = "us-east-1"
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "amd64/ubuntu:16.04"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = true
volumes = ["/cache"]
shm_size = 0
[runners.machine]
IdleCount = 0
IdleTime = 600
MaxBuilds = 100
MachineDriver = "amazonec2"
MachineName = "gitlab-docker-machine-%s"
Any ideas what the problem can be?
Have also noted this still-open similar issue.
My conf-file:
external_url "http://192.168.3.23" # note the use of a dotted ip
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'gitlab#myhome.com'
gitlab_rails['gitlab_email_display_name'] = 'gitlab'
#gitlab_rails['gitlab_email_reply_to'] = 'gitlab#myhome.com'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mail.home"
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_domain'] = "myhome.com"
mattermost_external_url 'http://192.168.3.23'
mattermost['gitlab_enable'] = true
mattermost['gitlab_secret'] = "4d1e<***>bdbfe"
mattermost['gitlab_id'] = "1c441<***>092df"
mattermost['gitlab_scope'] = ""
mattermost['gitlab_auth_endpoint'] = "http://192.168.3.23/oauth/authorize"
mattermost['gitlab_token_endpoint'] = "http://192.168.3.23/oauth/token"
mattermost['gitlab_user_api_endpoint'] = "http://192.168.3.23/api/v3/user"
# Shut down GitLab services on the Mattermost server
#gitlab_rails['enable'] = false
But now by the address 192.168.3.23 loading only gitlab.
GitLab Community Edition 8.4.4 9c31cc6!
How to start use gitlab and mattermost together?
Need use different url-address for GitLab and Mattermost.
extermanl_url "http://192.168.3.23"
...
mattermost_external_url "http://192.168.3.23:8065"
Solve here.
Due to the fact that modulepath is now deprecated in a implicit scope (IE main), how can I keep the modulepath for each of my environments? I would like to keep my modules inside of a global scope without having to specify an environment.
[main]
certname = {somecert}
dns_alt_names = puppet,{otherorgs}
vardir = /var/opt/lib/pe-puppet
logdir = /var/log/pe-puppet
rundir = /var/run/pe-puppet
modulepath = $confdir/environments/$environment/modules:/opt/puppet/share/puppet/modules
manifest = /etc/puppetlabs/puppet/environments/$environment/manifests/site.pp
server = {puppetserver}
user = pe-puppet
group = pe-puppet
archive_files = true
archive_file_server = {puppetserver}
[master]
certname = {puppetserver}
ca_name = 'Puppet CA generated on {puppetserver} at 2013-11-07 13:15:40 -0800'
reports = puppetdb,cimlog,console
node_terminus = console
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
storeconfigs_backend = puppetdb
storeconfigs = true
manifest=$confdir/environments/$environment/manifests/site.pp
[agent]
report = true
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
graph = true
pluginsync = true
environment = production
http_compression = true
splaylimit=1800
configtimeout=480
splay=true
It's not perfectly clear what you are asking here, but I shall assume that you care about retaining the ability to keep a global set of modules in /opt/puppet/share/puppet/modules that is independent of the respective environment.
Puppet allows this through the basemodulepath option.
[main]
environmentpath=$confdir/environments
basemodulepath=/opt/puppet/share/puppet/modules