I have setup the Gitlab self hosted instance having one runner registered as "linux runner manager" which is running jobs of Gitlab Pipelines on ECS fargate by using docker image as task, after job completed on Gitlab, that container will be destroyed.
I want to achieve Complete Autoscaling setup for runners of Gitlab where I have Linux, Macos & windows runner managers building these 3 different platforms pipelines based on docker containers inside ECS Fargate cluster.
I have achieved this setup in case of linux by following this official documentation of Gitlab.
Here is the desired rough diagram of what i have in mind
In current achieved setup I have these two instances, one gitlab master node and one linux manager instance which is explained in above diagram.
I have two files on runner manager
one is config.toml having connection gitlab and token stuff.
and the other one is fargate.toml having connection ecs fargate.
Lastly i have cluster ECS cluster having Fargate Arch.
When the job is being triggered on Gitlab Pipeline, it invoke container in ECS and create new task once job is completed it destroy the task, so in this way ECS cluster will have empty task bucket unless there is triggered pipeline active from gitlab.
The container image currently which I am using for building jobs on ECS is available here
I have gone through official documentation of ECS fargate as well
Seems like only window is supported.
I am wondering if whatever I am looking is even achievable in this case. i.e. is there any official docker files as gitlab runners for windows and macOS is available to support my current architecture? or any customization of files on runner managers end?
Related
I'm currently using resource: container in azure pipeline to pull my custom image tools from ACR, and use this image to create a container than can run several CLI commands on my pipeline.
Pulling this custom image tools takes so much time, roughly around 5mins and I want to avoid that as it is considered as a wasted time and a blockage since I do debugging most of the time.
Question: Is it possible to create an Azure Container Instance that constantly run my custom image tools then call this container inside my pipeline to run some cli commands?
I'm having a hard time finding documentation, so I'm not really sure if it is possible.
You can try to set up a self-hosted agent in your custom Docker image, and when running the container on ACI, it will install the agent and connect it to your Azure DevOps.
Then, you can use this self-hosted agent to run your pipeline job which needs to run the CLI commands. Since the self-hosted agent is hosted in the container, the job runs on the self-hosted agent will run in the container.
About how to configure set up self-hosted agent in Docker image, you can reference the document "Run a self-hosted agent in Docker".
I have following image configuration in my gitlab-ci.yml:
default:
image: registry.gitlab.com/gitlab-org/terraform-images/stable:latest
#image: curlimages/curl:latest
...
this works fine when I am deploying on https://gitlab.com/, however when you try to deploy my code along with above ci configuration I get following errors in my CICD:
This job is stuck because the project doesn't have any runners online assigned to it.
Go to project CI settings
My question is while using https://gitlab.com/ I didn't specifically assigned any runners to my project through settings. But now it seems I have supposed to do that.
Why is that?
And if this is necessary how can I do this?
When using gitlab.com -- the gitlab instance has shared runners configured available to run all untagged CI jobs.
On your own self-hosted gitlab you must either configure your own shared runners for your instance or register runners to your projects/groups.
You cannot use the gitlab.com shared runners on a self-hosted gitlab instance.
From scope of runners:
Shared Runners
Shared runners are available to every project in a GitLab instance.
Use shared runners when you have multiple jobs with similar requirements. Rather than having multiple runners idling for many projects, you can have a few runners that handle multiple projects.
If you are using a self-managed instance of GitLab:
Your administrator can install and register shared runners by going to your project’s Settings > CI/CD, expanding the Runners section, and clicking Show runner installation instructions. These instructions are also available in the documentation.
The administrator can also configure a maximum number of shared runner pipeline minutes for each group.
If you are using GitLab.com:
You can select from a list of shared runners that GitLab maintains.
The shared runners consume the pipelines minutes included with your account.
I suppose you technically would be able to use public GitLab runners for your self-hosted instance if you create an account on gitlab.com and setup CICD for external repos pointing to your self-hosted instance -- but your minutes would be a separate entitlement from your self-hosted license, among other serious limitations.
I am trying to install and configure spinnaker on azure kubernetes. I don't find any documentation clearly explaining the process.
Is there any documentation available on installing and configuring spinnaker on azure kubernetes?
Azure quickstart templates contains spinnaker Continuous Deployment to Kubernetes template that
allows you to deploy and configure a DevOps pipeline from an Azure
Container Registry to a Kubernetes cluster. It deploys an instance of
Jenkins on a Linux Ubuntu 14.04 LTS VM and an instance of Spinnaker on
the same Kubernetes cluster that your pipeline will target.
The Jenkins instance will include a basic pipeline that checks out a
user-provided git repository, builds the Docker container based on the
Dockerfile at the root of the repo, and pushes the image to the
provisioned Azure Container Registry. The Spinnaker instance will
include a basic pipeline that is triggered by any new tag in the
registry and deploys the image to the provisioned Kubernetes cluster.
also check spinnaker on azure kubernetes service
I would like to know if the community edition of GitLab (Self hosted) has support for a CI process that would include:
Creation of docker containers (docker-compose or Dockerfile)
Save custom docker images to a docker Registry in GitLab
Integrate ansible To manage configuration items
I checked the pricing / feature compare section of their website but I wasn’t able to find this info. Wondering if any fellow stack overflowers that are using this framework already can help out.
Thanks
You can sign up for a free GitLab CE account and try it out. Here's what I found:
Docker Images and Registry
GitLab CE comes with a Package Registry (Maven, NuGet, PyPi, etc.) and Container Registry (for Docker images). However, I don't think GitLab will provide free storage for your packages. The benefit is full integration in your CI/CD pipeline. You'll need to host your own server, as described in How to Build Docker Images and Host a Docker Image Repository with GitLab. The official docs are at GitLab Container Registry.
Once you've setup your Docker images/registry, you can configure GitLab CE to run various jobs in Docker containers. It's as easy as specifying the name of the image in an image field in gitlab-ci.yml. The images will run on an application called a Runner on a VM or bare metal. You can setup your own Runners in AWS/GCP/Azure or even on your own laptop, but GitLab CE also provides 2000 free pipeline-minutes per month hosted on GCP. Instructions for setting up GitLab Runners can be found at https://docs.gitlab.com/runner/.
Ansible Integration
Once you get your docker images/registry and runners set up, you can store the Ansible binaries and dependencies in the docker images. You can execute the playbooks from the script section of a job defined in gitlab-ci.yml. Using Ansible and GitLab a Infrastructure-for-Code. You can read some great tutorials here and here.
We are running AWS ECS/ECR with Gitlab CI as CI/CD.
Due to load increase we are searching for the best way to autoscale the runner on AWS.
I know Gitlab supports Autoscaling for its ci-runner. https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/
But I'm wondering if it would be possible to leverage the ECS cluster on AWS for this purpose. Has anybody ever set up the runner on an ECS cluster with Loadbalancer and Autoscaling on ECS side and could provide some insights regarding such a setup?
merci in advance
A
Since each CI task is usually run in a container (or other isolated environment), this would require the GitLab runner to natively talk to ECS to spin up new containers for jobs
I don't think it's gonna happen; GL supports Kubernetes, which is more versatile in that it's not tied to AWS, and also support EC2 autoscaling for AWS. ECS sounds like a lot of extra effort for a very narrow benefit.