In our project we have a number of repositories that are all needed to build a few different docker containers.
The Dockerfiles of all containers are located in a single repository.
I would like to be able to pull all repositories I need for the builds before building the containers. This would avoid pulling a repository multiple times which can take almost a minute for some of these.
Essentially what I would like to do is:
stages:
- initialize
- build
repo1 pull:
stage: initialize
script:
- git clone https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.instance/group/repo1.git
repo2 pull:
stage: initialize
script:
- git clone https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.instance/group/repo2.git
image1:
stage: build
script:
- cp repo1/ image1/
- cd image1
- docker build -t image1 --network=host .
- docker login -u gitlab-runner -p gitlab docker.registry
- docker push docker.registry/image1:latest
image2:
stage: build
script:
- cp repo2/ image2/
- cd image2
- docker build -t image2 --network=host .
- docker login -u gitlab-runner -p gitlab docker.registry
- docker push docker.registry/image2:latest
What is the best way to share the repositories over the stages while still allowing for parallel building within the stage?
I'm still learning Gitlab-ci but maybe cache would help?
Add this before you pull the repos and the runner should check if the repo is already in the cache.
cache:
paths:
- repo1/
- repo2/
Gitlab cache reference: https://docs.gitlab.com/ee/ci/caching/
Edit:
Or maybe using Git submodules for the dependant repos would be a good idea?
The runner can then be configured to fetch those submodules to the workspace and you should have access to them throughout the ci job.
You need to add a .gitmodules file that would looks something like this:
[submodule "repo1"]
path = repo1
url = ../repo1.git
[submodule "repo2"]
path = repo2
url = ../repo2.git
And then in your .gitlab-ci.yml
variables:
GIT_SUBMODULE_STRATEGY: normal
Gitlab submodule reference: https://docs.gitlab.com/ce/ci/git_submodules.html
Related
I want to run a pipeline that builds a docker image from an app and deploys it to Gitlab Registry and Docker Hub.
I'd like this pipeline to run only when the main branch gets a commit and when that commit has a message that is a "version". Examples of versions:
1.0.0
3.4.0
10.1.6
I have made the following gitlab-ci.yml, however, my pipeline never triggers. What am I doing wrong? I've already checked the regex and it's ok, so I'm guessing I'm doing something wrong with the Gitlab config?
image: docker:stable
variables:
PROJECT_NAME: "project"
BRANCH_NAME: "main"
IMAGE_NAME: "$PROJECT_NAME:$CI_COMMIT_MESSAGE"
LATEST_IMAGE_NAME: "$PROJECT_NAME:latest"
services:
- docker:19.03.12-dind
build_image:
script:
# Push to Gitlab registry
- docker build -t $CI_REGISTRY/username/$PROJECT_NAME/$IMAGE_NAME .
- docker build -t $CI_REGISTRY/username/$PROJECT_NAME/$LATEST_IMAGE_NAME .
- docker login $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD
- docker push $CI_REGISTRY/username/$PROJECT_NAME/$IMAGE_NAME
- docker push $CI_REGISTRY/username/$PROJECT_NAME/$LATEST_IMAGE_NAME
# Push to Docker hub
- docker build -t username/$IMAGE_NAME .
- docker build -t username/$LATEST_IMAGE_NAME .
- docker login -u username-p $DOCKER_HUB_TOKEN
- docker push username/$IMAGE_NAME
- docker push username/$LATEST_IMAGE_NAME
rules:
- if: $CI_COMMIT_MESSAGE =~ /^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$/
- if: $CI_PIPELINE_SOURCE == "$BRANCH_NAME"
The problem you're having is likely due to the $ marker. Often commit messages get formatted with additional newlines, which would make the pattern not match.
Alternatively, your pattern may work if using CI_COMMIT_TITLE which only includes the first line of the commit message.
I propose to you an alternative, that is tags.
A tag is a label that you attach on git commits. Examples of tags are: 1.0.1, 1.1.2, latest.
In order to tag a commit with a version, just issue:
git tag <version> (the newest commit is subtended here)
In order to trigger your pipeline for any tag, just use this:
rules:
- if: $CI_COMMIT_TAG
In this solution, we have decoupled the version issue from the message concern.
I have a remote server where I serve my project via Nginx. I am using Gitlab CI to automatize my deploy process and I have ran into a problem. When I push my commits to the master branch the gitlab-runner run nicely but the problem is that it removes my React build folder (it is ok, as I have put it into the .gitignore), but because it always removes my build folder my Nginx could not serve any files until the build finish, and a new build folder creaeted. Is is there any solution for this problem? It would be nice if I could cache my build file until the build process finish. I attached my gitlab.ci.yml. Thank's in advance!
image: docker:latest
services:
- docker:dind
stages:
- build
- deploy
variables:
GIT_SSL_NO_VERIFY: "1"
build-step:
stage: build
tags:
- shell
script:
- docker image prune -f
- docker-compose -f docker-compose.yml -f docker-compose.prod.yml build
deploy-step:
stage: deploy
tags:
- shell
script:
- docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d
It should be possible to use git fetch and to disable git clean when your deploy job starts. Here are links for the variables to do this:
https://docs.gitlab.com/ee/ci/yaml/#git-clean-flags
https://docs.gitlab.com/ee/ci/yaml/#git-strategy
It would look something like this:
deploy-step:
variables:
GIT_STRATEGY: fetch
GIT_CLEAN_FLAGS: none
stage: deploy
tags:
- shell
script:
- docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d
This should make GitLab use git fetch instead of git clone, and to not run any git clean ... commands. The build artifacts from your previous run should then not be removed.
There are some problems with this though. If something goes wrong with a build, you might end up in a situation where you will have to manually log into the server where the runner is to fix it. The reason that GitLab uses git clean is to prevent these types of problems.
A more proper solution is to use nginx to have a sort of dubble buffer. You can have two different build folders, change the config in nginx, and then send a signal to nginx to reload the config. nginx will then make sure to gracefully switch to the new version of your application, without any interruptions. Here is a link to someone that has done this:
https://syshero.org/2016-06-09-zero-downtime-deployments-using-nginx/
I am using Gitlab for repository and ci/cd.
Currently trying to set up pipeline that creates a docker image from a build stage.
All examples I have seen have a simple naming of image where e.g. the branch is used(master)
My question is if I want to tag the image based on the current tag in the repository how do I do this?
I am presuming I can use a Gitlab runner variable but do not see one to us
There are a lot of predefined variables in Gitlab CI. I think you are looking for CI_COMMIT_TAG.
So you could use it this way:
docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
So the image would look like registry.example.com/group/project:tag
As shell command is not possible yet with variables inside .gitlab-ci.yml, you may edit a build script that get the current tag and build the image inside that script
Both file at the root of your project :
build.sh :
#!/bin/sh
IMAGE="$CI_REGISTRY/$CI_PROJECT_PATH:$CI_COMMIT_REF_NAME-$(git describe --abbrev=0 --tags)"
docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
docker build --pull -t $IMAGE .
docker push $IMAGE
.gitlab-ci.yml :
image: docker:latest
services:
- docker:dind
stages:
- release
release:
stage: release
script:
- apk update && apk add git
- ./build.sh
I have a problem. I have two merge requests from two different branches to the master branch in a project.
Now I want to start an environment in GitLab for each merge request. I do this with a shell executor and I start docker container with docker run image_name where I mount the folder from the build process inside the container. Looks like this:
stages:
- deploy
deploy_stage:
stage: deploy
script:
- docker run -d --name ContainerName -v ${CI_PROJECT_DIR}:/var/www/html -e VIRTUAL_HOST=example.com php
environment:
name: review/$CI_COMMIT_REF_NAME
url: http://example.com
on_stop: stop_stage
tags:
- shell
except:
- master
stop_stage:
stage: deploy
variables:
GIT_STRATEGY: none
script:
- docker stop ContainerName
- docker rm ContainerName
when: manual
environment:
name: review/$CI_COMMIT_REF_NAME
action: stop
tags:
- shell
Now my problem is that one environment is runng and when a new job runs the checkout/code get overwritten by the new pipeline job and both environments have now the same code but they should be different.
Does anyone have a solution for me how I can configure the gitlab runner to have different checkout folder for each merge request?
I want to build and add a custom image (with ruby, node.js, bower, grunt, jekyll etc.) and tag it as 'myimage:1.0'. This image needs to be stored in gitlab container registry and then used in .gitlab-ci.yml as image: sachin.1.0.0. So that my build via gitlab ci will have everything preinstalled like node.js, etc.
Tried enough, How can this be done ?
Before you do this, you need to configure a gitlab runner which allows you to use docker build. You can configure this using the instructions here depending on your use case
Next, create a new repo in gitlab, let's call it sachin-image.
Inside the root of the git repo, add a Dockerfile with installation of everything you need.
Now, into this repo, add a .gitlab-ci.yml file like so:
---
before_script:
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN <my-docker-gitlab-registry-url>
stages:
- build
build_image:
stage: build
script:
- docker build -t gitlab.example.com/my/dockerimage/repo:latest .
- docker push gitlab.example/my/dockerimage/repo:latest
tags:
- docker_engine
At this point, you now have automated docker builds working in gitlab. In order to use this image in future gitlab builds, all you need to use the following image url:
gitlab.example.com/my/dockerimage/repo:latest