I use Gitlab runner and works fine for a single server. The gitlab-ci.yml is simple:
stages:
- test
- deploy
test:
stage: test
image: php
tags:
- docker
script:
- echo "Run tests..."
deploy:
stage: deploy
tags:
- shell
script:
- sh deploy.sh
As i said this is fine for a single server but to deploy same app on another server? I tried with same gitlab-runner config (same conf.toml) but then it was only updating one of them randomly.
Is there somehow gitlab Ci to be triggered by more than 1 runner and deploy all of them according gitlab-ci.yml?
You can register several runners (e.g. tagged serverA and serverB) from different servers and have multiple deployment jobs, each of them performed by a different runner. This is because you can set more than one tag in a job and only a runner having all the tags will be used.
stages:
- test
- deploy
test:
stage: test
image: php
tags:
- docker
script:
- echo "Run tests..."
deployA:
stage: deploy
tags:
- shell
- serverA
script:
- sh deploy.sh
deployB:
stage: deploy
tags:
- shell
- serverB
script:
- sh deploy.sh
However, take into account a situation when one of the deployment jobs fails - this would end up in you having two different versions of the code on the servers. Depending on your situation this might or might not be a problem.
Yes there is, just set up two jobs for the same stage:
stages:
- deploy
deploy:one:
stage: deploy
script:
- echo "Hello CI one"
deploy:two:
stage: deploy
script:
- echo "Hello CI two"
If necessary you can use tags on your runners to choose which one to use.
Since 2016, you now have Environments and deployments
Environments describe where code is deployed.
Each time GitLab CI/CD deploys a version of code to an environment, a deployment is created.
GitLab:
Provides a full history of deployments to each environment.
Tracks your deployments, so you always know what is deployed on your servers.
It does integrates well with Prometheis, and, with GitLab 13.11 (April 2021), you even have:
Update a deploy freeze period in the UI
In GitLab 13.2, we added the ability to create a deploy freeze period in the project’s CI/CD settings.
This capability helps teams avoid unintentional deployments, reduce uncertainty, and mitigate deployment risks.
However, it was not possible to update deploy freezes.
In GitLab 13.11, we are adding the ability to edit an existing deploy freeze. This way, you can update the freeze period to match your business needs.
See Documentation and Issue.
As shown in "gitlab-ci.yml deployment on multiple hosts", you can use YAML anchors to trigger parallel deployment on multiple environments, which means "multiple servers".
Related
We have multiple environments (staging, production...) and I don't want to put sentitive informations like database passwords inside the codebase. For this, I want to use environment variables provided by GitLab CI/CD.
However I don't know how to tell GitLab to run a different set of variables depending on my environment.
What I've done so far:
1- Create environments : Via UI (Project => Operations => Environments : Here I created 2 environments, STAGING and PRODUCTION
2- Create variables Via UI (Project => Settings => CI/CD => Variables : Here I created the variable DB_PASSWORD twice (with of course different values assigned) , one with environment scope set to STAGING, the other one to PRODUCTION.
Now what I want to do is run my project's pipeline. So I go to Project => CI/CD => Pipelines => Run Pipeline and here I expect GitLab CI to ask me if I would like to run my pipeline with the set of variables set for STAGING or PRODUCTION but it doesn't.
How I am supposed to tell GitLab that I want to run my pipeline using DB_PASSWORD variable with the value corresponding to the environment I want to target?
You need to specify the environment in your gitlab-ci.yml file, see here
Example from official gitlab docs:
stages:
- test
- build
- deploy
test:
stage: test
script: echo "Running tests"
build:
stage: build
script: echo "Building the app"
deploy_staging:
stage: deploy
script:
- echo "Deploy to staging server"
environment:
name: staging
url: https://staging.example.com
only:
- master
In this example when running deploy_staging the environment is set to staging and thus you can access the defined Variables for the environment, like so:
deploy_staging:
stage: deploy
script:
- echo "Deploy to staging server"
environment:
name: staging
url: https://staging.example.com
variables:
DB_PASS: ${DB_PASSWORD} # which is your defined variable within Gitlab CI
only:
- master
My CI has two main steps. Build and deploy. The result of build is that an artifact is uploaded to maven nexus. And currently manual deploy step just takes the latest artifact from nexus and deploys it.
stages:
- build
- deploy
full:
stage: build
image: ubuntu
script:
- // Build and upload to nexus here
deploy:
stage: deploy
script:
- // Take latest artifact from nexus and deploy
when: manual
But to me this doesn't seem to make that much sense to always deploy latest build from every pipeline. I think ideally deploy step of each pipeline should deploy the artifact that was build by the same pipelines build task. Otherwise deploy step of each pipeline will do exactly the same thing regardless when it is started.
So I have two questions.
1) How can I make my deploy step to deploy the version that was build by this run?
2) If I still want to keep the "deploy latest" functionality, then does gitlab support adding a task separate of each pipeline because as I explained this step doesn't make a lot of seance to be in pipeline? I imagine it being in a separate specific place.
Not too familiar with maven and nexus, but assuming you can name the artifact before you push it, you can add one of the built-in environment variables that dictates which pipeline it's from.
ie:
...
Build:
stage: build
script:
- ./buildAsNormal.sh > build$CI_PIPELINE_ID.extension
- ./pushAsNormal.sh
Deploy:
stage: deploy
script:
- ./deployAsNormal #(but specify the build$CI_PIPELINE_ID.extension file)
There are a lot of CI env variables you can use that are extremely useful. The full list of them is here. The difference with $CI_PIPELINE_ID and $CI_JOB_ID is that the pipeline id is constant for all jobs in the pipeline, no matter when they execute. That means the pipeline id will be the same even if you run a manual step a week after the automated steps. The job id is specific to each job.
Regarding your comment, the usage of artifacts: can solve your problem.
You can put the version number in a file and get the file in the next stage :
stages:
- build
- deploy
full:
stage: build
image: ubuntu
script:
- echo "1.0.0" > version
- // Build and upload to nexus here
artifacts:
paths:
- version
expire_in: 1 week
deploy:
stage: deploy
script:
- VERSION = $(cat version)
- // Take the artifact from nexus using VERSION variable and deploy
when: manual
An alternative is to build, push to nexus and use artifact: to pass the result of the build to the Deploy job :
stages:
- build
- deploy
full:
stage: build
image: ubuntu
script:
- // Build and put the result in out/ directory
- // Upload the result from out/ to nexus
artifacts:
paths:
- out/
expire_in: 1 week
deploy:
stage: deploy
script:
- // Take the artifact from in out/ directory and deploy it
when: manual
I have a free account of gitlab.
I also have a company account (not sure which plan).
I have the exact same project, a wrapper on EventStore.
In the CI pipeline I want to spin up a container with event store so that I can run some integration tests against it.
This is my .gitlab-ci.yml that restores, compiles, runs tests and publishes nuget packages
#Stages
stages:
- ci
- pack
#Global variables
variables:
GITLAB_RUNNER_DOTNET_CORE: mcr.microsoft.com/dotnet/core/sdk:2.2
EVENT_STORE: eventstore/eventstore:release-5.0.2
NUGET_REPOSITORY: $NEXUS_NUGET_REPOSITORY
NUGET_API_KEY: $NEXUS_API_KEY
NUGET_FOLDER_NAME: nupkgs
#Docker image
image: $GITLAB_RUNNER_DOTNET_CORE
#Jobs
ci:
stage: ci
services:
- $EVENT_STORE
variables:
# event store service params testing with standard ports
EVENTSTORE_INT_TCP_PORT: "1113"
EVENTSTORE_EXT_TCP_PORT: "1113"
EVENTSTORE_INT_HTTP_PORT: "2113"
EVENTSTORE_EXT_HTTP_PORT: "2113"
EVENTSTORE_EXT_HTTP_PREFIXES: "http://*:2113/"
script:
- dotnet restore --no-cache --force
- dotnet build --configuration Release
- dotnet vstest test/*Tests/bin/Release/**/*Tests.dll
pack-beta-nuget:
stage: pack
script:
- export VERSION_SUFFIX=beta$CI_PIPELINE_ID
- dotnet pack *.sln --configuration Release --output $NUGET_FOLDER_NAME --version-suffix $VERSION_SUFFIX --include-source --include-symbols -p:SymbolPackageFormat=snupkg
- dotnet nuget push **/*.nupkg --api-key $NUGET_API_KEY --source $NUGET_REPOSITORY
except:
- master
pack-nuget:
stage: pack
script:
- dotnet restore
- dotnet pack *.sln --configuration Release --output $NUGET_FOLDER_NAME
- dotnet nuget push **/*.nupkg --api-key $NUGET_API_KEY --source $NUGET_REPOSITORY
only:
- master
As you can see, I spin up the event store container.
From my integration tests I try to connect to the container within the CI using the following connection string:
"ConnectTo=tcp://admin:changeit#127.0.0.1:1113; HeartBeatTimeout=500;";
With my work account it works fine, there is a container listening on 127.0.0.1 on port 1113 and I can connect to it using the above connection string.
With my free personal account it is unable to connect.
Why?
I suspect it has something to do with the way docker is available on both gitlab CI runners, but why is different?
And more important, how can I configure event store on my personal CI pipeline on my free account so that I can connect to it if the localhost is not a valid host Uri for whatever reason?
Well, you have not provided any details but it seems you're using the Docker executor. In that case, services are not available on localhost but only accessible as service aliases.
This is an extract from the working CI file:
test:
stage: test
script:
- dotnet test
variables:
ASPNETCORE_ENVIRONMENT: Testing
EVENTSTORE_EXT_HTTP_PORT: 2113
EVENTSTORE_EXT_TCP_PORT: 1113
EVENTSTORE_RUN_PROJECTIONS: all
EVENTSTORE_START_STANDARD_PROJECTIONS: "true"
EventStore__ConnectionString: ConnectTo=tcp://admin:changeit#eventstore:1113
services:
- name: eventstore/eventstore:latest
alias: eventstore
only:
refs:
- branches
- tags
For this to work, your appsettings.Testing.json file needs to point to ConnectTo=tcp://admin:changeit#eventstore:1113.
If you want to keep using the appsettings file with the configuration that points to localhost, you can override the setting using env variable in the CI file. Just remember to add environment variables as the configuration source. The code snippet above has such an override, matching our settings structure:
{
"EventStore": {
"ConnectionString": "ConnectTo=whatever"
}
}
If you ever decide using the Kubernetes executor, you will need to revert to using localhost, because Kubernetes executor creates one pod per build with multiple containers, including all service containers. There's an open issue to support service aliases with Kubernetes runners, I think it will be like 12.9 or 13, pretty soon. That being said, using service aliases is a safe, future proof way of making it all work.
P.S. Just noticed that your setup works with one account and doesn't work with another. My guess would be that you either use different executors (Docker doesn't work and Kubernetes works) or different GitLab versions (if the service alias issue has already been fixed).
I run an end2end test in gitlab-CI , see https://docs.cypress.io/guides/guides/continuous-integration.html.
I run it after I deploy my app.
It works well but I want to change the base url in order to run it against my prod or my staging env. It is possible via an environment var passed to the test.
I don’t want to write a test job per environment, then I would like to get the environment URL via env var, but the $CI_ENVIRONMENT_URL is only available on the deploy job, not in the next one.
deploy-prod:
stage: deploy
script:
- some commands
environment:
name: prod
url: http://myprod.com
only:
- master
deploy-staging:
stage: deploy
script:
- some other commands
environment:
name: staging
url: http://mystaging.com
only:
- staging
test:
stage: after-deploy
script:
- CYPRESS_baseUrl=$CI_ENVIRONMENT_URL cypress run
I expect $CI_ENVIRONMENT_URL equal http://mystaging.com or http://myprod.com depending the previous deploy job has been run. But it is empty, seems $CI_ENVIRONMENT_URL is only available in deploy job.
Is it possible to pass a variable from on job to a next job?
You can use artifacts feature: write the $CI_ENVIRONMENT_URL in a file:
echo $CI_ENVIRONMENT_URL > environmentUrl.txt
save it as artifact, and then read it in the next job:
$CI_ENVIRONMENT_URL=`cat environmentUrl.txt`
We have a project hosted on an internal Gitlab installation.
The Pipeline of the project has 3 stages:
Build
Tests
Deploy
The objective is to hide or disable the Deploy stage when Tests fails
The problem is that we can't use artifacts because they are lost each time our machines reboot.
My question: Is there an alternative solution to artifacts to achieve this task?
The used .gitlab-ci.yml looks like this:
stages:
- build
- tests
- deploy
build_job:
stage: build
tags:
# - ....
before_script:
# - ....
script:
# - ....
when: manual
only:
- develop
- master
all_tests:
stage: tests
tags:
# - ....
before_script:
# - ....
script:
# - ....
when: manual
only:
- develop
- master
prod:
stage: deploy
tags:
# - ....
script:
# - ....
when: manual
environment: prod
I think you might have misunderstood the purpose of the built-in CI. The goal is to have building and testing all automated on each commit or at least every push. Having all tasks set to manual execution gives you almost no advantage over external CI tools like Jenkins or Bamboo. Your only advantage to local execution of the targets right now is having visibility in a central place.
That said there is no way to conditionally show or hide CI tasks, because it's against the basic idea. If you insist on your idea, you could look up the artifacts of the previous stages and abort the manual execution in case something is wrong.
The problem is that we can't use artifacts because they are lost each time our machines reboot
AFAIK artifacts are uploaded to the master and not saved on the runners. You should be fine having your artifacts passed from stage to stage.
By the way, the default for when is on_success which means to execute build only when all builds from prior stages succeed.