I have the following .gitlab-ci.yml file
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- ssh-add /root/gitlab-runner/.ssh/id_rsa
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
- apt-get update -qq && apt-get install -y -qq apt-utils sqlite3 libsqlite3-dev nodejs tree
- gem install bundler --no-ri --no-rdoc
- bundle install --jobs $(nproc) "${FLAGS[#]}"
- cp /root/gitlab-runner/.database.gitlab-ci.yml config/database.yml
- RAILS_ENV=test rake parallel:setup
rspec:
script:
- rake parallel:spec
The issue is that we have so many projects using the exact same before_script actions and these actions sometimes change, so we have to update this file for every project. Is there a way to automatically configure the runner to execute actions so that the .gitlab-ci.yml in this case becomes:
rspec:
script:
- rake parallel:spec
You can save all the before_script commands into a Bash script, store it on the server hosting the runner and then just reference it in all the projects:
before_script:
- /[path on the host]/script.sh
If you are using Docker, you can either include the file in your own image or use volumes to mount the host directory in the Docker container.
It would be a bit more complicated in case you have multiple runners on different servers.
Related
The following as the pipeline we are using to deploy the App to the Debian server.
stages:
- deploy
deploy-job: # This job runs in the deploy stage.
stage: deploy # It only runs when *both* jobs in the test stage complete successfully.
environment: production
image: node:latest
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$SSH_PRIVATE_KEY")
- mkdir ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_SERVER_HOSTKEYS" > ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- echo "Deploying application..."
- ssh $SSH_USER#$SSH_IP "cd $PROEJCT_PATH/$PROJECT_DIRECTORY_NAME && serve -s build"
- echo "Application successfully deployed."
But this throws the following Error Message:
file:///usr/local/lib/node_modules/serve/build/main.js:169
const ipAddress = request.socket.remoteAddress?.replace("::ffff:", "")
?? "unknown"; ^
SyntaxError: Unexpected token '.' at Loader.moduleStrategy
(internal/modules/esm/translators.js:133:18) at async link
(internal/modules/esm/module_job.js:42:21)
We had the same issue after installing node on the Debian Server but after updating it with nvm install 19.4.0it fixed the problem
The command serve -s build did work then on the server but it does not work in the pipeline.
We are discussing the possibility that the container is using is his own environment but we are not sure with that assumption.
Can some help and explain the problem.
The error comes from the "new" optional chaining javascript feature ?. which is supported from v14. However default debian 11 node version is v12. Thus you should update node to newest version
sudo apt remove node npm nodejs #remove them old node/npm
sudo snap install node --classic #see https://snapcraft.io/node
or use instead nvm like you have done; to install specific node version
i want to run a script that is needed for my test_integration and build stage. Is there a way to specify this in the before script so i don't have to write it out twice.
before_script:
stage: ['test_integration', 'build']
this does not seem to work i get the following error in gitlab ci linter.
Status: syntax is incorrect
Error: before_script config should be an array of strings
.gitlab-ci.yml
stages:
- security
- quality
- test
- build
- deploy
image: node:10.15.0
before_script:
stage: ['test_integration', 'build']
script:
- apt-get update
- apt-get -y install apt-transport-https ca-certificates curl gnupg2 software-properties-common
- curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
- add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
- apt-get update
- apt-get -y install docker-ce
- curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
- chmod +x /usr/local/bin/docker-compose
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
services:
- mongo
- docker:dind
security:
stage: security
script:
- npm audit
quality:
stage: quality
script:
- npm install
- npm run-script lint
test_unit:
stage: test
script:
- npm install
- npm run-script unit-test
test_integration:
stage: test
script:
- docker-compose -f CI/backend-service/docker-compose.yml up -d
- npm install
- npm run-script integration-test
build:
stage: build
script:
- npm install
- export VERSION=`git describe --tags --always`
- docker build -t $CI_REGISTRY_IMAGE:$VERSION .
- docker push $CI_REGISTRY_IMAGE
deploy:
stage: deploy
script: echo 'deploy'
The before_script syntax does not support a stages section. You could use before_script as you have done without the stages section, however the before_script stage would run for every single job in the pipeline.
Instead, what you could do is use GitLab's anchor's feature, which allows you to duplicate content across the .gitlab-ci file.
So in your scenario, it would look something like:
stages:
- security
- quality
- test
- build
- deploy
image: node:10.15.0
.before_script_template: &build_test-integration
before_script:
- apt-get update
- apt-get -y install apt-transport-https ca-certificates curl gnupg2 software-properties-common
- curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
- add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
- apt-get update
- apt-get -y install docker-ce
- curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
- chmod +x /usr/local/bin/docker-compose
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
services:
- mongo
- docker:dind
security:
stage: security
script:
- npm audit
quality:
stage: quality
script:
- npm install
- npm run-script lint
test_unit:
stage: test
script:
- npm install
- npm run-script unit-test
test_integration:
stage: test
<<: *build_test-integration
script:
- docker-compose -f CI/backend-service/docker-compose.yml up -d
- npm install
- npm run-script integration-test
build:
stage: build
<<: *build_test-integration
script:
- npm install
- export VERSION=`git describe --tags --always`
- docker build -t $CI_REGISTRY_IMAGE:$VERSION .
- docker push $CI_REGISTRY_IMAGE
deploy:
stage: deploy
script: echo 'deploy'
Edit: there is another way, instead of using anchors, you could also use extends syntax:
.before_script_template:
before_script:
- apt-get update
- apt-get -y install apt-transport-https ca-certificates curl gnupg2 software-properties-common
- curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
- add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
- apt-get update
- apt-get -y install docker-ce
- curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
- chmod +x /usr/local/bin/docker-compose
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
test_integration:
extends: .before_script_template
stage: test
script:
- docker-compose -f CI/backend-service/docker-compose.yml up -d
- npm install
- npm run-script integration-test
build:
extends: .before_script_template
stage: build
script:
- npm install
- export VERSION=`git describe --tags --always`
- docker build -t $CI_REGISTRY_IMAGE:$VERSION .
- docker push $CI_REGISTRY_IMAGE
etc
I'm currently facing a problem with the gitlab.com shared-runners. What I'm trying to archieve in my pipeline is:
- NPM install and using grunt to make some uncss, minimize and compress tasks
- Cleaning up
- Building a docker container with the app included
- Moving the container to gitlab registry
Unfortunateley I don't get it running since a long time! I tried a lot of different gitlab.ci configs - without success. The problem is, that I have to use the "image: docker:latest" to have all the docker-tools running. But then I don't have node and grunt installed in the container.
Also the other way around is not working. I was trying to use image: centos:latest and install docker manually - but this is also not working as I always just get a Failed to get D-Bus connection: Operation not permitted
Does anyone has some more experience on the gitlab-ci using docker build commands in a docker shared runner?
Any help is highly appreciated!!
Thank you
Jannik
Gitlab can be a bit tricky :) I dont have an example based on CentOS, but I have one based on Ubuntu if that helps you. Here is some copy paste of a working gitlab pipeline of mine which uses gulp (you should be easily able to adjust it to work with your grunt).
The .gitlab-ci.yml looks like this (adjust the CONTAINER... variables at the beginning):
variables:
CONTAINER_TEST_IMAGE: registry.gitlab.com/psono/psono-client:$CI_BUILD_REF_NAME
CONTAINER_RELEASE_IMAGE: registry.gitlab.com/psono/psono-client:latest
stages:
- build
docker-image:
stage: build
image: ubuntu:16.04
services:
- docker:dind
variables:
DOCKER_HOST: 'tcp://docker:2375'
script:
- sh ./var/build-ubuntu.sh
- docker info
- docker login -u gitlab-ci-token -p "$CI_BUILD_TOKEN" registry.gitlab.com
- docker build -t $CONTAINER_TEST_IMAGE .
- docker push $CONTAINER_TEST_IMAGE
in addition i have a this "./var/build-ubuntu.sh" which you can adjust a bit according to your needs, replace some ubuntu dependencies or switch gulp for grunt as needed:
#!/usr/bin/env bash
apt-get update && \
apt-get install -y libfontconfig zip nodejs npm git apt-transport-https ca-certificates curl openssl software-properties-common && \
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - && \
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable" && \
apt-get update && \
apt-get install -y docker-ce && \
ln -s /usr/bin/nodejs /usr/bin/node && \
npm install && \
node --version && \
npm --version
I'm trying to build Jekyll blog using gitlab runner (for gitlab pages). I get the following error: ERROR: Build failed: exit code 1. So far, everything worked. Link to project: https://gitlab.com/dash.plus/dashBlog
Just add
- apt-get update && apt-get install -y nodejs
And ofc
- bundle install
inside gitlab-cl.yaml
image: ruby:2.3
test:
stage: test
script:
- gem install jekyll
- bundle install
- apt-get update && apt-get install -y nodejs
- bundle exec jekyll -d test/
artifacts:
paths:
- test
except:
- master
pages:
stage: deploy
script:
- gem install jekyll
- bundle install
- apt-get update && apt-get install -y nodejs
- bundle exec jekyll -d public/
artifacts:
paths:
- public
only:
- master
I have the following configuration as .gitlab-ci.yml
but I found out after successfully pass build stage (which
would create a virtualenv called venv), it seems that
in test stage you would get a brand new environment(there's
no venv directory at all). So I wonder should I put setup
script in before_script therefor it would run in each phase(build/test/deploy). Is it a right way to do it ?
before_script:
- uname -r
types:
- build
- test
- deploy
job_install:
type: build
script:
- apt-get update
- apt-get install -y libncurses5-dev
- apt-get install -y libxml2-dev libxslt1-dev
- apt-get install -y python-dev libffi-dev libssl-dev
- apt-get install -y python-virtualenv
- apt-get install -y python-pip
- virtualenv --no-site-packages venv
- source venv/bin/activate
- pip install -q -r requirements.txt
- ls -al
only:
- master
job_test:
type: test
script:
- ls -al
- source venv/bin/activate
- cp crawler/settings.sample.py crawler/settings.py
- cd crawler
- py.test -s -v
only:
- master
adasd
Gitlab CI jobs supposed to be independent, because they could run on different runners. It is not issue. There two ways to pass files between stages:
The right way. Using artefacts.
The wrong way. Using cache. With cache key "hack". Still need same runner.
So yes, supposed by gitlab way to have everything your job depends on in before script.
Artifacts example:
artifacts:
when: on_success
expire_in: 1 mos
paths:
- some_project_files/
Cache example:
cache:
key: "$CI_BUILD_REF_NAME"
untracked: true
paths:
- node_modules/
- src/bower_components/
For correct running environment i suggest using docker with image containing apt-get dependencies. And use artefacts for passing job results between jobs. Note that artefact also uploaded to gitlab web interface and being able to download them. So if they are quite heavy use small expire_in time, for removing them after all jobs done.