gitlab ci cache/keep golang packages between stages - gitlab

I use gitlab-ci to test, compile and deploy a small golang application but the problem is that the stages take longer than necessary because they have to fetch all of the dependencies every time.
How can I keep the golang dependencies between two stages (test and build)?
This is part of my current gitlab-ci config:
test:
stage: test
script:
# get dependencies
- go get github.com/foobar/...
- go get github.com/foobar2/...
# ...
- go tool vet -composites=false -shadow=true *.go
- go test -race $(go list ./... | grep -v /vendor/)
compile:
stage: build
script:
# getting the same dependencies again
- go get github.com/foobar/...
- go get github.com/foobar2/...
# ...
- go build -race -ldflags "-extldflags '-static'" -o foobar
artifacts:
paths:
- foobar

As mentioned by Yan Foto, you can only use paths that are within the project workspace. But you can move the $GOPATH to be inside your project, as suggested by extrawurst blog.
test:
image: golang:1.11
cache:
paths:
- .cache
script:
- mkdir -p .cache
- export GOPATH="$CI_PROJECT_DIR/.cache"
- make test

This is a pretty tricky task, as GitLab does not allow caching outside the project directory. A quick and dirty task would be to copy the contents of $GOPATH under some directory inside the project (say _GO), cache it and copy it upon each stage start back to $GOPATH:
after_script:
- cp -R $GOPATH ./_GO || :
before_script:
- cp -R _GO $GOPATH
cache:
untracked: true
key: "$CI_BUILD_REF_NAME"
paths:
- _GO/
WARNING: This is just a (rather ugly) workaround and I haven't tested it myself. It should only exhibit a possible solution.

Related

How to build specific dist folder by specific branch in GitLab CI/CD

How to build specific dist folder by specific branch in GitLab CI/CD ?
I got a main project that includes two folders,I want to build the project if I change the specific folder,such as when I changed code in Web1 folder ,then the CI/CD will be triggered and only build Web1 folder,not build all folders in the same time?
stages:
- build
build:
stage: build
tags:
- test
only:
- test
script:
# - Web1
- cd Web1
- npm install
- npm run build
- rm - rf /home/user/test/web1
- cp - r../dist/web1 /home/user/test/web1
- cd..
# - Web2
- cd Web2
- npm install
- npm run build
- rm - rf /home/user/test/web2
- cp - r../dist/web2 /home/user/test/web2
- cd..
You should create two different jobs for this, which are triggered differently with the help of the keyword changes.
One job should be triggered when changes are made in the "Web1" folder
build_web1:
stage: build
only:
changes:
- Web1/**/*
script:
- ...
and the other when changes are made in the "Web2" folder
build_web2:
stage: build
only:
changes:
- Web2/**/*
script:
- ...

"npm ci" command causing longer build time on gitlab than the "npm i"

Due to certain suggestions received from different articles, we have decided to use the "npm ci" to install the node dependencies from package-lock.json file to avoid breaking changes instead of using the "npm install" script.
But after making this change in .gitlab-ci.yml file, the builds are taking much more time to get install the dependencies. It has been increased from 7 minutes to around 23 minutes.
As per the attached screenshot, it looks like taking more time in removing the existing node_modules folder before installation -
Below are some details from the script file -
image: docker:latest
# When using dind, it's wise to use the overlayfs driver for
# improved performance.
variables:
DOCKER_DRIVER: overlay2
services:
- docker:dind
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- node_modules/
stages:
- test and build
# - documentation-server
- deploy
variables:
GIT_STRATEGY: clone
# ELECTRON_SKIP_BINARY_DOWNLOAD: 1
build:library:
image: trion/ng-cli-karma
stage: test and build
only:
- master
- /^.*/#library_name
script:
- echo _auth=${NPM_TOKEN} >> .npmrc
- mkdir -p dist/core
- cd dist/core
- npm init -y
- cd ../..
- ls -al /hugo
- npm ci
Any help or suggestions would be really helpful to fix this issue.

How do I correctly expand variables in .gitlab-ci.yml?

In my .gitlab-ci.yml file I try and set a variable containing a timestamp in the before_script section.
I would then like to expand that variable and append it to the archive I'm creating for my build.
The file goes roughly like this:
#.gitlab-ci.yml
image: node:14.4.0-buster
before_script:
- export DATETIME=$(date "+%Y%m%d%H%M%S")
stages:
#- test # not relevant for this question
- build
- deploy
build:
stage: build
script:
- npm install
- npm run build
- ls -la build
- tar cvfJ build_${DATETIME}.tar.xz build/
- sha1sum build_${DATETIME}.tar.xz
artifacts:
paths:
- build_${DATETIME}.tar.xz
deploy:
image: node:14.4.0-buster
stage: deploy
script:
- sha1sum build_${DATETIME}.tar.xz
- tar xvfJ build_${DATETIME}.tar.xz
# do the actual deploy
only:
- master
The deploy stage fails at sha1sum. The output is:
$ sha1sum build_${DATETIME}.tar.xz
sha1sum: build_20200702165854.tar.xz: No such file or directory
This shows that the expansion is done correctly, yet something is wrong.
What am I missing?
The before_script is run at the start of each job, and so the export DATETIME=$(date "+%Y%m%d%H%M%S") would be different for both stages.
It would probably be better to use ${CI_COMMIT_SHORT_SHA} instead for example.

Gitlab Pages throw 404 when accessed

I have a group project with the following name (hosted in Gitlab): gitlab.com/my-group/my-project.
I have generated coverage reports during testing and saved them as artifacts using Gitlab CI. Here is Gitlab CI config:
test:
stage: test
image: node:11
before_script:
- npm install -g yarn
- yarn
cache:
paths:
- node_modules/
script:
- yarn lint
- yarn test --all --coverage src/
except:
- tags
artifacts:
paths:
- coverage/
coverage: '/Statements\s+\:\s+(\d+\.\d+)%/'
deploy-pages:
stage: deploy
dependencies:
- test
script:
- mv coverage/ public/
artifacts:
paths:
- public/
expire_in: 30 days
except:
- tags
When I open deploy stage job, I can see the artifact being created. Here is the screenshot: . All the files are under /public directory in the artifact.
Now, when I go to: https://my-group.gitlab.io/my-project, I keep getting 404.
I am not sure what step I am missing here. Can someone shed some light on this issue for me?
Thanks!
There are three basic requirements for the project itself:
project must be named group.gitlab.io (if you want it to be the base domain)
job must create artifact in public directory
job must be called pages
Most likely it's the last one that needs fixing since your job is currently called deploy-pages. Simply rename that to pages.
You'll know when you got everything working because under Settings > Pages, it will tell you the link where it's published to.

Gitlab-Ci. Transfer maven m2 local repository between stages

Good day collegues. I use gitlab ci in production. I have a lot of stages.
1)Build artifact
2)Deploy to external server
3)Using jfrog cli to deploy to artifactory
I have a problem with caching maven local repository. My runners download all dependicies in first step(build) and do the same in the last step(deploy to artifactory). Also my runner delete all data from m2 folder before final stage:
Removing .m2/antlr/
Removing .m2/aopalliance/
Removing .m2/asm/
Removing .m2/avalon-framework/
Removing .m2/backport-util-concurrent/
Removing .m2/ch/
Removing .m2/classworlds/
Removing .m2/com/
Removing .m2/commons-beanutils/
Removing .m2/commons-chain/
Removing .m2/commons-cli/
Removing .m2/commons-codec/
Removing .m2/commons-collections/
Removing .m2/commons-digester/
Removing .m2/commons-io/
Removing .m2/commons-lang/
Removing .m2/commons-logging/
Removing .m2/commons-validator/
Removing .m2/dom4j/
Removing .m2/io/
Removing .m2/javax/
Removing .m2/junit/
Removing .m2/log4j/
Removing .m2/logkit/
Removing .m2/net/
Removing .m2/org/
Removing .m2/oro/
Removing .m2/sslext/
Removing .m2/xml-apis/
Removing .m2/xmlunit/
Removing jfrog
Removing target/
my gitlav-ci yaml(withou second step):
stages:
- build
- deploy-artifactory
variables:
MAVEN_OPTS: "-Dmaven.repo.local=${CI_PROJECT_DIR}/.m2"
MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
cache:
key: "$CI_JOB_NAME"
paths:
- .m2/
build:
image: maven:latest
stage: build
tags:
- build
script:
- adduser --disabled-password --gecos '' mynonrootuser
- chmod --recursive 777 .
- git version
- su --command='mvn versions:set -DgenerateBackupPoms=false -DnewVersion="$CI_COMMIT_REF_SLUG-SNAPSHOT"' mynonrootuser
- su --command='mvn $MAVEN_CLI_OPTS clean install -B -f ./pom.xml' mynonrootuser
artifacts:
expire_in: 5 mins
paths:
- target/*.jar
only:
- develop
deploy-artifactory-snapshot:
retry: 2
image: maven:latest
stage: deploy-artifactory
tags:
- release
before_script:
- curl -fL https://getcli.jfrog.io | sh
- ./jfrog rt config --url=${ARTIFACTORY_URL} --user=${ARTIFACTORY_USER} --password=${ARTIFACTORY_PASSWORD}
- ./jfrog rt c show
- export M2_HOME=/usr/share/maven
- sed -i 's,MAVEN_REPO_SNAPSHOT_DEPLOYER,'"$MAVEN_REPO_SNAPSHOT_DEPLOYER"',g' configuration.yml
- sed -i 's,MAVEN_REPO_RELEASES_DEPLOYER,'"$MAVEN_REPO_RELEASES_DEPLOYER"',g' configuration.yml
- sed -i 's,MAVEN_REPO_SNAPSHOT_RESOLVER,'"$MAVEN_REPO_SNAPSHOT_RESOLVER"',g' configuration.yml
- sed -i 's,MAVEN_REPO_RELEASES_RESOLVER,'"$MAVEN_REPO_RELEASES_RESOLVER"',g' configuration.yml
script:
- ./jfrog rt mvn "versions:set -DgenerateBackupPoms=false -Dartifactory.publish.artifacts=false -DnewVersion="$CI_COMMIT_REF_SLUG-SNAPSHOT"" configuration.yml --build-name=scdfrestrunner --build-number=$CI_JOB_ID
- ./jfrog rt mvn "clean install" configuration.yml --build-name=scdfrestrunner --build-number=$CI_JOB_ID
- ./jfrog rt bce scdfrestrunner $CI_JOB_ID
- ./jfrog rt bp scdfrestrunner $CI_JOB_ID
only:
- develop
There are several ways to handle dependencies between build jobs/stages.
caching, guide is available here.
e.g cache section for m2
# Cache modules in between jobs
cache:
key: ${CI_COMMIT_REF_SLUG} # cache is for per branch
paths:
- ${CI_PROJECT_DIR}/.m2
caching also can be carried out using something like minio (open source s3 like storage solution). Integration with gitlab guide is available here. But since caching requires zipping and unzipping large amounts of artifacts may introduce slowness (but obviously way speed than without cache). To avoid that problem docker volumes can be used.
docker volume
basically we can use a out of the box docker volume say ~/.m2 mounted from host machine location. If you using in kubernetes cluster you can leverage kubnernets volume solutions for this. You have to customize your build image bit for support mounting dependency dir, following is an example of mvn build image. This approach is better because you can't cache outside of your build dir.
e.g. docker image
FROM maven:3.5.4-jdk-8
# m2 dir on docker container
ENV MAVEN_OPTS "-Dmaven.repo.local=/.m2/repository"
ENV MAVEN_CLI_OPTS "-s /usr/local/.m2/settings.xml --batch-mode"
ADD .m2 /usr/local/.m2 # copy settings.xml to build image
# same as the m2 dir specified in MAVEN_OPTS
VOLUME /.m2
Artifacts
If you just want to pass your build dir e.g. /target, you can use gitlab artifacts solution, guide is available here.

Resources