Gitlab CI exit code 1 using with flake8 - python-3.x

I don't understand why Job finished with exit code 1 in gitlab CI. If I don't use flake8 (f.ex run script line echo "hello world" > $FOLDER_NAME/my_test.txt) all good.
But I see that flake8 found errors in directory:
...
$ mkdir -p $FOLDER_NAME
$ flake8 --max-line-length=120 --ignore=W605,W504 --tee --output-file=$FOLDER_NAME/$LOG_NAME $CHECKING_FOLDER
./framework/tests/test_5_helper.py:30:30: W292 no newline at end of file
./framework/tests/test_1_start.py:2:1: F401 'pprint.pprint' imported but unused
Cleaning up file based variables
00:00
ERROR: Job failed: exit code 1
yml-file:
stages:
- check
pep8_check:
stage: check
image: python:3.8-alpine
variables:
FOLDER_NAME: 'logs'
LOG_NAME: 'linter.log'
CHECKING_FOLDER: './framework/tests'
when: always
before_script:
- python -m pip install --upgrade pip
- pip install flake8
- export
- mkdir -p $FOLDER_NAME
script:
- flake8 --max-line-length=120 --ignore=W605,W504 --tee --output-file=$FOLDER_NAME/$LOG_NAME $CHECKING_FOLDER
artifacts:
expire_in: 7 days
paths:
- $FOLDER_NAME/

Flake8 finds 2 errors so exits with 1, this makes the GitLab pipeline fail.
You have a few options:
if you want GitLab to ignore any error flake8 may find, then you can just add the parameter --exit-zero, this will make flake8 exit with 0 which makes the GitLab pipeline successful
if you want to ignore those specific errors from your output:
./framework/tests/test_5_helper.py:30:30: W292 no newline at end of file
./framework/tests/test_1_start.py:2:1: F401 'pprint.pprint' imported but unused
then you just have to add those to the ignore list like you did for others:
change this --ignore=W605,W504 to this --ignore=W605,W504,W292,F401
you can also go and "fix/amend/change" your code so flake8 stops flagging those lines when parsing your source code
In any case reading the help with flake8 --help may give some more ideas on how to tackle these corner cases depending on what you want to achieve.
Also see here the lists of error/warning/violation codes E***, W***, F***:
https://pycodestyle.pycqa.org/en/latest/intro.html#error-codes
https://flake8.pycqa.org/en/latest/user/error-codes.html

Related

.gitlab-ci.yml extends Keyword does not Include Earlier Section

I am trying to use the extends keyword in the .gitlab-ci.yml of a GitLab Python project. It's not working, and I can't figure out why not.
I am using GitLab's CI/CD framework to test my Python project. The project has unit tests written with pytest and the following Dockerfile.
# syntax=docker/dockerfile:1
FROM python:3.9
WORKDIR /install
COPY . .
RUN pip install --no-cache-dir --upgrade .
EXPOSE 8000
CMD ["uvicorn", "sample_api.api:app"]
When I have the following .gitlab-ci.yml, GitLab's CI/CD system starts the python:3.9.16-slim-buster image and successfully runs the test job.
include:
- template: Auto-DevOps.gitlab-ci.yml
test:
stage: test
image: python:3.9.16-slim-buster
before_script:
- pip install .
script:
- pytest tests/unit
services: []
However, the test job fails when I change it to use the extends keyword like so.
include:
- template: Auto-DevOps.gitlab-ci.yml
.tests:
stage: test
image: python:3.9.16-slim-buster
before_script:
- pip install .
services: []
test:
extends: .tests
script:
- pytest tests/unit
The log of the failed test job looks like this.
...
Preparing the "docker" executor
00:11
Using Docker executor with image gliderlabs/herokuish:latest ...
...
Executing "step_script" stage of the job script
00:03
Using docker image sha256:686c154e24a2373406bdf9c8f44904b5dbe5cd36060c61d3da137086389d18d3 for gliderlabs/herokuish:latest with digest gliderlabs/herokuish#sha256:5d5914135908a234c20eec80daaa6a386bfa74293310bc0c79148fe7a7e4a926 ...
$ pip install .
/bin/bash: line 154: pip: command not found
Cleaning up project directory and file based variables
00:02
ERROR: Job failed: exit code 1
It is failing because the default herokuish:latest image is being used instead of python:3.9.16-slim-buster. It appears that the .tests section is never used.
I assume there's something wrong with my syntax, but it seems so simple and I can't figure out what it is.

PyLint Doesn't work on GitLab (Python 3.X)

I'm having a problem with Pylint on gitlab. For some reason, it crashes on the middle of the process of analysing the code..
But doesn't say nothing to me, only crashes..
Here are some log:
core/db/sql/pgsql/__init__.py:162:8: E1101: Instance of 'Session' has no 'close' member (no-member)
/core/db/sql/pgsql/__init__.py:165:4: C0116: Missing function or method docstring (missing-function-docstring)
/core/db/sql/pgsql/__init__.py:166:8: E1101: Instance of 'Session' has no 'commit' member (no-member)
/core/db/sql/pgsql/__init__.py:168:4: C0116: Missing function or method docstring (missing-function-docstring)
/core/db/sql/pgsql/__init__.py:169:8: E1101: Instance of 'Session' has no 'rollback' member (no-member)
/core/db/sql/pgsql/__init__.py:171:4: C0116: Missing function or method docstring (missing-function-docstring)
/core/db/sql/pgsql/__init__.py:174:4: C0116: Missing function or method docstring (missing-function-docstring)/bin/bash: line 98: 43 Segmentation fault (core dumped) pylint /
44 Done | tee ./pylint/pylint.log
The following messages were raised:
- fatal message issued
- error message issued
- refactor message issued
Fatal messages detected. Failing...
Running after_script
00:02
196
Uploading artifacts for failed job
00:02
198 ERROR: Job failed: exit code 1
Here are the .gitlab-cy.yml configuration file:
image: "python:3.8.5"
# Change pip's cache directory to be inside the project directory since we can
# only cache local items.
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache"
cache:
paths:
- .cache/pip
- venv/
before_script:
- python --version
- pip install virtualenv
- virtualenv venv
- source venv/bin/activate
- pip install -r requirements.txt
stages:
- Static Analysis
- Test
flake8:
stage: Static Analysis
script:
- flake8 project/
pylint:
stage: Static Analysis
script:
- mkdir ./pylint
- pylint project/ | tee ./pylint/pylint.log || pylint-exit $?
- PYLINT_SCORE=$(sed -n 's/^Your code has been rated at \([-0-9.]*\)\/.*/\1/p' ./pylint/pylint.log)
- anybadge --label=Pylint --file=pylint/pylint.svg --value=$PYLINT_SCORE 2=red 4=orange 8=yellow 10=green
- echo "Pylint score is $PYLINT_SCORE"
artifacts:
paths:
- ./pylint/
coverage:
stage: Test
script:
- coverage erase
- $(coverage run -m unittest discover -s tests | tee cov.log || exit 0)
- coverage report -m
coverage: '/TOTAL.+ ([0-9]{1,3}%)/'
artifacts:
paths:
- ./pylint/
when: always
With this erros codes, I didn't find nothing on web..
And I have others projects on gitlab with the same configuration, and they run without any problem.
Does anyone already seen this?
Thanks!!
I do not see how are you executing the pylint. Please provide your gitlab-ci.yml file. But I had the very similar problem and I solved it by adding pylint-exit library:
pylint --disable=W0611,C0411,C0301 tests > ./pylint/tests.log || pylint-exit $?

How do I create an artifact so it will be available for download in .gitlab-ci.yml

I have a GitLab ci pipeline and I am not sure how to get it to generate an artifact with the binary file that happened in the build stage.
Here is my yml file...
stages:
- test
- build
- art
image: golang:1.9.2
variables:
BIN_NAME: example
ARTIFACTS_DIR: artifacts
GO_PROJECT: example
before_script:
- mkdir -p ${GOPATH}/src/${GO_PROJECT}
- mkdir -p ${CI_PROJECT_DIR}/${ARTIFACTS_DIR}
- go get -u github.com/golang/dep/cmd/dep
- cp -r ${CI_PROJECT_DIR}/* ${GOPATH}/src/${GO_PROJECT}/
- cd ${GOPATH}/src/${GO_PROJECT}
test:
stage: test
script:
# Run all tests
go test -run ''
build:
stage: build
script:
# Compile and name the binary as `hello`
- go build -o hello
# Execute the binary
- ./hello
art:
script:
artifacts:
paths:
- ./hello
The test and build phases run fine but the art stage does not when it is added to the yml file.
I have found lots of examples on line but finding it hard to convert them to my exact situation.
All I want to for the artifact to appear as a download on the pipeline like in this link.
Downloading artifacts
after trying solution suggested i get the following...
$ go build -o hello
$ ./hello
Heldfgdfglo 2
Uploading artifacts...
WARNING: ./hello: no matching files
ERROR: No files to upload
Job succeeded
Tried adding..
GOPATH: /go
and...
- cd ${GOPATH}/src/${GO_PROJECT}
now getting following error...
Uploading artifacts...
WARNING: /go/src/example/hello: no matching files
ERROR: No files to upload
Job succeeded
output shared as requested...
mkdir -p ${GOPATH}/src/${GO_PROJECT}
$ mkdir -p ${CI_PROJECT_DIR}/${ARTIFACTS_DIR}
$ go get -u github.com/golang/dep/cmd/dep
$ cp -r ${CI_PROJECT_DIR}/* ${GOPATH}/src/${GO_PROJECT}/
$ cd ${GOPATH}/src/${GO_PROJECT}
$ go build -o hello
$ pwd
/go/src/example
$ ls -l hello
-rwxr-xr-x. 1 root root 1859961 Jun 19 08:27 hello
$ ./hello
Heldfgdfglo 2
Uploading artifacts...
WARNING: /go/src/example/hello: no matching files
ERROR: No files to upload
Job succeeded
./hello is not matching your artifact path because you changed the directory before running your script.
You need to move the generated executable to the original working directory of the gitlab runner, because artifact paths can only be relative to the build directory:
build:
stage: build
script:
# Compile and name the binary as `hello`
- go build -o hello
# Execute the binary
- ./hello
# Move to gitlab build directory
- mv ./hello ${CI_PROJECT_DIR}
artifacts:
paths:
- ./hello
See https://gitlab.com/gitlab-org/gitlab-ce/issues/15530
You need to specify your artifact paths in the job that creates them, since every job triggers a new, empty environment (more or less considering the cache):
build:
stage: build
script:
# Compile and name the binary as `hello`
- go build -o hello
# Execute the binary
- ./hello
artifacts:
paths:
- ./hello

Gitlab - How to add badge based on jobs pipeline

My goal is to show badges (ex : ) based on pipeline results.
I have a private gitlab ce omnibus instance with the following .gitlab-ci.yml :
image: python:3.6
stages:
- lint
- test
before_script:
- python -V
- pip install pipenv
- pipenv install --dev
lint:
stage: lint
script:
- pipenv run pylint --output-format=text --load-plugins pylint_django project/ | tee pylint.txt
- score=$(sed -n 's/^Your code has been rated at \([-0-9.]*\)\/.*/\1/p' pylint.txt)
- echo "Pylint score was $score"
- ls
- pwd
- pipenv run anybadge --value=$score --file=pylint.svg pylint
artifacts:
paths:
- pylint.svg
test:
stage: test
script:
- pipenv run python manage.py test
So I thought that I would store the image in the artifacts of the lint job and display it via the badge feature.
But I encounter the following issue : when I browse https://example.com/[group]/[project]/-/jobs/[ID]/artifacts/file/pylint.svg, instead of seeing the badge I have the following message :
The image could not be displayed because it is stored as a job artifact. You can download it instead.
And anyways, I feel like this is the wrong way, because even if I could get the image, there don't seems to be a way to get the image from the last job since gitlab URL for badges images only supports %{project_path}, %{project_id}, %{default_branch}, %{commit_sha}
So how one would add badge to a gitlab project based on a svg generated from results in a gitlab pipeline ?
My guess is that I could push to a .badge folder but that doesn't sound like a clean solution.
You can indeed get the artifact(s) for the latest job (see documentation here), but the trick is that you need to use a slightly different URL:
https://example.com/[group]/[project]/-/jobs/artifacts/[ref]/raw/pylint.svg?job=lint
where [ref] is the reference to your branch/commit/tag.
Speaking of badge placeholders available in Gitlab, you can potentially put %{default_branch} or %{commit_sha} into [ref]. This won't allow you to get the correct badge for every branch, but at least your default branch will get one.
Please also note that ?job=lint query parameter is required, without it the URL won't work.

gitlab ci cache/keep golang packages between stages

I use gitlab-ci to test, compile and deploy a small golang application but the problem is that the stages take longer than necessary because they have to fetch all of the dependencies every time.
How can I keep the golang dependencies between two stages (test and build)?
This is part of my current gitlab-ci config:
test:
stage: test
script:
# get dependencies
- go get github.com/foobar/...
- go get github.com/foobar2/...
# ...
- go tool vet -composites=false -shadow=true *.go
- go test -race $(go list ./... | grep -v /vendor/)
compile:
stage: build
script:
# getting the same dependencies again
- go get github.com/foobar/...
- go get github.com/foobar2/...
# ...
- go build -race -ldflags "-extldflags '-static'" -o foobar
artifacts:
paths:
- foobar
As mentioned by Yan Foto, you can only use paths that are within the project workspace. But you can move the $GOPATH to be inside your project, as suggested by extrawurst blog.
test:
image: golang:1.11
cache:
paths:
- .cache
script:
- mkdir -p .cache
- export GOPATH="$CI_PROJECT_DIR/.cache"
- make test
This is a pretty tricky task, as GitLab does not allow caching outside the project directory. A quick and dirty task would be to copy the contents of $GOPATH under some directory inside the project (say _GO), cache it and copy it upon each stage start back to $GOPATH:
after_script:
- cp -R $GOPATH ./_GO || :
before_script:
- cp -R _GO $GOPATH
cache:
untracked: true
key: "$CI_BUILD_REF_NAME"
paths:
- _GO/
WARNING: This is just a (rather ugly) workaround and I haven't tested it myself. It should only exhibit a possible solution.

Resources