I have an OpenAPI yaml file that I have built in the Swagger Editor tool. I would now like to build a static site that looks exactly the same as that.
I would like to incorporate this into a Gitlab CI/CD pipeline so that any changes are automatically updated.
Currently, I am using the redoc-cli and I really don't like the way it looks. Also, there is no "Try it out" functionality.
I have been searching all day for the necessary components to build that static site, but there do not seem to be any concrete answers.
I am currently build my GitLab Pages site like this:
pages:
stage: deploy
script:
- mkdir ~/.npm-global
- npm config set prefix '~/.npm-global'
- echo "export PATH=~/.npm-global/bin:$PATH" > ~/.profile
- source ~/.profile
- npm install -g redoc-cli --verbose
- redoc-cli bundle -o public/index.html swagger.yaml
artifacts:
paths:
- public
only:
- master
Related
I want to deploy a static page using GitLab repos with plain HTML/CSS (actually SCSS). As far as I learnt, a static page needs at least .gitlab-ci.yml and /public folder. The file .gitlab-ci.yml will have a minimum requirement like this: (an example from official doc)
pages:
stage: deploy
script:
- mkdir .public
- cp -r * .public
- mv .public public
artifacts:
paths:
- public
only:
- master
And my question is lying in the script line.
(I assume the script below will create a hidden folder name .public and copy all the file in it then move it from .public to public folder. Please correct me if I'm wrong.)
script:
- mkdir .public
- cp -r * .public
- mv .public public
To me, it's similar to shell-script of Linux. It's also confirmed in GitLab doc that it's run by the Runner. But the problem is, how do I know how many shell-scripts are installed in GitLab? And is it possible to make one?
I would like to make 2 folders: src and public. The GitLab CI will run the script and compile SCSS from src then move it to public.
I'm using gitlab.com by the way.
So a few things to consider. Each job in gitlab is run in a container. Generally you specify which one you want to use. Pages is a special case though so you don't have to care about the image for the container.
The pages job will populate your public folder. But you can alter the gitlab-ci.yml file and add steps.
This would build an app using node:
build_stuff:
stage: build
image: node:11
before_script:
- npm install
script:
- npm run build
artifacts:
paths:
- build
pages:
stage: deploy
script:
- mkdir .public
- cp -r build/ .public
- mv .public public
artifacts:
paths:
- public
only:
- master
Formatting is off
Things to note. The first step is running the build steps to generate all the assets for your output folder. It is then storing anything declared in the artifacts block, in this case the build folder, and passing it on to the next job. Adjust this step accordingly to what you need to build your app.
The only thing I altered in the second step is that you copy the contents of the build folder instead of the entire repo into the .public folder. Adjust this to your needs as well.
As for shell scripts, there are none present except for the ones you bring to the repository. The default runner supports Bash so you can execute bash commands just as you would in your terminal.
If you create the file foo.sh in your repo and do bash foo.sh it will execute the script, if it's executable. Remember to chmod it before pushing it.
There are no "shell-scripts installed in Gitlab". Gitlab supports several shells and the script part in your example are just pure bash commands. Since you are most probably using the default docker runner you can execute bash commands from the script part, run scripts in other languages that are in your repo, install packages on the docker container and even prepare and run your own docker images.
My goal is to show badges (ex : ) based on pipeline results.
I have a private gitlab ce omnibus instance with the following .gitlab-ci.yml :
image: python:3.6
stages:
- lint
- test
before_script:
- python -V
- pip install pipenv
- pipenv install --dev
lint:
stage: lint
script:
- pipenv run pylint --output-format=text --load-plugins pylint_django project/ | tee pylint.txt
- score=$(sed -n 's/^Your code has been rated at \([-0-9.]*\)\/.*/\1/p' pylint.txt)
- echo "Pylint score was $score"
- ls
- pwd
- pipenv run anybadge --value=$score --file=pylint.svg pylint
artifacts:
paths:
- pylint.svg
test:
stage: test
script:
- pipenv run python manage.py test
So I thought that I would store the image in the artifacts of the lint job and display it via the badge feature.
But I encounter the following issue : when I browse https://example.com/[group]/[project]/-/jobs/[ID]/artifacts/file/pylint.svg, instead of seeing the badge I have the following message :
The image could not be displayed because it is stored as a job artifact. You can download it instead.
And anyways, I feel like this is the wrong way, because even if I could get the image, there don't seems to be a way to get the image from the last job since gitlab URL for badges images only supports %{project_path}, %{project_id}, %{default_branch}, %{commit_sha}
So how one would add badge to a gitlab project based on a svg generated from results in a gitlab pipeline ?
My guess is that I could push to a .badge folder but that doesn't sound like a clean solution.
You can indeed get the artifact(s) for the latest job (see documentation here), but the trick is that you need to use a slightly different URL:
https://example.com/[group]/[project]/-/jobs/artifacts/[ref]/raw/pylint.svg?job=lint
where [ref] is the reference to your branch/commit/tag.
Speaking of badge placeholders available in Gitlab, you can potentially put %{default_branch} or %{commit_sha} into [ref]. This won't allow you to get the correct badge for every branch, but at least your default branch will get one.
Please also note that ?job=lint query parameter is required, without it the URL won't work.
I have a project hosted on Gitlab. The project website is inside the pages branch and is a jekyll based site.
My .gitlab-ci.yml looks like
pages:
script:
- gem install jekyll
- jekyll build -d public/
artifacts:
paths:
- public
only:
- pages
image: node:latest
cache:
paths:
- node_modules/
before_script:
- npm install -g gulp-cli
- npm install
test:
script:
- gulp test
When I pushed this configuration file to master, the pipeline executed only the test job and not pages job. I thought maybe pushing to master didn't invoke this job because only specifies pages branch. Then I tried pushing to pages branch but to no avail.
How can I trigger the pages job?
You're right to assume that the only constraint makes the job run only on the ref's or branches specified in the only clause.
See https://docs.gitlab.com/ce/ci/yaml/README.html#only-and-except
It could be that there's a conflict because the branch and the job have the same name. Could you try renaming the job to something different just to test?
I'd try a couple of things.
First, I'd put in this stages snippet at the top of the YML:
stages:
- test
- pages
This explicitly tells the CI to run the pages stage after the test stage is successful.
If that doesn't work, then, I'd remove the only tag and see what happens.
Complementing #rex answer's:
You can do either:
pages:
script:
- gem install jekyll
- jekyll build -d public/
artifacts:
paths:
- public
Which will deploy your site regardless the branch name, or:
pages:
script:
- gem install jekyll
- jekyll build -d public/
artifacts:
paths:
- public
only:
- master # or whatever branch you want to deploy Pages from
Which will deploy Pages from master.
Pls let me know if this helps :)
Gitlab has functionality to generade badges about build status and coverage percentage.
Is it possible to create custom badge to display Pylint results?
Or just display this results in README.md?
I already have CI job for Pylint
I have written a python badge generation package that produces badges very visually similar to the main badge services. It is highly flexible, you can import and use in your python code, or run from the command line.
I use this in GitLab CI to display pylint and coverage scores.
There are other ways to do this using shields.io (see other answer from kubouch), but this approach can be used in situations where you may not have external internet access, such as in a corporate / enterprise setting where firewalls or proxies are blocking internet access.
GitLab CI Setup
1. Generate the badge
My CI pipeline has a step that runs pylint, and I used sed to extract the score from the output text. I then use anybadge (details below) to generate a pylint score badge, and save it as public/pylint.svg.
pylint:
stage: test
script:
- pylint --rcfile=.pylintrc --output-format=text <LIST-OF-FILES-TO-RUN-PYLINT-AGAINST> | tee pylint.txt
- score=$(sed -n 's/^Your code has been rated at \([-0-9.]*\)\/.*/\1/p' pylint.txt)
- echo "Pylint score was $score"
- anybadge --value=$score --file=public/pylint.svg pylint
If pylint generates a non-zero rc then GitLab will see that as a command error and the job will fail, meaning no badge is generated, and missing image will show where the badge is used.
NOTE: pylint WILL OFTEN generate non-zero return codes since it uses the exit code to communicate the status of the lint check. I suggest using something like pylint-exit to handle pylint return codes in CI pipelines.
2. Register badge as pipeline artifact
I register the generated badge file as an artifact in the CI job by including this in the .gitlab-ci.yml:
pylint:
...
- echo "Pylint score was $score"
- anybadge --value=$score --file=public/pylint.svg pylint
artifacts:
paths:
- public/pylint.svg
3. Publish badge to GitLab Pages
I include a pages publish step, which deploys everything in the public directory to GitLab pages:
pages:
stage: deploy
artifacts:
paths:
- public
only:
- master
4. Include badge in README.md
When the master pipeline runs for the project, the pylint.svg file is published to GitLab Pages, and I can then reference the image from my project README.md so that the latest pylint badge is displayed.
If you are using https://gitlab.com for your project then the URL for the svg artifact will usually be something like this (replace NAMESPACE with your username, or group name if your project is under a group - more details here):
https://NAMESPACE.gitlab.io/pyling.svg
In your README.md you can include an image with:
If you want to make the image into a link you can use:
[](LINKTARGET)
Let me know if you need more information on any of the setup.
Anybadge Python Package
Here's some more info on the anybadge Python package:
You can set the badge label and value, and you can set the color based on thresholds. There are pre-built settings for pylint, coverage, and pipeline success, but you can create any badge you like.
Here is a link to the github project with more detailed documentation: https://github.com/jongracecox/anybadge
Install with pip install anybadge
Example python code:
import anybadge
# Define thresholds: <2=red, <4=orange <8=yellow <10=green
thresholds = {2: 'red',
4: 'orange',
6: 'yellow',
10: 'green'}
badge = anybadge.Badge('pylint', 2.22, thresholds=thresholds)
badge.write_badge('pylint.svg')
Example command line use:
anybadge --label pylint --value 2.22 --file pylint.svg 2=red 4=orange 8=yellow 10=green
Update 2019
Using GitLab Pages is no longer required
It is now possible to directly access to the latest articfact, which simplify the workaround.
Use a dedicated pylint artifact instead of public, and remove the unnecessary deploy step (or edit it if already used):
pylint:
stage: test
before_script:
- pip install pylint pylint-exit anybadge
script:
- mkdir ./pylint
- pylint --output-format=text . | tee ./pylint/pylint.log || pylint-exit $?
- PYLINT_SCORE=$(sed -n 's/^Your code has been rated at \([-0-9.]*\)\/.*/\1/p' ./pylint/pylint.log)
- anybadge --label=Pylint --file=pylint/pylint.svg --value=$PYLINT_SCORE 2=red 4=orange 8=yellow 10=green
- echo "Pylint score is $PYLINT_SCORE"
artifacts:
paths:
- ./pylint/
Note that here I copy the Pylint log file in the folder artifact, in this way it will be accessible without looking at the pipeline logs.
The badge image will then be available at https://gitlab.example.com/john-doe/foo/-/jobs/artifacts/master/raw/pylint/pylint.svg?job=pylint, and the Pylint log at https://gitlab.example.com/john-doe/foo/-/jobs/artifacts/master/raw/pylint/pylint.log?job=pylint.
2. You can use GitLab's builtin badges instead of images in README
GitLab can now include badges in a projet or group, that will be displayed in the project header.
Got to Settings / General / Badges, then create a new badge by setting its link and image link, as described above:
If you don't want to use the README, gitlab pages, anybadge or dropbox you can use https://img.shields.io/badge/lint%20score-$score-blue.svg to 'create' a badge (which is just an URL) and change the badge image URL via the gitlab API.
Details and the lint stage of my .gitlab-ci.yml
I developed a workaround solution to real-time per-job badges.
It is not Pylint specific but the approach is general and you can easily modify it into what you need.
This example repo (branch badges) creates a custom badge for each CI job. There is also a complete walkthrough so I won't copy-paste it here.
The core idea is (assuming you're now inside a running CI job):
Create a badge (e.g. fetch it from shields.io into a file).
Upload the badge file to some real-time storage from where it can be linked (e.g. Dropbox).
Dropbox supports calling its API via HTTP requests (see this).
Thus, all the above can be done using e.g. curl or Python requests - basic tools.
You just need to pass the Dropbox access token as secret variable and save the file under the same name to overwrite the old badge.
Then, you can then directly link the Dropbox badge wherever you need.
There are some tricks to it so be sure to check my example repo if you want to use it.
For me it works quite well and seems to be fast.
The advantage of this method is that you don't have to mess with GitLab Pages.
Instead of publishing on Pages you put it to Dropbox.
That is a simple file transfer called by HTTP request.
No more to that.
Tutorial
Add the below file - .gitlab-ci.yml to your GitLab repository:
pylint:
stage: test
image: python:3.7-slim
before_script:
- mkdir -p public/badges public/lint
- echo undefined > public/badges/$CI_JOB_NAME.score
- pip install pylint-gitlab
script:
- pylint --exit-zero --output-format=text $(find -type f -name "*.py" ! -path "**/.venv/**") | tee /tmp/pylint.txt
- sed -n 's/^Your code has been rated at \([-0-9.]*\)\/.*/\1/p' /tmp/pylint.txt > public/badges/$CI_JOB_NAME.score
- pylint --exit-zero --output-format=pylint_gitlab.GitlabCodeClimateReporter $(find -type f -name "*.py" ! -path "**/.venv/**") > codeclimate.json
- pylint --exit-zero --output-format=pylint_gitlab.GitlabPagesHtmlReporter $(find -type f -name "*.py" ! -path "**/.venv/**") > public/lint/index.html
after_script:
- anybadge --overwrite --label $CI_JOB_NAME --value=$(cat public/badges/$CI_JOB_NAME.score) --file=public/badges/$CI_JOB_NAME.svg 4=red 6=orange 8=yellow 10=green
- |
echo "Your score is: $(cat public/badges/$CI_JOB_NAME.score)"
artifacts:
paths:
- public
reports:
codequality: codeclimate.json
when: always
pages:
stage: deploy
image: alpine:latest
script:
- echo
artifacts:
paths:
- public
only:
refs:
- master
Substitute the below variables accordingly depending upon your project structure. For example, if your repository is located at company_intern/john/robot_code and you added the .gitlab-ci.yml file to your main branch, then:
$GROUP = company_intern
$SUBGROUP = john
$PROJECT_NAME = robot_code
$BRANCH = main
# Substitute the above variables in the badge
[](https://gitlab.com/$GROUP/$SUBGROUP/$PROJECT_NAME/-/jobs/artifacts/$BRANCH/browse/public/lint?job=pylint)
Your badge has now been integrated! To verify the lining process locally before committing it directly:
# To lint all the python files in the directory:
pylint --exit-zero --output-format=text $(find -type f -name "*.py" ! -path "**/.venv/**")
# To lint a specific file, say foo.py:
pylint --exit-zero --output-format=text foo.py
References:
pylint-gitlab ยท PyPI
If you use flake8 to run pylint, then an easy way to generate a badge is to use genbadge. This simple commandline tool provides capabilities to generate badges for tests, coverage, and flake8.
Simply run
genbadge flake8 -i flake8stats.txt
to generate the badge from a flake8 statistics file, such as this one: . You can then use the badge to provide a quick link to the HTML report generated by flake8-html. See documentation for details (I'm the author by the way !).
I'm setting up Gitlab CI to create an asciidoc page from a Swagger YAML specification.
To this end, I've set up Gitlab to execute the swagger maven plugin and save the results to the public artifacts folder, like this:
enter codedeploy:jdk8:
stage: test
script:
- 'mvn $MAVEN_CLI_OPTS swagger2markup:convertSwagger2markup'
#only:
# - master
# Archive up the built documentation site.
artifacts:
paths:
- target/asciidoc
image: maven:3.3.9-jdk-8
pages:
image: busybox:latest
stage: deploy
script:
- mv target/asciidoc public
dependencies:
- deploy:jdk8
artifacts:
paths:
- public
#only:
# - master
My question is how to automatically upload the resulting asciidoc artifacts as wiki pages in my Gitlab project. I can browse the public artifacts folder but Gitlab doesn't render the .adoc files in there, it only offers to download them.
Update. I found Gitlab issue 18106 which is a feature request to "allow runners to push via their CI token". I think this feature would be the answer to my question, or does anyone have a better idea?