In the gitlab documentation some URL's are described for the purpose of downloading artifacts from pipelines HERE. They seem to have forgotten to describe HOW to download artifacts given these URLs.
Can it be done in a simple way? Or do I have to install e.g. wget, create a token, define a token, use a token?
If someone could give an example that would be great.
The documentation referenced in the question is supposed to be an REST API call, using the Job Artifact API:
GET /projects/:id/jobs/artifacts/:ref_name/download?job=name
To use this in a script definition inside .gitlab-ci.yml, you can use either:
The JOB-TOKEN header with the GitLab-provided CI_JOB_TOKEN variable.
For example, the following job downloads the artifacts of the test job of the main branch.
The command is wrapped in single quotes because it contains a colon (:):
artifact_download:
stage: test
script:
- 'curl --location --output artifacts.zip --header "JOB-TOKEN: $CI_JOB_TOKEN" "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/jobs/artifacts/main/download?job=test"'
Or the job_token attribute with the GitLab-provided CI_JOB_TOKEN variable.
For example, the following job downloads the artifacts of the test job of the main branch:
artifact_download:
stage: test
script:
- 'curl --location --output artifacts.zip >"https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/jobs/artifacts/main/download?job=test&job_token=$CI_JOB_TOKEN"'
But the artifact: directive is meant to store data in the job workspace, for a new iteration of the job to get back, in the same folder.
No "download" involved, as illustrated in the article "GitLab CI: Cache and Artifacts explained by example" by Anton Yakutovich.
As such, no curl/wget/TOKEN should be needed to access an artifact stored by a previous job execution.
Related
I am stuck on the gitlab ci/cd pipeline code because I can't get the Merge_Request creator name in a predefined variable.
Please help me for getting Merge_Request creator / author name in gitlab ci/cd
I need some Merge_Request author name in gitlab ci/cd pipeline
You can get merge request author by using API and predefined variables. Check this endpoint /projects/:id/merge_requests/:merge_request_iid (https://docs.gitlab.com/ee/api/merge_requests.html#get-single-mr). Instead of attributes id and merge_request_iid you should use variables CI_PROJECT_ID and CI_MERGE_REQUEST_IID. Then you can filter response by jq tool.
Something like this:
curl -X GET -H "PRIVATE-TOKEN: ${ACESS_TOKEN}" ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/merge_requests/${CI_MERGE_REQUEST_IID} | jq .author.username
Is it possible to get GitLab to report out what it's actually running in the output log. For example, the following .gitlab-ci.yml
variables:
MAVEN_CLI_OPTS: >-
-s $CI_PROJECT_DIR/.m2/settings.xml
--batch-mode
--errors
--fail-at-end
--show-version
-DinstallAtEnd=true
-DdeployAtEnd=true
compile-test-package:
stage: package
script:
- mvn ${MAVEN_CLI_OPTS} package
The run log then shows this
...
mvn ${MAVEN_CLI_OPTS} package
...
But I really would like it to give the specific details like
...
mvn -s path/to/my/project/.m2/settings.xml --batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true package
...
The best I've learned on this is that GitLab doesn't have a way to report the full details of what its executing. As Michael Delgado mentioned, putting an echo in to output will give you a method to do this. However use this with cation cause protected values could be exposed using this method.
I'm still struggling with variable replacement but at least now I can see what GitLab is replacing and not.
Can someone explain to me in simple terms what the release stage actually does, and why that is useful?
As for as I understand the official documentation, all it does is create some kind of tag?!
EDIT: (responding to wedm)
I am referring to the stage described on the official gitlab documentation on https://docs.gitlab.com/ee/ci/yaml/#release
This page describes all the details, but I don't get the big picture: Why are we doing this, what are we trying to accomplish here?
I have taken over a project that has a "release" stage after the deploy stage:
release:
stage: release
image: registry.gitlab.com/gitlab-org/release-cli:latest
needs:
- build-release-env
rules:
- if: "$CI_COMMIT_TAG =~ /^v/"
variables:
ASSET_URL_BUILD: "${CI_PROJECT_URL}/-/jobs/artifacts/${CI_COMMIT_SHA}/download?job=build-www"
ASSET_URL_BUNDLE: "${CI_PROJECT_URL}/-/jobs/artifacts/${CI_COMMIT_SHA}/download?job=docker-image"
script:
- echo "Creating release for ${CI_COMMIT_TAG} (${CI_COMMIT_SHA})"
- cat ./CHANGELOG.latest.md
- |
release-cli create --name "Release ${CI_COMMIT_TAG}" --tag-name ${CI_COMMIT_TAG} \
--description ./CHANGELOG.latest.md \
--assets-link "{\"name\":\"Complete Build\",\"url\":\"${ASSET_URL_BUILD}\"}" \
--assets-link "{\"name\":\"Docker image\",\"url\":\"${ASSET_URL_BUNDLE}\"}" \
--ref ${CI_COMMIT_SHA}
Quoted from the GitLab documentation about releases (emphasis mine):
To introduce a checkpoint in your source code history, you can assign a Git tag at the moment of release. However, in most cases, your users need more than just the raw source code. They need compiled objects or other assets output by your CI/CD system.
One common example would be to attach built binaries to a release.
Concerning your given example: from the script part it looks like it's creating a gitlab release for the project it's run on. Doing that programmatically gives some advantages, as not all options (e.g. timed releases) are available via the web ui of gitlab.
I'm trying to retrieve a build file using the gitlab API. This file was created and stored as an artifact from an upstream pipeline. Running
curl -o download --location --header 'PRIVATE-TOKEN:{MY_API_TOKEN}' https://gitlab.foo.com/api/v4/projects/{PROJECT_ID}/jobs/artifacts/{REF_BRANCH}/download?job={JOB_NAME}
on my local machine gives me a proper build file once I run unzip download. However in the runner, the same command returns a much smaller file which I can't unzip. I've checked that the environment variables that are passed in the runner are right.
job in .gitlab-ci.yml
deploy_production_environment:
stage: deploy_prod
image:
name: banst/awscli
script:
- apk --no-cache add curl
- apk add unzip
- echo $JOB_ID
- echo $FE_BUILD_TOKEN
- echo "https://gitlab.foo.com/api/v4/projects/${PROJECT_ID}/jobs/artifacts/${CI_COMMIT_REF_NAME}/download?job=build_prod"
- aws configure set region us-east-1
- "curl -o download --location --header 'PRIVATE-TOKEN:${FE_BUILD_TOKEN}' https://gitlab.foo.com/api/v4/projects/${PROJECT_ID}/jobs/artifacts/${CI_COMMIT_REF_NAME}/download?job=build_prod"
- ls -l
- unzip download
- aws s3 cp build s3://$S3_BUCKET_PROD --recursive
gitlab job output:
`
output from my local terminal:
Why does the API call from inside the runner consistently result in this much smaller (corrupted?) file while the same call pulls the zip file down correctly on my local machine?
The first check to do when a curl brings back a "small" file it to read its content.
Often, the file is not so much corrupted but includes a text-based error message in it, which can give a clue as to the actual issue.
Adding -v to the curl command can also help illustrating the issue during the curl process (when executed in the context of the GitLab job).
Thank you to VonC for the debugging help, recommending the -v flag to the curl command. It turns out that the single quotes around 'PRIVATE-TOKEN:${FE_BUILD_TOKEN}' prevented the variable from being parsed to its correct string value which was giving a 401 'Permission Denied' error. Removing the single quotes did the trick.
I am planning to run some validations against the pull request in a CI pipeline and, based on the validation results, I wish to automatically create an issue and assign it to developers.
Is this achievable in a GitLab pipeline?
Thanks!
You should be able to call the New Issue API through a curl call (as seen here)
curl --request POST --header "PRIVATE-TOKEN: PlainTextToken" "https://gitlab.com/api/v4/projects/:id/issues?title=a title"
You would need to add a private token as a masked variable.
If you just need to create a single issue, go with the cURL command. However, if you require more complex logic, the python-gitlab library can be a useful tool in your arsenal.
To create a project issue:
import gitlab
import os
gl = gitlab.Gitlab(os.environ['CI_SERVER_URL'], private_token=os.environ['PRIVATE_TOKEN'])
project = gl.projects.get(os.environ['CI_PROJECT_ID'])
issue_details = {
'title': f'Validation failed in {os.environ["CI_PROJECT_NAME"]}',
'description': f'Pipeline: {os.environ["CI_PIPELINE_URL"]}',
'assignee_ids': [111, 222]
}
issue = project.issues.create(issue_details )
This assumes you have created a masked variable called PRIVATE_TOKEN. The other variables are pre-defined variables.
You can add the above code and other logic to a Python script and call it in gitlab-ci.yml like this:
# Use whatever image you need, but make sure it has Python installed
image: python:3.7
...
handle-failure:
when: on_failure
before_script:
- pip install -r ./cicd/gitlab/requirements.txt
script:
- python -m ./cicd/gitlab/create_issue.py