I want to automatically load test results of my gitlab pipeline (basically one xml file) into my project management application. No direct connection between these two is possible. However the gitlab API offers the following options to download artifacts of a pipeline:
all artifacts of the latest successful pipeline run (selected by job name)
GET /projects/:id/jobs/artifacts/:ref_name/download?job=name`
all artifacts of a specific job (selected by job id)
GET /projects/:id/jobs/:job_id/artifacts
a single artifact file (selected by job id)
GET /projects/:id/jobs/:job_id/artifacts/*artifact_path
My current situation is following:
I have test reports which are saved inside the job artifacts when running the pipeline. The artifacts are created on every run of the pipeline independent of its outcome
gitlab-ci.yaml
...
artifacts:
when: always
...
The artifact I am trying to download has a dynamic name
./reports/junit/test-results-${CI_JOB_ID}.xml
If I now want to download the latest test results to a different server than the gitlab server, I have to realize that I don't know the latest job ID, which means:
I can't access the artifact directly because it has a dynamic name
I can't access the artifacts of a specific job
I can access the artifacts of the latest job, but only if it was successful
This leaves me with the situation, that i only get to download the latest test results, if nothing went wrong while testing. To put it mildly, this is suboptimal.
Is there some way to download the artifacts from the latest job run (without knowing the job ID), independent of its outcome?
Is there some way to download the artifacts from the latest job run
(without knowing the job ID), independent of its outcome?
In order to achieve this we will use the Gitlab API in combination with the jq package.
Let's break down this question into components.
Firstly, we need to find out the id of the last executed pipeline for this project. https://docs.gitlab.com/ee/api/pipelines.html#list-project-pipelines
GET /projects/:id/pipelines
For this call you will need your access token, if you don't have one already check
https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html#create-a-personal-access-token
https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html#create-a-project-access-token
You will also need the Project ID
LAST_PIPELINE_ID=$(curl -s --header "PRIVATE-TOKEN: <access_token>" https://gitlab.com/api/v4/projects/<project_id>/pipelines | jq '.[0].id')
Next we will retrieve the job id by providing the job name by using the following API
https://docs.gitlab.com/ee/api/jobs.html#list-pipeline-jobs
GET /projects/:id/pipelines/:pipeline_id/jobs
In your case you need to change the following example with your job's name, in this example let's call it my_job
JOB_ID=$(curl -s --header "PRIVATE-TOKEN: <access_token>" https://gitlab.com/api/v4/projects/<project_id>/pipelines/$LAST_PIPELINE_ID/jobs | jq '.[] | select(.name=="my_job") | .id')
Now we are ready to actually retrieve the artifacts, with the following API
GET /projects/:id/jobs/:job_id/artifacts
https://docs.gitlab.com/ee/api/job_artifacts.html#get-job-artifacts
wget -U "Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.17 (KHTML,like Gecko) Ubuntu/11.04 Chromium/11.0.654.0 Chrome/11.0.654.0 Safari/534.17" --header "PRIVATE-TOKEN: <access_token>" "https://gitlab.com/api/v4/projects/<project_id>/jobs/$JOB_ID/artifacts" -O artifacts.zip
The artifacts are available as artifacts.zip in the folder you executed wget from
Combining them here for clarity
LAST_PIPELINE_ID=$(curl -s --header "PRIVATE-TOKEN: <access_token>" https://gitlab.com/api/v4/projects/<project_id>/pipelines | jq '.[0].id')
JOB_ID=$(curl -s --header "PRIVATE-TOKEN: <access_token>" https://gitlab.com/api/v4/projects/<project_id>/pipelines/$LAST_PIPELINE_ID/jobs | jq '.[] | select(.name=="my_job") | .id')
wget -U "Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.17 (KHTML,like Gecko) Ubuntu/11.04 Chromium/11.0.654.0 Chrome/11.0.654.0 Safari/534.17" --header "PRIVATE-TOKEN: <access_token>" "https://gitlab.com/api/v4/projects/<project_id>/jobs/$JOB_ID/artifacts" -O artifacts.zip
Related
I established a CI/CD pipeline with GitLab. I've created a variable to make an automated TAG called ADD_TAG.
I've put the below script
REF=$$2
TAG_NAME=$$1
TOKEN=$$ANDROID_CHANGELOG_PRIVATE_TOKEN
URL="$${CI_SERVER_URL}/api/v4/projects/$${CI_PROJECT_ID}/repository/tags"
PARAMS="tag_name=$${TAG_NAME}&ref=$${REF}"
curl --fail --request POST --header "PRIVATE-TOKEN: $${TOKEN}" "$${URL}?$${PARAMS}"
but I got the below error
curl: (3) URL using bad/illegal format or missing URL
Please help me to solve it.
Use single $ when referencing a variables.
https://docs.gitlab.com/ee/ci/variables/index.html#predefined-cicd-variables
For example:
URL="$${CI_SERVER_URL}/api/v4/projects/$${CI_PROJECT_ID}/repository/tags"
should be
URL="${CI_SERVER_URL}/api/v4/projects/${CI_PROJECT_ID}/repository/tags"
I am having a workflow with a task that is dependant on external application execution (not residing in Databricks). After external application finishes, how to update the status of a task to complete. Currently, Jobs API doesn't support status updates.
Runs Cancel is available
curl --netrc --request POST \
https://<databricks-instance>/api/2.0/jobs/runs/cancel \
--data '{ "run_id": <run-id> }'
More Details
In the gitlab documentation some URL's are described for the purpose of downloading artifacts from pipelines HERE. They seem to have forgotten to describe HOW to download artifacts given these URLs.
Can it be done in a simple way? Or do I have to install e.g. wget, create a token, define a token, use a token?
If someone could give an example that would be great.
The documentation referenced in the question is supposed to be an REST API call, using the Job Artifact API:
GET /projects/:id/jobs/artifacts/:ref_name/download?job=name
To use this in a script definition inside .gitlab-ci.yml, you can use either:
The JOB-TOKEN header with the GitLab-provided CI_JOB_TOKEN variable.
For example, the following job downloads the artifacts of the test job of the main branch.
The command is wrapped in single quotes because it contains a colon (:):
artifact_download:
stage: test
script:
- 'curl --location --output artifacts.zip --header "JOB-TOKEN: $CI_JOB_TOKEN" "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/jobs/artifacts/main/download?job=test"'
Or the job_token attribute with the GitLab-provided CI_JOB_TOKEN variable.
For example, the following job downloads the artifacts of the test job of the main branch:
artifact_download:
stage: test
script:
- 'curl --location --output artifacts.zip >"https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/jobs/artifacts/main/download?job=test&job_token=$CI_JOB_TOKEN"'
But the artifact: directive is meant to store data in the job workspace, for a new iteration of the job to get back, in the same folder.
No "download" involved, as illustrated in the article "GitLab CI: Cache and Artifacts explained by example" by Anton Yakutovich.
As such, no curl/wget/TOKEN should be needed to access an artifact stored by a previous job execution.
I'm trying to retrieve a build file using the gitlab API. This file was created and stored as an artifact from an upstream pipeline. Running
curl -o download --location --header 'PRIVATE-TOKEN:{MY_API_TOKEN}' https://gitlab.foo.com/api/v4/projects/{PROJECT_ID}/jobs/artifacts/{REF_BRANCH}/download?job={JOB_NAME}
on my local machine gives me a proper build file once I run unzip download. However in the runner, the same command returns a much smaller file which I can't unzip. I've checked that the environment variables that are passed in the runner are right.
job in .gitlab-ci.yml
deploy_production_environment:
stage: deploy_prod
image:
name: banst/awscli
script:
- apk --no-cache add curl
- apk add unzip
- echo $JOB_ID
- echo $FE_BUILD_TOKEN
- echo "https://gitlab.foo.com/api/v4/projects/${PROJECT_ID}/jobs/artifacts/${CI_COMMIT_REF_NAME}/download?job=build_prod"
- aws configure set region us-east-1
- "curl -o download --location --header 'PRIVATE-TOKEN:${FE_BUILD_TOKEN}' https://gitlab.foo.com/api/v4/projects/${PROJECT_ID}/jobs/artifacts/${CI_COMMIT_REF_NAME}/download?job=build_prod"
- ls -l
- unzip download
- aws s3 cp build s3://$S3_BUCKET_PROD --recursive
gitlab job output:
`
output from my local terminal:
Why does the API call from inside the runner consistently result in this much smaller (corrupted?) file while the same call pulls the zip file down correctly on my local machine?
The first check to do when a curl brings back a "small" file it to read its content.
Often, the file is not so much corrupted but includes a text-based error message in it, which can give a clue as to the actual issue.
Adding -v to the curl command can also help illustrating the issue during the curl process (when executed in the context of the GitLab job).
Thank you to VonC for the debugging help, recommending the -v flag to the curl command. It turns out that the single quotes around 'PRIVATE-TOKEN:${FE_BUILD_TOKEN}' prevented the variable from being parsed to its correct string value which was giving a 401 'Permission Denied' error. Removing the single quotes did the trick.
This question already has answers here:
'Including' private project file using `$CI_JOB_TOKEN`
(2 answers)
Closed 3 years ago.
I am trying to make a request to the api of my gitlab host to resolve a tag release information, from what i have seen on the internet is that i could be able to get this information with curl using the pre-defined variables in the job so i could get what i am looking for.
Right now how the curl is being executed is like this:
curl -H "PRIVATE-TOKEN: $CI_JOB_TOKEN" $CI_API_V4_URL/projects/$CI_PROJECT_ID/repository/tags/$CI_COMMIT_TAG
The response of this request is 404: Unauthorized. I have also tried using CI_BUILD_TOKEN environment variable but i also get the same result.
The repository on which i am creating these jobs is not a private, just internal. So i don't know how to make this work.
In case it's necessary, here's how the job pipeline is coded:
fetch-tag-info:
stage: setup
only:
- tags
image: ubuntu:18.04
before_script:
- apt-get update
- apt-get install curl jq -y
script:
- "curl -H \"PRIVATE-TOKEN: $CI_JOB_TOKEN\" $CI_API_V4_URL/projects/$CI_PROJECT_ID/repository/tags/$CI_COMMIT_TAG -o release-notes.json"
You can use CI_JOB_TOKEN only in public project with everyone has access. See open issue in GitLab: https://gitlab.com/gitlab-org/gitlab-foss/issues/29566
Meanwhile, you can use the solution mentioned here: https://stackoverflow.com/a/44469417/6076269, or here: https://stackoverflow.com/a/51005977/6076269