In Gitlab, how can I programatically download the artifacts issued at end of a CI pipeline?
It is easy to download it via the UI but how can I get it through API? In other words, is it possible to access it via a token or something similar?
It is possible through the API as in https://docs.gitlab.com/ee/api/jobs.html#get-job-artifacts
GET /projects/:id/jobs/:job_id/artifacts
Example requests:
Using the PRIVATE-TOKEN header:
curl --location --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects/1/jobs/8/artifacts"
Using the JOB-TOKEN header (only inside .gitlab-ci.yml):
curl --location --header "JOB-TOKEN: $CI_JOB_TOKEN" "https://gitlab.example.com/api/v4/projects/1/jobs/8/artifacts"
Using the job_token parameter (only inside .gitlab-ci.yml):
curl --location --form "job-token=$CI_JOB_TOKEN" "https://gitlab.example.com/api/v4/projects/1/jobs/8/artifacts"
This works for me:
#!/bin/bash
GITLAB_URL="https://gitlab.example.com"
GITLAB_ARTIFACT_TOKEN="<token>"
group="<group>"
project="<project>"
branch="<branch>"
job="<job>"
outZipFile="$project.zip"
outHeadersFile="$outZipFile.httpheaders"
etagArgs=()
# The following is unfortunately not yet supported by GitLab; the returned Etag never changes:
#if [[ -f "$outHeadersFile" ]] && [[ -f "$outZipFile" ]]; then
# etag=$(grep etag < "$outHeadersFile" | cut -f2 -d' ')
# if [[ -n "$etag" ]]; then
# etagArgs=("--header" "If-None-Match: $etag")
# echo "using etag: $etag"
# fi
#fi
response=$(curl "$GITLAB_URL/api/v4/projects/${group}%2F${project}/jobs/artifacts/$branch/download?job=$job" \
--silent \
-w "%{http_code}\n" \
-D "$outHeadersFile" \
-o "$outZipFile.tmp" \
--header "PRIVATE-TOKEN: $GITLAB_ARTIFACT_TOKEN" \
"${etagArgs[#]}")
if [[ "$response" == 4* ]] || [[ "$response" == 5* ]]; then
echo "ERROR - Http status: $response"
rm "$outZipFile.tmp"
exit 1
elif [[ "$response" == 304 ]]; then
echo "$project is up-to-date"
else
echo "update $outZipFile"
mv "$outZipFile.tmp" "$outZipFile"
fi
This worked for me.
Created a new personal access token with API scopes.
Use the token to download it via curl command as shown below.
curl --location --header "PRIVATE-TOKEN:MY_PRIVATE_TOKEN" "https://it-gitlab.cloud.net/api/v4/projects/projectId/jobs/jobId/artifacts" --output watcher
Related
This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
Closed 11 months ago.
I want to have a curl command like below
curl --location --request POST 'https://abcd.com/api/v4/projects/<projectId>/triggers' \
--header 'PRIVATE-TOKEN: <your_access_token>' \
--form 'description="my description"'
Now I wrote a shell script function to generate it dynamically
it need projectId, token, and description as a pramter
callApi(){
while IFS="," read -r -a users; do
for u in "${users[#]}"
do
url="'https://abcd.com/api/v4/projects/$1/triggers'"
echo $url
header="'PRIVATE-TOKEN: $2'"
echo $header
desc="'description=$u token'"
echo $desc
tk=$(curl --location --request POST $url \
--header $header \
--form $desc)
echo $tk
done
done <<< $(cat $3)
}
callApi "<projectId>" "<token>" ./users.csv
It echo perfectly
But
It thorws error
Don't use both double and single quotes like that. You are adding literal single quotes to the url (and other variables) which, as you have discovered, breaks.
Use double quotes if you need to allow command or parameter substitution, single quotes otherwise. Double quote your variables everywhere you dereference them.
Use indentation for readability.
Useless use of cat.
I've captured the function parameters at the start of the function for visibility: I did not even notice the $1 buried in the URL.
callApi() {
local project=$1 token=$2 userfile=$3
while IFS="," read -r -a users; do
for u in "${users[#]}"; do
url="https://abcd.com/api/v4/projects/${project}/triggers"
echo "$url"
header="PRIVATE-TOKEN: ${token}"
echo "$header"
desc="description=$u token"
echo "$desc"
tk=$(
curl --location \
--request POST \
--header "$header" \
--form "$desc" \
"$url"
)
echo "$tk"
done
done < "$userfile"
}
There is a secure file store built into Azure DevOps, available here: https://dev.azure.com/{organization}/{project}/_library?itemType=SecureFiles
I want to upload a file into that secure storage from a pipeline (because the file is generated by the pipeline). I understand that currently there is no dedicated task type to do this, so I have to do it via the Azure DevOps REST API.
How can I do that?
This bash scripts seems to be working for me:
set -e
name="my-file"
token="$(System.AccessToken)"
base_url="https://dev.azure.com/{organization}/{project}/_apis"
local_file="/path/to/local/file"
echo "Check if secure file $name exists"
existing_id="$(curl --fail -s -L -u ":${token}" "${base_url}/distributedtask/securefiles?api-version=5.0-preview.1" | jq -r ".value[] | select(.name == \"$name\").id " )"
if [ -n "$existing_id" ]; then
echo "Delete existing secure file: $existing_id"
curl --fail -v -L -X DELETE -u ":${token}" "${base_url}/distributedtask/securefiles/$existing_id?api-version=5.0-preview.1"
fi
echo "Uploading secure file as: $name"
curl --data-raw "#${local_file}" -u ":${token}" --fail -v -L -X POST -H "Content-Type: application/octet-stream" "${base_url}/distributedtask/securefiles?api-version=5.0-preview.1&name=${name}"
NOTE: I've used the API call examples that I've found here: https://documenter.getpostman.com/view/10072318/SzfAyS4s#3f75659d-4461-4efe-9ba3-77d5112f0bbe
Is it possible to download a file from GitLab using the API? I am using CentOS 6 commandline. The documentation for the API says "Get file from repository" but it is only to get the metadata and not the file itself. The example they give is:
curl --request GET --header 'PRIVATE-TOKEN: <your_access_token>' 'https://gitlab.example.com/api/v4/projects/13083/repository/files/test%2Epy/raw?ref=master'
If I use the raw option, it gives me the contents of the file, but it saves the name with as test%2Epy/raw?ref=master
How do I get it to save as test.py?
Append > test.py to curl as below:
curl --request GET --header 'PRIVATE-TOKEN: ' 'https://gitlab.example.com/api/v4/projects/13083/repository/files/test%2Epy/raw?ref=master' > test.py
It's also possible to use the group and project name instead of the project-id:
response=$(curl "$GITLAB_URL/api/v4/projects/<group>%2F<project>/repository/files/<folder>%2Ftest%2E.py/raw?ref=master" \
--silent \
-o "test.py" \
--header "PRIVATE-TOKEN: $GITLAB_TOKEN")
if [[ $response == 4* ]] || [[ $response == 5* ]]; then
echo ERROR - Http status: "$response"
exit 1
fi
It's important to URL encode the group + project path and the file path as well.
I'm having a strange problem that's driving me crazy!
The task in hand is to start one set of files during the first login of "root" user and another set of files during the second login of the same user. I decided to use the ".profile" and ".bashrc" files and to reload the ".bashrc" file towards the end of the task happening during the first login.
During the first login, I create a private key and certificate signing request, and call an API to get the certificate. I store this certificate and private key in a file location and then modify the ".bashrc" to invoke the second set of files, which make use of this certificate and key to authenticate an application to run.
The problem is that the certificate and key are overwritten and become null randomly after the first boot. I've attached the code below for your review.
FIRST SET OF FILES
".profile" script
# .bash_profile
umask 022
if [ -f ~/.bashrc ]; then
source ~/.bashrc
fi
".bashrc" script
/myFolder/backgroundTask1.sh &
/myFolder/certificateGenerator.sh
backgroundTask1.sh script
pipe=/myFolder/testpipe
if [[ ! -p $pipe ]]; then
mkfifo $pipe
fi
while true
do
## Do some status LED blinking task here
done &
while true
do
if read line < $pipe; then
if [[ "$line" == 'success' ]]; then
## Kill the background LED blinking task created in the above while loop
kill $!
rm $pipe
exit
elif [[ "$line" == 'failed' ]]; then
kill $!
rm $pipe
exit
fi
fi
done
certificateGenerator.sh script
PLEASE NOTE THE LAST FEW LINES WHERE I MODIFY THE BASHRC SCRIPT
Please also note the files /anotherFolder/myKey.key and /anotherFolder/myCert.crt
#!/bin/bash
## Named pipe location for communicating to backgroundTask1
pipe=/myFolder/testpipe
openssl req -new -newkey rsa:2048 -nodes -out certificateSigningRequest.csr -keyout /anotherFolder/myKey.key -subj "/C=myCountry/ST=myState/L=myCity/O=myCompany/OU=myOU/CN=myDevice"
cert_req=$(<$certificateSigningRequest.csr)
## Get AD token from Azure for talking to my custom API hosted on Azure
response=$(curl -o - -s -w "%{http_code}\n" -X POST \
https://login.microsoftonline.com/myCompany.onmicrosoft.com/oauth2/token \
-H 'content-type: application/x-www-form-urlencoded' \
-d 'grant_type=client_credentials&resource=https%3A%2F%2Fmanagement.core.windows.net%2F&client_id=myClientID&client_secret=mySecret')
if [ $?==0 ]; then
status=$(echo $response | tail -c 4)
body=${response::-3}
token=$(echo $body | jq -r '.access_token')
fi
## Send CSR to my custom API to get certificate
response=$(jq -n --arg csr "$cert_req" \
'{
cert: {
csr: $csr
}
}' |
curl -o - -s -w "%{http_code}\n" -X POST \
https://myCustomAPI.azurewebsites.net/api/v1/customEndpoint \
-H "authorization: Bearer $token" \
-H "content-type: application/json" \
-d #-
)
## Parse the response to find out if the request succeeded
if [ $?==0 ]; then
destCertDir=/anotherFolder/myCert.crt
status=$(echo $response | tail -c 4)
body=${response::-3}
cert=$(echo $body | jq -r '.certificate')
if [ "$status" == "$http_success" ]; then
echo "$cert" > "$destCertDir"
## Change .bashrc for next boot
echo '/myFolder/backgroundTask2.sh &' > ~/.bashrc
echo '/myFolder/applicationAuthenticator.sh' >> ~/.bashrc
echo "success" > $pipe
exit
fi
fi
SECOND SET OF FILES
".profile" script
# .bash_profile
umask 022
if [ -f ~/.bashrc ]; then
source ~/.bashrc
fi
".bashrc" script
/myFolder/backgroundTask2.sh &
/myFolder/applicationAuthenticator.sh
backgroundTask2.sh script
pipe=/myFolder/testpipe2
if [[ ! -p $pipe ]]; then
mkfifo $pipe
fi
while true
do
## Do some status LED blinking task here
done &
while true
do
if read line < $pipe; then
if [[ "$line" == 'success' ]]; then
## Kill the background LED blinking task created in the above while loop
kill $!
rm $pipe
exit
elif [[ "$line" == 'failed' ]]; then
kill $!
rm $pipe
exit
fi
fi
done
applicationAuthenticator.sh script
PLEASE NOTE HOW I MODIFY BASHRC TO STARTUP NORMAL FROM NEXT REBOOT TOWARDS THE END OF THIS SCRIPT
#!/bin/bash
## Named pipe location for communicating to backgroundTask2
pipe=/myFolder/testpipe2
response=$(curl https://myProduct/myCustomAPI.com \
--cert /anotherFoler/myCert.crt --key /anotherFolder/myKey.key \
-H 'Content-Type: application/x-www-form-urlencoded; charset=utf-8' \
-d 'data=xxx')
if [[ $response == 204 ]; then
echo '' > ~/.bashrc
echo "success" > $pipe
exit
else
echo "failed" > $pipe
exit
fi
Problem
Even thought the first set of files create the key and certificate, they are overwritten to NULL after the first reboot.
To make sure that they exist before reboot, I go to the location "/anotherFolder" and check the files physically. They have the full key and certificate before reboot. When I reboot and see that the script fails, the same key and certificate files (which had actual data before reboot) now have NULL values.
I wrote the following script:
#!/bin/bash
if [ $# -ne 1 ];then
echo "Usage: ./script <input-file>"
exit 1
fi
while read user pass; do
curl -iL --data-urlencode user="$user" --data-urlencode password="$pass" http://foo.com/signin 1>/dev/null 2>&1
if [ $? -eq 0 ];then
echo "ok"
elif [ $? -ne 0 ]; then
echo "failed"
fi
done < $1
question:
Whenever I run with even wrong user & pass the result is ok for me...
How can I be sure that my parameters are correct or not?
Thanks
It’s because you are getting output from your curl command. Typing that command with random user/pass gets this:
$ curl -iL --data-urlencode user=BLAHBLAH --data-urlencode password=BLAH http://foo.com/signin
HTTP/1.1 301 Moved Permanently
Server: nginx/1.0.5
Date: Wed, 19 Feb 2014 05:53:36 GMT
Content-Type: text/html
Content-Length: 184
Connection: keep-alive
Location: http://www.foo.com/signin
. . .
. . .
<body>
<!-- This file lives in public/500.html -->
<div class="dialog">
<h1>We're sorry, but something went wrong.</h1>
</div>
</body>
</html>
Hence,
$ echo $?
0
But modify the URL to garbage:
$ curl -iL --data-urlencode user=BLAHBLAH --data-urlencode password=BLAH http://foof.com/signin
curl: (6) Could not resolve host: foof.com
$ echo $?
6
Even when the login fails, the HTTP server will still return a page with an error message. As curl is able to retrieve this page it will finish successfully.
In order to get curl to fail on server errors, you need the --fail parameter. Although this may not be fail-safe according to the curl man page, it is worth a try.
If --fail does not work, you could parse the header in the output of your curl request or have a look at the --write-out parameter.