Is it possible to download a file from GitLab using the API? I am using CentOS 6 commandline. The documentation for the API says "Get file from repository" but it is only to get the metadata and not the file itself. The example they give is:
curl --request GET --header 'PRIVATE-TOKEN: <your_access_token>' 'https://gitlab.example.com/api/v4/projects/13083/repository/files/test%2Epy/raw?ref=master'
If I use the raw option, it gives me the contents of the file, but it saves the name with as test%2Epy/raw?ref=master
How do I get it to save as test.py?
Append > test.py to curl as below:
curl --request GET --header 'PRIVATE-TOKEN: ' 'https://gitlab.example.com/api/v4/projects/13083/repository/files/test%2Epy/raw?ref=master' > test.py
It's also possible to use the group and project name instead of the project-id:
response=$(curl "$GITLAB_URL/api/v4/projects/<group>%2F<project>/repository/files/<folder>%2Ftest%2E.py/raw?ref=master" \
--silent \
-o "test.py" \
--header "PRIVATE-TOKEN: $GITLAB_TOKEN")
if [[ $response == 4* ]] || [[ $response == 5* ]]; then
echo ERROR - Http status: "$response"
exit 1
fi
It's important to URL encode the group + project path and the file path as well.
Related
I need to call a REST API using a curl command inside a gitlab-ci script.
There are some variables that need to be replaced in the curl command, but if I use same variable, looks like it is not replaced by its value.
Below a piece of my gitlab-ci that works fine without any variable:
create_changelog:
stage: create_changelog
variables:
REDMINE_API_KEY: "cbf456d6c74ae85bfbcddfd130c75af4f0977448"
script:
- |
curl -X PUT \
"https://redmine.example.com/projects/flutter-2/wiki/abc1.xml" \
-H "cache-control: no-cache" \
-H "content-type: application/xml" \
-H "x-redmine-api-key: cbf456d6c74ae85bfbcddfd130c75af4f0977448" \
-d "<?xml version=\"1.0\"?><wiki_page><text>Example</text><parent title=\"Release\"/></wiki_page>"
But in case a replace, for example, the apk_key by a variable, like below, I dont get the wiki page created on the server.
create_changelog:
stage: create_changelog
variables:
REDMINE_API_KEY: "cbf456d6c74ae85bfbcddfd130c75af4f0977448"
script:
- |
curl -X PUT \
"https://redmine.example.com/projects/flutter-2/wiki/abc1.xml" \
-H "cache-control: no-cache" \
-H "content-type: application/xml" \
-H "x-redmine-api-key: ${REDMINE_API_KEY}" \
-d "<?xml version=\"1.0\"?><wiki_page><text>Example</text><parent title=\"Release\"/></wiki_page>"
How can I use variable in the curl command above ?
This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
Closed 11 months ago.
I want to have a curl command like below
curl --location --request POST 'https://abcd.com/api/v4/projects/<projectId>/triggers' \
--header 'PRIVATE-TOKEN: <your_access_token>' \
--form 'description="my description"'
Now I wrote a shell script function to generate it dynamically
it need projectId, token, and description as a pramter
callApi(){
while IFS="," read -r -a users; do
for u in "${users[#]}"
do
url="'https://abcd.com/api/v4/projects/$1/triggers'"
echo $url
header="'PRIVATE-TOKEN: $2'"
echo $header
desc="'description=$u token'"
echo $desc
tk=$(curl --location --request POST $url \
--header $header \
--form $desc)
echo $tk
done
done <<< $(cat $3)
}
callApi "<projectId>" "<token>" ./users.csv
It echo perfectly
But
It thorws error
Don't use both double and single quotes like that. You are adding literal single quotes to the url (and other variables) which, as you have discovered, breaks.
Use double quotes if you need to allow command or parameter substitution, single quotes otherwise. Double quote your variables everywhere you dereference them.
Use indentation for readability.
Useless use of cat.
I've captured the function parameters at the start of the function for visibility: I did not even notice the $1 buried in the URL.
callApi() {
local project=$1 token=$2 userfile=$3
while IFS="," read -r -a users; do
for u in "${users[#]}"; do
url="https://abcd.com/api/v4/projects/${project}/triggers"
echo "$url"
header="PRIVATE-TOKEN: ${token}"
echo "$header"
desc="description=$u token"
echo "$desc"
tk=$(
curl --location \
--request POST \
--header "$header" \
--form "$desc" \
"$url"
)
echo "$tk"
done
done < "$userfile"
}
In Gitlab, how can I programatically download the artifacts issued at end of a CI pipeline?
It is easy to download it via the UI but how can I get it through API? In other words, is it possible to access it via a token or something similar?
It is possible through the API as in https://docs.gitlab.com/ee/api/jobs.html#get-job-artifacts
GET /projects/:id/jobs/:job_id/artifacts
Example requests:
Using the PRIVATE-TOKEN header:
curl --location --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects/1/jobs/8/artifacts"
Using the JOB-TOKEN header (only inside .gitlab-ci.yml):
curl --location --header "JOB-TOKEN: $CI_JOB_TOKEN" "https://gitlab.example.com/api/v4/projects/1/jobs/8/artifacts"
Using the job_token parameter (only inside .gitlab-ci.yml):
curl --location --form "job-token=$CI_JOB_TOKEN" "https://gitlab.example.com/api/v4/projects/1/jobs/8/artifacts"
This works for me:
#!/bin/bash
GITLAB_URL="https://gitlab.example.com"
GITLAB_ARTIFACT_TOKEN="<token>"
group="<group>"
project="<project>"
branch="<branch>"
job="<job>"
outZipFile="$project.zip"
outHeadersFile="$outZipFile.httpheaders"
etagArgs=()
# The following is unfortunately not yet supported by GitLab; the returned Etag never changes:
#if [[ -f "$outHeadersFile" ]] && [[ -f "$outZipFile" ]]; then
# etag=$(grep etag < "$outHeadersFile" | cut -f2 -d' ')
# if [[ -n "$etag" ]]; then
# etagArgs=("--header" "If-None-Match: $etag")
# echo "using etag: $etag"
# fi
#fi
response=$(curl "$GITLAB_URL/api/v4/projects/${group}%2F${project}/jobs/artifacts/$branch/download?job=$job" \
--silent \
-w "%{http_code}\n" \
-D "$outHeadersFile" \
-o "$outZipFile.tmp" \
--header "PRIVATE-TOKEN: $GITLAB_ARTIFACT_TOKEN" \
"${etagArgs[#]}")
if [[ "$response" == 4* ]] || [[ "$response" == 5* ]]; then
echo "ERROR - Http status: $response"
rm "$outZipFile.tmp"
exit 1
elif [[ "$response" == 304 ]]; then
echo "$project is up-to-date"
else
echo "update $outZipFile"
mv "$outZipFile.tmp" "$outZipFile"
fi
This worked for me.
Created a new personal access token with API scopes.
Use the token to download it via curl command as shown below.
curl --location --header "PRIVATE-TOKEN:MY_PRIVATE_TOKEN" "https://it-gitlab.cloud.net/api/v4/projects/projectId/jobs/jobId/artifacts" --output watcher
I am able to upload a local file to a remote server using the command
curl -T abc.pom -uusername:password URL
But I am not able to upload a remote file to that URL. The command I am using is this
curl -T remote-file-url -uusername:password URL
Is it not possible to do this? Is downloading it and then uploading it again the only option here?
My approach:
TF=/tmp/temp && curl <REMOTE_FILE_URL> -o $TF && curl -T $TF <UPLOAD_URL> && rm -f $TF
It might be possible to pipe the content of file from 1st to 2nd cURL but then the second one has to prepare the HTML form-encoded body by itself. The -T is a shorthand for this - it creates the form and populates it directly:
curl <REMOTE_FILE_URL> | curl -i -X POST -H "Content-Type: multipart/form-data" -d #- <UPLOAD_URL>
You may curl directly from the remote host by sending a command through ssh to the destination host :
ssh user#destination_host 'curl -o destination_file_path remote_file_url'
I am using cURL to post a XML file on Linux as follows:
curl -X POST --header "Content-Type: text/xml" -d #test.xml "https://www.example.com"
How can I check the status of the cURL command to see if the file was posted or not?
Thanks for any help.
I'm not sure if I got you but you can basically check the return value of the curl command. The return value of the last command is stored in the variable $?.
Example:
curl -X POST --header "Content-Type: text/xml" -d #test.xml "https://www.example.com"
ret=$? # store return value for later usage in the error message
if [ $ret != 0 ] ; then
echo "POST failed with exit code $ret"
fi
This list of possible error codes can be found at the bottom of the man page. They are very helpful for debugging.
You can get the response status of your operation using -w %{http_code}
curl -s -o out.txt -w %{http_code} http://www.example.com/
In this example -s means silent mode, and -o out.txt means to save the response(usually html) into a file.
For this above command, you'l have output 200 when its success.