Sending cURL command through Gitlab CICD file - gitlab

I have a .gitlab-ci.yml file. I need to send a cURL command to a REST API (documentation here). When I hardcode parameters like bearer, roomid it works and the message is received on the client (in this example a Webex Teams bot).
The below works (static entries for bearer, roomid and a message)
before_script:
- export roomid
- export bearer
.send_message:
send:
- |
curl --location --request POST 'https://webexapis.com/v1/messages' \
--header 'Authorization: Bearer M***0f' \
--header 'Content-Type: application/json' \
--data-raw '{
"roomId" : "Y***Ri",
"markdown" : "# Message formatted in markdown",
}'
stages:
- convert
convert:
stage: convert
script:
- python3 convert.py -t dna_sites.xlsx
- !reference [.send_message, send]
variables:
msg: "Conversion complete"
However, when I want to make it more generic by using values stored in Gitlab CICD variable section (bearer, roomid) or elsewhere in the .gitlab-ci..yml file (msg), I cannot get it to work. Therefor I'm referencing them through $ sign ($bearer, $roomid and $msg)
before_script:
- export roomid
- export bearer
.send_message:
send:
- |
curl --location --request POST 'https://webexapis.com/v1/messages' \
--header 'Authorization: Bearer $bearer' \
--header 'Content-Type: application/json' \
--data-raw '{
"roomId" : "$roomid",
"markdown" : "$msg",
}'
stages:
- convert
convert:
stage: convert
script:
- python3 convert.py -t dna_sites.xlsx
- !reference [.send_message, send]
variables:
msg: "Conversion complete"
My guess is I'm doing something wrong with the quotes ' or ".

Related

I am getting "TypeError: openai.createEdit is not a function" while trying to use the Create edit function, can anyone point if there's a mistake?

I tried the the same in a node.js web app and in postman, I have added the code and curl for respectively below:
Node JS:
const response = await openai.createEdit("text-davinci-002", {
input: "What day of the wek is it?",
instruction: "Fix the spelling mistakes" });
CURL:
curl --location --request POST 'https://api.openai.com/v1/engines/text-davinci-002/edits'
--header 'Content-Type: application/json'
--header 'Authorization: Bearer API_KEY'
--data-raw '{
"input": "What day of the wek is it?",
"instruction": "Fix the spelling mistakes"
}'

Gitlab CICD: use functions inside gitlab-ci.yml

I have a .gitlab-ci.yml file which allows needs to execute the same function for every step. I have the following and this works.
image:
name: hashicorp/terraform
before_script:
- export MYDATE=$(date "+%d/%m/%y - %H:%M:%S")
stages:
- validate
- plan
validate:
stage: validate
script:
- terraform validate
- 'curl --request POST --header "Authorization: Bearer $bearer" --form "text=$MYDATE $msg" https://api.teams.com/v1/messages'
variables:
msg: "Example1"
plan:
stage: plan
script:
- terraform validate
- 'curl --request POST --header "Authorization: Bearer $bearer" --form "text=$MYDATE $msg" https://api.teams.com/v1/messages'
variables:
msg: "Example2"
Given it is always the same curl command, I wanted to use a function which I declare once and can use in every step. Something along the lines of below snippet.
image:
name: hashicorp/terraform
before_script:
- export MYDATE=$(date "+%d/%m/%y - %H:%M:%S")
.send_message: &send_message
script:
- 'curl --request POST --header "Authorization: Bearer $bearer" --form "text=$MYDATE $msg" https://api.teams.com/v1/messages'
stages:
- validate
- plan
validate:
stage: validate
script:
- terraform validate
- &send_message
variables:
msg: "Example1"
plan:
stage: plan
script:
- terraform validate
- &send_message
variables:
msg: "Example2"
How could I use such a function in a .gitlab-ci.yml file.
you can used include with !reference such as:
functions.yml
.send_message:
script:
- 'curl --request POST --header "Authorization: Bearer $bearer" --form "text=$MYDATE $msg" https://api.teams.com/v1/messages'
.gitlab-ci.yml
include:
- local: functions.yml
default:
image:
name: hashicorp/terraform
before_script:
- export MYDATE=$(date "+%d/%m/%y - %H:%M:%S")
stages:
- validate
- plan
validate:
stage: validate
script:
- terraform validate
- !reference [.send_message, script]
variables:
msg: "Example1"
plan:
stage: plan
script:
- terraform validate
- !reference [.send_message, script]
variables:
msg: "Example2"
ref: https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html#reference-tags
You can also use a regular old bash function defined at the top:
before_script:
- export MYDATE=$(date "+%d/%m/%y - %H:%M:%S")
- send_bearer () { terraform validate; curl --request POST --header "Authorization: Bearer $bearer" --form "text=$MYDATE $1" https://api.teams.com/v1/messages; }
...
validate:
stage: validate
script:
- send_bearer $msg
variables:
msg: "Example1"
plan:
stage: plan
script:
- send_bearer $msg
variables:
msg: "Example2"

Error in Xray Rest API call for importing Test Execution Result

I know this query has been answered in so many post but those have not helped me. I did research, and tried, but still facing issue in making an API call to import test execution result.
Approach I took:
Created Test(Test Details: Cucumber), Test Precondition, Test Set, Test Plan and Test Execution
Exported Test using "Xray - Export to Cucumber" option
Added this in my BDD-Cucumber framework, executed and it has generated me cucumber.json file after execution
Trying API call using postman
/api/v1/import/execution/cucumber
curl --location --request POST 'https://xray.cloud.xpand-it.com/api/v1/import/execution/cucumber' \
--header 'Authorization: Bearer $token’ \
--header 'Content-Type: application/json' \
--data-binary '#/Users/aranjan/Downloads/cucumber.json'
Error:
{ "error": "Error creating Test Execution - Team is required."}
Now, this means it is trying to create new instead of update existing
Then, I used
/api/v1/import/execution/cucumber/multipart
curl --location --request POST 'https://xray.cloud.xpand-it.com/api/v1/import/execution/cucumber/multipart' \
--header 'Authorization: Bearer $token’ \
--form 'info=#/Users/aranjan/Downloads/xrayresultimport.json' \
--form 'result=#/Users/aranjan/Downloads/cucumber.json'
Error:
{ "error": "Unexpected field (result)"}
xrayresultimport.json
{
"fields": {
"project": {
"key": "HYP"
},
"customfield_10962": [
"Team","TeamQAAuto"
],
"issuetype": {
"id": "10722"
}
}
}
/api/v1/import/execution
curl --location --request POST 'https://xray.cloud.xpand-it.com/api/v1/import/execution' \
--header 'Authorization: Bearer $token’ \
--header 'Content-Type: application/json' \
--data-raw '{
"testExecutionKey": "HYP-3313",
"info" : {
"startDate" : "2020-09-25T11:47:35+01:00",
"finishDate" : "2020-09-25T11:53:00+01:00",
"testPlanKey" : "HYP-3341"
},
"tests" : [
{
"testKey" : "HYP-3330",
"start" : "2020-09-25T11:47:35+01:00",
"finish" : "2020-09-25T11:50:56+01:00",
"comment" : "Successful execution",
"status" : "PASSED"
}
]
}'
{ "error": "Error updating Test Execution - Issue update failed!"}
Agenda:
I want to import the execution result in my existing Test Execution.
I request you to guide me here.
Thanks in advance.
Currently, if you use the multipart endpoint it will always create new Test Executions.
The multipart request has a minor typo: you should have "results" instead of "result". An example would be something like this:
curl -H "Content-Type: multipart/form-data" -X POST -F info=#xrayresultimport.json -F results=#cucumber.json -H "Authorization: Bearer $token" https://xray.cloud.xpand-it.com/api/v2/import/execution/cucumber/multipart
That should make it work :)
Note: concerning the last example you gave, there could be several causes for it, including restrictions on Jira side. That would be analyzed by the Xray support team.

GitLab CI How to POST JSON data to a url within a CI job using here-document?

Hoping someone can help. I am experiencing difficulty making a curl POST request with JSON data from a gitlab CI job.
The curl request works fine in a local terminal session, (N.B I did not use double quotes in terminal session). If I do not escape double quotes in the gitlab CI yaml I get the error curl: (3) [globbing] nested brace in column 112
If I escape the double quotes in the GitLab CI job, as shown below I get the error:
curl: (3) [globbing] unmatched brace in column 1
In all cases I get the error /bin/bash: line 134: warning: here-document at line 134 delimited by end-of-file (wanted `EOF')
Is it possible to POST JSON data using here-documents from a GitLab CI job?
.gitlab-ci.yml job extract
release:
image: node:12-stretch-slim
stage: release
before_script:
- apt-get update && apt-get install -y curl git jq
script:
- version=$(git describe --tags | cut -d'-' -f 1 | sed 's/^v*//')
- echo "generating release for version ${version}"
- npm pack
# - >
# url=$(curl
# --header "Private-Token: $API_TOKEN"
# -F "file=#${CI_PROJECT_DIR}/objectdetection-plugin-${version}.tgz" "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/uploads"
# |
# jq '.url')
- url="http://www.example.com"
- echo "Retrieved url from file upload as ${url}"
- echo "The full url would be ${CI_PROJECT_URL}/${url}"
- >
curl -X POST https://requestbin.io/1f84by61
--header 'Content-Type: application/json; charset=utf-8'
--data-binary #- << EOF
{
\"name\": \"Release v${version}\",
\"tag_name\": \"v${version}\",
\"ref\": \"v${version}\",
\"description\": \"Test\",
\"assets\": {
\"links\": [
{
\"name\": \"objectdetection-plugin-source\",
\"url\": \"CI_PROJECT_URL/${url}\",
\"filepath\": \"${url}\",
\"link_type\": \"other\"
}
]
}
}
EOF
when: manual
only:
- /^release-.*$/
Solved it after reading this
Using |- preserves newlines within the command and does not append a newline at the end of the command string. Used this principle to save the JSON data to a variable and then referenced the variable in the subsequent curl command.
Below I have included the script:
release:
image: node:12-stretch-slim
stage: release
before_script:
- apt-get update && apt-get install -y curl git jq
script:
- git fetch --prune --unshallow
- version=$(git describe --tags | cut -d'-' -f 1 | sed 's/^v*//')
- npm pack
- >
url=$(curl --silent --show-error
--request POST
--header "Private-Token: $API_TOKEN"
-F "file=#${CI_PROJECT_DIR}/objectdetection-plugin-${version}.tgz" "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/uploads"
|
jq --raw-output --monochrome-output '.url')
- |-
PAYLOAD=$(cat << JSON
{
"name": "Release v$version",
"tag_name": "v$version",
"ref": "v$version",
"description": "$(sed -zE 's/\r\n|\n/\\n/g' < CHANGELOG.md)",
"assets": {
"links": [
{
"name": "objectdetection-plugin-source",
"url": "$CI_PROJECT_URL$url",
"filepath": "$url",
"link_type": "other"
}
]
}
}
JSON
)
- echo "$PAYLOAD"
- >
http_response=$(curl --silent --show-error --write-out "%{http_code}" -o response.txt
--request POST "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/releases"
--header 'Content-Type: application/json'
--header 'Accept: application/json'
--header "Private-Token: ${API_TOKEN}"
--data-binary "${PAYLOAD}")
- |-
if [ $http_response != "201" ]; then
exit 1
else
echo "Server returned:"
cat response.txt
fi
when: manual
allow_failure: false
only:
- /^release-.*$/
I was trying to send a message to my Google Chat, WebHook URL, and the above solution didn't work.
So I used this instead
'curl --silent --show-error --location --request POST ${GCHAT_DEVOPS_WEBHOOK} --header ''Content-Type: application/json; charset=UTF-8'' --data-raw ''{"text":"Hey <users/all>: A new build has been deployed to *''${APP_ENVIRONMENT}''* via ''${CI_JOB_NAME}'' - *''${APP_BRANCH}''*"}'''

parse error: Invalid numeric literal at line 1, column 8

I am trying to get a token to some site by using curl. It looks like request is done correctly because I have to wait a bit for response however something is during deserialization because I always got error: parse error: Invalid numeric literal at line 1, column 8
This is how script looks like:
TOKEN=$(curl --request POST \
--url 'https://${DOMAIN_NAME}/getmy/token' \
--header 'content-type: application/json' \
--data '{"grant_type":"password", "username":"${USER_EMAIL}",
"password":"${USER_PASSWORD}",
"audience":"https://localhost:8443/my-composite-service", "scope":"openid
email test:read test:write", "client_id": "${CLIENT_ID}",
"client_secret": "${CLIENT_SECRET}"}' -s | jq -r .access_token)
Is it because of jq?
What is more I am sure that env variables are there, even with hard coded values the same error will be thrown.
Thank you in advance
Some hints:
Do not put everything in one line, make it readable instead.
Structure your code with functions.
Do error handling.
Use Bash's debugging functionality.
Do not build JSON with string concatenation, use JQ instead, because only JQ quotes JSON data correctly. A password may contain quoting characters.
An example:
set -eu
set -x
USER_EMAIL="user#domain.org"
USER_PASSWORD="password"
CLIENT_ID="id"
CLIENT_SECRET="secret"
DOMAIN_NAME="domain.org"
data()
{
local template='
{
"grant_type": "password",
"username": $username,
"password": $password,
"audience": "https://localhost:8443/my-composite-service",
"scope": "openid email test:read test:write",
"client_id": $client_id,
"client_secret": $client_secret
}'
if jq <<<null -c \
--arg username "${USER_EMAIL}" \
--arg password "${USER_PASSWORD}" \
--arg client_id "${CLIENT_ID}" \
--arg client_secret "${CLIENT_SECRET}" \
"$template"
then
return
else
printf "ERROR: Can not format request data." >&2
exit 1
fi
}
post()
{
if curl --request POST \
--url 'https://${DOMAIN_NAME}/getmy/token' \
--header 'content-type: application/json' \
--data "$1" \
-s
then
return
else
printf "ERROR: Can not send post request." >&2
exit 1
fi
}
token()
{
if jq -r .access_token
then
return
else
printf "ERROR: Can not parse JSON response." >&2
exit 1
fi
}
TOKEN="$(post "$(data)" | token)"

Resources