I have this script in my gitlab yml file:
script:
- "curl -X POST -H 'Content-type: application/json' --data '{\"tag\":\"${CI_COMMIT_TAG}\", \"projectId\":\"${PROJECT_ID}\"}' http://localhost:1337/slack"**
Let's say when I create a tag named 'testing-v1' on gitlab, the variables $CI_COMMIT_TAG should be 'testing-v1', but instead it is showing as it is.
It's not replacing.
If I hard code the tag name and project id like this
- "curl -X POST -H 'Content-type: application/json' --data '{\"tag\":\"testing-v1\", \"projectId\":\"1111111\"}' http://localhost:1337/slack"**
It's working.
If you copy-paste your curl line (without the first and last double-quote) in shellcheck.net, you will be pointed to SC2016
Expressions don't expand in single quotes, use double quotes for that.
Single quotes prevent expansion of everything, including variables and command substitution.
If you want to use the values of variables and such, use double quotes instead.
Note that if you have other items that needs single quoting, you can use both in a single word:
echo '$1 USD is '"$rate GBP"
See an example of a working gitlab-ci.yml with curl here:
image: ruby:2.3
stages:
- clear-cache
CLOUDFLARE_URL: https://api.cloudflare.com/client/v4/zones/$zone/purge_cache
EMAIL: "X-Auth-Email: $email"
AUTH: "X-Auth-Key: $key"
CONTENT: 'Content-Type: application/json'
clear-cache:
stage: clear-cache
script:
- curl -X POST "${CLOUDFLARE_URL}" -H "${EMAIL}" -H "${AUTH}" -H "${CONTENT}" --data '{"purge_everything":true}'
only:
- master
Related
I need to call a REST API using a curl command inside a gitlab-ci script.
There are some variables that need to be replaced in the curl command, but if I use same variable, looks like it is not replaced by its value.
Below a piece of my gitlab-ci that works fine without any variable:
create_changelog:
stage: create_changelog
variables:
REDMINE_API_KEY: "cbf456d6c74ae85bfbcddfd130c75af4f0977448"
script:
- |
curl -X PUT \
"https://redmine.example.com/projects/flutter-2/wiki/abc1.xml" \
-H "cache-control: no-cache" \
-H "content-type: application/xml" \
-H "x-redmine-api-key: cbf456d6c74ae85bfbcddfd130c75af4f0977448" \
-d "<?xml version=\"1.0\"?><wiki_page><text>Example</text><parent title=\"Release\"/></wiki_page>"
But in case a replace, for example, the apk_key by a variable, like below, I dont get the wiki page created on the server.
create_changelog:
stage: create_changelog
variables:
REDMINE_API_KEY: "cbf456d6c74ae85bfbcddfd130c75af4f0977448"
script:
- |
curl -X PUT \
"https://redmine.example.com/projects/flutter-2/wiki/abc1.xml" \
-H "cache-control: no-cache" \
-H "content-type: application/xml" \
-H "x-redmine-api-key: ${REDMINE_API_KEY}" \
-d "<?xml version=\"1.0\"?><wiki_page><text>Example</text><parent title=\"Release\"/></wiki_page>"
How can I use variable in the curl command above ?
This question already has answers here:
Build a JSON string with Bash variables
(17 answers)
Closed 2 years ago.
I am working on simple curl POST script and I have problem with string interpolation of a variable that is initialized with data from a script. Let me describe it:
#!/bin/bash
EMAIL=$(source ./random_email_generator.sh) #this generates some random string
echo "$EMAIL" #here email is printed well
curl --insecure --header "Content-Type: application/json" \
--request POST \
--data '{"email":'"$EMAIL"',"password":"validPassword"}' \ #problem lies here because email is always empty string
http:/ticketing.dev/api/users/signup
To sum up, there is a lot of quotes and double quotes here and it is abit mixed up, how can I resolve value of EMAIL in place of json body?
Thanks in advance
The variable is being interpolated properly. The problem is that you're not wrapping it in double quotes, so you're creating invalid JSON.
curl --insecure --header "Content-Type: application/json" \
--request POST \
--data '{"email":"'"$EMAIL"'","password":"validPassword"}' \
http:/ticketing.dev/api/users/signup
When you want to see what the command actually looks like, put echo before curl or run the script with bash -x. You would have seen
--data {"email":foo#bar.com,"password":"validPassword"}
It would be better if you used the jq tool to manipulate JSON in bash scripts.
Below my .gitlab-ci.yml file:
image: docker:latest
services:
- docker:dind
stages:
- deploy_dev_env
Deploy DEV Environment:
stage: deploy_dev_env
script:
- curl -v -H "Content-Type: multipart/form-data" -X PUT -F uploadInput=#schema/schema.xml -F overwrite=true -F xmlaEnabledFlag=true -F parameters="DataSource=outputDS" -F parameters="EnableXmla=true" -u $PENTAHO_DEPLOY_USER:$PENTAHO_DEPLOY_PASSWORD http://$PENTAHO_HOST/pentaho/plugin/data-access/api/datasource/analysis/catalog/sca
This very simple script is causing an error because the colons(:) in the "Content-Type: multipart/form-data" piece.
Running the CI Lint in the script I get the following:
Status: syntax is incorrect
jobs:deploy dev environment:script config should be a string or an array containing strings and arrays of strings
If I replace "Content-Type: multipart/form-data" by "Content-Type multipart/form-data" (removed the ":"), I get I correct syntax for my .gitlab-ci.yml file.
Is this a bug or should I re-write my curl command in a different way ?
Docs warn about special characters in yaml script:
Note: Sometimes, script commands will need to be wrapped in single or double quotes. For example, commands that contain a colon (:) need to be wrapped in quotes so that the YAML parser knows to interpret the whole thing as a string rather than a “key: value” pair. Be careful when using special characters: :, {, }, [, ], ,, &, *, #, ?, |, -, <, >, =, !, %, #, `.
One of the option would be to use yaml block scalar
Deploy DEV Environment:
stage: deploy_dev_env
script: >
curl -v -H "Content-Type: multipart/form-data" -X PUT -F uploadInput=#schema/schema.xml -F overwrite=true -F xmlaEnabledFlag=true -F parameters="DataSource=outputDS" -F parameters="EnableXmla=true" -u $PENTAHO_DEPLOY_USER:$PENTAHO_DEPLOY_PASSWORD http://$PENTAHO_HOST/pentaho/plugin/data-access/api/datasource/analysis/catalog/sca
or use some other way to escape colon in yaml.
I faced this issue due to yaml syntax before and the workaround i found was
Store the "Content-Type: application/json" as an environment variable in the CICD settings
for example
Key: HEADER
Value: Content-Type: application/json
and in your script
- curl -v -H "$HEADER" -X PUT -F uploadInput=#schema/schema.xml -F overwrite=true -F xmlaEnabledFlag=true -F parameters="DataSource=outputDS" -F parameters="EnableXmla=true" -u $PENTAHO_DEPLOY_USER:$PENTAHO_DEPLOY_PASSWORD http://$PENTAHO_HOST/pentaho/plugin/data-access/api/datasource/analysis/catalog/sca
After going through several options, I found below simple solution
Use like this,
Content_Type="Content-Type:"
export RESULT="$(echo "$Content_Type" multipart/form-data)"
echo "Result is " $RESULT
Use $RESULT as variable there
My question is very simple. I want to use environment variables in a cURL command sth similar to this:
curl -k -X POST -H 'Content-Type: application/json' -d '{"username":"$USERNAME","password":"$PASSWORD"}'
When I run the command $USERNAME is passed to the command as a "$USERNAME" string not the value of the variable. Is there a way to escape this situation?
Thanks.
Single quotes inhibit variable substitution, so use double quotes. The inner double quotes must then be escaped.
... -d "{\"username\":\"$USERNAME\",\"password\":\"$PASSWORD\"}"
Since this answer was written in 2015, it has become clear that this technique is insufficient to properly create JSON:
$ USERNAME=person1
$ PASSWORD="some \"gnarly 'password"
$ echo "{\"username\":\"$USERNAME\",\"password\":\"$PASSWORD\"}"
{"username":"person1","password":"some "gnarly 'password"}
$ echo "{\"username\":\"$USERNAME\",\"password\":\"$PASSWORD\"}" | jq .
parse error: Invalid numeric literal at line 1, column 47
The quoting problem are clear. The (shell) solutions are not
Current best practice: use a JSON-specific tool to create JSON:
jq
$ jq -n -c --arg username "$USERNAME" --arg password "$PASSWORD" '$ARGS.named'
{"username":"person1","password":"some \"gnarly 'password"}
jo
$ jo "username=$USERNAME" "password=$PASSWORD"
{"username":"person1","password":"some \"gnarly 'password"}
And with curl:
json=$( jq -n -c --arg username "$USERNAME" --arg password "$PASSWORD" '$ARGS.named' )
# or
json=$( jo "username=$USERNAME" "password=$PASSWORD" )
# then
curl ... -d "$json"
For less quoting, read from standard input instead.
curl -k -X POST -H 'Content-Type: application/json' -d #- <<EOF
{ "username": "$USERNAME", "password": "$PASSWORD"}
EOF
-d #foo reads from a file named foo. If you use - as the file name, it reads from standard input. Here, standard input is supplied from a here document, which is treated as a double-quoted string without actually enclosing it in double quotes.
curl -k -X POST -H 'Content-Type: application/json' -d '{"username":"'$USERNAME'","password":"'$PASSWORD'"}'
Here the variable are placed outside of "'" quotes and will be expanded by shell (just like in echo $USERNAME). For example assuming that USRNAME=xxx and PASSWORD=yyy the argv[7] string passed to curl is {"username":"xxx","password":"yyy"}
And yes, this will not work when $USERNAME or $PASSWORD contain space characters.
Our: curl -k -X POST -H 'Content-Type: application/json' -d '{"username":"'"$USERNAME"'","password":"'"$PASSWORD"'"}'
You can wrap the environment variables with "'" and you should keep the single quote for the external json object.
e.g
-d '{"username":"'"$USERNAME"'","password":"'"$PASSWORD"'"}'
I use the following line to create database:
curl -X POST 'http://10.1.1.1:8086/db?u=root&p=root' -d '{name: test1}
if i try to do it from shell script:
ip=10.1.1.1
curl -X POST 'http://$ip:8086/db?u=root&p=root' -d '{name: test1}'
i have a problem with shell variable substitution within single quotas, if i try to use them within double quotas:
curl -X POST "http://$ip:8086/db?u=root&p=root" -d '{name: test1}'
variable is expanded to the right value, printing in terminal
curl -X POST "http://10.1.21.1:8086/db?u=root&p=root" -d '{name: test1}': **No such file or directory**
What would be the right solution to this problem?
Try this:
ip=10.1.1.1
curl -X POST 'http://'"$ip"':8086/db?u=root&p=root' -d '{name: test1}'