This question already has answers here:
Build a JSON string with Bash variables
(17 answers)
Closed 2 years ago.
I am working on simple curl POST script and I have problem with string interpolation of a variable that is initialized with data from a script. Let me describe it:
#!/bin/bash
EMAIL=$(source ./random_email_generator.sh) #this generates some random string
echo "$EMAIL" #here email is printed well
curl --insecure --header "Content-Type: application/json" \
--request POST \
--data '{"email":'"$EMAIL"',"password":"validPassword"}' \ #problem lies here because email is always empty string
http:/ticketing.dev/api/users/signup
To sum up, there is a lot of quotes and double quotes here and it is abit mixed up, how can I resolve value of EMAIL in place of json body?
Thanks in advance
The variable is being interpolated properly. The problem is that you're not wrapping it in double quotes, so you're creating invalid JSON.
curl --insecure --header "Content-Type: application/json" \
--request POST \
--data '{"email":"'"$EMAIL"'","password":"validPassword"}' \
http:/ticketing.dev/api/users/signup
When you want to see what the command actually looks like, put echo before curl or run the script with bash -x. You would have seen
--data {"email":foo#bar.com,"password":"validPassword"}
It would be better if you used the jq tool to manipulate JSON in bash scripts.
Related
This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
Closed 11 months ago.
I want to have a curl command like below
curl --location --request POST 'https://abcd.com/api/v4/projects/<projectId>/triggers' \
--header 'PRIVATE-TOKEN: <your_access_token>' \
--form 'description="my description"'
Now I wrote a shell script function to generate it dynamically
it need projectId, token, and description as a pramter
callApi(){
while IFS="," read -r -a users; do
for u in "${users[#]}"
do
url="'https://abcd.com/api/v4/projects/$1/triggers'"
echo $url
header="'PRIVATE-TOKEN: $2'"
echo $header
desc="'description=$u token'"
echo $desc
tk=$(curl --location --request POST $url \
--header $header \
--form $desc)
echo $tk
done
done <<< $(cat $3)
}
callApi "<projectId>" "<token>" ./users.csv
It echo perfectly
But
It thorws error
Don't use both double and single quotes like that. You are adding literal single quotes to the url (and other variables) which, as you have discovered, breaks.
Use double quotes if you need to allow command or parameter substitution, single quotes otherwise. Double quote your variables everywhere you dereference them.
Use indentation for readability.
Useless use of cat.
I've captured the function parameters at the start of the function for visibility: I did not even notice the $1 buried in the URL.
callApi() {
local project=$1 token=$2 userfile=$3
while IFS="," read -r -a users; do
for u in "${users[#]}"; do
url="https://abcd.com/api/v4/projects/${project}/triggers"
echo "$url"
header="PRIVATE-TOKEN: ${token}"
echo "$header"
desc="description=$u token"
echo "$desc"
tk=$(
curl --location \
--request POST \
--header "$header" \
--form "$desc" \
"$url"
)
echo "$tk"
done
done < "$userfile"
}
Could someone tell me how to pass the output of one CURL GET to another CURL POST? I mean something like this:
curl --header "Content-Type: application/json" --request POST --data "{\"specification\": curl --header "Content-Type: application/json" --request GET "https://user:pass#anotherhost"}" "https://user:pass#localhost"
Use jq to craft JSON in the shell.
response="$(curl --header "Content-Type: application/json" --request GET "https://user:pass#anotherhost")"
data="$(jq "{specification: .}" <<< "$response")"
curl --header "Content-Type: application/json" --request POST --data "$data" "https://user:pass#localhost"
Keep in mind, that passwords in command line arguments are public to the host.
It looks like you need to use command substitution. The result would be something like this.
curl --header "Content-Type: application/json" --request POST --data "{\"specification\": $(curl --header "Content-Type: application/json" --request GET "https://user:pass#anotherhost")}" "https://user:pass#localhost"
$() is used for command substitution and it invokes a subshell. The command in the parentheses (a.k.a. round brackets) of $() is executed in a subshell and the output is then placed in the original command.
So the curl GET will be executed in a subshell and the result will be passed to the curl POST
You can read more about it at the link below:
https://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html
I have this script in my gitlab yml file:
script:
- "curl -X POST -H 'Content-type: application/json' --data '{\"tag\":\"${CI_COMMIT_TAG}\", \"projectId\":\"${PROJECT_ID}\"}' http://localhost:1337/slack"**
Let's say when I create a tag named 'testing-v1' on gitlab, the variables $CI_COMMIT_TAG should be 'testing-v1', but instead it is showing as it is.
It's not replacing.
If I hard code the tag name and project id like this
- "curl -X POST -H 'Content-type: application/json' --data '{\"tag\":\"testing-v1\", \"projectId\":\"1111111\"}' http://localhost:1337/slack"**
It's working.
If you copy-paste your curl line (without the first and last double-quote) in shellcheck.net, you will be pointed to SC2016
Expressions don't expand in single quotes, use double quotes for that.
Single quotes prevent expansion of everything, including variables and command substitution.
If you want to use the values of variables and such, use double quotes instead.
Note that if you have other items that needs single quoting, you can use both in a single word:
echo '$1 USD is '"$rate GBP"
See an example of a working gitlab-ci.yml with curl here:
image: ruby:2.3
stages:
- clear-cache
CLOUDFLARE_URL: https://api.cloudflare.com/client/v4/zones/$zone/purge_cache
EMAIL: "X-Auth-Email: $email"
AUTH: "X-Auth-Key: $key"
CONTENT: 'Content-Type: application/json'
clear-cache:
stage: clear-cache
script:
- curl -X POST "${CLOUDFLARE_URL}" -H "${EMAIL}" -H "${AUTH}" -H "${CONTENT}" --data '{"purge_everything":true}'
only:
- master
My question is very simple. I want to use environment variables in a cURL command sth similar to this:
curl -k -X POST -H 'Content-Type: application/json' -d '{"username":"$USERNAME","password":"$PASSWORD"}'
When I run the command $USERNAME is passed to the command as a "$USERNAME" string not the value of the variable. Is there a way to escape this situation?
Thanks.
Single quotes inhibit variable substitution, so use double quotes. The inner double quotes must then be escaped.
... -d "{\"username\":\"$USERNAME\",\"password\":\"$PASSWORD\"}"
Since this answer was written in 2015, it has become clear that this technique is insufficient to properly create JSON:
$ USERNAME=person1
$ PASSWORD="some \"gnarly 'password"
$ echo "{\"username\":\"$USERNAME\",\"password\":\"$PASSWORD\"}"
{"username":"person1","password":"some "gnarly 'password"}
$ echo "{\"username\":\"$USERNAME\",\"password\":\"$PASSWORD\"}" | jq .
parse error: Invalid numeric literal at line 1, column 47
The quoting problem are clear. The (shell) solutions are not
Current best practice: use a JSON-specific tool to create JSON:
jq
$ jq -n -c --arg username "$USERNAME" --arg password "$PASSWORD" '$ARGS.named'
{"username":"person1","password":"some \"gnarly 'password"}
jo
$ jo "username=$USERNAME" "password=$PASSWORD"
{"username":"person1","password":"some \"gnarly 'password"}
And with curl:
json=$( jq -n -c --arg username "$USERNAME" --arg password "$PASSWORD" '$ARGS.named' )
# or
json=$( jo "username=$USERNAME" "password=$PASSWORD" )
# then
curl ... -d "$json"
For less quoting, read from standard input instead.
curl -k -X POST -H 'Content-Type: application/json' -d #- <<EOF
{ "username": "$USERNAME", "password": "$PASSWORD"}
EOF
-d #foo reads from a file named foo. If you use - as the file name, it reads from standard input. Here, standard input is supplied from a here document, which is treated as a double-quoted string without actually enclosing it in double quotes.
curl -k -X POST -H 'Content-Type: application/json' -d '{"username":"'$USERNAME'","password":"'$PASSWORD'"}'
Here the variable are placed outside of "'" quotes and will be expanded by shell (just like in echo $USERNAME). For example assuming that USRNAME=xxx and PASSWORD=yyy the argv[7] string passed to curl is {"username":"xxx","password":"yyy"}
And yes, this will not work when $USERNAME or $PASSWORD contain space characters.
Our: curl -k -X POST -H 'Content-Type: application/json' -d '{"username":"'"$USERNAME"'","password":"'"$PASSWORD"'"}'
You can wrap the environment variables with "'" and you should keep the single quote for the external json object.
e.g
-d '{"username":"'"$USERNAME"'","password":"'"$PASSWORD"'"}'
I use the following line to create database:
curl -X POST 'http://10.1.1.1:8086/db?u=root&p=root' -d '{name: test1}
if i try to do it from shell script:
ip=10.1.1.1
curl -X POST 'http://$ip:8086/db?u=root&p=root' -d '{name: test1}'
i have a problem with shell variable substitution within single quotas, if i try to use them within double quotas:
curl -X POST "http://$ip:8086/db?u=root&p=root" -d '{name: test1}'
variable is expanded to the right value, printing in terminal
curl -X POST "http://10.1.21.1:8086/db?u=root&p=root" -d '{name: test1}': **No such file or directory**
What would be the right solution to this problem?
Try this:
ip=10.1.1.1
curl -X POST 'http://'"$ip"':8086/db?u=root&p=root' -d '{name: test1}'