What is the proper way to send this ,./'; '[]}{":?><|\\ as form-data value in curl. I'm doing this
curl --location --request POST 'https://postman-echo.com/post' \
--form 'more=",./'\'';[]}{\":?><|\\\\'"
right now and it gives different result, apparently only 2 backslashes in the response which is supposed to be 4 in total
Response snippet here
Solved!
Apparently this was the problem with my fish shell which was escaping the trailing double quotes. When i ran the same request on bash it was successful.
probably a shell is interpreting your cmd-line and each \\ pair gets reduced to a single \
(revision, even more explicit) :
using echo to show results,
echo more=",./';[]}{\":?><|\\\\\\\\"
adjust if needed, then copy the more="..." part to your curl cmd-line
Related
I have problem to execute curl request in the directory has comma in the name using bash command line.
curl --request POST --form "file=#$PWD/input_file" http://HOSTURL.com > output_file
if the directory name is
"test" works
"test test" works
"test, test" doesn't work.
I tried many ways to escape characters like quotations, back slush, changing IFS... but still getting error "failed creating formpost data".
Could someone advise how I should treat such directory names?
This looks like a case curl isn't designed to handle. However, by passing the filename on stdin, you can avoid needing it to correctly parse that value at all.
curl --request POST --form "file=#-" http://HOSTURL.com <input_file >output_file
In How can I use environment variables in body of a curl PUT request?, I was given the great advise to always use " using doing environment variables.
Say I want do following query:
curl -XPUT http://"${HOST}"/create/"${USER}" -d'{"user":"'"${USER}"'"}'
I enclosed ${USER} between " to ensure that spaces in the user name are possible. I did the same for ${HOST}, although that was strictly not required, since hostnames cannot contain spaces as far as I know.
I am wondering if the following request is equal to the previous request:
curl -XPUT "http://${HOST}/create/${USER}" -d'{"user":"'"${USER}"'"}'
Are they equal? Which one is preferred/most standard?
Yes, they are equal.
I'd prefer
curl -XPUT "http://${HOST}/create/${USER}" -d"{\"user\":\"${USER}\"}"
first because:
it is shorter as #Ryan said in comment
second literal is more readable when in one chunk rather than concatenatig two styles of quotes
some editors will highlight them in more readable way (for example vim )
As you've seen, dealing with quoting conventions in Bash when you have arbitrary data is difficult. However, there's a third way of quoting in cases like this than can make life a lot easier: "here documents".
Using <<TOKEN in a shell command indicates that the lines after the command will be read as the standard input to the command, terminated with TOKEN. Within the here document the usual quoting characters lose their special meaning and are interpreted literally, but variable substitution still happens normally.
To demonstrate, start a netcat "server" to display requests in one terminal with
nc -kl localhost 8888
Now, in another terminal, run this shell script:
name="Alice's Restaurant"
password="quote is ' and doublequote is \\\"."
curl -XPUT http://localhost:8888/create/user --data-binary #- <<EOF
{
"name": "$name",
"password": "$password",
"created": "$(date --iso-8601)"
}
EOF
When a --data argument is given # that requests that curl read the data from the filename specified immediately after the #, and using - as the filename reads from stdin.
Note that here I use --data-binary to make the server's output easier to understand; in production use you'd want to use --data-urlencode or, if the server accepts data in another format, ensure you're setting the Content-type header to that format rather than leaving it at the default application/x-www-form-urlencoded.
When you run the above, you'll see the following in your netcat terminal:
PUT /create/user HTTP/1.1
Host: localhost:8888
User-Agent: curl/7.52.1
Accept: */*
Content-Length: 112
Content-Type: application/x-www-form-urlencoded
{
"name": "Alice's Restaurant",
"password": "quote is ' and doublequote is \".",
"created": "2018-02-20"
}
As you can see, normal quoting characters are not treated specially, you need not do any special quoting on individual shell variables that get expanded within the here document, and you can even use $() to run shell commands whose output will be substituted within the document.
(By the way, I specified the double quote within the password variable as \\\", setting it to \" in the variable after shell interpolation of a double-quoted string, because that's necessary to produce valid JSON. Oh, you can never escape the quoting issues.)
This question already has answers here:
Difference between single and double quotes in Bash
(7 answers)
Closed 5 years ago.
I have a problem that I just cannot seem to solve.
I need to make a curl request to a given URL, and the URL requires a dollar sign in it.
So, for example:
www.example.com/mypath/function&$filter=whatever
Now, I can execute this just fine from the command line if I put the URl in single ticks, or if I escape the dollar sign with a backslash, and then put it in a double quote.
Obviously, there is a problem if you do not do either of the above, because bash will see the '$' and will interpret anything after it to mean a variable name.
So when I try:
URL="www.example.com/mypath/function&\\\$filter=whatever"
MYOUTPUT=$(curl -s --header "Authorization: $HEADER" "$URL")
it doesn't work right.
When I try
URL="www.example.com/mypath/function&\$filter=whatever"
MYOUTPUT=$(curl -s --header "Authorization: $HEADER" "'$URL'")
it doesn't work right.
What am I doing wrong?
I can tell it's not working right because the server is not responding the same way in the script as it does in the command line. The site responds in a certain default manner if the query isn't done right, and I always get the default response through the script.
If you put a literal $ in a string, bash will not try to interpret it in future expansions:
URL='www.example.com/mypath/function&$filter=whatever'
MYOUTPUT=$(curl -s --header "Authorization: $HEADER" "$URL")
I have a script that has SMS forwarded to it and posts some of that data to a PHP script. Below is my Bash script:
#!/bin/bash
# Script to post data to Top up processor
curl --request POST 'http://127.0.0.1/user//topup/process.php' --data "receipt=$1" --data "username=$9"
So to run it:
./mpesa_topup.sh sms_message
But the SMS server forwards the message with single quotes:
./mpesa_topup.sh 'sms_message'
The script ends up "parsing the entire SMS as 1 positional parameter. Here is a debug of what happens when the sms server runs the script.
root#sms:/var/lib/playsms/sms_command/1# bash -x mpesa_topup.sh 'JJA88QHC22 Confirmed.on 101015 at 9:49 PMKsh25.00 received from 254712345678 SOME BODY.New Account balance is Ksh25.00'
+ curl --request POST http://10.5.1.2/topup/process.php --data 'receipt=JJA88QHC22 Confirmed.on 101015 at 9:49 PMKsh25.00 received from 254722227332 JOTHAM KIIRU.New Account balance is Ksh25.00' --data username=
root#sms:/var/lib/playsms/sms_command/1#
Is there a way to remove/ignore the opening and closing single quotes in the Bash script?
PS : I am not a coder, gotten where I am with help from my friend Google.
It seems like you want the first and ninth word out of the single argument you are sent. You can do something like this:
$ set -- 'JJA88QHC22 Confirmed.on 101015 at 9:49 PMKsh25.00 received from 254712345678 SOME BODY.New Account balance is Ksh25.00'
$ echo $1
JJA88QHC22 Confirmed.on 101015 at 9:49 PMKsh25.00 received from 254712345678 SOME BODY.New Account balance is Ksh25.00
$ set -f # a
$ set -- $1 # b
$ set +f # c
$ echo $1
JJA88QHC22
$ echo $9
254712345678
The key is (b) where we omit the double quotes around the variable. This allows the shell to perform word-splitting on the value of the variable.
The shell will also attempt to perform glob-pattern expansion, unless you tell it not to, which I do in (a), and then turn that back on in (c).
You can solve this simply by putting your main command inside a function, and calling it again.
Your server is invoking your script with simple quotes, which transform your arguments in one single argument ($1).
If you treat this arg and call your_function() inside the script, you solved!
Here goes the example:
#!/bin/bash
# Script to post data to Top up processor
args=$1
your_function(){
curl --request POST 'http://127.0.0.1/user//topup/process.php' --data "receipt=$1" --data "username=$9"
}
your_function $1
Yes but that won't help. In the end, your code passes the whole SMS as a single string to curl because of --data "receipt=$1". If you only remove the quotes, that would become --data "receipt=JJA88QHC22" and the rest (like the amount) would be missing.
Your problem is that the input was multiple lines of text and that got somehow mangled. The solution is to parse the SMS. Since money is involved, you probably don't want any mistakes. That's why I would use a real programming language like Python or Java. But if you want to use BASH, this might work until an attacker starts sending you SMS to steal money:
# Split first parameter into $1...$n
set -- $1
recepient="$1"
# $2: Confirmed.on
# $3: 101015
# $4: at
# $5: 9:49
# $6: PMKsh25.00
amount=$(echo $6 | sed -e s/^(AM|PM)//) # sed removed the AM/PM at the beginning
# $7: received
# $8: from
sender="$7 $8 $9" # 254722227332 JOTHAM KIIRU.
Small Note: I removed the http:// from infront each link, because stackoverflow isn't allowing me to post it in original way.
I wrote a script which access to a webpage, to catch a URL and download it. One of the urls makes curl stop working and the whole URLS in the list to the same.
The script works as following:-
PAGE=$(curl -sL pageurl)
FILE_URL=$(echo $PAGE | sed -e 's/^.*<a href=\"\(.*\)\">\(.*\) alt="File" \/><\/a>.*$/\1/')
The FILE_URL VALUE is
URL/files/PartOne - Booke (Coll).pdf
webprod25.megashares.com/index.php?d01=3109985&lccdl=9e8e091ef33dd103&d01go=1&fln=/adobe reader exe.rar
AND SO One for others
When curl tried to catch this url it shows the following error using the debug mode of bash
++ curl -sOL 'webprod37.megashares.com/index.php?d01=3109985&lccdl=9e8e091ef33dd103&d01go=1&fln=/adobe' reader exe.rar fileshare273.depositfiles.com/auth-13023763920cd7ec18a0fdbfa8b62d35-188.165.197.50-43792102-7713641/FS273-7/PageMaker.rar -sOLJg fileshare601.depositfiles.com/auth-1302376689013d421df6c01e7f64c8d2-188.165.197.50-43801594-82379659/FS601-2/Adobe_Flash_Player_v10.3.180.65.2.rar -sOLJg 'webprod37.megashares.com/index.php?d01=de48789&lccdl=9e8e091ef33dd103&d01go=1&fln=/KAZAMIZA.COM.Adobe.Flash' Player-10.3.180.65.Beta-2.JUDGMENT DAY.rar bellatrix.oron.com/spzsttzwytpflwd76j3ne2moukomuhcdxg6llddfztqa2ztd7cplwwp457h3mxuacq3pbxzs/An-Beat - Mentally Insine '(Original' 'Mix).mp3'
curl: option -: is unknown
curl: try 'curl --help' or 'curl --manual' for more information
The quote marks the curl put it itself, I tried to do some workarounds like escaping url but it not works.
The basic problem seems to be that you are using $() expansion for something that looks to me like a multi line value. You should try iterating over each line.
The other problem looks like one of improper quoting of URLs containing spaces. There's a lone dash (-) in "An-Beat - Mentally Insine"
Oh, one more problem: The sed part to catch the href="..." contents only works if there's exactly one href on the line. If there are two or more, your \(.*\) will match everything else up to the last href. You should use something like href="\([^"]*\)", matching "any number of non-doublequotes followed by a doublequote".
Quote your variables as in:
pageurl='the url'
PAGE=$(curl -sL "$pageurl")
FILE_URL=$(echo "$PAGE" | sed -e 's/^.*<a href=\"\(.*\)\">\(.*\) alt="File" \/><\/a>.*$/\1/')
Otherwise, shell expansion will occur. The error "option -: is unknown" comes from the final part:
An-Beat - Mentally Insine
Because you didn't apply quotes to it, it got parsed as arguments, which you can clearly see in the syntax-highlighted code.