I'm using the google-http-client for a project at work and when I do some requests I have the following thing printed on my console.
curl -v --compressed -X POST -H 'Accept-Encoding: gzip' -H 'User-Agent: Google-HTTP-Java-Client/1.23.0 (gzip)' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -d '#-' -- 'http://example.com' << $$$
I was wondering what << $$$ mean.
If I try to run this command into a linux terminal seems that << $$$ makes the console to wait for more input. If that's the case, how can I specify to the terminal that I'm done feeding inputs to it?
Later edit: I have found that curl arguments -d #- implies that data will be red from the stdin.
This is a "here-document" with an unusual end marker.
A here-document is a type of redirection, and usually looks like
utility <<MARKER
document
content
goes here
MARKER
That is, it feeds a document delimited by MARKER to the utility on its standard input.
This is like utility <file where file contains the lines in the here-document, except that the shell will do variable expansion and command substitution on the text of the document (this may be prevented by quoting the marker as either \MARKER or 'MARKER' at the start).
The here-document marker can be any word, but $$$ is a highly unusual choice of word for it. As $ has a special meaning in the shell, using $ in the marker is, or may be, confusing to the reader.
If you type
somecommand <<stuff
in the shell, the shell expects you to give the rest of the contents of the here-document, and then the word stuff on a line by itself. That's how you signal end of input in a here-document.
Related
What is the proper way to send this ,./'; '[]}{":?><|\\ as form-data value in curl. I'm doing this
curl --location --request POST 'https://postman-echo.com/post' \
--form 'more=",./'\'';[]}{\":?><|\\\\'"
right now and it gives different result, apparently only 2 backslashes in the response which is supposed to be 4 in total
Response snippet here
Solved!
Apparently this was the problem with my fish shell which was escaping the trailing double quotes. When i ran the same request on bash it was successful.
probably a shell is interpreting your cmd-line and each \\ pair gets reduced to a single \
(revision, even more explicit) :
using echo to show results,
echo more=",./';[]}{\":?><|\\\\\\\\"
adjust if needed, then copy the more="..." part to your curl cmd-line
In How can I use environment variables in body of a curl PUT request?, I was given the great advise to always use " using doing environment variables.
Say I want do following query:
curl -XPUT http://"${HOST}"/create/"${USER}" -d'{"user":"'"${USER}"'"}'
I enclosed ${USER} between " to ensure that spaces in the user name are possible. I did the same for ${HOST}, although that was strictly not required, since hostnames cannot contain spaces as far as I know.
I am wondering if the following request is equal to the previous request:
curl -XPUT "http://${HOST}/create/${USER}" -d'{"user":"'"${USER}"'"}'
Are they equal? Which one is preferred/most standard?
Yes, they are equal.
I'd prefer
curl -XPUT "http://${HOST}/create/${USER}" -d"{\"user\":\"${USER}\"}"
first because:
it is shorter as #Ryan said in comment
second literal is more readable when in one chunk rather than concatenatig two styles of quotes
some editors will highlight them in more readable way (for example vim )
As you've seen, dealing with quoting conventions in Bash when you have arbitrary data is difficult. However, there's a third way of quoting in cases like this than can make life a lot easier: "here documents".
Using <<TOKEN in a shell command indicates that the lines after the command will be read as the standard input to the command, terminated with TOKEN. Within the here document the usual quoting characters lose their special meaning and are interpreted literally, but variable substitution still happens normally.
To demonstrate, start a netcat "server" to display requests in one terminal with
nc -kl localhost 8888
Now, in another terminal, run this shell script:
name="Alice's Restaurant"
password="quote is ' and doublequote is \\\"."
curl -XPUT http://localhost:8888/create/user --data-binary #- <<EOF
{
"name": "$name",
"password": "$password",
"created": "$(date --iso-8601)"
}
EOF
When a --data argument is given # that requests that curl read the data from the filename specified immediately after the #, and using - as the filename reads from stdin.
Note that here I use --data-binary to make the server's output easier to understand; in production use you'd want to use --data-urlencode or, if the server accepts data in another format, ensure you're setting the Content-type header to that format rather than leaving it at the default application/x-www-form-urlencoded.
When you run the above, you'll see the following in your netcat terminal:
PUT /create/user HTTP/1.1
Host: localhost:8888
User-Agent: curl/7.52.1
Accept: */*
Content-Length: 112
Content-Type: application/x-www-form-urlencoded
{
"name": "Alice's Restaurant",
"password": "quote is ' and doublequote is \".",
"created": "2018-02-20"
}
As you can see, normal quoting characters are not treated specially, you need not do any special quoting on individual shell variables that get expanded within the here document, and you can even use $() to run shell commands whose output will be substituted within the document.
(By the way, I specified the double quote within the password variable as \\\", setting it to \" in the variable after shell interpolation of a double-quoted string, because that's necessary to produce valid JSON. Oh, you can never escape the quoting issues.)
I have a script that has SMS forwarded to it and posts some of that data to a PHP script. Below is my Bash script:
#!/bin/bash
# Script to post data to Top up processor
curl --request POST 'http://127.0.0.1/user//topup/process.php' --data "receipt=$1" --data "username=$9"
So to run it:
./mpesa_topup.sh sms_message
But the SMS server forwards the message with single quotes:
./mpesa_topup.sh 'sms_message'
The script ends up "parsing the entire SMS as 1 positional parameter. Here is a debug of what happens when the sms server runs the script.
root#sms:/var/lib/playsms/sms_command/1# bash -x mpesa_topup.sh 'JJA88QHC22 Confirmed.on 101015 at 9:49 PMKsh25.00 received from 254712345678 SOME BODY.New Account balance is Ksh25.00'
+ curl --request POST http://10.5.1.2/topup/process.php --data 'receipt=JJA88QHC22 Confirmed.on 101015 at 9:49 PMKsh25.00 received from 254722227332 JOTHAM KIIRU.New Account balance is Ksh25.00' --data username=
root#sms:/var/lib/playsms/sms_command/1#
Is there a way to remove/ignore the opening and closing single quotes in the Bash script?
PS : I am not a coder, gotten where I am with help from my friend Google.
It seems like you want the first and ninth word out of the single argument you are sent. You can do something like this:
$ set -- 'JJA88QHC22 Confirmed.on 101015 at 9:49 PMKsh25.00 received from 254712345678 SOME BODY.New Account balance is Ksh25.00'
$ echo $1
JJA88QHC22 Confirmed.on 101015 at 9:49 PMKsh25.00 received from 254712345678 SOME BODY.New Account balance is Ksh25.00
$ set -f # a
$ set -- $1 # b
$ set +f # c
$ echo $1
JJA88QHC22
$ echo $9
254712345678
The key is (b) where we omit the double quotes around the variable. This allows the shell to perform word-splitting on the value of the variable.
The shell will also attempt to perform glob-pattern expansion, unless you tell it not to, which I do in (a), and then turn that back on in (c).
You can solve this simply by putting your main command inside a function, and calling it again.
Your server is invoking your script with simple quotes, which transform your arguments in one single argument ($1).
If you treat this arg and call your_function() inside the script, you solved!
Here goes the example:
#!/bin/bash
# Script to post data to Top up processor
args=$1
your_function(){
curl --request POST 'http://127.0.0.1/user//topup/process.php' --data "receipt=$1" --data "username=$9"
}
your_function $1
Yes but that won't help. In the end, your code passes the whole SMS as a single string to curl because of --data "receipt=$1". If you only remove the quotes, that would become --data "receipt=JJA88QHC22" and the rest (like the amount) would be missing.
Your problem is that the input was multiple lines of text and that got somehow mangled. The solution is to parse the SMS. Since money is involved, you probably don't want any mistakes. That's why I would use a real programming language like Python or Java. But if you want to use BASH, this might work until an attacker starts sending you SMS to steal money:
# Split first parameter into $1...$n
set -- $1
recepient="$1"
# $2: Confirmed.on
# $3: 101015
# $4: at
# $5: 9:49
# $6: PMKsh25.00
amount=$(echo $6 | sed -e s/^(AM|PM)//) # sed removed the AM/PM at the beginning
# $7: received
# $8: from
sender="$7 $8 $9" # 254722227332 JOTHAM KIIRU.
I want to get some data from a HTTP server. What it sends me depends on what I put in a POST request.
What I put in the INPUT_TEXT field is a sequence of words. When I run the following command, I get good looking output.
$ curl http://localhost:59125/process -d INPUT_TEXT="here are some words"
I want a bash script to take some string as a command line argument, and pass it appropriately to curl. The first thing I tried was to put the following in a script:
sentence=$1
command="curl http://localhost:59125/process -d INPUT_TEXT=\"${sentence}\""
$command
I then run the script like so:
$ ./script "here are some words"
But then I get a curl Couldn't resolve host error for each of "are", "some", and "words". It would seem that "here" got correctly treated as the INPUT_TEXT, but the rest of the words were then considered to be hosts, and not part of the option.
So I tried:
command=("curl" "http://localhost:59125/process" "-d" "INPUT_TEXT='$sentence'")
${command[#]}
I got the same output as the first script. I finally got what I wanted with:
result=$(curl http://localhost:59125/process -d INPUT_TEXT="${sentence}")
echo $result
I'm still unsure as to what the distinction is. In the first two cases, when I echoed out the contents of command, I get exactly what I input from the interactive Bash prompt, which had worked fine. What caused the difference?
The following will work:
command=("curl" "http://localhost:59125/process"
"-d" "INPUT_TEXT=$sentence")
"${command[#]}"
That has two changes from yours:
I removed the incorrect quotes around $sentence since you don't want to send quotes to the server (as far as I can see).
I put double-quotes around the use of "${command[#]}". Without the double quotes, the array's elements are concatenated with spaces between them and then the result is word-split. With double quotes, the individual array elements are used as individual words.
The second point is well-explained in the bash FAQ and a bunch of SO answers dealing with quotes.
The important thing to understand is that quotes only quote when a command is parsed. A quote which is a character in a variable is just a character; it is not reinterpreted when the value of the variable expanded. Whitespace in the variable is used for word-splitting if the variable expansion is unquoted; the fact that the whitespace was quoted in the the command which defined the variable is completely irrelevant. In this sense, bash is just the same as any other programming language.
I'm trying to POST to the HTTP gateway of an SMS provider (Sybase 365) using CURL from a Linux shell script.
I need to pass the following data (note the [ ] and LF characters)
[MSISDN]
List=+12345678
[MESSAGE]
Text=Hello
[END]
If I submit a file using the -F parameter, CURL removes the LF e.g.
curl -F #myfile "http://www.sybase.com/..."
results in this at the server (which is rejected)
[MSISDN]List=+12345678[MESSAGE]Text=Hello[END]
Is there anything I can do to avoid this or do I need an alternative tool?
I'm using a file containing my data for testing but I'd like to avoid that in practice and POST directly from the script.
Try using --data-binary instead of -d(ata-ascii).
From the manual:
--data-binary (HTTP) This posts data in a similar manner as --data-ascii does, although when using this option the entire context of the posted data is kept as-is.
If you want to post a binary file without the strip-newlines feature of the --data-ascii option, this is for you. If this option is used several times, the ones following the first will append data.
ETA: oops, I should read the question more closely. You're using -F, not -d. But --data-binary may be still be worth a shot.
Probably a silly thought, but I don't suppose it actually requires CRLF instead of just LF?
Alternatively, have you tried using the --data-binary option instead of -F?
I've got this working using -d
request=`printf "[MSISDN]\nList=$number\n[MESSAGE]\nText=$message\n[END]\n"`
response=`curl -s -u $username:$password -d "$request" http://www.sybase.com/...`
Curiously, if I use -d #myfile (where myfile contains LF separated text), it doesn't work.
I also tried --data-binary without success.
curl "url" --data-binary #myfile
posts new lines in the data [tested on curl 7.12.1]