Backslash in JSON on request body for ExpressJS- with body-parser - node.js

Everything looks like OK unless a backslash char in JSON comes from client to the Express server with body-parser.
Request body is like this (We put this onto Postman (raw value) and Swagger-UI as is:
{
"username": "admin",
"password": "abc%\-xyz"
}
Postman's curl code is like this:
curl -X POST \
http://localhost:3000/login \
-H 'Content-Type: application/json' \
-H 'Postman-Token: ef96edfd-dc43-47fe-89c0-76e562fa206d' \
-H 'cache-control: no-cache' \
-d '{
"username": "admin",
"password": "abc%\-xyz"
}'
Swagger-UI's curl code:
curl -X POST "http://localhost:3000/login" -H "accept: application/json" -H "Content-Type: application/json" -d "{ \"username\": \"admin\", \"password\": \"abc%\\-xyz\"}"
I . don't know what's the difference between Swagger-UI and Postman but it looks like there is one more backslash in Swagger's curl. But there is no difference on the server side.
Indeed, I cannot control the request body which comes from the client and the passwords can have such characters like backslash. App always gives this error:
SyntaxError: Unexpected number in JSON at position 45
at JSON.parse (<anonymous>)
at parse (/path/to/project/node_modules/body-parser/lib/types/json.js:93:19)
at /path/to/project/node_modules/body-parser/lib/read.js:121:18
at invokeCallback (/path/to/project/node_modules/raw-body/index.js:224:16)
at done (/path/to/project/node_modules/raw-body/index.js:213:7)
at IncomingMessage.onEnd (/path/to/project/node_modules/raw-body/index.js:273:7)
at IncomingMessage.emit (events.js:189:13)
at endReadableNT (_stream_readable.js:1103:12)
at process._tickCallback (internal/process/next_tick.js:63:19)
If JSON becomes with two backslashes (\):
{
"username": "admin",
"password": "abc%\\-xyz"
}
Then there is no error anymore on the server side (inside of the route.post() function) but the body.password has still two backslashes: abc%\\-xyz
As it was stated on the comments, it may be a problem of parsing.
What is the correct way to deal with this?
bodyParser.json({strict: false}) doesn't help

You are just trying to fix broken clients.
The fact that your server already accepts passwords with a backslash shows this.
If you turn on strict mode for your Jason parser, everything becomes moredictable, as the parser no longer tries to make educated guesses on how it thinks your data should look. This also means that you now have to send JSON from your postman client, instead of your self-invented protocol, so send "test\\" when your password is \.
This is because a slash (and a double quote) have special meaning when used in a JSON string, and thus they get special treatment, somethings that other characters don't get. The same goes for when someones password contains a ", if this is the case, a client must send \" to your server.

Related

trying to curl with a file into request and failing [duplicate]

I need to make a POST request via cURL from the command line. Data for this request is located in a file. I know that via PUT this could be done with the --upload-file option.
curl host:port/post-file -H "Content-Type: text/xml" --data "contents_of_file"
You're looking for the --data-binary argument:
curl -i -X POST host:port/post-file \
-H "Content-Type: text/xml" \
--data-binary "#path/to/file"
In the example above, -i prints out all the headers so that you can see what's going on, and -X POST makes it explicit that this is a post. Both of these can be safely omitted without changing the behaviour on the wire. The path to the file needs to be preceded by an # symbol, so curl knows to read from a file.
I need to make a POST request via Curl from the command line. Data for this request is located in a file...
All you need to do is have the --data argument start with a #:
curl -H "Content-Type: text/xml" --data "#path_of_file" host:port/post-file-path
For example, if you have the data in a file called stuff.xml then you would do something like:
curl -H "Content-Type: text/xml" --data "#stuff.xml" host:port/post-file-path
The stuff.xml filename can be replaced with a relative or full path to the file: #../xml/stuff.xml, #/var/tmp/stuff.xml, ...
If you are using form data to upload file,in which a parameter name must be specified , you can use:
curl -X POST -i -F "parametername=#filename" -F "additional_parm=param2" host:port/xxx
Most of answers are perfect here, but when I landed here for my particular problem, I have to upload binary file (XLSX spread sheet) using POST method, I see one thing missing, i.e. usually its not just file you load, you may have more form data elements, like comment to file or tags to file etc as was my case. Hence, I would like to add it here as it was my use case, so that it could help others.
curl -POST -F comment=mycomment -F file_type=XLSX -F file_data=#/your/path/to/file.XLSX http://yourhost.example.com/api/example_url
I was having a similar issue in passing the file as a param. Using -F allowed the file to be passed as form data, but the content type of the file was application/octet-stream. My endpoint was expecting text/csv.
You are able to set the MIME type of the file with the following syntax:
-F 'file=#path/to/file;type=<MIME_TYPE>
So the full cURL command would look like this for a CSV file:
curl -X POST -F 'file=#path/to/file.csv;type=text/csv' https://test.com
There is good documentation on this and other options here: https://catonmat.net/cookbooks/curl/make-post-request#post-form-data
I had to use a HTTP connection, because on HTTPS there is default file size limit.
https://techcommunity.microsoft.com/t5/IIS-Support-Blog/Solution-for-Request-Entity-Too-Large-error/ba-p/501134
curl -i -X 'POST' -F 'file=#/home/testeincremental.xlsx' 'http://example.com/upload.aspx?user=example&password=example123&type=XLSX'

Update/Set Default Headers in the Curl

I have a scenario in which I call multiple API requests with the same headers.
Headers would usually have:
Authorization: Bearer ASADFW&YUWFIANDJDDAjd8sa87sad7sa8dhsahdusabhdw
Content-Type: text/html;charset=UTF-8
etc...
I need to set these headers key-value pair as a default so whenever I curl the APIs these get picked.
Is there a way to do this?
I usually just keep a text file open with the Curl command, something like:
curl -X POST -H "Authorization: Bearer ASADFW&YUWFIANDJDDAjd8sa87sad7sa8dhsahdusabhdw"
-H "Content-Type: text/html;charset=UTF-8" -d 'blah'
http://some_url.com:8080
Then, just copy and paste the above into a terminal and make the changes you want. Note that I have formatted the above on multiple lines for reading purposes only. In practice, the entire Curl command should be on a single line.

invalid UTF-8 JSON error on Couchdb Find request using regex

I have a problem when sending a POST request to my couchdb database using curl.
curl -X POST http://admin:password#localhost:3005/fisconet/_find -d '{"selector":{"fields":{"keywords":{"$regex":".*évasion fiscale.*"}}}}' -H "Content-Type: application/json"
This is the error I get:
{"error":"bad_request","reason":"invalid UTF-8 JSON"}
If I remove the accent of the regular expression {"$regex":".*evasion fiscale.*"}, I don't get the error but I don't find what I'm looking for.
Couldn't find anything on the internet that worked.
Thanks

What is the reason for curl --data-urlencode error on running curl from within a Windows batch script?

I am using the Text Analytics service (find the language) of the Azure portal and a Windows batch script with Curl which is sending JSON data:
The following Windows batch script using curl and the --data option works:
set data="{'documents':[{'id':1,'text':'your are my guest'}]}"
curl -X POST %endpoint% ^
-H "Content-Type: application/json" ^
--data %data%
However, it does not work when I want to use the --data-urlencode option with the same code.
The output error is:
Request body must be present
I have tried many things (double quote, escape, and so on), but nothing works.
Some ideas?
The solution was already described here
The trick is to use a ('Notepad++' UTF-8 without BOM) file, say data.txt, with the --data-binary option :
curl -X POST %endpoint% ^
-H "Content-Type: application/json" ^
--data-binary #data.txt
Inside the file data.txt :
{'documents':[{'id':1,'text':'your are my guest'}]}
But it works with any language (chinese, hindi, etc...), so you don't have to encode anything. For instance, you could have use, in the data.txt file
{'documents':[{'id':2,'text':'大都会区有它自己的当地路边快餐口味'}]}

How to send a simple HTML file from a bash script with the curl command using SendGrid's SMTP API?

I am new to bash scripting. I have spent hours searching for a solution..
#!/bin/bash
# EMAIL_TO, FROM_EMAIL, etc variables are initialized here
# ...
# ...
maildata='{"personalizations": [{"to": [{"email": "'${EMAIL_TO}'"}]}],"from": {"email": "'${FROM_EMAIL}'",
"name": "'${FROM_NAME}'"},"subject": "'${SUBJECT}'","content": [{"type": "text/html", "value": "'${bodyHTML}'"}]}'
curl --url https://api.sendgrid.com/v3/mail/send \
--header 'Authorization: Bearer '$SENDGRID_API_KEY \
--header 'Content-Type: application/json' \
--data "'$maildata'"
The snippet above works fine whenever bodyHTML is set here, inside the script.
But I want an external html file to be sent in this manner.
So, the question:
How can I send, for example, "mail.html" using the above scheme?
(How to set the bodyHTML variable? What command(s) to use?
bodyHTML=$(cat "mail.html") # results in an error message on executing the curl line)
The issue seems to have to do with the bodyHTML (initialized either from file or within script) containing double quotation marks. As in , for example. But what to do with it...? Perhaps I need to first serialize HTML as JSON within the script..? How?
i guess bodyHTML needs proper json-encoding, try
maildata='{"personalizations": [{"to": [{"email": "'${EMAIL_TO}'"}]}],"from": {"email": "'${FROM_EMAIL}'",
"name": "'${FROM_NAME}'"},"subject": "'${SUBJECT}'","content": [{"type": "text/html", "value": '$(printf "%s" "${bodyHTML}" | php -r 'echo json_encode(stream_get_contents(STDIN));')'}]}'
... ofc they should probably ALL be properly json-encoded, not just the HTML, but the html seems like the thing most likely to break your json here..

Resources