I'm developing a backend for an existing application which sends Gzipped data to an endpoint on Netlify Functions. However, I wasn't able to get the body as a Buffer instead of a string.
This means that Zlib is unable to decompress the body since invalid bytes have been replaced by the Unicode replacement character and they turn into EF BF BD. This is my code:
import { Handler } from "#netlify/functions";
export const handler: Handler = async (event, context) => {
console.log(Buffer.from(event.body).toString("hex"));
const extractedData = zlib.gunzipSync(event.body).toString();
} // The problem is that "event.body" is a string.
This is reproducible by using this command. It sends "Hello World!" compressed with Gzip:
base64 -d <<< "H4sIAAAAAAAA//NIzcnJVwjPL8pJUQQAoxwpHAwAAAA=" | curl -X POST --data-binary #- "http://localhost:8888/.netlify/functions/post"
However, my code doesn't produce the expected output because of the replacements:
Expected: 1f8b08000000000000fff348cdc9c95708cf2fca49510400a31c291c0c000000
Received: 1fefbfbd08000000000000efbfbdefbfbd48efbfbdefbfbdefbfbd5708efbfbd2fefbfbd49510400efbfbd1c291c0c000000
Is there any way to access the raw body of the request, preferably as a Buffer?
After some research of how it's done in AWS, I found something who resolve my issue.
The discord sign system need rawBody as stripe, that should do the same:
const rawBody = String(event.body).replace(/'/g, '\'').replace(/\\'/g,"'")
I found this with this by reading this comment: https://stackoverflow.com/a/71582521/5511370
And with the description in the doc of AWS of the function, I was able to do it in JS:
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html
scapes the characters in a string using JavaScript string rules.
Note
This function will turn any regular single quotes (') into escaped ones (\').
However, the escaped single quotes are not valid in JSON.
Thus, when the output from this function is used in a JSON property,
you must turn any escaped single quotes (\') back to regular single quotes (').
This is shown in the following example:
$util.escapeJavaScript(data).replaceAll("\\'","'")
Related
I'm trying to implement SimplePay (OTP) payment gateway in a NodeJS project.
So far I'm only exploring and following the technical documentation, and try to put together the test start request as per the documentation, but I cannot get the same signature as it's expected in the docs.
Documentation (examples are with their PHP SDK):
How to create the signature:
signature = codeBase64(hmacWithSha384(merchantKey, message))
(so create a HMCAC SHA384 hash of the whole JSON sent in the post body (this is the message), and use the merchantKey, then base64 encode the has result)
merchantKey:
FxDa5w314kLlNseq2sKuVwaqZshZT5d6
message (should be the whole JSON sent in the POST body):
{"salt":"c1ca1d0e9fc2323b3dda7cf145e36f5e","merchant":"PUBLICTESTHUF","orderRef":"101010516348232058105","currency":"HUF","customerEmail":"sdk_test#otpmobil.com","language":"HU","sdkVersion":"SimplePayV2.1_Payment_PHP_SDK_2.0.7_190701:dd236896400d7463677a82a47f53e36e","methods":\["CARD"\],"total":"25","timeout":"2021-10-30T12:30:11+00:00","url":"https://sdk.simplepay.hu/back.php"}
Expected outcome of the signature according to the docs:
gcDJ8J7TyT1rC/Ygj/8CihXaLwniMWRav09QSEMQUnv5TbYaEDvQAuBE1mW3plvZ
Now I've tried a lot of things, but can't seem to get the same signature.
Here's my code, and questions:
const bodyAsText = `{"salt":"c1ca1d0e9fc2323b3dda7cf145e36f5e","merchant":"PUBLICTESTHUF","orderRef":"101010516348232058105","currency":"HUF","customerEmail":"sdk_test#otpmobil.com","language":"HU","sdkVersion":"SimplePayV2.1_Payment_PHP_SDK_2.0.7_190701:dd236896400d7463677a82a47f53e36e","methods":\["CARD"\],"total":"25","timeout":"2021-10-30T12:30:11+00:00","url":"https://sdk.simplepay.hu/back.php"}\`;
const message = JSON.parse(bodyAsText);
const merchantKey = "FxDa5w314kLlNseq2sKuVwaqZshZT5d6";
const signature = createHmac("sha384", merchantKey)
.update(JSON.stringify(message))
.digest("base64");
console.log("Your signature is: ", signature);\
I've also tried this way:
const signature = createHmac("sha384", merchantKey) .update(JSON.stringify(message)) .digest().toString('base64');
But the result is the same:
dwvuWq4Hyec9BtriL4N+LC42u4G10PcHcpcvCM9TIxL7VJJFBvIi5jQLev86Pedu
I've tried submitting the request via postman. for the same JSON in body, I get signature error with my signature, and success with the one written in docs, which is kind of expected, I just don't get what I'm missing.
The issue is caused by the URL in the message and differences between PHP (to which the documentation refers) and JavaScript/NodeJS (what you are using).
In PHP, \/ is not treated as an escape sequence and is interpreted as \/. In JavaScript, \/ is treated as an escape sequence and interpreted as /. For the input to be interpreted as \/, \/ (or /) must be replaced with \\/ in the message string, e.g.:
const crypto = require('crypto');
const merchantKey = "FxDa5w314kLlNseq2sKuVwaqZshZT5d6";
const body = {"salt":"c1ca1d0e9fc2323b3dda7cf145e36f5e","merchant":"PUBLICTESTHUF","orderRef":"101010516348232058105","currency":"HUF","customerEmail":"sdk_test#otpmobil.com","language":"HU","sdkVersion":"SimplePayV2.1_Payment_PHP_SDK_2.0.7_190701:dd236896400d7463677a82a47f53e36e","methods":["CARD"],"total":"25","timeout":"2021-10-30T12:30:11+00:00","url":"https:\/\/sdk.simplepay.hu\/back.php"}
const message = JSON.stringify(body).replace(/\//g, "\\\/");
const signature = crypto.createHmac("sha384", merchantKey)
.update(message)
.digest("base64");
console.log("Your signature is: ", signature); // Your signature is: gcDJ8J7TyT1rC/Ygj/8CihXaLwniMWRav09QSEMQUnv5TbYaEDvQAuBE1mW3plvZ
Note that this fix is to be made in the message string, not in the JavaScript object, where \/ would be interpreted as /, and \\/ as \\/ (and not as \/).
Also note that there are two typos in the JavaScript object you posted for the message: The backslashes before the two square brackets must be removed.
I created a NodeJS application which should get some data from an external API-Server. That server provides its data only as 'Content-Type: text/plain;charset=ISO-8859-1'. I have got that information through the Header-Data of the server.
Now the problem for me is that special characters like 'ä', 'ö' or 'ü' are shown as �.
I tried to convert them with Iconv to UTF-8, but then I got these things '�'...
My question is, what am I doing wrong?
For testing I use Postman. These are the steps I do to test everything:
Use Postman to trigger my NodeJS application
The App requests data from the API-Server
API-Server sends Data to NodeJS App
My App prints out the raw response-data of the API, which already has those strange characters �
The App then tries to convert them with Iconv to UTF-8, where it shows me now this '�' characters
Another strange thing:
When I connect Postman directly to the API-Server, the special characters get shown as they have too without problems. Therefore i guess my application causes the problem but I cannot see where or why...
// Javascript Code:
try {
const response = await axios.get(
URL
{
params: params,
headers: headers
}
);
var iconv = new Iconv('ISO-8859-1', 'UTF-8');
var converted = await iconv.convert(response.data);
return converted.toString('UTF-8');
} catch (error) {
throw new Error(error);
}
So after some deeper research I came up with the solution to my problem.
The cause of all trouble seems to lie within the post-process of axios or something similar. It is the step close after data is received and convertet to text and shortly before the response is generated for my nodejs-application.
What I did was to define the "responseType" of the GET-method of axios as an "ArrayBuffer". Therefore an adjustment in axios was necessary like so:
var resArBuffer = await axios.get(
URL,
{
responseType: 'arraybuffer',
params: params,
headers: headers
}
);
Since JavaScript is awesome, the ArrayBuffer provides a toString() method itself to convert the data from ArrayBuffer to String by own definitions:
var response = resArBuffer.data.toString("latin1");
Another thing worth mentioning is the fact that I used "latin1" instead of "ISO-8859-1". Don't ask me why, some sources even recommended to use "cp1252" instead, but "latin1" workend for me here.
Unfortunately that was not enough yet since I needed the text in UTF-8 format. Using "toString('utf-8')" itself was the wrong way too since it would still print the "�"-Symbols. The workaround was simple. I used "Buffer.from(...)" to convert the "latin1" defined text into a "utf-8" text:
var text = Buffer.from(response, 'utf-8').toString();
Now I get the desired UTF-8 converted text I needed. I hope this thread helps anyone else outhere since thse informations hwere spread in many different threads for me.
I have the following code:
const notifications = await axios.get(url)
const ctype = notifications.headers["content-type"];
The ctype receives "text/json; charset=iso-8859-1"
And my string is like this: "'Ol� Matheus, est� pendente.',"
How can I decode from iso-8859-1 to utf-8 without getting those erros?
Thanks
text/json; charset=iso-8859-1 is not a valid standard content-type. text/json is wrong and JSON must be UTF-8.
So the best way to get around this at least on the server, is to first get a buffer (does axios support returning buffers?), converting it to a UTF-8 string (the only legal Javascript string) and only then run JSON.parse on it.
Pseudo-code:
// be warned that I don't know axios, I assume this is possible but it's
// not the right syntax, i just made it up.
const notificationsBuffer = await axios.get(url, {return: 'buffer'});
// Once you have the buffer, this line _should_ be correct.
const notifications = JSON.parse(notificationBuffer.toString('ISO-8859-1'));
I'm using NodeJS with Express, and when I use foreign characters in the URL, they automatically get encoded.
How do I decode it back to the original string?
Before calling NodeJS, I escape characters.
So the string: אובמה
Becomes %u05D0%u05D5%u05D1%u05DE%u05D4
The entire URL now looks like: http://localhost:32323/?query=%u05D0%u05D5%u05D1%u05DE%u05D4
Now in my NodeJS, I get the escaped string %u05D0%u05D5%u05D1%u05DE%u05D4.
This is the relevant code:
var url_parts = url.parse(req.url, true);
var params = url_parts.query;
var query = params.query; // '%u05D0%u05D5%u05D1%u05DE%u05D4'
I've tried url and querystring libraries but nothing seems to fit my case.
querystring.unescape(query); // still '%u05D0%u05D5%u05D1%u05DE%u05D4'
Update 16/03/18
escape and unescape are deprecated.
Use:
encodeURIComponent('אובמה') // %D7%90%D7%95%D7%91%D7%9E%D7%94
decodeURIComponent('%D7%90%D7%95%D7%91%D7%9E%D7%94') // אובמה
Old answer
unescape('%u05D0%u05D5%u05D1%u05DE%u05D4') gives "אובמה"
Try:
var querystring = unescape(query);
You should use decodeURI() and encodeURI() to encode/decode a URL with foreign characters.
Usage:
var query = 'http://google.com';
query = encodeURI(query);
query = decodeURI(query); // http://google.com
Reference on MDN:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/decodeURI
Decoding query parameters from a URL
decodeURIComponent cannot be used directly to parse query parameters from a URL. It needs a bit of preparation.
function decodeQueryParam(p) {
return decodeURIComponent(p.replace(/\+/g, ' '));
}
console.log(decodeQueryParam('search+query%20%28correct%29'));
// 'search query (correct)'
SOURCE: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/decodeURIComponent#decoding_query_parameters_from_a_url
#**update may-2020**
# how to use an encoder/decoder on node js
### I am writing my answer due to I have noisy data I spend 4 hour to fixed
data email input = myemail#gmail.com
data URL input = /us/home
```
decodeURI function that only decodes a URL special character
email output =>myemail%40gmail.com
url output => %2Fus%2F
using decodeURIComponent
email output = > myemail#gmail.com
url output => /us/
```
here some clarification where you can use decodeURI and decodeURIComponent a fucntion
/*Making http request to the api (Git hub)
create request
parse responce
wrap in a function
*/
var https = require("https");
var username = 'lynndor';
//CREATING AN OBJECT
var options = {
host: 'api.github.com',
path: ' /users/'+ username +'/repos',
method: 'GET'
};
var request = https.request(options, function(responce){
var body = ''
responce.on("data", function(chunk){
body += chunk.toString('utf8')
});
responce.on("end", function(){
console.log("Body", body);
});
});
request.end();
Im trying to create a request to the git hub api, the aim is to get the list repository for the specified you, but i keep getting the above mentioned error, please help
for other situation can be helpful
JavaScript encodeURI() Function
var uri = "my test.asp?name=ståle&car=saab";
var res = encodeURI(uri);
Your "path" variable contains space
path: ' /users/'+ username +'/repos',
Instead it should be
path: '/users/'+ username +'/repos',
Use encodeURIComponent() to encode uri
and decodeURIComponent() to decode uri
Its because there are reserved characters in your uri. You will need to encode uri using inbuilt javascript function encodeURIComponent()
var options = {
host: 'api.github.com',
path: encodeURIComponent('/users/'+ username +'/repos'),
method: 'GET'
};
to decode encoded uri component you can use decodeURIComponent(url)
Typically, you do not want to use encodeURI() directly. Instead, use fixedEncodeURI(). To quote MDN encodeURI() Documentation...
If one wishes to follow the more recent RFC3986 for URLs, which makes square brackets reserved (for IPv6) and thus not encoded when forming something which could be part of a URL (such as a host), the following code snippet may help:
function fixedEncodeURI(str) {
return encodeURI(str).replace(/%5B/g, '[').replace(/%5D/g, ']');
}
There is a similar issue with encodeURIComponent() (source: MDN encodeURIComponent() Documentation), as well as a similar fixedEncodeURIComponent() function. These should be used, rather than the actual encodeURI() or encodeURIComponent() function calls...
To be more stringent in adhering to RFC 3986 (which reserves !, ', (, ), and *), even though these characters have no formalized URI delimiting uses, the following can be safely used:
function fixedEncodeURIComponent(str) {
return encodeURIComponent(str).replace(/[!'()*]/g, function(c) {
return '%' + c.charCodeAt(0).toString(16);
});
}
I was getting this error while trying to hit Elasticsearch's API. For me, it was due to Chinese characters in the Document's Title (in the Request I was sending). Switching to all English characters fixed the issue.
Sometimes browser inspector uses abbreviation of long JSON object.
In my case, the data included unescaped characters such as '…' which should not be in http request.