I have a problem to verify a string create by crypto.createHmac with Node.js.
I made some test, first in PHP - everything is OK but I can't find the correct way todo this in Node.js:
PHP CODE:
$jsonData = '"face_url":"https:\/\/"';
echo($jsonData);
echo("\n");
$client_secret = 'kqm6FksaIT';
echo hash_hmac("sha256", $jsonData, $client_secret);
Result:
"face_url":"https:\/\/"
34a4eb09a639c9b80713158ae89e7e8311586e6e6d76e09967f4e42a24759b3e
With Node.js, I have a problem with the interpretation of the string:
var crypto = require('crypto');
var str = '"face_url":"https:\/\/"';
console.log(str);
//OK
var buf1 = crypto.createHmac('sha256','kqm6FksaIT').update(str);
var v = buf1.digest('hex');
console.log(v);
//END
RESULT:
"face_url":"https://"
eb502c4711a6d926eeec7830ff34e021ed62c91e574f383f6534fdd30857a907
=> FAIL.
As you can see, the interpretation of the string is different "face_url":"https:\/\/"** VS **"face_url":"https://"**
I have tried a lot of things, Buffer.From base64, utf8, JSON.stringify, JSON.parse but I can't find a solution.
If you try with another string like: '"face_url":"https"' it's OK Result is the same.
I try to validate the key received in a Netatmo POST packet who contain:
"face_url":"https:\/\/netatmocameraimage.blob.core
You can find an implementation of netatmo webhook in PHP here:
https://github.com/Netatmo/Netatmo-API-PHP/blob/master/Examples/Webhook_Server_Example.php
After reflexion, the only difference between codes was the interpretation of request.body.
In PHP, it seems to be in plain text.
Nodejs parses the request in JSON format ...
After that supposition, i made some test with NodeJS this morning, i configured the expres server with the following option:
var express = require('express');
var crypto = require('crypto');
var app = express();
var bodyParser = require('body-parser');
app.use(bodyParser.text({type:"*/*"}));
After that, the string appears correctly with these famous "/" :
console.log RESULT :
,"face_url":"https://netatmocameraimage.blob.core.windows.net/production/
And voila! The HMAC is now CORRECT!
The HMAC from NETATMO is calculated on brut text and not from JSON!
In the PHP code, only the escape sequences \\ and \' are recognized in a single quoted expression, in all other cases the backslash is interpreted as a literal backslash, i.e. \/ is interpreted as a literal backslash followed by a literal slash (see here, sec. Single quoted). This explains the output of the PHP code:
$jsonData = '"face_url":"https:\/\/"';
...
Output:
"face_url":"https:\/\/"
34a4eb09a639c9b80713158ae89e7e8311586e6e6d76e09967f4e42a24759b3e
In JavaScript, the backslash is ignored for characters that do not represent an escape sequence, (see here, last passage), i.e. an \/ is equivalent to a literal slash. This explains the output of the JavaScript code:
var str = '"face_url":"https:\/\/"';
...
Output:
"face_url":"https://"
eb502c4711a6d926eeec7830ff34e021ed62c91e574f383f6534fdd30857a907
So in order for the JavaScript code to give the same result as the PHP, the backslash must be masked:
var str = '"face_url":"https:\\/\\/"';
...
Output:
"face_url":"https:\/\/"
34a4eb09a639c9b80713158ae89e7e8311586e6e6d76e09967f4e42a24759b3e
Presumably the string with the \/ is the result of a JSON serialization in PHP with json_encode(), which escapes the / by default, i.e. converts it to \/, see also here. In JavaScript, / is simply serialized as /. Note that in PHP the escaping of / can be disabled with JSON_UNESCAPED_SLASHES, see also here.
Related
I am working with a system that syncs files between two vendors. The tooling is written in Javascript and does a transformation on file names before sending it to the destination. I am trying to fix a bug in it that is failing to properly compare file names between the origin and destination.
The script uses the file name to check if it's on destination
For example:
The following file name contains a special character that has different encoding between source and destination.
source: Chinchón.jpg // hex code: ó
destination : Chinchón.jpg // hex code: 0xf3
The function that does the transformation is:
export const normalizeText = (text:string) => text
.normalize('NFC')
.replace(/\p{Diacritic}/gu, "")
.replace(/\u{2019}/gu, "'")
.replace(/\u{ff1a}/gu, ":")
.trim()
and the comparison is happening just like the following:
const array1 = ['Chinchón.jpg'];
console.log(array1.includes('Chinchón.jpg')); // false
Do I reverse the transformation before comparing? what's the best way to do that?
If i got your question right:
// prepare dictionary
const rawDictionary = ['Chinchón.jpg']
const dictionary = rawDictionary.map(x => normalizeText(x))
...
const rawComparant = 'Chinchón.jpg'
const comparant = normalizeText(rawComparant)
console.log(rawSources.includes(comparant))
when there is no chinese character, php and node output the same result.
but when this is chinese character, the output of php is correct, the output of node is not correct
const crypto = require('crypto');
function encodeDesECB(textToEncode, keyString) {
var key = new Buffer(keyString.substring(0, 8), 'utf8');
var cipher = crypto.createCipheriv('des-ecb', key, '');
cipher.setAutoPadding(true);
var c = cipher.update(textToEncode, 'utf8', 'base64');
c += cipher.final('base64');
return c;
}
console.log(encodeDesECB(`{"key":"test"}`, 'MIGfMA0G'))
console.log(encodeDesECB(`{"key":"测试"}`, 'MIGfMA0G'))
node output
6RQdIBxccCUFE+cXPODJzg==
6RQdIBxccCWXTmivfit9AOfoJRziuDf4
php output
6RQdIBxccCUFE+cXPODJzg==
6RQdIBxccCXFCRVbubGaolfSr4q5iUgw
The problem is not the encryption, but a different JSON serialization of the plaintext.
In the PHP code, json_encode() converts the characters as a Unicode escape sequence, i.e. the encoding returns {"key":"\u6d4b\u8bd5"}. In the NodeJS code, however, {"key": "测试"} is applied.
This means that different plaintexts are encrypted in the end. Therefore, for the same ciphertext, a byte-level identical plaintext must be used.
If Unicode escape sequences are to be applied in the NodeJS code (as in the PHP code), an appropriate conversion is necessary. For this the jsesc package can be used:
const jsesc = require('jsesc');
...
console.log(encodeDesECB(jsesc(`{\"key\":\"测试\"}`, {'lowercaseHex': true}), 'MIGfMA0G')); // 6RQdIBxccCXFCRVbubGaolfSr4q5iUgw
now returns the result of the posted PHP code.
If the Unicode characters are to be used unmasked in the PHP code (as in the NodeJS code), an appropriate conversion is necessary. For this the flag JSON_UNESCAPED_UNICODE can be set in json_encode():
$data = json_encode($data, JSON_UNESCAPED_UNICODE); // 6RQdIBxccCWXTmivfit9AOfoJRziuDf4
now returns the result of the posted NodeJS code.
I am using GMAIL api to send email from the nodejs api. I am rendering the raw body using the following utility function
message += '[DEFAULT EMOJI 😆]'
const str = [
'Content-Type: text/html; charset="UTF-8"\n',
'MIME-Version: 1.0\n',
'Content-Transfer-Encoding: 7bit\n',
'to: ',
to,
'\n',
'from: ',
from.name,
' <',
from.address,
'>',
'\n',
'subject: ',
subject + '[DEFAULT EMOJI 😆]',
'\n\n',
message
].join('');
return Buffer.alloc(str.length, str).toString('base64').replace(/\+/g, '-').replace(/\//g, '_');
The code i have used to send the email is
const r = await gmail.users.messages.send({
auth,
userId: "me",
requestBody: {
raw: makeEmailBody(
thread.send_to,
{
address: user.from_email,
name: user.from_name,
},
campaign.subject,
campaign.template,
thread.id
),
},
});
The emojis are being rendered in the body but not working in subject. See the pic below
Left one is from Gmail in Google Chrome on Desktop and right one is from Gmail App in Mobile
Your utility may benefit from several improvements (we will get to the emoji problem):
First, make it RFC822 compliant by separating lines with CRLF (\r\n).
Be careful with Content-Transfer-Encoding header, set it to 7bit is easiest, but may not be generic enough (quoted-printable is likely the better option).
Now to the emoji problem:
You need to make sure the subject is correctly encoded separately from the body to be able to pass the emoji. According to RFC13420, you can use either Base64 or quoted-printable encoding on a subject to create an encoded-word, described as:
"=" "?" charset "?" encoding "?" encoded-text "?" "="
Where encoding is either Q for quoted-printable and B for Base64 encoding.
Note that the resulting encoded subject string must not be longer than 76 chars in length, which reserves 75 chars for the string and 1 for the separator (to use multiple words, separate them with space or newline [CRLF as well]).
So, set your charset to utf-8, encoding to Q, encode the actual subject with something like below1, and you are half way done:
/**
* #summary RFC 1342 header encoding
* #see {#link https://www.rfc-editor.org/rfc/rfc1342}
*/
class HeaderEncoder {
/**
* #summary encode using Q encoding
*/
static quotedPrintable(str: string, encoding = "utf-8") {
let encoded = "";
for (const char of str) {
const cp = char.codePointAt(0);
encoded += `=${cp.toString(16)}`;
}
return `=?${encoding}?Q?${encoded}?=`;
}
}
Now, the fun part. I was working on a GAS project that had to leverage the Gmail API directly (after all, that is what the client library does under the hood). Even with the correct encoding, an attempt to pass something like a "Beep! \u{1F697}" resulted in an incorrectly parsed subject.
Turns out you need to leverage fromCodePoint operating on byte array or buffer from the original string. This snippet should suffice (don't forget to apply only to multibyte chars):
const escape = (u: string) => String.fromCodePoint(...Buffer.from(u));
0 This is the initial RFC, it would be more appropriate to refer to RFC 2047. Also, see RFC 2231 for including locale info in the header (and some more obscure extensions).
1 If the char falls into a range of printable US-ASCII, it can be left as-is, but due to an extensive set of rules, I recommend sticking to 48-57 (numbers), 65-90 (uppercase) and 97-122 (lowercase) ranges.
I know there are a billion regex questions on stackoverflow, but I can't understand why my uri matcher isn't working in node.
I have the following:
var uri = "file:tmp.db?mode=ro"
function parseuri2db(uri){
var regex = new RegExp("(?:file:)(.*)(?:\\?.*)");
let dbname = uri.match(regex)
return dbname
}
I'm trying to identify only the database name, which I expect to be:
After an uncaptured file: group
Before an optional ? + parameters to end of string.
While I'm using:
var regex1 = new RegExp("(?:file:)(.*)(?:\\?.*)");
I thought the answer was actually more like:
var regex2 = new RegExp("(?:file:)(.*)(?:\\??.*)");
With a 0 or 1 ? quantifier on the \\? literal. But the latter fails.
Anyway, my result is:
console.log(parseuri2db(conf.db_in.filename))
[ 'file:tmp.db?mode=ro',
'tmp.db',
index: 0,
input: 'file:tmp.db?mode=ro' ]
Which seems to be capturing the whole string in the first argument, rather than just the single capture group I asked for.
My questions are:
What am I doing wrong that I'm getting multiple captures?
How can I rephrase this to capture my capture groups with names?
I expected something like the following to work for (2):
function parseuri2db(uri){
// var regex = new RegExp("(?:file:)(.*)(?:\\?.*)");
// let dbname = uri.match(regex)
var regex = new RegExp("(?<protocol>file:)(?<fname>.*)(<params>\\?.*)");
let [, protocol, fname, params] = uri.match(regex)
return dbname
}
console.log(parseuri2db(conf.db_in.filename))
But:
SyntaxError: Invalid regular expression: /(?<protocol>file:)(?<fname>.*)(<params>\?.*)/: Invalid group
Update 1
Answer to my first question is that I needed to not capture the ? literal in the second capture group:
"(?:file:)([^?]*)(?:\\??.*)"
That particular node regex library does not support groups.
I just discovered that Node (tested: v0.8.23, current git: v0.11.3-pre) ignores any decoding errors in its Buffer handling, silently replacing any non-utf8 characters with '\ufffd' (the Unicode REPLACEMENT CHARACTER) instead of throwing an exception about the non-utf8 input. As a consequence, fs.readFile, process.stdin.setEncoding and friends mask a large class of bad input errors for you.
Example which doesn't fail but really ought to:
> notValidUTF8 = new Buffer([ 128 ], 'binary')
<Buffer 80>
> decodedAsUTF8 = notValidUTF8.toString('utf8') // no exception thrown here!
'�'
> decodedAsUTF8 === '\ufffd'
true
'\ufffd' is a perfectly valid character that can occur in legal utf8 (as the sequence ef bf bd), so it is non-trivial to monkey-patch in error handling based on this showing up in the result.
Digging a little deeper, it looks like this stems from node just deferring to v8's strings and that those in turn have the above behaviour, v8 not having any external world full of foreign-encoded data.
Are there node modules or otherwise that let me catch utf-8 decode errors, preferrably with context about where the error was discovered in the input string or buffer?
I hope you solved the problem in those years, I had a similar one and eventually solved with this ugly trick:
function isValidUTF8(buf){
return Buffer.compare(new Buffer(buf.toString(),'utf8') , buf) === 0;
}
which converts the buffer back and forth and check it stays the same.
The 'utf8' encoding can be omitted.
Then we have:
> isValidUTF8(new Buffer('this is valid, 指事字 eè we hope','utf8'))
true
> isValidUTF8(new Buffer([128]))
false
> isValidUTF8(new Buffer('\ufffd'))
true
where the '\ufffd' character is correctly considered as valid utf8.
UPDATE: now this works in JXcore, too
From node 8.3 on, you can use util.TextDecoder to solve this cleanly:
const util = require('util')
const td = new util.TextDecoder('utf8', {fatal:true})
td.decode(Buffer.from('foo')) // works!
td.decode(Buffer.from([ 128 ], 'binary')) // throws TypeError
This will also work in some browsers by using TextDecoder in the global namespace.
As Josh C. said above: "npmjs.org/package/encoding"
From the npm website: "encoding is a simple wrapper around node-iconv and iconv-lite to convert strings from one encoding to another."
Download:
$ npm install encoding
Example Usage
var result = encoding.convert(new Buffer([ 128 ], 'binary'), "utf8");
console.log(result); //<Buffer 80>
Visit the site: npm - encoding