NodeJS RSA prehashed sign - node.js

Question

I'm running NodeJS 8.12.0 and have to set a signature over a hash, without re-hashing it, performing a raw signature. Or, in other words, encrypt the hashed value with the private key.
const crypto = require('crypto');
// 4096 bits key.
let pk = "<BASE64 DER>";
let pub = "<BASE64 DER>";
// Transform them to PEM.
pk = `-----BEGIN PRIVATE KEY-----\n${pk.replace('\n', '')}\n-----END PRIVATE KEY-----\n`;
pub = `-----BEGIN PUBLIC KEY-----\n${pub.replace('\n', '')}\n-----END PUBLIC KEY-----\n`;
// Load the data to sign and set the signature.
const fingerprint = Buffer.from('2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824','hex');
const signature = crypto.privateEncrypt({
key: pk,
padding: crypto.constants.RSA_PKCS1_PADDING
},
fingerprint
);
// Unfortunately, the server is not able to verify the signature...
console.log(signature.toString('hex'));
So I took a look at raw encryption of the message hash with my private key and ended up with some sort of EMSA encoding and followed these steps:
Apply the hash function to the message
Encode the algorithm ID for the hash function and the hash into ASN.1 value of DigestInfo (Appendix A.2.4)
Generate an octet string PS consisting of emLen - tLen - 3 octets of 0xff
Concatenate PS, the DER encoded value T, and other padding to form the encoded message EM as EM = 0x00 || 0x01 || PS || 0x00 || T
So, working that out
// 2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824
H = 2c f2 4d ba 5f b0 a3 0e 26 e8 3b 2a c5 b9 e2 9e 1b 16 1e 5c 1f a7 42 5e 73 04 33 62 93 8b 98 24
emLen = 512
T = 30 31 30 0d 06 09 60 86 48 01 65 03 04 02 01 05 00 04 20 2c f2 4d ba 5f b0 a3 0e 26 e8 3b 2a c5 b9 e2 9e 1b 16 1e 5c 1f a7 42 5e 73 04 33 62 93 8b 98 24
PS = 04 06 02 00 33 ff ff ff
// 00010406020033ffffff003031300d0609608648016503040201050004202cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824
EM = 00 01 04 06 02 00 33 ff ff ff 00 30 31 30 0d 06 09 60 86 48 01 65 03 04 02 01 05 00 04 20 2c f2 4d ba 5f b0 a3 0e 26 e8 3b 2a c5 b9 e2 9e 1b 16 1e 5c 1f a7 42 5e 73 04 33 62 93 8b 98 24
But when I put that into the privateEncrypt, I do not get the correct output either. Can anyone help me here?

Answer 1

You are trying to do the PKCS1-padding (RSASSA-PKCS1-V1_5) manually. But this isn't necessary. It's enough to concatenate the Hash-ID (here for SHA-256) and your data (fingerprint), the rest is done by the implicitly selected padding (crypto.constants.RSA_PKCS1_PADDING), i.e.
// Signing
var fingerprint = Buffer.from('2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824','hex');
var id = Buffer.from([0x30, 0x31, 0x30, 0x0d, 0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x01, 0x05, 0x00, 0x04, 0x20]);
var allData = Buffer.concat([id, fingerprint]);
var signature = crypto.privateEncrypt(privateKey, allData); // crypto.constants.RSA_PKCS1_PADDING by default
// Verifying with createVerify
var verify = crypto.createVerify('RSA-SHA256');
verify.update('<the signed data>');
var verified = verify.verify(publicKey, signature); // provides true
// Verifying with publicDecrypt
var decryptedFingerprint = crypto.publicDecrypt(publicKey, Buffer.from(signature)).slice(-32); // provides fingerprint
Note: If you want to do the padding manually, you have to set the byte sequence 0x00 || 0x01 || PS || 0x00 before the allData-buffer. PS consists of as many 0xff-bytes as necessary to reach the key length (in bytes). In addition, the flag
crypto.constants.RSA_NO_PADDING must be set explicitly in the privateEncrypt-call. However, this isn't necessary, since the result is the same. The details are described in RFC 8017, Section 8.2.1