I've been trying to encrypt some messages using "crypto" library in NodeJS, and I'm getting the following error:
(node:31732) UnhandledPromiseRejectionWarning: Error: error:0607F08A:digital envelope routines:EVP_EncryptFinal_ex:data not multiple of block length
at Cipheriv.final (internal/crypto/cipher.js:164:28)
at self.test (...)
self.test = async function(info, object) {
let message = {
info: info,
object: object
};
let iv = crypto.randomBytes(16)
let key = Buffer.from(config.key, 'utf8');
let cipher = crypto.createCipheriv("aes-128-ecb", key, '');
cipher.setAutoPadding(false)
let encrypted = cipher.update(JSON.stringify(message));
encrypted = Buffer.concat([iv, encrypted, cipher.final()]);
encrypted = encrypted.toString('base64');
console.log(encrypted);
}
The error is originating from the cipher.final() call as seen in the stack above.
I can't figure out what this error says and how to resolve it. Unfortunately due to constraints (I'm trying to send encrypted data over UDP) I'm not able to use algorithms like CBC (messages are not received in the same order they are encrypted).
Any help is greatly appreciated!
cipher.setAutoPadding(false) sets padding to false, and ECB and CBC only operate on full blocks - which is the reason why padding is required for anything that is not a multiple of the block size. You should remove the line (preferred) or create your own padding (and fall in a trap of inventing your own crypto).
Note that both ECB and CBC are inherently vulnerable to plaintext / padding oracle attacks. ECB is insecure anyway and it doesn't use an IV. For transport mode security you need a MAC or you should use an authenticated cipher. Transport security is hard to accomplish, try DTLS.
Related
I am starting to use crypto module for my NodeJS project.
My code is simple as below:
const { createCipheriv, randomBytes, createDecipheriv } = require('crypto');
const key = randomBytes(32);
const iv = randomBytes(16);
const cipher = createCipheriv('aes256', key, iv);
const decipher = createDecipheriv('aes256', key, iv);
const cipherTest = (message) => {
const encryptedMessage = cipher.update(message, 'utf8', 'hex') + cipher.final('hex');
const decryptedMessage = decipher.update(encryptedMessage, 'hex', 'utf-8') + decipher.final('utf8');
return decryptedMessage.toString('utf-8')
}
The cipherTest function returns the same result as input. Correct!
However, If I create the cipher and decipher functions separately as below , It can not decipher the encryptedMessage.
const encryptMessage = (message) => {
const encryptedMessage = cipher.update(message, 'utf8', 'hex') + cipher.final('hex');
return encryptedMessage;
}
const decryptMessage = (encryptedMessage) => {
const decryptedMessage = decipher.update(encryptedMessage, 'hex', 'utf-8') + decipher.final('utf8');
return decryptedMessage;
}
Could someone have a look? I am so thankful for that.
Ciphers are algorithms, but they contain state. Furthermore, they usually require an IV as input, which needs to be randomized for CBC which is used in this case. That IV needs to change for each ciphertext, assuming that the key remains the same.
As such, you should generally keep to the following structure in pseudo-code:
MessageEncryptor {
Key key
constructor(Key key) {
if (key = bad format) {
throw error // fail fast
this.key = key
}
String encryptMessage(String message) {
Cipher cipher = Cipher.Create(key)
Bytes iv = Rng.createBytes(cipher.blockSize)
cipher.setIV(iv)
Bytes encodedMessage = UTF8.encode(message)
// lower level runtimes may require padding for CBC mode
Bytes ciphertext = cipher.encrypt(encodedMessage)
ciphertext = Bytes.concat(ciphertext, cipher.final())
Bytes ciphertextMessage = Bytes.concat(iv, ciphertext)
// use HMAC here to append an authentication tag
String encodedCiphertextMessage = Base64.encode(ciphertextMessage)
return encodedCiphertextMessage
}
String decryptMessage(String encodedCiphertextMessage) {
Cipher cipher = Cipher.Create(key)
Bytes ciphertextMessage = Base64.decode(encodedCiphertextMessage)
// use HMAC here to verify an authentication tag
Bytes iv = Bytes.sub(0, cipher.blockSizeBytes)
cipher.setIV(iv)
Bytes encodedMessage = cipher.decrypt(ciphertextMessage, start = iv.Size)
encodedMessage = Bytes.concat(encodedMessage, cipher.final())
// lower level runtimes may require unpadding here
String message = UTF8.decode(encodedMessage)
return message
}
}
As you can see it is imperative that you:
only use the Cipher construct within the method bodies; these object carry state and are generally not thread-safe as they are mutable;
rely on the key as a field so that the message encryptor always uses the same key;
create a new MessageEncryptor instance in case you want to use another key;
create a new IV per message during encryption and extract it from the ciphertext message during decryption.
Notes:
if your scheme can handle bytes then there is no need to encode the ciphertextMessage;
the IV may have a different size or may not require random values for other modes such as GCM;
using an authenticated mode such as GCM is much safer, but hashing the IV + ciphertext using HMAC is also an option (use a separate key if you are unsure about possible issues for reusing the encryption key);
UTF8.encode and UTF8.decode can simply be replaced by any other codec in case your message is not a simple string (UTF8 is highly recommended for strings though);
a lot of libraries will have shortcuts for encoding and such, so your code may be shorter;
there are also pre-made container formats such as CMS or NaCL which are probably more secure and more flexible than any scheme you can come up with;
CBC, as used in this example, is vulnerable to plaintext oracle attacks including padding oracle attacks
for larger messages it makes more sense to use streaming for all encoding, decoding, encryption and decryption operations (the methods shown here have a high and unnecessary memory requirement);
create these kind of classes only for specific types of messages, and create a protocol + protocol description if you do so; most crypto API's don't need wrapper classes, it was created with this kind of flexibility on purpose;
The MessageEncryptor still is very lightweight - actually it has only one key as state (sometimes having a an additional key and an algorithm or two as well). If you are going to expand with fields that can change value or state such as a buffer then you want to create a new instance of the class each time you use it from a different thread. Otherwise you want to clear the state after each successful message encryption / decryption.
You may want to destroy the key securely once you've finished using it.
I'm looking for a cipher algorithm which doesn't require an initialization vector. I use the NodeJS function crypto.createCipheriv which documents at least the option to pass no initialization vector.
If the cipher does not need an initialization vector, iv may be null.
const encryptionAlgoritm = 'aes-192-cbc';
const encryptionKey = '...';
const cipher = createCipheriv(encryptionAlgoritm, encryptionKey, null);
// Error: Missing IV for cipher aes-192-cbc
Which OpenSSL algorithm can be used without the iv parameter?
Background: I don't need to decrypt the encrypted value. I want to pass the encrypted value to the database and also query the database with the encrypted value. Therefore the encryption algorithm needs to be bijective.
I need to write the reverse (encryption) of the following decryption function:
const crypto = require('crypto');
let AESDecrypt = (data, key) => {
const decoded = Buffer.from(data, 'binary');
const nonce = decoded.slice(0, 16);
const ciphertext = decoded.slice(16, decoded.length - 16);
const tag = decoded.slice(decoded.length - 16);
let decipher = crypto.createDecipheriv('aes-256-gcm', key, nonce);
decipher.setAuthTag(tag)
decipher.setAutoPadding(false);
try {
let plaintext = decipher.update(ciphertext, 'binary', 'binary');
plaintext += decipher.final('binary');
return Buffer.from(plaintext, 'binary');
} catch (ex) {
console.log('AES Decrypt Failed. Exception: ', ex);
throw ex;
}
}
This above function allows me to properly decrypt encrypted buffers following the spec:
| Nonce/IV (First 16 bytes) | Ciphertext | Authentication Tag (Last 16 bytes) |
The reason why AESDecrypt is written the way it (auth tag as the last 16 bytes) is because that is how the default standard library implementations of AES encrypts data in both Java and Go. I need to be able to bidirectionally decrypt/encrypt between Go, Java, and Node.js. The crypto library based encryption in Node.js does not put the auth tag anywhere, and it is left to the developer how they want to store it to pass to setAuthTag() during decryption. In the above code, I am baking the tag directly into the final encrypted buffer.
So the AES Encryption function I wrote needed to meet the above circumstances (without having to modify AESDecrypt since it is working properly) and I have the following code which is not working for me:
let AESEncrypt = (data, key) => {
const nonce = 'BfVsfgErXsbfiA00'; // Do not copy paste this line in production code (https://crypto.stackexchange.com/questions/26790/how-bad-it-is-using-the-same-iv-twice-with-aes-gcm)
const encoded = Buffer.from(data, 'binary');
const cipher = crypto.createCipheriv('aes-256-gcm', key, nonce);
try {
let encrypted = nonce;
encrypted += cipher.update(encoded, 'binary', 'binary')
encrypted += cipher.final('binary');
const tag = cipher.getAuthTag();
encrypted += tag;
return Buffer.from(encrypted, 'binary');
} catch (ex) {
console.log('AES Encrypt Failed. Exception: ', ex);
throw ex;
}
}
I am aware hardcoding the nonce is insecure. I have it this way to make it easier to compare properly encrypted files with my broken implementation using a binary file diff program like vbindiff
The more I looked at this in different ways, the more confounding this problem has become for me.
I am actually quite used to implementing 256-bit AES GCM based encryption/decryption, and have properly working implementations in Go and Java. Furthermore, because of certain circumstances, I had a working implementation of AES decryption in Node.js months ago.
I know this to be true because I can decrypt in Node.js, files that I encrypted in Java and Go. I put up a quick repository that contains the source code implementations of a Go server written just for this purpose and the broken Node.js code.
For easy access for people that understand Node.js, but not Go, I put up the following Go server web interface for encrypting and decrypting using the above algorithm hosted at https://go-aes.voiceit.io/. You can confirm my Node.js decrypt function works just fine by encrypting a file of your choice at https://go-aes.voiceit.io/, and decrypting the file using decrypt.js (Please look at the README for more information on how to run this if you need to confirm this works properly.)
Furthermore, I know this issue is specifically with the following lines of AESEncrypt:
const tag = cipher.getAuthTag();
encrypted += tag;
Running vbindiff against the same file encrypted in Go and Node.js, The files started showing differences only in the last 16 bytes (where the auth tag get's written). In other words, the nonce and the encrypted payload is identical in Go and Node.js.
Since the getAuthTag() is so simple, and I believe I am using it correctly, I have no idea what I could even change at this point. Hence, I have also considered the remote possibility that this is a bug in the standard library. However, I figured I'd try Stackoverflow first before posting an Github Issue as its most likely something I'm doing wrong.
I have a slightly more expanded description of the code, and proof of how I know what is working works, in the repo I set up to try to get help o solve this problem.
Thank you in advance.
Further info: Node: v14.15.4 Go: go version go1.15.6 darwin/amd64
In the NodeJS code, the ciphertext is generated as a binary string, i.e. using the binary/latin1 or ISO-8859-1 encoding. ISO-8859-1 is a single byte charset which uniquely assigns each value between 0x00 and 0xFF to a specific character, and therefore allows the conversion of arbitrary binary data into a string without corruption, see also here.
In contrast, the authentication tag is not returned as a binary string by cipher.getAuthTag(), but as a buffer.
When concatenating both parts with:
encrypted += tag;
the buffer is converted into a string implicitly using buf.toString(), which applies UTF-8 encoding by default.
Unlike ISO-8859-1, UTF-8 is a multi byte charset that defines specific byte sequences between 1 and 4 bytes in length that are assigned to characters, s. UTF-8 table. In arbitrary binary data (such as the authentication tag) there are generally byte sequences that are not defined for UTF-8 and therefore invalid. Invalid bytes are represented by the Unicode replacement character with the code point U+FFFD during conversion (see also the comment by #dave_thompson_085). This corrupts the data because the original values are lost. Thus UTF-8 encoding is not suitable for converting arbitrary binary data into a string.
During the subsequent conversion into a buffer with the single byte charset binary/latin1 with:
return Buffer.from(encrypted, 'binary');
only the last byte (0xFD) of the replacement character is taken into account.
The bytes marked in the screenshot (0xBB, 0xA7, 0xEA etc.) are all invalid UTF-8 byte sequences, s. UTF-8 table, and are therefore replaced by the NodeJS code with 0xFD, resulting in a corrupted tag.
To fix the bug, the tag must be converted with binary/latin1, i.e. consistent with the encoding of the ciphertext:
let AESEncrypt = (data, key) => {
const nonce = 'BfVsfgErXsbfiA00'; // Static IV for test purposes only
const encoded = Buffer.from(data, 'binary');
const cipher = crypto.createCipheriv('aes-256-gcm', key, nonce);
let encrypted = nonce;
encrypted += cipher.update(encoded, 'binary', 'binary');
encrypted += cipher.final('binary');
const tag = cipher.getAuthTag().toString('binary'); // Fix: Decode with binary/latin1!
encrypted += tag;
return Buffer.from(encrypted, 'binary');
}
Please note that in the update() call the input encoding (the 2nd 'binary' parameter) is ignored, since encoded is a buffer.
Alternatively, the buffers can be concatenated instead of the binary/latin1 converted strings:
let AESEncrypt_withBuffer = (data, key) => {
const nonce = 'BfVsfgErXsbfiA00'; // Static IV for test purposes only
const encoded = Buffer.from(data, 'binary');
const cipher = crypto.createCipheriv('aes-256-gcm', key, nonce);
return Buffer.concat([ // Fix: Concatenate buffers!
Buffer.from(nonce, 'binary'),
cipher.update(encoded),
cipher.final(),
cipher.getAuthTag()
]);
}
For the GCM mode, a nonce length of 12 bytes is recommended by NIST for performance and compatibility reasons, see here, chapter 5.2.1.1 and here. The Go code (via NewGCMWithNonceSize()) and the NodeJS code apply a nonce length of 16 bytes different from this.
I want to encrypt a user's data with AES-256 to store it securely in my database. However, I have the problem that the key must be 32 characters long. But the passwords of my users are usually much shorter. Is there a way how I can "extend" the length of the passwords?
I also thought about the fact that human-made passwords are usually weak. So I would need some kind of function that "links" the password to the encryption key?
Here is my code, which I use to encrypt and decrypt:
const crypto = require('crypto');
const algorithm = 'aes-256-cbc';
const key; //Here I would get the password of the user
function encrypt(text) {
const iv = crypto.randomBytes(16);
let cipher = crypto.createCipheriv('aes-256-cbc', Buffer.from(key), iv);
let encrypted = cipher.update(text);
encrypted = Buffer.concat([encrypted, cipher.final()]);
return { iv: iv.toString('hex'), encryptedData: encrypted.toString('hex') };
}
function decrypt(text) {
let iv = Buffer.from(text.iv, 'hex');
let encryptedText = Buffer.from(text.encryptedData, 'hex');
let decipher = crypto.createDecipheriv('aes-256-cbc', Buffer.from(key), iv);
let decrypted = decipher.update(encryptedText);
decrypted = Buffer.concat([decrypted, decipher.final()]);
return decrypted.toString();
}
Many thanks for answers in advance.
Update 1.0:
After some research I have found the following code: (Source)
const crypto = require('crypto');
// Uses the PBKDF2 algorithm to stretch the string 's' to an arbitrary size,
// in a way that is completely deterministic yet impossible to guess without
// knowing the original string
function stretchString(s, outputLength) {
var salt = crypto.randomBytes(16);
return crypto.pbkdf2Sync(s, salt, 100000, outputLength, 'sha512');
}
// Stretches the password in order to generate a key (for encrypting)
// and a large salt (for hashing)
function keyFromPassword(password) {
// We need 32 bytes for the key
const keyPlusHashingSalt = stretchString(password, 32 + 16);
return {
cipherKey: keyPlusHashingSalt.slice(0, 32),
hashingSalt: keyPlusHashingSalt.slice(16)
};
}
If I got everything right, this should solve my problem: From any password I can generate a secure encryption key with a given length using the above function. The same password always generates the same encryption key with the function keyFromPassword(password), right?
Update 2.0:
Thanks to #President James K. Polk, who gave me some important tips, I have now updated my code. I hope that everything is fine now.
You should not be using your user's passwords directly as keys. Instead, you can use them as input to a key-derivation algorithm like pkbdf2.
As this paper on PKBDF2 explains:
Unfortunately, user-chosen passwords are generally short and
lack enough entropy [11], [21], [18]. For these reasons, they cannot
be directly used as a key to implement secure cryptographic systems. A possible solution to this issue is to adopt a key derivation
function (KDF), that is a function which takes a source of initial
keying material and derives from it one or more pseudorandom keys.
A library for computing pkbdf2 for Node.js, for example is found here.
Always add a salt to the text, and append it to the end of the text. The salt can be a randomly generated string of arbitrary length and can easily satisfy the 32-char length limit. In addition, adding salts before encryption strengthens the encryption.
Problem: We have to encrypt a certain column on a certain table (Postgres). It has to be decryptable in SQL queries and in our nodejs/sequelize application layer. The encryption can happen in either layer, but it must be decodable from either.
The issue I'm running into (and I'm sure it's user error) is that if I encrypt in the db I can only decrypt in the db, and the same for node.
I've tried using PGP_SYM_ENCRYPT and ENCRYPT in postgres and crypto and crypto-js/aes in node. I've gotten it to the point where it's decrypting without an error, but returns gibberish.
A few things I've tried so far (test key is thirtytwocharsthirtytwocharsplus):
set() {
this.setDataValue('field', seq.cast(seq.fn('PGP_SYM_ENCRYPT', val,
config.AES_KEY), 'text'))
}
This properly writes the field such that PGP_SYM_DECRYPT will decrypt it, but there's (apparently?) no way to tell Sequelize to wrap the field name with a function call so it's a lot of extra js that I feel is avoidable
const decipher = crypto.createDecipher('aes256', config.AES_KEY)
decipher.setAutoPadding(false);
return decipher.update(new Buffer(this.getDataValue('field', 'binary'), 'binary', 'ascii')) + decipher.final('ascii')
This will decode the field but returns gibberish (�Mq��8Ya�b) instead of the value (test)
aes.encrypt('test', config.AES_KEY)
aes.decrypt(field, config.AES_KEY).toString(CryptoJS.enc.Utf8)
This encrypts fine, decrypts fine, but Postgres errors when trying to decrypt (using either PGP_SYM_DECRYPT or DECRYPT). Casting the resulting field to ::TEXT and pasting it into an online AES Decrypter returns the expected value.
I really want to avoid having to add a bunch of boilerplate to our node repositories/queries, and I really feel like this should work. Using the same crypto algorithm should yield the same results
Any nudge or pointer would be greatly appreciated
Postgres has rather unclear documentation about the raw encryption functions. After a few tries and failures, I managed to replicate most of logic logic in nodejs.
Here is the program I used.
const crypto = require('crypto');
const iv = Buffer.alloc(16); // zeroed-out iv
function encrypt(plainText, algorithm, key) {
const cipher = crypto.createCipheriv(algorithm, key, iv);
let encrypted = cipher.update(plainText, 'utf8', 'base64');
encrypted += cipher.final('base64');
return encrypted;
}
function decrypt(encrypted, algorithm, key) {
const decrypt = crypto.createDecipheriv(algorithm, key, iv);
let text = decrypt.update(encrypted, 'base64', 'utf8');
text += decrypt.final('utf8')
return text;
}
const originalText = "hello world";
const userKey = 'abcd'
const algorithm = 'aes-128-cbc';
const paddedKey = Buffer.concat([Buffer.from(userKey), Buffer.alloc(12)]); // make it 128 bits key
const hw = encrypt(originalText, algorithm, paddedKey);
console.log("original", originalText);
console.log("encrypted:", hw);
console.log("decoded: ", decrypt(hw, algorithm, paddedKey).toString());
Also here is a list of things not documented for the raw functions of postgres:
the key will be auto padded to match one of the 3 lengths: 128 bits, 192 bits, 256 bits
algorithm is automatically upgraded, when key length exceeds the limit. e.g. if key exceeds 128bits, aes-192-cbc will be used to encrypt
if key exceeds 256 bits, it will be truncated to 256 bits.
It would be easier to replicate it in application language (either Javascript or Java), if Postgres has proper documentation of these functions.
Ok, I got it working, hopefully properly
What I did was:
Encrypting with crypto.createCipheriv('aes-256-cbc', new Buffer(config.AES_KEY), iv) in node, encrypt_iv in pgsql and storing as hex in the db, and decrypting with crypto.createDecipheriv/decrypt_iv into text/utf8
I don't know what part I was missing, but between specifying aes256, using the iv methods, and flipping juggling hex/text it seems to be working.
👍