I'm encrypting a a text file locally using the crypto module from node.js and upload the file to IPFS. I'm trying to then download the file and decrypt it. But once I upload the encrypted file to IPFS, it seems to change to a buffer.
It looks somewhat like this before upload (I only copied the first line):
N�&��6-d�9L% 9���E��k�ir�C��ڤ|%B5-(���i�
...
...
And it looks like this after download:
{"type":"Buffer","data":[1,78,211,38,190,164,54,25,...])
I tried to convert it to multiple encodings, but that doesn't seem to solve anything. Is there a way to upload it in a different format (which stays the same on IPFS) or is there a way to convert the buffer back and decrypt it?
This is the code I use for encryption:
export const encrypt = (buffer, key) => {
const algorithm = 'aes-256-ctr';
const iv = crypto.randomBytes(16);
const cipher = crypto.createCipheriv(algorithm, key, iv);
const result = Buffer.concat([iv, cipher.update(buffer), cipher.final()]);
return result;
};
Related
I am working out a custom hybrid encryption system. I've got symmetric encryption & asymmetric encryption & decryption all handled server-side. All I need to work out now is symmetric decryption.
I got some trouble because my client is sending symmetric key, iv & data all in string format (after asymmetric decryption), but CryptoJS is very touchy with it's encoding. It's also very confusing and vague as far as documentation goes- at least for a relatively new developer. I just can't figure out what encoding CryptoJS wants for each argument. I figure I should have guessed right by now, but no.
Docs
Some help I've gotten previously
I'm requesting help getting the encoding right so that I can decrypt with the following. And thanks a lot for any assistance.
Example of data after asymmetric decryption as per below (throw away keys):
symmetricKey: bDKJVr5wFtQZaPrs4ZoMkP2RjtaYpXo5HHKbzrNELs8=,
symmetricNonce: Z8q66bFkbEqQiVbrUrts+A==,
dataToReceive: "hX/BFO7b+6eYV1zt3+hu3o5g61PFB4V3myyU8tI3W7I="
exports.transportSecurityDecryption = async function mmTransportSecurityDecryption(dataToReceive, keys) {
const JSEncrypt = require('node-jsencrypt');
const CryptoJS = require("crypto-js");
// Asymmetrically decrypt symmetric cypher data with server private key
const privateKeyQuery = new Parse.Query("ServerPrivateKey");
const keyQueryResult = await privateKeyQuery.find({useMasterKey: true});
const object = keyQueryResult[0];
const serverPrivateKey = object.get("key");
const crypt = new JSEncrypt();
crypt.setPrivateKey(serverPrivateKey);
let decryptedDataString = crypt.decrypt(keys);
let decryptedData = JSON.parse(decryptedDataString);
// Symmetrically decrypt transit data
let symmetricKey = decryptedData.symmetricKey;
let symmetricNonce = decryptedData.symmetricNonce;
// Works perfectly till here <---
var decrypted = CryptoJS.AES.decrypt(
CryptoJS.enc.Hex.parse(dataToReceive),
CryptoJS.enc.Utf8.parse(symmetricKey),
{iv: CryptoJS.enc.Hex.parse(symmetricNonce)}
);
return decrypted.toString(CryptoJS.enc.Utf8);
}
You are using the wrong encoders for data, key and IV. All three are Base64 encoded (and not hex or Utf8). So apply the Base64 encoder.
The ciphertext must be passed to CryptoJS.AES.decrypt() as a CipherParams object or alternatively Base64 encoded, which is implicitly converted to a CipherParams object.
When both are fixed, the plain text is: "[\"001\",\"001\"]".
var symmetricKey = "bDKJVr5wFtQZaPrs4ZoMkP2RjtaYpXo5HHKbzrNELs8="
var symmetricNonce = "Z8q66bFkbEqQiVbrUrts+A=="
var dataToReceive = "hX/BFO7b+6eYV1zt3+hu3o5g61PFB4V3myyU8tI3W7I="
var decrypted = CryptoJS.AES.decrypt(
dataToReceive, // pass Base64 encoded
//{ciphertext: CryptoJS.enc.Base64.parse(dataToReceive)}, // pass as CipherParams object, works also
CryptoJS.enc.Base64.parse(symmetricKey),
{iv: CryptoJS.enc.Base64.parse(symmetricNonce)}
);
console.log(decrypted.toString(CryptoJS.enc.Utf8));
<script src="https://cdnjs.cloudflare.com/ajax/libs/crypto-js/4.1.1/crypto-js.min.js"></script>
I need to write the reverse (encryption) of the following decryption function:
const crypto = require('crypto');
let AESDecrypt = (data, key) => {
const decoded = Buffer.from(data, 'binary');
const nonce = decoded.slice(0, 16);
const ciphertext = decoded.slice(16, decoded.length - 16);
const tag = decoded.slice(decoded.length - 16);
let decipher = crypto.createDecipheriv('aes-256-gcm', key, nonce);
decipher.setAuthTag(tag)
decipher.setAutoPadding(false);
try {
let plaintext = decipher.update(ciphertext, 'binary', 'binary');
plaintext += decipher.final('binary');
return Buffer.from(plaintext, 'binary');
} catch (ex) {
console.log('AES Decrypt Failed. Exception: ', ex);
throw ex;
}
}
This above function allows me to properly decrypt encrypted buffers following the spec:
| Nonce/IV (First 16 bytes) | Ciphertext | Authentication Tag (Last 16 bytes) |
The reason why AESDecrypt is written the way it (auth tag as the last 16 bytes) is because that is how the default standard library implementations of AES encrypts data in both Java and Go. I need to be able to bidirectionally decrypt/encrypt between Go, Java, and Node.js. The crypto library based encryption in Node.js does not put the auth tag anywhere, and it is left to the developer how they want to store it to pass to setAuthTag() during decryption. In the above code, I am baking the tag directly into the final encrypted buffer.
So the AES Encryption function I wrote needed to meet the above circumstances (without having to modify AESDecrypt since it is working properly) and I have the following code which is not working for me:
let AESEncrypt = (data, key) => {
const nonce = 'BfVsfgErXsbfiA00'; // Do not copy paste this line in production code (https://crypto.stackexchange.com/questions/26790/how-bad-it-is-using-the-same-iv-twice-with-aes-gcm)
const encoded = Buffer.from(data, 'binary');
const cipher = crypto.createCipheriv('aes-256-gcm', key, nonce);
try {
let encrypted = nonce;
encrypted += cipher.update(encoded, 'binary', 'binary')
encrypted += cipher.final('binary');
const tag = cipher.getAuthTag();
encrypted += tag;
return Buffer.from(encrypted, 'binary');
} catch (ex) {
console.log('AES Encrypt Failed. Exception: ', ex);
throw ex;
}
}
I am aware hardcoding the nonce is insecure. I have it this way to make it easier to compare properly encrypted files with my broken implementation using a binary file diff program like vbindiff
The more I looked at this in different ways, the more confounding this problem has become for me.
I am actually quite used to implementing 256-bit AES GCM based encryption/decryption, and have properly working implementations in Go and Java. Furthermore, because of certain circumstances, I had a working implementation of AES decryption in Node.js months ago.
I know this to be true because I can decrypt in Node.js, files that I encrypted in Java and Go. I put up a quick repository that contains the source code implementations of a Go server written just for this purpose and the broken Node.js code.
For easy access for people that understand Node.js, but not Go, I put up the following Go server web interface for encrypting and decrypting using the above algorithm hosted at https://go-aes.voiceit.io/. You can confirm my Node.js decrypt function works just fine by encrypting a file of your choice at https://go-aes.voiceit.io/, and decrypting the file using decrypt.js (Please look at the README for more information on how to run this if you need to confirm this works properly.)
Furthermore, I know this issue is specifically with the following lines of AESEncrypt:
const tag = cipher.getAuthTag();
encrypted += tag;
Running vbindiff against the same file encrypted in Go and Node.js, The files started showing differences only in the last 16 bytes (where the auth tag get's written). In other words, the nonce and the encrypted payload is identical in Go and Node.js.
Since the getAuthTag() is so simple, and I believe I am using it correctly, I have no idea what I could even change at this point. Hence, I have also considered the remote possibility that this is a bug in the standard library. However, I figured I'd try Stackoverflow first before posting an Github Issue as its most likely something I'm doing wrong.
I have a slightly more expanded description of the code, and proof of how I know what is working works, in the repo I set up to try to get help o solve this problem.
Thank you in advance.
Further info: Node: v14.15.4 Go: go version go1.15.6 darwin/amd64
In the NodeJS code, the ciphertext is generated as a binary string, i.e. using the binary/latin1 or ISO-8859-1 encoding. ISO-8859-1 is a single byte charset which uniquely assigns each value between 0x00 and 0xFF to a specific character, and therefore allows the conversion of arbitrary binary data into a string without corruption, see also here.
In contrast, the authentication tag is not returned as a binary string by cipher.getAuthTag(), but as a buffer.
When concatenating both parts with:
encrypted += tag;
the buffer is converted into a string implicitly using buf.toString(), which applies UTF-8 encoding by default.
Unlike ISO-8859-1, UTF-8 is a multi byte charset that defines specific byte sequences between 1 and 4 bytes in length that are assigned to characters, s. UTF-8 table. In arbitrary binary data (such as the authentication tag) there are generally byte sequences that are not defined for UTF-8 and therefore invalid. Invalid bytes are represented by the Unicode replacement character with the code point U+FFFD during conversion (see also the comment by #dave_thompson_085). This corrupts the data because the original values are lost. Thus UTF-8 encoding is not suitable for converting arbitrary binary data into a string.
During the subsequent conversion into a buffer with the single byte charset binary/latin1 with:
return Buffer.from(encrypted, 'binary');
only the last byte (0xFD) of the replacement character is taken into account.
The bytes marked in the screenshot (0xBB, 0xA7, 0xEA etc.) are all invalid UTF-8 byte sequences, s. UTF-8 table, and are therefore replaced by the NodeJS code with 0xFD, resulting in a corrupted tag.
To fix the bug, the tag must be converted with binary/latin1, i.e. consistent with the encoding of the ciphertext:
let AESEncrypt = (data, key) => {
const nonce = 'BfVsfgErXsbfiA00'; // Static IV for test purposes only
const encoded = Buffer.from(data, 'binary');
const cipher = crypto.createCipheriv('aes-256-gcm', key, nonce);
let encrypted = nonce;
encrypted += cipher.update(encoded, 'binary', 'binary');
encrypted += cipher.final('binary');
const tag = cipher.getAuthTag().toString('binary'); // Fix: Decode with binary/latin1!
encrypted += tag;
return Buffer.from(encrypted, 'binary');
}
Please note that in the update() call the input encoding (the 2nd 'binary' parameter) is ignored, since encoded is a buffer.
Alternatively, the buffers can be concatenated instead of the binary/latin1 converted strings:
let AESEncrypt_withBuffer = (data, key) => {
const nonce = 'BfVsfgErXsbfiA00'; // Static IV for test purposes only
const encoded = Buffer.from(data, 'binary');
const cipher = crypto.createCipheriv('aes-256-gcm', key, nonce);
return Buffer.concat([ // Fix: Concatenate buffers!
Buffer.from(nonce, 'binary'),
cipher.update(encoded),
cipher.final(),
cipher.getAuthTag()
]);
}
For the GCM mode, a nonce length of 12 bytes is recommended by NIST for performance and compatibility reasons, see here, chapter 5.2.1.1 and here. The Go code (via NewGCMWithNonceSize()) and the NodeJS code apply a nonce length of 16 bytes different from this.
I have text of the form crypto.randomBytes(30).toString("hex") that I need encrypted.
Below is the encrypt and decrypt algorithms that I use.
import crypto from "crypto";
const ALGORITHM = "aes-256-ctr";
const IV_LENGTH = 16;
const ENCRYPTION_KEY = crypto.randomBytes(32);
export const encrypt = (text: string) => {
const iv = crypto.randomBytes(IV_LENGTH);
const cipher = crypto.createCipheriv(ALGORITHM, ENCRYPTION_KEY, iv);
const encryptedText = cipher.update(text, "utf8", "base64") + cipher.final("base64");
return `${iv.toString("hex")}:${encryptedText}`;
};
export const decrypt = (text: string) => {
const textParts = text.split(":");
const iv = Buffer.from(textParts.shift(), "hex");
const decipher = crypto.createDecipheriv(ALGORITHM, ENCRYPTION_KEY, iv);
const encryptedText = Buffer.from(textParts.join(":"), "base64");
return decipher.update(encryptedText, "base64", "utf8") + decipher.final("utf8");
};
I run node in my terminal and am able to mess around with these functions in my repl-like environment.
When I am within that node session, I see the following:
const encryptedText = encrypt("0e1819ff39ce47ec80488896a16520bc6b8fcd7d55dc918c96c61ff8e426")
// Output: "9fa7486458345eae2b46687a81a9fcf5:LOrlVD06eotggmIPAq0z9yzP/EHoeQyZyK6IiBYKZMIWvWYLekmSe73OjlgXdWJVOrcTyWS/eP3UU2yv"
const decryptedText = decrypt(encryptedText);
// Output: "0e1819ff39ce47ec80488896a16520bc6b8fcd7d55dc918c96c61ff8e426"
Just like I want!
If I exit the node session, and open a new node session and copy and paste to decrypt the same string I get the following:
const decryptedText = decrypt(ENCRYPTED_TEXT_FROM_ABOVE)
// Output: "�Z<�\r����S78V��z|Z\u0013��\u001a}�����#ߩ����Ɣh���*����y\b�\u001d���l'�m�'�"
Why is this happening? What changed? Clearly it seems like the Node no longer knows how to display the characters or something. I don't know what encoding it is now.
I came across this because I store the encrypted data in Postgres and upon retrieving it, I sometimes need to decrypt it. For some reason, when I restart the node session it forgets how to read it.
The interesting thing is I can decrypt(encrypt("another string")) => "another string" in the new node terminal and it'll work, but the original string no longer does.
The decryption step is failing here since you are generating a new key for each session in the line:
const ENCRYPTION_KEY = crypto.randomBytes(32);
If you log the key like so:
console.log( { key: ENCRYPTION_KEY.toString("hex") });
You'll see the key is different for each run. So it makes sense that we fail to decrypt the encrypted data from a previous session!
If you change to using a fixed key:
const ENCRYPTION_KEY = Buffer.from("8b3d2068cf410479451eef41fe07d43e62ec80b962ae30cd99f7698499acfd61", "hex");
The output from each session should be decrypted in the next one.
Of course we won't want to leave keys in code, so it would be best to use an environment variable for this purpose.
I tried decrypting a file from s3 which was uploaded by unload command from redshift with AES symmetric key encryption.
If we use the AWS java sdk to download with aes key given to the s3 client it works fine.But if we try to manually decrypt it after downloading the file it gives javax.crypto.BadPaddingException: Given final block not properly padded error.
The reason for manually decrypting the file is i want to decrypt the file using node.js and as far as i know there is no sdk in node that can do this directly.
Node.js code that i tried:
var AWS = require('aws-sdk');
var fs = require('fs');
var crypto = require('crypto');
var CryptoJS = require("crypto-js");
var algorithm = 'aes256';
var inputEncoding = 'hex';
var outputEncoding = 'utf-8';
var key = "symmetric key base 64"; //prod
var data = fs.readFileSync('/tmp/files/myfile');
console.log(data);
var decipher = crypto.createDecipher(algorithm,key);
var deciphered = decipher.update(data, inputEncoding, outputEncoding);
console.log(deciphered);
deciphered += decipher.final(outputEncoding);
console.log(deciphered);
When i try this i get this error: Error: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
So Redshift uses the envelope encryption the same way as the AWS SDK uses envelope encryption to store files on S3. So in order to decrypt the file you should:
Get the encrypted data key and the iv from the S3 object metadata (x-amz-meta-x-amz-key and x-amz-meta-x-amz-iv respectively)
Decrypt that x-amz-meta-x-amz-key value using your symmetric key using AES256 ECB mode => var decipher = crypto.createDecipheriv('AES-128-ECB',key,'');
Then decrypt '0000_part_00' using AES256 CBC mode with iv set to iv from step1 and the key set to the result of step 2. => crypto.createDecipheriv('aes-128-cbc', key, iv)
Remove padding (should be able to use cipher.setAutoPadding(true) if Node.js Crypo, what's the default padding for AES? is correct)
I haven't coded it in nodejs but I have successfully used these steps in Python
step2 step 3+4
How safe is it to encrypt your files with a cipher like AES-256-CBC with the checksum of the same file, used as a key? Is that a risk, or is it safe?
Like this steps:
Generating the checksum of a file
Encrypting file with checksum of the file
Saving encrypted file on a server or disk
Encrypting checksum with a RSA/ECC Public key
Saving encrypted checksum on a server or disk
Decrypting:
Decrypting checksum with RSA/ECC Private key
Using checksum to decrypt file
Are my ideas safe?
CodesInChaos answered my question with a URL. I want to redirect you to:
https://crypto.stackexchange.com/questions/729/is-convergent-encryption-really-secure
https://security.stackexchange.com/questions/7142/online-backup-how-could-encryption-and-de-duplication-be-compatible
for encripting an file to checksum is more or less easy; you can use the library like cripto-js
npm install cripto-js
And if you use typescript
npm install #types/criptojs
After that you already install this library, you must convert the file (image) to MD5 and then, convert the STRING generate for MD5 to base64.
You can run some this:
new Promise((resolve, reject) => {
const reader = new FileReader();
reader.readAsDataURL(file);
reader.onload = (event: any) => {
let binari = event.target.result;
let md5 = criptoJS.MD5(binari).toString();
const encodedWord = criptoJS.enc.Utf8.parse(md5);
const encoded = criptoJS.enc.Base64.stringify(encodedWord);
resolve(reader.result);
};
reader.onerror = error => reject(error);
});
and yes, i am run typescript.
I hope this it work for you. :) happy coding