I'm attempting to replicate some python encryption code in node.js using the built in crypto library. To test, I'm encrypting the data using the existing python script and then attempting to decrypt using node.js.
I have everything working except for one problem, doing the decryption results in a truncated initial decrypted result unless I grab extra data, which then results in a truncated final result.
I'm very new to the security side of things, so apologize in advance if my vernacular is off.
Python encryption logic:
encryptor = AES.new(key, AES.MODE_CBC, IV)
<# Header logic, like including digest, salt, and IV #>
for rec in vect:
chunk = rec.pack() # Just adds disparate pieces of data into a contiguous bytearray of length 176
encChunk = encryptor.encrypt(chunk)
outfile.write(encChunk)
Node decryption logic:
let offset = 0;
let derivedKey = crypto.pbkdf2Sync(secret, salt, iterations, 32, 'sha256');
let decryptor = crypto.createDecipheriv('aes-256-cbc', derivedKey, iv);
let chunk = data.slice(offset, (offset + RECORD_LEN))
while(chunk.length > 0) {
let clearChunk = decryptor.update(chunk);
// unpack clearChunk and do something with that data
offset += RECORD_LEN;
chunk = data.slice(offset, offset + RECORD_LEN);
}
I would expect my initial result to print something like this to hex:
54722e34d8b2bf158db6b533e315764f87a07bbfbcf1bd6df0529e56b6a6ae0f123412341234123400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
And it gets close, expect it cuts off the final 16 bytes (in example above the final 32 "0's" would be missing). This shifts all following decryptions by those 16 bytes, meaning those 32 "0's" are added to the front of the next decrypted chunk.
If I add 16 bytes to the initial chunk size (meaning actually grab more data, not just shift the offset) then this solves everything on the front end, but results in the final chunk losing it's last 16 bytes of data.
One thing that seems weird to me: The initial chunk has a length of 176, but the decrypted results has a length of 160. All other chunks have lengths 176 before and after decryption. I'm assuming I'm doing something wrong with how I'm initializing the decryptor which is causing it to expect an extra 16 bytes of data at the beginning, but can't for the life of me figure out what.
I must be close since the decrypted data is correct, minus the mystery shifting, even when reading in large amounts of data. Just need to figure out this final step.
Short version based on your updated code: if you are absolutely certain that every block will be 176 bytes (i.e. a multiple of 16), then you can add cipher.setAutoPadding(false) to your Node code. If that's not true, or for more about why, read on.
At the end of your decryption, you need to call decryptor.final to get the final block.
If you have all the data together, you can decrypt it in one call:
let clearChunk = decryptor.update(chunk) + decryptor.final()
update() exists so that you can pass data to the decryptor in chunks. For example, if you had a very large file, you may not want a full copy of the encrypted data plus a full copy of the decrypted data in memory at the same time. You can therefore read encrypted data from the file in chunks, pass it to update(), and write out the decrypted data in chunks.
The input data using CBC mode must be a multiple of 16 bytes long. To ensure this, we typically use PKCS7 padding. That will pad out your input data to a multiple of 16. If it's already a multiple of 16 it will add an extra block of 16 bytes. The padding value is the number of padding values. So if your block is 12 bytes long, it will be padded with 04040404. If it's a multiple of 16, then the padding is 16 bytes of 0x10. This padding system lets the decryptor validate that it's removing the right amount of padding. This is likely what's causing your 176/160 issue.
This padding issue is why there's a final() call. The system needs to know which block is the last block so it can remove the padding. So the first call to update() will always return one fewer blocks than you pass in, since it's holding onto it until it knows whether it's the last block.
Looking at your Python code, I think it's not padding at all (most Python libraries I'm familiar with don't pad automatically). As long as the input is certain to be a multiple of 16, that's ok. But the default for Node is to expect padding. If you know that your size will always be a multiple of 16, then you can change the Node side with cipher.setAutoPadding(false). If you don't know for certain that the input size will always be a multiple of 16, then you need to add a pad() call on the Python side for the final block.
Related
Am following the instructions on https://datatracker.ietf.org/doc/html/rfc5649#section-3 and I have gotten to a point where I need to generate the LSB(32,A) for the Alternative Initial Value (AIV). Am using NodeJS with buffers to implement the algorithm. My understanding is 32-bits === buffer.length == 4 or in other words, length 4 of buffer is the 32-bits referenced in the article. I have padded the key after converting it buffer then padding with the length value of 8 - (length % 8) with 0s as value as indicated in the article. Now the thing I have not been able to figure out is getting the value of 32-bit MLI. How do I get the MLI, I just know its Message Length Indicator but thats all I know about it.
Example:
const key = Buffer.from('base64 key', 'base64');
const kek = Buffer.from('A65959A6', 'hex');
Now here I have only MSB(32, A) but not LSB(32, A), how do I get the value, and is there anything I am doing wrong, please help I have already spent alot of time trying to figure this out.
Scenario: Let's say my key length is 75, now I have to pad the remaining 5 characters for it to be multiples of 8, now how do I generate LSB(32, A) in this case?
My server will be sending many IDs to the browser/user, and, for that session, the user might operate using those IDs. Between users, and between multiple sessions of a single user, I need IDs to be encrypted, so they cannot be traced except within the context of a single session. Due to the number of IDs that will be used in each session, it is not reasonable to dereference or hash them and store lookup tables for each session.
The IDs are effectively UUIDs, unique 128 bit values. The server will encrypt them in the context of a session, and when the user queries using them, and the server can decrypt those values within the context of that same session. I would like for the encrypted output to also be 128 bits in length (for example, so they could be rendered as UUIDs even in their encrypted state). What is the best way for me to achieve this?
This is my sample code, demonstrating that I can encrypt 16 bytes (the size of a block), but the cipher extends it to 2 blocks, doubling the size to 32 bytes, when I finalize it. I think because it is OK for a value to be encrypted the same way twice in the context of a single session, it is acceptable to reuse the same IV for each item; so the server stores a key and IV for the session, and can encrypt and decrypt all of the IDs with those.
async function sampleCrypt() {
const algorithm = 'aes-128-cbc';
crypto.scrypt("samplePassword", "salty", 16, (err, key) => {
const iv: Buffer = crypto.randomBytes(16);
const cipher: crypto.Cipher = crypto.createCipheriv(algorithm, key, iv);
const inbuffer = Buffer.allocUnsafe(16);
inbuffer.writeUInt32BE(1960); // just some sample data
console.log(cipher.update(inbuffer, undefined, 'hex')); // loads the whole buffer in
console.log(cipher.final('hex'));
});
}
sampleCrypt();
/* Sample output:
83134f7dc2f9b175bd70a7dd0512eaf7
9495e0cfceab0439fddc92f3fffa48c2
*/
Please advise if I have made any incorrect assumptions here as well. Thanks!
Block ciphers such as AES require plaintexts whose lengths are an integer multiple of the blocksize (16 bytes for AES). If this isn't the case, padding must be used. NodeJS applies PKCS7 padding by default. Here a complete padding block is appended if the plaintext length is already an integer multiple of the blocksize. This is the reason why in your case a 16 bytes plaintext results in a 32 bytes ciphertext. But since the plaintext is always exactly one block long, there is actually no need for padding. In NodeJS the padding can be disabled with cipher.setAutoPadding(false), so in your case plaintext and ciphertext are both 16 bytes long.
A block cipher only encrypts one block. To encrypt longer plaintexts, an operation mode must be used, e.g. CBC as in the posted code. Generally, these operation modes use an initialization vector (IV) whose size is equal to the blocksize (16 bytes for AES). The IV must meet certain conditions, e.g. a key/IV pair may only be used once. Since the IV isn't secret it's usually placed before the ciphertext. In your case, this would result in a 32 bytes result (IV + ciphertext). The condition mentioned also means that the concept you use (one key/IV pair for all encryptions) is inherently insecure.
An operating mode that doesn't require an IV is ECB. ECB generates the same ciphertext for the same plaintext, which generallay allows conclusions from the ciphertext to the plaintext. This problem doesn't exist for a mode with an IV. Therefore ECB is more insecure compared to a mode with an IV. However, the severity of this insecurity ultimately depends on the characteristics of the plaintext and the particular application, and the respective requirements determine whether this disadvantage is tolerable or not. 1-block plaintexts containing a GUID are less vulnerable in this respect than multi-block plaintexts with some message content, so ECB may be an option here.
With disabled padding and ECB mode a 16 bytes plaintext results in a 16 bytes ciphertext, as the following TypeScript code demonstrates:
import * as crypto from "crypto";
const algorithm:string = 'aes-128-ecb';
const key:Buffer = crypto.randomBytes(16);
// Encryption
const plaintextEnc:Buffer = Buffer.from('0123456789012345');
const cipherEnc:crypto.Cipher = crypto.createCipheriv(algorithm, key, null);
cipherEnc.setAutoPadding(false);
const ciphertext:Buffer = Buffer.concat([cipherEnc.update(plaintextEnc), cipherEnc.final()]);
console.log(ciphertext.toString('hex'));
// Decryption
const cipherDec:crypto.Decipher = crypto.createDecipheriv(algorithm, key, null);
cipherDec.setAutoPadding(false);
const plaintextDec:Buffer = Buffer.concat([cipherDec.update(ciphertext), cipherDec.final()]);
console.log(plaintextDec.toString('hex'));
If the limitation to 16 bytes is dropped, GCM would be a recommendable mode that provides besides confidentiality also authenticity and integrity. GCM uses a 12 bytes IV (nonce) and generates a tag (typically 16 bytes) that is used for authentication. In your case, the result (IV + ciphertext + tag) would have a length of 44 bytes. Note that if a key/IV pair is used more than once for GCM, security is lost.
I have data that was encrypted in PHP as follows:
mcrypt_encrypt(MCRYPT_RIJNDAEL_256, SECRET, $data, MCRYPT_MODE_CBC, $iv)
I need to decrypt this data in a Python 3 application. I am trying to use PyCrypto but I am open to other libraries. I expect the following to work:
decryptor = AES.new(key, mode, IV=IV)
plain = decryptor.decrypt(ciphertext)
My initialization vector is 32 bytes, and the following exception is thrown:
ValueError: IV must be 16 bytes long
How can I set PyCrypto to use a 32 byte initialization vector and 32 byte block size?
Alternatively, is there a different library that I can use to decrypt the data?
Thanks to the comments I implemented a suitable solution. I modified rijndael.py in the linked duplicate question to accept bytes rather than strings. I then use it as follows to decrypt 32-byte blocks with the 32-byte initialization vectors.
from rijndael import rijndael
iv = b'myInitializationVectorfoobarfoob'
key = b'myKeyfoobarfoobarfoobarfoobarfoo'
text = b'myCipherTextFoobarfoobarfoobarfo'
r = rijndael(key, block_size=32)
plaintext = r.decrypt(text)
l = ''.join([chr(a ^ b) for a, b in zip(plaintext.encode('latin-1'), iv)])
print(l)
Note that using this rather than PyCrypto is only necessary because libmcrypt incorrectly sets the data block sizes, and thus the initialization vector sizes, to be equal to the key sizes. As far as I understand, data block sizes should always be 128 bits for AES-Rijndael.
I've written a small command line utility to encrypt single files with AES, using Python 3. As I'm sure we all know, AES works on 16-byte blocks, so if I want to encrypt a file that isn't exactly a multiple of 16, then I'll have to pad the file to make it a multiple of 16. PKCS #7 padding scheme says that I should pad the last chunk with N bytes all of value N. This is how I do that in my encryption function.
for chunk in getChunks(plainFile, chunkSizeBytes):
padLength = ((AES.block_size - len(chunk)) % AES.block_size)
# We have to have padding!
if padLength == 0:
padLength = 16
pad = chr(padLength) * padLength
chunk += pad.encode('utf-8')
# Write the encrypted chunk to an output file.
cipherFile.write(en.encrypt(chunk))
However, I'm unsure about how I should read this data from that last chunk of a decrypted file. Is there a way to read in files in reverse order? What's the correct way to do this?
I should pad the last chunk with N bytes all of value N.
In this sentence, the first N is equal to the second N, which means the value of the byte determines how much characters you need to remove for decoding.
For example, if you only have 9 characters in your last chunk, pad with 7 characters of value 7 (7 turns out to be the BEL character, but that doesn't matter).
I'm trying to base64 encode a huge input file and end up with an text output file, and I'm trying to find out whether it's possible to encode the input file bit-by-bit, or whether I need to encode the entire thing at once.
This will be done on the AS/400 (iSeries), if that makes any difference. I'm using my own base64 encoding routine (written in RPG) which works excellently, and, were it not a case of size limitations, would be fine.
It's not possible bit-by-bit but 3 bytes at a time, or multiples of 3 bytes at time will do!.
In other words if you split your input file in "chunks" which size(s) is (are) multiples of 3 bytes, you can encode the chunks separately and piece together the resulting B64-encoded pieces together (in the corresponding orde, of course. Note that the last chuink needn't be exactly a multiple of 3 bytes in size, depending on the modulo 3 value of its size its corresponding B64 value will have a few of these padding characters (typically the equal sign) but that's ok, as thiswill be the only piece that has (and needs) such padding.
In the decoding direction, it is the same idea except that you need to split the B64-encoded data in multiples of 4 bytes. Decode them in parallel / individually as desired and re-piece the original data by appending the decoded parts together (again in the same order).
Example:
"File" contents =
"Never argue with the data." (Jimmy Neutron).
Straight encoding = Ik5ldmVyIGFyZ3VlIHdpdGggdGhlIGRhdGEuIiAoSmltbXkgTmV1dHJvbik=
Now, in chunks:
"Never argue --> Ik5ldmVyIGFyZ3Vl
with the --> IHdpdGggdGhl
data." (Jimmy Neutron) --> IGRhdGEuIiAoSmltbXkgTmV1dHJvbik=
As you see piece in that order the 3 encoded chunks amount the same as the code produced for the whole file.
Decoding is done similarly, with arbitrary chuncked sized provided they are multiples of 4 bytes. There is absolutely not need to have any kind of correspondance between the sizes used for encoding. (although standardizing to one single size for each direction (say 300 and 400) may makes things more uniform and easier to manage.
It is a trivial effort to split any given bytestream into chunks.
You can base64 any chunk of bytes without problem.
The problem you are faced with is that unless you place specific requirements on your chunks (multiples of 3 bytes), the sequence of base64-encoded chunks will be different than the actual output you want.
In C#, this is one (sloppy) way you could do it lazily. The execution is actually deferred until string.Concat is called, so you can do anything you want with the chunked strings. (If you plug this into LINQPad you will see the output)
void Main()
{
var data = "lorum ipsum etc lol this is an example!!";
var bytes = Encoding.ASCII.GetBytes(data);
var testFinal = Convert.ToBase64String(bytes);
var chunkedBytes = bytes.Chunk(3);
var base64chunks = chunkedBytes.Select(i => Convert.ToBase64String(i.ToArray()));
var final = string.Concat(base64chunks);
testFinal.Dump(); //output
final.Dump(); //output
}
public static class Extensions
{
public static IEnumerable<IEnumerable<T>> Chunk<T>(this IEnumerable<T> list, int chunkSize)
{
while(list.Take(1).Count() > 0)
{
yield return list.Take(chunkSize);
list = list.Skip(chunkSize);
}
}
}
Output
bG9ydW0gaXBzdW0gZXRjIGxvbCB0aGlzIGlzIGFuIGV4YW1wbGUhIQ==
bG9ydW0gaXBzdW0gZXRjIGxvbCB0aGlzIGlzIGFuIGV4YW1wbGUhIQ==
Hmmm, if you wrote the base64 conversion yourself you should have noticed the obvious thing: each sequence of 3 octets is represented by 4 characters in base64.
So you can split the base64 data at every multiple of four characters, and it will be possible to convert these chunks back to their original bits.
I don't know how character files and byte files are handled on an AS/400, but if it has both concepts, this should be very easy.
are text files limited in the length of each line?
are text files line-oriented, or are they just character streams?
how many bits does one byte have?
are byte files padded at the end, so that one can only create files that span whole disk sectors?
If you can answer all these questions, what exact difficulties do you have left?