I've started some work of which requires some quality random bytes, such as 32 at a time for an initialising vector for certain cryptographic applications. My issue is, this may be called upon multiple times simultaneously and I cannot afford the block /dev/random issues to wait for more collection of entropy.
I could use it to seed other algorithms, for example what /dev/urandom may do - however I do not trust what I cannot understand, I do not have any readily available resource on its method nor do I know if it remains the same between many kernel versions, I prefer a well defined method of some sort.
Are you aware of any methods you can think of over standard PRNGs that would be suited enough to use for (simultaneous) key generation and alike?
Would certain ciphers such as RC4 with a large seed be sufficient to generate random output? (I've seen a /dev/frandom implementation that uses this, however am not entirely sure of it.)
If it means anything, I am on a headless Debian server, reason of lack of entropy gathering.
The response is simple: use /dev/urandom, not /dev/random. /dev/urandom is cryptographically secure, and will not block. The "superiority" of /dev/random over /dev/urandom exist only in a specific theoretical setting which makes no sense if the random bytes are to be used with just about any "normal" cryptographic algorithm, such as encryption or signatures.
See this for more details.
(Trust me, I am a cryptographer.)
Consider using a hardware random number generator. For example, the entropy key or Whirlygig. Using /dev/urandom instead will avoid blocking but may (depending on your level of paranoia) degrade security (you'll output more random bits than you have input entropy, so in theory the output is predictable - this isn't a problem if you're just using it for IVs however)
On a modern CPU with AES hardware acceleration, you can easily reach more than 1 GiB/s of random data by encrypting a string of zeros using a random password (from /dev/urandom), as shown by another answer on serverfault. Note that the random password is passed as a pipe, so that it doesn't show up in the process list.
On my machine, this approach is roughly 100 times faster than /dev/urandom:
$ openssl enc -aes-256-ctr -pass file:<(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64) -nosalt < /dev/zero | pv > /dev/null
11.2GiB 0:00:09 [1.23GiB/s] [ <=> ]
$
$ # Let's look at /dev/urandom for comparison:
$ pv < /dev/urandom > /dev/null
48MiB 0:00:04 [12.4MiB/s] [ <=> ]
If you put this in a shell script, you can easily pipe it into other processes:
$ cat ~/.bin/fast_random
#!/bin/bash
openssl enc -aes-256-ctr \
-pass file:<(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64) \
-nosalt < /dev/zero
Related
Is there a way to control the output of md5sum command, Isn't it possible to set how many bits the md5 output should be.
$ echo $var | md5sum -b
45819a662f88d80bfb35ad703ca8676f
but the output is too long, I want it to be shorter and I don't want to omit some part of it.
No. An MD5 hash is 128 bits long; you cannot make it shorter without leaving part of it out.
MD5 hash is always 128-bit (or 16 bytes).
Is there any method in OpenSSL which is an equivalent of Crypto++'s SecByteBlock?
Something that clears the memory before freeing it among other things of securing the memory block with sensitive information. Is there any way of securing the RSA struct in memory?
Is there any method in OpenSSL which is an equivalent of Crypto++'s SecByteBlock?
A SecByteBlock is a class that takes advantage of OOP by combining data with the operations to act on the data (lots of hand waiving). OpenSSL is a C library, and it does not have most of the goodies related to OOP.
In OpenSSL, you would use OPENSSL_cleanse. Here are some one-liner uses of it in OpenSSL:
$ grep -R cleanse * | grep -v doc
...
apps/apps.c: OPENSSL_cleanse(buff, (unsigned int)bufsiz);
apps/apps.c: OPENSSL_cleanse(buf, (unsigned int)bufsiz);
apps/apps.c: OPENSSL_cleanse(buf, (unsigned int)bufsiz);
apps/ca.c: OPENSSL_cleanse(key, strlen(key));
apps/dgst.c: OPENSSL_cleanse(buf, BUFSIZE);
apps/enc.c: OPENSSL_cleanse(str, SIZE);
apps/enc.c: OPENSSL_cleanse(str, strlen(str));
...
Is there any way of securing the RSA struct in memory?
RSA_free calls OPENSSL_cleanse internally. So the structure is zeroized when its discarded. According to the OpenSSL man page on RSA_new and RSA_free:
RSA_free() frees the RSA structure and its components. The key is erased before the memory is returned to the system.
But you should probably to define your requirements for "secure in memory." If your requirements include wrapping, then no, OpenSSL does not provide it. But neither does Crypto++.
$ echo 'this is text' > text.1
$ openssl enc -aes-256-cbc -a -k "thisisapassword" -in text.1 -out text.enc
$ openssl enc -d -aes-256-cbc -a -k "thisisapassword" -in text.enc -out text.2
$ cat text.2
this is text
I can do this with openssl. Now, how do I do the same in m2crypto. Documentation is lacking this. I looked at the snv test cases, still nothing there. I found one sample, http://passingcuriosity.com/2009/aes-encryption-in-python-with-m2crypto/ (changed to aes_256_cbc), and it will encrypted/descrypt it's own strings, but it cannot decrypt anything made with openssl, and anything it encrypts isn't decryptable from openssl.
I need to be able enc/dec with aes-256-cbc as have many files already encrypted with this and we have many other systems in place that also handle the aes-256-cbc output just fine.
We use password phrases only, with no IV. So setting the IV to \0 * 16 makes sense, but I'm not sure if this is also part of the problem.
Anyone have any working samples of doing AES 256 that is compatible with m2crypto?
I will also be trying some additional libraries and seeing if they work any better.
Part of the problem is that the openssl created file contains 16 bytes of prepended salt information Salted__xxxxxxxx. So, these must be extracted first, then decryption may occur. The next problem is to take original password, sprinkle in the salt, and take the generated key from that and make the key/iv pair for decryption. I have been able to make the first round of they key in hash, but being 256 bit, it needs two rounds to be successful. The problem is creating the second round of hash.
It should also be mentioned that we are locked into python 2.4 so some of the future key routines that are introduced do not work for us.
I have been trying for about an hour now to find an elegant solution to this problem. My goal is basically to write a bandwidth control pipe command which I could re-use in various situations (not just for network transfers, I know about scp -l 1234). What I would like to do is:
Delay for X seconds.
Read Y amount (or less than Y if there isn't enough) data from pipe.
Write the read data to standard output.
Where:
X could be 1..n.
Y could be 1 Byte up to some high value.
My problem is:
It must support binary data which Bash can't handle well.
Roads I've taken or at least thought of:
Using a while read data construct, it filters all white characters in the encoding your using.
Using dd bs=1 count=1 and looping. dd doesn't seem to have different exit codes for when there were something in if and not. Which makes it harder to know when to stop looping. This method should work if I redirect standard error to a temporary file, read it to check if something was transfered (as it's in the statistics printed on stderr) and repeat. But I suspect that it's extremely slow if used on large amounts of data and if it's possible I'd like to skip creating any temporary files.
Any ideas or suggestions on how to solve this as cleanly as possible using Bash?
may be pv -qL RATE ?
-L RATE, --rate-limit RATE
Limit the transfer to a maximum of RATE bytes per second. A
suffix of "k", "m", "g", or "t" can be added to denote kilobytes
(*1024), megabytes, and so on.
It's not much elegant but you can use some redirection trick to catch the number of bytes copied by dd and then use it as the exit condition for a while loop:
while [ -z "$byte_copied" ] || [ "$byte_copied" -ne 0 ]; do
sleep $X;
byte_copied=$(dd bs=$Y count=1 2>&1 >&4 | awk '$2 == "byte"{print $1}');
done 4>&1
However, if your intent is to limit the transfer throughput, I suggest you to use pv.
Do you have to do it in bash? Can you just use an existing program such as cstream?
cstream meets your goal of a bandwidth controlled pipe command, but doesn't necessarily meet your other criteria with regard to your specific algorithm or implementation language.
What about using head -c ?
cat /dev/zero | head -c 10 > test.out
Gives you a nice 10 bytes file.
The problem is not about randomness itself (we have rand), but in cryptographically secure PRNG. What can be used on Linux, or ideally POSIX? Does NSS have something useful?
Clarification: I know about /dev/random, but it may run out of entropy pool. And I'm not sure whether /dev/urandom is guaranteed to be cryptographically secure.
Use /dev/random (requires user input, eg mouse movements) or /dev/urandom. The latter has an entropy pool and doesn't require any user input unless the pool is empty.
You can read from the pool like this:
char buf[100];
FILE *fp;
if (fp = fopen("/dev/urandom", "r")) {
fread(&buf, sizeof(char), 100, fp);
fclose(fp);
}
Or something like that.
From Wikipedia (my italics):
A counterpart to /dev/random is /dev/urandom ("unlocked" random source) which reuses the internal pool to produce more pseudo-random bits. This means that the call will not block, but the output may contain less entropy than the corresponding read from /dev/random. The intent is to serve as a cryptographically secure pseudorandom number generator. This may be used for less secure applications.
The /dev/random device is intended to be a source of cryptographically secure bits.