Algorithm, such as MD5 do you know that any amount of symbols that give it a certain number of coding or hash, and yet reversible (ie symmetrical) is?
You're probably looking for encryption. Hashing loses information by design.
A reversible hash becomes regular compression (eg. ZIP - .Net has classes for that) and/or encryption (eg. AES, included in .Net as well). You can't compress a long piece of text into a short hash and hope to be able to reverse it, since information would become lost - see Shannon's theorem, which specifies the limits on how much you can compress information.
Related
Do I need both MD5 and SHA-1 values to be sure the downloaded file is
a) Untouched by hackers. For example, when I need to download some app's .iso via torrents
and
b) Not corrupted during technical issues? For example, some unstable network connection during download.
Or, probably, SHA-1 value will be enough for both checks?
Also, is SHA-1 (without MD5) enough to be sure that some file downloaded years ago and stored somewhere on my HDD haven't degradated?
From a security perspective MD-5 is utterly broken.
SHA-1 is considered suspicious, and avoided for most uses if at all possible. For new projects: don't use it at all.
SHA-2 (aka SHA-256, SHA-512, etc.) is still widely used for fast hashes.
SHA-3 is the future since 2012, nothing is stopping you from using it already. I see little reason not to use it for new projects.
What's the problem with older ones:
Their resistance to finding collisions is below par: This is an attacker creating 2 contents that have the same hash. These are constructed at the same time. This problem is there for MD5 and SHA-1, and it's BAD, but requires the attacker creating both versions (and then they can do a switch at any time they want undetected).
Their resistance to length extension attacks is relatively weak. This is especially true for MD5, but SHA-1 and even SHA-2 to some degree suffer from it.
When is it not a problem: to ensure your disk has not produced an error: and hash will do, even a simple CRC32 will work wonders (and I'd recommend the simpler CRC check), or a RAID array, as these can fix errors, not just detect them.
Use both ?
Well if you have to find a collision on one hash and have that same set of plaintexts also produce a collision on another hash, is probably more difficult. This approach has been used in the past, The original PGP did something like it. If I'm not mistaken it had a number of things it calculated, one of them simply the length (which would prevent the extension attack above).
So yes, it likely adds something, but the way md5 and SHA-1 and SHA-2 work internally is quite similar, and that's the worrisome part: they are too much alike to be sure just how much it adds against a highly sophisticated attacker (think the level of the NSA and their counterparts).
So why not use one of the more modern versions of SHA-2, or even better SHA-3 ? They've no known weaknesses and have been peer-reviewed heavily. As such for any commercial level use, they should be more than enough.
Refs:
https://en.wikipedia.org/wiki/Length_extension_attack
https://en.wikipedia.org/wiki/Collision_attack
https://stackoverflow.com/questions/tagged/sha-3
I know the list of all hash functions is too long. I just want to know the most popular ones which are used in day to day IT practical tasks. I know MD5, SHA1, SHA2 (256 and 512) are really popular. Is there any other hash function I can add to these 5 algorithms?
I want to develop a hash Tool and I just want to include those algorithms that developers really need.
MD5, SHA-1 - Commonly used, used to be secure, but no longer collision resistant
SHA-2 - Commonly used, secure. It's a family of functions with different output size.
SHA-3 - Not yet specified, but will probably become popular after that. Wait for the spec. Will be a family of functions.
CRC32 - Not secure, but really common as checksum
MD4, RIPEMD160 - Haven't seen those for hashing files, but they're still around in some other contexts. MD4 is broken, some older members of the RIPEMD family are broken, but RIPEMD160 is still secure. Only place I've seen whirlpool is TrueCrypt's KDF.
TTH / TigerTreeHash - Used in some filesharing contexts, still secure but security margin grows thin
ED2K - Used in some filesharing contexts, MD4 based, broken collision resistance
Skein, Blake2 - Skein is a SHA-3 finalist, Blake2 is derived from one. Relatively fast in software and occasionally used but not really common. As a contributor to Blake2 I hope it gets more popular :)
Beyond the hashes you named CRC32 is really common, and TTH/ED2K are used in a filesharing context but rarely elsewhere. Haven't seen much of the other hashes in a file hashing context.
Most widely used (and defined in standards for SSL/TLS, OpenPGP, SSH) are:
CRC32 - simple checksum, used in ZIP, OpenPGP and number of other
standards.
MD2, MD5 - too old and weak MD5 - old and considered weak.
SHA1 - standard de facto, used almost everywhere (DSA algorithm is
used only with SHA1, that's also wide usage area).
SHA224/256/384/512 - should supersede SHA1, and is used with DSA keys
larger than 1024 bits, and ECDSA signatures
RipeMD160 - used in OpenPGP, and some X.509 certificates.
There are also other hash algorithms (you can get the full list on wikipedia), but most likely you'll never meet them in real life.
bcrypt and scrypt. These are meant for password hashing.
bcrypt has been around for quite a long time, and it's considered safe. scrypt is a newer one, and it applies some memory intensive operations to prevent brute-force attacks with GPU.
In case you just want to add hash functions in your tool, irrespective of security, then MD-4 and NIST SHA-1 and SHA-2 competition finalists can be implemented.
For newer and more secure hash functions, SHA-3 winner (Keccak) can be implemented.
NIST hash function competition
SHA-3
First you need to decide if you want fast, insecure hash functions, or slow, secure ones.
Of these the best are currently:
Fast: CRC32 on SSE4.2/armv7 HW, Murmur3, CityHash, FNV
Secure: SHA-3 (Keccak), SHA-2, BLAKE2
See https://code.google.com/p/smhasher/w/list for a testing framework of some popular ones.
[Edit note: prev. had bcrypt, scrypt as secure+slow hash functions, but they are just password hash functions]
I suggest you to study about DES and TDES, they do encryption with key and will be good choice for you if u need to encrypt/decrypt data with public / private key.
My software is using AES Rijndael.
I am using a SHA-256 hash to generate a key from a string with an arbitrary length, and then passing this as both the private and public key since in this instance I do not need to differentiate between the two.
How do I protect my key from being hacked out of the executable?
I know not to use a literal but instead generate the key at runtime with some predetermined steps, but all the same the key will still be in memory right before its sent on to the AES initialization function and so can quite easily be retrieved then.
AES is obviously very secure, but what good does that do me if someone breaks the executable instead?
Is there some common practise when solving this problem?
This can't be done. This is the basic problem with e.g. DRM scheme's on PC's: they need to have the key in memory, so it can be extracted. You can maybe obscure it while it is not in use, but that's about it. And if your application is popular and distributed, then somebody will crack you delicious scheme. That's why some companies use dongles or TPM chips for high value applications.
There is something - very complex in mathematical theory - called "whitebox cryptography". In this case, the AES algorithm is modified in a way, that it builds up the secret during encryption. I do not know exactly, how this is achieved, but this one does not need to have a initialized secret, but the secret is part of the algorithm.
An attacker might see, that your AES implementation is a bit "different" but at no time in execution the key is visible in memory. The only chance an attacker will have, is to copy the whole whitebox code but it is really hard to reverse engineer this - he would just be able to use it. Anyway depending on the way you use the AES, this might be enough to break in.
I want to encrypt a binary file. My goal is that to prevent anyone to read the file who doesn't have the password.
Which is the better solution, AES or Blowfish with the same key length? We can assume that the attacker has great resources (softwares, knowledge, money) for cracking the file.
Probably AES. Blowfish was the direct predecessor to Twofish. Twofish was Bruce Schneier's entry into the competition that produced AES. It was judged as inferior to an entry named Rijndael, which was what became AES.
Interesting aside: at one point in the competition, all the entrants were asked to give their opinion of how the ciphers ranked. It's probably no surprise that each team picked its own entry as the best -- but every other team picked Rijndael as the second best.
That said, there are some basic differences in the basic goals of Blowfish vs. AES that can (arguably) favor Blowfish in terms of absolute security. In particular, Blowfish attempts to make a brute-force (key-exhaustion) attack difficult by making the initial key setup a fairly slow operation. For a normal user, this is of little consequence (it's still less than a millisecond) but if you're trying out millions of keys per second to break it, the difference is quite substantial.
In the end, I don't see that as a major advantage, however. I'd generally recommend AES. My next choices would probably be Serpent, MARS and Twofish in that order. Blowfish would come somewhere after those (though there are a couple of others that I'd probably recommend ahead of Blowfish).
It is a not-often-acknowledged fact that the block size of a block cipher is also an important security consideration (though nowhere near as important as the key size).
Blowfish (and most other block ciphers of the same era, like 3DES and IDEA) have a 64 bit block size, which is considered insufficient for the large file sizes which are common these days (the larger the file, and the smaller the block size, the higher the probability of a repeated block in the ciphertext - and such repeated blocks are extremely useful in cryptanalysis).
AES, on the other hand, has a 128 bit block size. This consideration alone is justification to use AES instead of Blowfish.
In terms of the algorithms themselves I would go with AES, for the simple reason is that it's been accepted by NIST and will be peer reviewed and cryptanalyzed for years. However I would suggest that in practical applications, unless you're storing some file that the government wants to keep secret (in which case the NSA would probably supply you with a better algorithm than both AES and Blowfish), using either of these algorithms won't make too much of a difference. All the security should be in the key, and both of these algorithms are resistant to brute force attacks. Blowfish has only shown to be weak on implementations that don't make use of the full 16 rounds. And while AES is newer, that fact should make you lean more towards BlowFish (if you were only taking age into consideration). Think of it this way, BlowFish has been around since the 90's and nobody (that we know of) has broken it yet....
Here is what I would pose to you... instead of looking at these two algorithms and trying to choose between the algorithm, why don't you look at your key generation scheme. A potential attacker who wants to decrypt your file is not going to sit there and come up with a theoretical set of keys that can be used and then do a brute force attack that can take months. Instead he is going to exploit something else, such as attacking your server hardware, reverse engineering your assembly to see the key, trying to find some config file that has the key in it, or maybe blackmailing your friend to copy a file from your computer. Those are going to be where you are most vulnerable, not the algorithm.
AES.
(I also am assuming you mean twofish not the much older and weaker blowfish)
Both (AES & twofish) are good algorithms. However even if they were equal or twofish was slightly ahead on technical merit I would STILL chose AES.
Why? Publicity. AES is THE standard for government encryption and thus millions of other entities also use it. A talented cryptanalyst simply gets more "bang for the buck" finding a flaw in AES then it does for the much less know and used twofish.
Obscurity provides no protection in encryption. More bodies looking, studying, probing, attacking an algorithm is always better. You want the most "vetted" algorithm possible and right now that is AES. If an algorithm isn't subject to intense and continual scrutiny you should place a lower confidence of it's strength. Sure twofish hasn't been compromised. Is that because of the strength of the cipher or simply because not enough people have taken a close look ..... YET
The algorithm choice probably doesn't matter that much. I'd use AES since it's been better researched. What's much more important is choosing the right operation mode and key derivation function.
You might want to take a look at the TrueCrypt format specification for inspiration if you want fast random access. If you don't need random access than XTS isn't the optimal mode, since it has weaknesses other modes don't. And you might want to add some kind of integrity check(or message authentication code) too.
I know this answer violates the terms of your question, but I think the correct answer to your intent is simply this: use whichever algorithm allows you the longest key length, then make sure you choose a really good key. Minor differences in the performance of most well regarded algorithms (cryptographically and chronologically) are overwhelmed by a few extra bits of a key.
Both algorithms (AES and twofish) are considered very secure. This has been widely covered in other answers.
However, since AES is much widely used now in 2016, it has been specifically hardware-accelerated in several platforms such as ARM and x86. While not significantly faster than twofish before hardware acceleration, AES is now much faster thanks to the dedicated CPU instructions.
We have client server based app which saves user related data into a zip file and sets the passwd to the zip file programatically. Just wondering if it could be considered as secure.
Thanks
N
The "classic" encryption for Zip files is considered to be weak. It is breakable, quickly, by known methods. See: "A Known Plaintext Attack on the PKZIP Stream Cipher" for the original paper, by Biham and Kocher, from 1994. Yes, 16 years ago.
More recently there have been other exploits described, for example, the paper
Yet another Plaintext Attack on ZIP's Encryption Scheme (WinZIP) says that a classic-zip encrypted file with 3 entries, and created by WinZip, can be cracked in 2 hours on a "pentium". This was based on an exploit of a weakness in the random number generator then-current WinZip v9.0 tool. I'm sure it would go much faster now, on current processors, but at the same time, I'm pretty sure WinZip, now at v12.0, has fixed this problem in their random number generator. Nevertheless, even without the specific-to-WinZip-v9 exploit, classic ZIP encryption remains weak.
This weak zip encryption that has been cracked is also known as "ZIP 2.0 encryption" or "PKZIP encryption".
Many modern ZIP toolkits also support AES encryption of ZIP entries. This is considered to be strong encryption, and is quite secure (** See note). WinZip, XCeed, and DotNetZip are three such tools that support reading and writing zip files with this encryption level. Among the three, DotNetZip is the only free option.
You didn't mention the library you use to programmatically produce the zip file. If you use DotNetZip, producing an AES-encrypted ZIP file in C# is as easy as this:
using (var zip = new ZipFile())
{
zip.AddFile("MySensitiveFile.doc");
zip.Encryption = EncryptionAlgorithm.WinZipAes128;
zip.Password = "Very.Secret!";
zip.Save("MyEncryptedArchive.zip");
}
** note: Yoshi has published a paper entitled Attacking and Repairing the WinZip Encryption Scheme, describing exploits of WinZip's AES encryption to argue that WinZip's AES encryption is not secure. However, the exploits he describes rely on social-engineering or previous compromises or both. For example, the primary exploit described in the paper involves an attacker intercepting the encrypted zip file, modifying it, sending the modified copy to its intended recipient, getting the recipient to attempt to decrypt it and then send the result of that encryption back to the attacker, who can then decrypt the original file. This so-called "exploit" involves numerous leaps of faith, piled on the previous compromise of intercepted communication in both directions. No one has described any structural exploits of WinZip AES, on par with the exploits of ZIP classic encryption.
use 7zip, that has better password security - and also tick the 'encrypt filenames' option
Secure to what level? There are programs out there that can crack the password encryption on a zip file very quickly so if it has to withstand any sort of effort, then no.
If it's just a matter of ensuring that someone with a password can open it and to keep away casual prying eyes, then maybe.
If you want to have some halfway reasonably security I'd look into zipping up the data and then running it through proper encryption software like gpg.
You should ask a couple of question to yourself.
Where are you storing the zip files?
Which permissions are associated to the zip file?
Is the password a strong password?
Usually, it's a good habit to store user data into a folder that is out of the webroot, not directly accessible.
Password generators are also available and they should be used.