Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I recently came across a system where all of the DB connections were managed by routines obscured in various ways, including base 64 encoding, md5sums and various other techniques.
Why is security through obscurity a bad idea?
Security through obscurity would be burying your money under a tree. The only thing that makes it safe is no one knows it's there. Real security is putting it behind a lock or combination, say in a safe. You can put the safe on the street corner because what makes it secure is that no one can get inside it but you.
As mentioned by #ThomasPadron-McCarty below in a comment below:
If someone discovers the password, you can just change the password, which is easy. If someone finds the location, you need to dig up the money and move it somewhere else, which is much more work. And if you use security by obscurity in a program, you would have to rewrite the program.
Security through obscurity can be said to be bad because it often implies that the obscurity is being used as the principal means of security. Obscurity is fine until it is discovered, but once someone has worked out your particular obscurity, then your system is vulnerable again. Given the persistence of attackers, this equates to no security at all.
Obscurity should never be used as an alternative to proper security techniques.
Obscurity as a means of hiding your source code to prevent copying is another subject. I'm rather split on that topic; I can understand why you might wish to do that, personally I've never been in a situation where it would be wanted.
Security through obscurity is an interesting topic. It is (rightly) maligned as a substitute for effective security. A typical principle in cryptography is that a message is unknown but the contents are not. Algorithms for encyrption are typically widely published, analyzed by mathematicians and, after a time, some confidence is built up in their effectivness but there is never a guarantee that they're effective.
Some people hide their cryptographic algorithms but this is considered a dangerous practice because then such algorithms haven't gone through the same scrutiny. Only organisations like the NSA, which has a significant budget and staff of mathematicians, can get away with this kind of approach.
One of the more interesting developments in recent years has been the risk of steganography, which is the practice is hiding message in images, sound files or some other medium. The biggest problem in steganalysis is identifying whether or not a message is there or not, making this security through obscurity.
Last year I came across a story that Researchers Calculate Capacity of a Steganographic Channel but the really interesting thing about this is:
Studying a stego-channel in this way
leads to some counter-intuitive
results: for example, in certain
circumstances, doubling the number of
algorithms looking for hidden data can
increase the capacity of the
steganographic channel.
In other words, the more algorithms you use to identify messages the less effective it becomes, which goes against the normal criticism of security through obscurity.
Interesting stuff.
The main reason it is a bad idea is that it does not FIX the underlying problems, just attempts to hide them. Sooner or later, the problems will be discovered.
Also, extra encryption will incur additional overhead.
Finally excessive obscurity (like using checksums) makes maintenance a nightmare.
Better security alternatives is to eliminate potential weaknesses in your code such as enforced inputs to prevent injection attacks.
One factor the ability to recover from a security breach. If someone discovers your password, just reset it. But if someone uncovers your obscure scheme, you're hosed.
Using obscurity as all these people agree is not security, its buying yourself time. That said having a decent security system implemented then adding an extra layer of obscurity is still useful. Lets say tomorrow someone finds an unbeatable crack/hole in the ssh service that can't be patched immediately.
As a rule I've implemented in house... all public facing servers expose only the ports needed ( http/https ) and nothing more. One public facing server then will have ssh exposed to the internet at some obscure high numbered port and a port scanning trigger setup to block any IP's that try to find it.
Obscurity has its place in the world of security, but not as the first and last line of defense. In the example above, I don't get any script/bot attacks on ssh because they don't want to spend the time searching for a non-standard ssh service port and if they do, their unlikely to find it before another layer of security steps in and cuts them off.
All of the forms of security available are actually forms of security through obscurity. Each method increases in complexity and provides better security but they all rely on some algorithm and one or more keys to restore the encrypted data. "Security through obscurity" as most call it is when someone chooses one of the simplest and easiest to crack algorithms.
Algorithms such as character shifting are easy to implement and easy to crack, that's why they are a bad idea. It's probably better than nothing, but it will, at most, only stop a casual glance at the data from being easily read.
There are excellent resources on the Internet you can use to educate yourself about all of the available encryption methods and their strengths and weaknesses.
Security is about letting people in or keeping them out depending on what they know, who they are, or what they have. Currently, biometrics aren't good at finding who you are, and there's always going to be problems with it (fingerprint readers for somebody who's been in a bad accident, forged fingerprints, etc.). So, actually, much of security is about obfuscating something.
Good security is about keeping the stuff you have to keep secret to a minimum. If you've got a properly encrypted AES channel, you can let the bad guys see everything about it except the password, and you're safe. This means you have a much smaller area open to attack, and can concentrate on securing the passwords. (Not that that's trivial.)
In order to do that, you have to have confidence in everything but the password. This normally means using industry-standard crypto that numerous experts have looked at. Anybody can create a cipher they can't break, but not everybody can make a cipher Bruce Schneier can't break. Since there's a thorough lack of theoretical foundations for cipher security, the security of a cipher is determined by having a lot of very smart and knowledgeable people try to come up with attacks, even if they're not practical (attacks on ciphers always get better, never worse). This means the crypto algorithm needs to be widely known. I have very strong confidence in the Advanced Encryption Standard, and almost none in a proprietary algorithm Joe wrote and obfuscated.
However, there's been problems with implementations of crypto algorithms. It's easy to inadvertantly leave holes whereby the key can be found, or other mischief done. It happened with an alternate signature field for PGP, and weaknesses with SSL implemented on Debian Linux. It's even happened to OpenBSD, which is probably the most secure operating system readily available (I think it's up to two exploits in ten years). Therefore, these should be done by a reputable company, and I'd feel better if the implementations were open source. (Closed source won't stop a determined attacker, but it'll make it harder for random good guys to find holes to be closed.)
Therefore, if I wanted security, I'd try to have my system as reliable as possible, which means as open as possible except for the password.
Layering security by obscurity on top of an already secure system might help some, but if the system's secure it won't be necessary, and if it's insecure the best thing is to make it secure. Think of obscurity like the less reputable forms of "alternative medicine" - it is very unlikely to help much, and while it's unlikely to hurt much by itself it may make the patient less likely to see a competent doctor or computer security specialist, whichever.
Lastly, I'd like to make a completely unsolicited and disinterested plug for Bruce Schneier's blog, as nothing more than an interested reader. I've learned a lot about security from it.
One of the best ways of evaluating, testing or improving a security product is to have it banged on by a large, clever peer group.
Products that rely for their security on being a "black box" can't have the benefit of this kind of test. Of course, being a "black box" always invites the suspicion (often justified) that they wouldn't stand up to that kind of scrutiny anyway.
I argued in one case that password protection is really security through obscurity. The only security I can think of that wouldn't be STO is some sort of biometric security.
Besides that bit of semantics and nit picking, STO (Security through obscurity) is obviously bad in any case where you need real security. However, there might be cases where it doesn't matter. I'll often XOR pad a text file i don't want anyone reading. But I don't really care if they do, i'd just prefer that it not be read. In that case, it doesn't matter, and an XOR pad is a perfect example of an easy to find out STO.
It is almost never a good idea. It is the same to say, is it a good idea to drive without seatbelt? Of course you can find some cases where it fits, but the anwser due to experience seems obvious.
Weak encryption will only deter the least motivated hackers, so it isn't valueless, it just isn't very valuable, especially when strong encryption, like AES, is available.
Security through obscurity is based on the assumption that you are smart and your users are stupid. If that assumption is based on arrogance, and not empirical data, then your users- and hackers-- will determine how to invoke the hidden method, bring up the unlinked page, decompile and extract the plain text password from the .dll, etc.
That said, providing comprehensive meta-data to users is not a good idea, and obscuring is perfectly valid technique as long as you back it up with encryption, authorization, authentication and all those other principles of security.
If the OS is Windows, look at using the Data Protection API (DPAPI). It is not security by obscurity, and is a good way to store login credentials for an unattended process. As pretty much everyone is saying here, security through obscurity doesn't give you much protection.
http://msdn.microsoft.com/en-us/library/ms995355.aspx
http://msdn.microsoft.com/en-us/library/ms998280.aspx
The one point I have to add which hasn't been touched on yet is the incredible ability of the internet to smash security through obscurity.
As has been shown time and time again, if your only defense is that "nobody knows the back door/bug/exploit is there", then all it takes is for one person to stumble across it and, within minutes, hundreds of people will know. The next day, pretty much everyone who wants to know, will. Ouch.
Related
I am developing a Windows Form Application in C#.I have heard that one should not use built in methods and functions in code since hackers have deep understanding of such built in methods and know how to fail them Instead one should always use his/her own functions and methods and if not then call built in functions intelligently from those newly made functions.How much is that true?
A supporting example in favour of my argument is that I have seen developer always develope there own made encryption algorithm like AES,DES,RC4 and Hash functions since they believe that built in encryption algorithm have many times backdoor in them.
What?! No, no, no! Whoever told you this is just wrong.
There is a common fallacy that published source code is more vulnerable to "h4ckerz" because it is available for anyone to spot the flaws in. However, I'm glad you mentioned crypto, because this is an area where this line of reasoning really stands out as the fallacy it is.
One of the most popular questions of all time on https://security.stackexchange.com/ is about a developer (in the OP he was given the pseudonym "Dave") who shared this fear of published code. Dave, like the developer you saw, was trying to homebrew his own encryption algorithm. Here's one of the most popular comments in that thread:
Dave has a fundamentally false premise, that the security of an algorithm relies on (even partially) its obscurity - that's not the case. The security of a hashing algorithm relies on the limits of our understanding of mathematics, and, to a lesser extent, the hardware ability to brute-force it. Once Dave accepts this reality (and it really is reality, read the Wikipedia article on hashing), it's a question of who is smarter - Dave by himself, or a large group of specialists devoted to this very particular problem. (emphasis added)
As a matter of fact, as it stands now, the top two memes on Security.SE are "Don't roll your own" and "Don't be a Dave".
While this has all been about crypto, this applies in general to most open-source software. The chance that a backdoor will get found and fixed goes up with each new set of eyes laid on the code. This should be a simple and uncontroversial premise: the more people are looking for something, the higher the chance it will be found. Yes, this applies to malicious users looking for exploits. However, it also applies to power users, white hat hackers, security researchers, cryptographers, professional developers, and others working for "good", which generally (hopefully) outnumber those working for "evil". This also implicitly relies on the false premise that hackers need to see the source code to do bad things. This should be obviously false based on the sheer number of proprietary systems whose source code has never been published (various Microsoft and Adobe programs come to mind) which have been inundated with vulnerabilities for years. Maybe having source code to read makes the hacker's job easier, but maybe not -- is it easier to pore over source code looking for an attack vector or to just use scanning tools and scripts against a compiled binary?
tl;dr Don't be a Dave. Rolling your own means you have to be the best at everything to succeed, instead of taking a sampling of the best the community has to offer.
Heartbleed
In your comment, you rebut:
Then why was the Heartbleed bug in openSSL not found and corrected [earlier]?
Because no one was looking at it. That's the sad truth. Here's the difference -- what happened once someone did find it? Now tens of thousands of security researchers, crypto experts, and others are looking at it. Suppose the same kind of vulnerability existed in one of the proprietary products I mentioned earlier, which it very well could. Once it's caught (if it's caught), ask yourself:
Could the team of programmers at the company responsible benefit from the help of the entire worldwide community of security experts, cryptographers, and other analysts right now?
If a bug this critical were discovered (and that's a big if!) in your software, would you be prepared to deal with the fallout caused by your custom implementation?
Unless you know of specific failure modes or weaknesses of the built-in methods your application would use and know how to minimize or eliminate them, it is probably better to use the methods provided by the language or library designers, which will often be both more efficient and more secure than what an average programmer would come up with on the fly for a particular project.
Your example absolutely does not support your view: developing your own encryption algorithm without some serious background in the domain and review by cryptanalysts, and then employing it in security-critical code, is a recipe for disaster. Even developing your own custom implementation of an industry standard encryption algorithm can present problems, and almost certainly will if you are inexperienced at it.
When it comes to password security there are things that people agree on like storing salted hashes of passwords, which gives a statistical defense against a compromised model and data.
What some don't seem to agree with are what to do with the salts. There are an infinite number of techniques you can do to try to protect the salts as well, but many experts will suggest that they are just pointless obfuscation of the security model, and that over time the model will be exposed, and I find myself disagreeing with this, but I may not be understanding the other viewpoint.
What I don't understand is if your model is compromised, why should you assume a full compromise instead of a partial? If your security model is distributed across different pieces of infrastructure which aren't equally secured, a partial compromise might not even be a problem. (If you do something like, say, encrypt the salt and retrieve the encryption-key from a more highly secured environment that is less likely to be compromised)
I'm assuming that a compromise isn't always 100% here. I've never had a system hacked nor have I ever hacked so I don't have the complete picture.
The big difference between obfuscation and encryption is that what is obfuscated can be clarified without the need for anything beyond the obfuscated information - as long as you can figure out the type of data you are looking at, then you can find a way to clarify it.
Encryption, meanwhile, can only be decrypted with the right key/password/cracking tool. Encrypted data without those things is very secure.
Consequently obfuscation can at best slow down a determined attacker, although it might put off a less determined one. There are many easier and more maintainable things you can do around the security of your system and your servers to harden them against less determined attackers, so the cost of obfuscation in terms of additional work for you and your systems is rarely justified in my opinion.
To assume a compromise is not 100% when one occurs is a very risky endeavour as what you are actually assuming is that you are smarter and more aware of your system than your attacker. You may be right, but if you assume it and you are wrong then you are in very serious trouble indeed. You have to prove it, and given the problems with proving a negative it is safer to act as though you have been compromised completely when a compromise occurs and to design your systems so that they are as secure as possible even in the situation where a total compromise has occurred.
if your model is compromised, why should you assume a full compromise
instead of a partial?
You should always assume the worst case in security. This is the safest approach.
a partial compromise might not even be a problem
Perhaps. But are you really sure?
In your OP you are basing your analysis on too many assumptions. Perhaps in a very specific setup your evaluation that a partial compromise might not be a problem might be justifiable.
But then again you assume that the attacker is not good enough to find a hole.
Don't base your security model on assumptions.
Security through obscurity, is, at best, a gamble. Does it pay to store the salt somewhere else? Probably. But you have to look at the cost versus reward when you start getting into these multi-step security setups. I would say, if your application requires it, you will know it. Otherwise, adding more steps into the application for the sake of having super intense security may only give yourself headaches when things go wrong. Not everyone needs NASA level security for their systems. That isn't to say security isn't important. It's just that you need to temper it to your current environment.
I've read quite a few times how I shouldn't use cryptography if I'm not an expert. Basically both Jeff and Eric tell you the same:
Cryptography is difficult, better buy the security solution from experts than doing it yourself.
I completely agree, for a start it's incredibly difficult to perceive all possible paths an scenario might take, all the possible attacks against it and against your solution... but then When should we use it?
I will face in a few months with the task of providing a security solution to a preexisting solution we have. That is, we exchange data between servers, second phase of the project is providing good security to it. Buying a third party solution will eat up the budget anyway so ... When is it good to use cryptography for a security solution? Even if you are not a TOP expert.
Edit: To clarify due to some comments.
The project is based on data transport across network locations, the current implementation allows for a security layer to be placed before transport and we can make any changes in implementation we like (assuming reasonable changes, the architecture is well design so changes should have an acceptable impact). The question revolves around this phrase from Eric Lippert:
I don’t know nearly enough about cryptography to safely design or implement a crypto-based security system.
We're not talking about reinventing the wheel, I had in mind a certain schema when I designed the system that implied secure key exchange, encryption and decryption and some other "counter measures" (man in the middle, etc) using C# .NET and the included cryptography primitives, but I'm by no means an expert in the field so when I read that, I of course start doubting myself. Am I even capable of implementing a secure system? Would it always be parts of the system that will be insecure unless I subcontract that part?
I think this blog posting (not mine!) gives some good guidelines.
Other than that there are some things you should never do unless you're an expert. This is stuff like implementing your own crypto algorithm (or your own version of a published algorithm). It's just crazy to do that yourself! (When there's CAPI, JCE, OpenSSL, ....)
Beyond that though if you're 'inventing' anything it's almost certainly wrong. In the Coding Horror post you linked to - the main mistake to my mind is that he's doing it a very low level and you just don't need to. If you were encrypting things in Java (I'm not so familiar with .NET) you could use Jasypt which uses strong default algorithms and parameters and doesn't require you to know about ECB and CBC (though, arguably, you should anyway just because...).
There is going to be a prebuilt system for just about anything you're going to want to do with crypto. If you're storing keys then theres KeyCzar, in other cases theres Jasypt. The point is if you're doing anything 'unusual' with crypto - you shouldn't be; if you're doing something not 'unusual' then you don't need to do the crypto yourself. Don't invent a new way to store keys, generate keys from passwords, verify signatures etc - it's not necessary, it's complicated and you'll almost certainly make a mistake unless you're very very careful...
So... I don't think you necessarily need to be afraid of encrypting things but be aware that if you're specifying algorithms and parameters to those algorithms directly in your code it is probably not good. There are exceptions to any rule but as in the blog post I linked above - if you type AES into your code you're doing it wrong!
The key "take-away" from the Matasano blog post is right at the end (note that TLS is a more precise name for SSL):
THOMAS PTACEK
GPG for data at rest. TLS for data in
motion.
NATE LAWSON
You can also use Guttman's cryptlib,
which has a sane API. Or Google
Keyczar. They both have really simple
interfaces, and they try to make it
hard to do the wrong thing. What we
need are fewer libraries with higher
level interfaces. But we also need
more testing for those libraries.
The rule of thumb with cryptography isn't that you shouldn't use it if you're not an expert; rather, it's that you shouldn't re-invent the wheel unless you're an expert. In other words, use existing implementations / libraries / algorithms as much as possible. For example, don't write your own cryptographic authentication algorithm, or come up with yet another way to store keys.
As for when to use it: whenever you have data that needs to be protected from having others see it. Beyond that, it comes down to which algorithms / approaches are best: SSL vs. IPsec vs. symmetric vs. PKI, etc.
Also, a word of advice: key management is often the most challenging part of any comprehensive cryptographic solution.
You have things backwards: first you must specify your actual requirements in detail ("provide a security solution" is meaningless marketing drivel). Then you look for ways to satisfy those specific requirements; croptography will satisfy some of them.
Example of requirements that cryptography can satisfy:
Protect data sent over publich channels from spying
Protect data against tampering (or rather, detect manipulated data)
Allow servers and clients as well as users to prove their identity to each other
You need to go through the same process as for any other requirement. What is the problem being solved, what is the outcome the users are looking for, how is the solution proposed going to be supported going forward, what are the timescales involved. Sometimes there is an off the shelf solution that does the job, sometimes what you want needs to be developed as a custom solution, and sometimes you'll choose a custom solution as it will work out more cost effective than an off the shelf one.
The same is true with security requirements, but the added complexity is that to do any sort of custom solution requires additional expertise in the technical teams (development, support etc). There is also the issue that the solution may need to be not only secure but recognised as secure. This may be far easier to achieve with an off the shelf solution.
And RickNZ is absolutely right - don't forget key management. Consider this right at the outset as part of the decision making process.
The question I would start by asking, is what are you trying to achieve.
If you are trying to just secure the transmission of the data from server a to server b, then there are a number of mechanisms you could use, which would require little work, such as SSL.
However if you are trying to secure all of the data stored in the application that is a far more difficult, although if it is a requirement, then I would suggest that any cryptography, regardless of how easy to break, is better than none.
As someone who has been asked to do similar things, you face a daunting number of questions in implementing your system. There are major difference between securing a system and implementing cryptography systems.
Implementing a cryptography system is very difficult and experts routinely get it wrong, both in theory and practice. A famous theoretical failure was the knapsack cryptosystem which has been largely abandoned due to the Lenstra–Lenstra–Lovász lattice basis reduction algorithm. On the other side, we saw in the last year how an incorrect seed in Debian's random number generator opened up any key generated by the OS. You want to use a prepackaged cryptosystem, not because its an "experts-only" field, but because you want a community tested and supported system. Almost every cryptographic algorithm I know of has bounds that assume certain tasks to be hard, and if those tasks turn out to be computable (as in the LLL algorithm) the whole system becomes useless over night.
But, I believe, the real heart of the question is how to use things in order to make a secure system. While there are many libraries out there to generate keys, cipher the text, and so on, there are very few systems that implement the entire package. But as always security boils down to two concepts: worth of protection and circle of trust.
If you are guarding the Hope diamond, you spend a lot of money designing a system to protect it, employ a constant force to watch it, and hire crackers to continually try to break in. If you are just discouraging bored teenagers from reading your email, you hack something up in an hour and you don't use that address for secret company documents.
Additionally managing the circle of trust is just as difficult of a task. If your circle includes tech savvy, like-minded friends, you make a system and give them a large amount of trust with the system. If it includes many levels of trust, such as users, admins, and so on, you have a tiered system. Since you have to manage more and more interactions with a larger circle, the bugs in the larger system become more weaknesses to hack and thus you must be extremely careful in designing this system.
Now to answer your question. You hire a security expert the moment the item you're protecting is valuable enough and your circle of trust includes those you cannot trust. You don't design cryptography systems unless you do it for a living and have a community to break them, it is a full time academic discipline. If you want to hack for fun, remember that it is only for fun and don't let the value of what you are protecting get too high.
Pay for security (of which cryptography is a part but only a part) what it is worth but no more. So your first task is to decide what your security is worth, or or how much various states of security are worth. Then invite whoever holds the budget to select which state to aim for and therefore how much to spend.
No absolutes here, it's all relative.
Why buy cryptography? It's one of the most developed area in open source software of great quality :) See for example TrueCrypt or OpenSSL
There is a good chance that whatever you need cryptography for there is already a good quality, reputable open source project for it! (And if you can see the source you can see what they did; I once saw an article about a commercial software supposed to "encrypt" a file that simply xorred every byte with a fixed value!)
And, also, why would you want to re-invent the wheel? It's unlikely that with no cryptography background you will do better or even come close to the current algorithms such as AES.
I think it totally depends on what you are trying to achieve.
Does the data need to be stored encrypted at either end or does it just need to be encrypted whilst in transit?
How are you transferring the data? FTP, HTTP etc?
Probably not a good idea to have security as a second phase as by that point presumably you've been moving data around insecurely for a period of time?
I'm modifying existing security code. The specifications are pretty clear, there is example code, but I'm no cryptographic expert. In fact, the example code has a disclaimer saying, in effect, "Don't use this code verbatim."
While auditing the code I'm to modify (which is supposedly feature complete) I ran across this little gem which is used in generating the challenge:
static uint16 randomSeed;
...
uint16 GetRandomValue(void)
{
return randomSeed++;/* This is not a good example of very random generation :o) */
}
Of course, the first thing I immediately did was pass it around the office so we could all get a laugh.
The programmer who produced this code knew it wasn't a good algorithm (as indicated by the comment), but I don't think they understood the security implications. They didn't even bother to call it in the main loop so it would at least turn into a free running counter - still not ideal, but worlds beyond this.
However, I know that the code I produce is going to similarly cause a real security guru to chuckle or quake.
What are the most common security problems, specific to cryptography, that I need to understand?
What are some good resources that will give me suitable knowledge about what I should know beyond common mistakes?
-Adam
Don't try to roll your own - use a standard library if at all possible. Subtle changes to security code can have a huge impact that aren't easy to spot, but can open security holes. For example, two modified lines to one library opened a hole that wasn't readily apparent for quite some time.
Applied Cryptography is an excellent book to help you understand crypto and code. It goes over a lot of fundamentals, like how block ciphers work, and why choosing a poor cipher mode will make your code useless even if you're using a perfectly implemented version of AES.
Some things to watch out for:
Poor Sources of Randomness
Trying to design your own algorithm or protocol - don't do it, ever.
Not getting it code reviewed. Preferably by publishing it online.
Not using a well established library and trying to write it yourself.
Crypto as a panacea - encrypting data does not magically make it safe
Key Management. These days it's often easier to steal the key with a side-channel attack than to attack the crypto.
Your question shows one of the more common ones: poor sources of randomness. It doesn't matter if you use a 256 bit key if they bits aren't random enough.
Number 2 is probably assuming that you can design a system better than the experts. This is an area where a quality implementation of a standard is almost certainly going to be better than innovation. Remember, it took 3 major versions before SSL was really secure. We think.
IMHO, there are four levels of attacks you should be aware of:
social engineering attacks. You should train your users not to do stupid things and write your software such that it is difficult for users to do stupid things. I don't know of any good reference about this stuff.
don't execute arbitrary code (buffer overflows, xss exploits, sql injection are all grouped here). The minimal thing to do in order to learn about this is to read Writing Secure Code from someone at MS and watching the How to Break Web Software google tech talk. This should also teach you a bit about defense in depth.
logical attacks. If your code is manipulating plain-text, certificates, signatures, cipher-texts, public keys or any other cryptographic objects, you should be aware that handling them in bad ways can lead to bad things. Minimal things you should be aware about include offline&online dictionary attacks, replay attacks, man-in-the-middle attacks. The starting point to learning about this and generally a very good reference for you is http://www.soe.ucsc.edu/~abadi/Papers/gep-ieee.ps
cryptographic attacks. Cryptographic vulnerabilities include:
stuff you can avoid: bad random number generation, usage of a broken hash function, broken implementation of security primitive (e.g. engineer forgets a -1 somewhere in the code, which renders the encryption function reversible)
stuff you cannot avoid except by being as up-to-date as possible: new attack against a hash function or an encryption function (see e.g. recent MD5 talk), new attack technique (see e.g. recent attacks against protocols that send encrypted voice over the network)
A good reference in general should be Applied Cryptography.
Also, it is very worrying to me that stuff that goes on a mobile device which is probably locked and difficult to update is written by someone who is asking about security on stackoverflow. I believe your case would one of the few cases where you need an external (good) consultant that helps you get the details right. Even if you hire a security consultant, which I recommend you to do, please also read the above (minimalistic) references.
What are the most common security problems, specific to cryptography, that I need to understand?
Easy - you(1) are not smart enough to come up with your own algorithm.
(1) And by you, I mean you, me and everyone else reading this site...except for possibly Alan Kay and Jon Skeet.
I'm not a crypto guy either, but S-boxes can be troublesome when messed with (and they do make a difference). You also need a real source of entropy, not just a PRNG (no matter how random it looks). PRNGs are useless. Next, you should ensure the entropy source isn't deterministic and that it can't be tampered with.
My humble advice is: stick with known crypto algorithms, unless you're an expert and understand the risks. You could be better off using some tested, publicly-available open source / public domain code.
Anyone visiting a torrent tracker is sure to find droves of "cracked" programs ranging from simple shareware to software suites costing thousands of dollars. It seems that as long as the program does not rely on a remote service (e.g. an MMORPG) that any built-in copy protection or user authentication is useless.
Is it effectively not possible to prevent a cracker from circumventing the copy protection? Why?
No, it's not really possible to prevent it. You can make it extremely difficult - some Starforce versions apparently accomplished that, at the expense of seriously pissing off a number of "users" (victims might be more accurate).
Your code is running on their system and they can do whatever they want with it. Attach a debugger, modify memory, whatever. That's just how it is.
Spore appears to be an elegant example of where draconian efforts in this direction have not only totally failed to prevent it from being shared around P2P networks etc, but has significantly harmed the image of the product and almost certainly the sales.
Also worth noting that users may need to crack copy protection for their own use; I recall playing Diablo on my laptop some years back, which had no internal optical drive. So I dropped in a no-cd crack, and was then entertained for several hours on a long plane flight. Forcing that kind of check, and hence users to work around it is a misfeature of the stupidest kind.
It is impossible to stop it without breaking your product. The proof:
Given: The people you are trying to prevent from hacking/stealing will inevitably be much more technically sophisticated than a large portion of your market.
Given: Your product will be used by some members of the public.
Given: Using your product requires access to it's data on some level.
Therefore, You have to released you encrypt-key/copy protection method/program data to the public in enough of a fashion that the data has been seen in it's useable/unencrypted form.
Therefore, you have in some fashion made your data accessible to pirates.
Therefore, your data will be more easily accessible to the hackers than your legitimate audience.
Therefore, ANYTHING past the most simplistic protection method will end up treating your legitimate audience like pirates and alienating them
Or in short, the way the end user sees it:
Because it's a fixed defense against a thinking opponent.
The military theorists beat this one to death how many millennia ago ?
Copy-protection is like security -- it's impossible to achieve 100% perfection but you can add layers that make it successively more difficult to crack.
Most applications have some point where they ask (themselves), "Is the license valid?" The hacker just needs to find that point and alter the compiled code to return "yes." Alternatively, crackers can use brute-force to try different license keys until one works. There's also social factors -- once one person buys the tool they might post a valid license code on the Internet.
So, code obfuscation makes it more difficult (but not impossible) to find the code to alter. Digital signing of the binaries makes it more difficult to change the code, but still not impossible. Brute-force methods can be combated with long license codes with lots of error-correction bits. Social attacks can be mitigated by requiring a name, email, and phone number that is part of the license code itself. I've used that method to great effect.
Good luck!
Sorry to bust in on an ancient thread, but this is what we do for a living and we're really really good at it. It's all we do. So some of the information here is wrong and I want to set the record straight.
Theoretically uncrackable protection is not only possible it's what we sell. The basic model the major copy protection vendors (including us) follow is to use encryption of the exe and dlls and a secret key to decrypt at runtime.
There are three components:
Very strong encryption: we use AES 128-bit encryption which is effectively immune to a brute force attack. Some day when quantum computers are common it might be possible to break it but it's unreasonable to assume you will crack this strength encryption to copy software as opposed to national secrets.
Secure key storage: if a cracker can get the key to the encryption, you're hosed. The only way to GUARANTEE a key can't be stolen is to store it on a secure device. We use a dongle (it comes in many flavors but the OS always just sees it as a removable flash drive). The dongle stores the key on a smart card chip which is hardened against side channel attacks like DPA. The key generation is tied to multiple factors which are non-deterministic and dynamic so no single key/master crack is possible. The communication between the key storage and the runtime on the computer is also encrypted so a man-in-the-middle attack is thwarted.
Debugger detection: Basically you want to stop a cracker from taking a snapshot of memory (after decryption) and making an executable out of that. Some of the stuff we do to prevent this is secret, but in general we allow for debugger detection and lock the license when a debugger is present (this is an optional setting). We also never completely decrypt the entire program in memory so you can never get all the code by "stealing" memory.
We have a full time cryptologist who can crack just about anybody's protection system. He spends all his time studying how to crack software so we can prevent it. So you don't think this is just a cheap shill for what we do, we're not unique: other companies such as SafeNet and Arxan Technologies can do some very strong protection as well.
A lot of software-only or obfuscation schemes are easy to crack since the cracker can just identify the program entry point and branch around any any license checking or other stuff the ISV has put in to try to prevent piracy. Some people even with dongles will throw up a dialog when the license isn't found--setting a breakpoint on that error will give the cracker a nice place in the assembly code to do a patch. Again, this requires unencrypted machine code to be available--something you don't get if you do strong encryption of the .exe.
One last thing: I think we're unique in that we've had several open contests where we provided a system to people and invited them to crack it. We've had some pretty hefty cash prizes but no one has yet cracked our system. If an ISV takes our system and implements it incorrectly it's no different from putting a great padlock on your front door attached to a cheap hasp with wood screws--easy to circumvent. But if you use our tools as we suggest we believe your software cannot be cracked.
HTH.
The difference between security and copy-protection is that with security, you are protecting an asset from an attacker while allowing access by an authorized user. With copy protection, the attacker and the authorized user are the same person. That makes perfect copy protection impossible.
I think given enough time a would-be cracker can circumvent any copy-protection, even ones using callbacks to remote servers. All it takes is redirecting all outgoing traffic through a box that will filter those requests, and respond with the appropriate messages.
On a long enough timeline, the survival rate of copy protection systems is 0. Everything is reverse-engineerable with enough time and knowledge.
Perhaps you should focus on ways of making your software be more attractive with real, registered, uncracked versions. Superior customer service, perks for registration, etc. reward legitimate users.
Basically history has shown us the most you can buy with copy protection is a little time. Fundamentally since there is data you want someone to see one way, there is a way to get to that data. Since there is a way someone can exploit that way to get to the data.
The only thing that any copy protection or encryption for that matter can do is make it very hard to get at something. If someone is motivated enough there is always the brute force way of getting around things.
But more importantly, in the computer software space we have tons of tools that let us see how things are working, and once you get the method of how the copy protection works then its a very simple matter to get what you want.
The other issue is that copy protection for the most part just frustrates your users who are paying for your software. Take a look at the open source model they don't bother and some folks are making a ton of money encouraging people to copy their software.
"Trying to make bits uncopyable is like trying to make water not wet." -- Bruce Schneier
Copy protection and other forms of digital restrictions management are inherently breakable, because it is not possible to make a stream of bits visible to a computer while simultaneously preventing that computer from copying them. It just can't be done.
As others have pointed out, copy protection only serves to punish legitimate customers. I have no desire to play Spore, but if I did, I'd likely buy it but then install the cracked version because it's actually a better product for its lack of the system-damaging SecuROM or property-depriving activation scheme.
}} Why?
You can buy the most expensive safe in the world, and use it to to protect something. Once you give away the combination to open the safe, you have lost your security.
The same is true for software, if you want people to use your product you must given them the ability to open the proverbial safe and access the contents, obfuscating the method to open the lock doesn't help. You have granted them the ability to open it.
You can either trust your customers/users, or you can waste inordinate amounts of time and resource trying to defeat them instead of providing the features they want to pay for.
It just doesn't pay to bother. Really. If you don't protect your software, and it's good, undoubtedly someone will pirate it. The barrier will be low, of course. But the time you save from not bothering will be time you can invest in your product, marketing, customer relationships, etc., building your customer base for the long term.
If you do spend the time on protecting your product instead of developing it, you'll definitely reduce piracy. But now your competitors may be able to develop features that you didn't have time for, and you may very well end up selling less, even in the short term.
As others point out, you can easily end up frustrating real and legitimate users more than you frustrate the crooks. Always keep your paying users in mind when you develop a circumvention technique.
If your software is wanted, you have no hope against the army of bored 17 year old's. :)
In the case of personal copying/non-commercial copyright infringement, the key factor would appear to be the relationship between the price of the item and the ease of copying it. You can increase the difficulty to copy it, but with diminishing returns as highlighted by some of the previous answers. The other tack to take would be to lower the price until even the effort to download it via bittorrent is more cumbersome than simply buying it.
There are actually many successful examples where an author has found a sweet spot of pricing that has certainly resulted in a large profit for themselves. Trying to chase a 100% unauthorized copy prevention is a lost cause, you only need to get a large group of customers willing to pay instead of downloading illegaly. The very thing that makes pirating softweare inexpensive is also what makes it inexpensive to publish software.
There's an easy way, I'm amazed you haven't said so in the answers above.
Move the copy protection to a secured area (understand your server in your secure lab).
Your server will receive random number from clients (check that the number wasn't used before), encrypt some ever evolving binary code / computation results with clients' number and your private key and send it back.
No hacker can circumvent this since they don't have access to your server code.
What I'm describing is basically webservice other SSL, that's where most company goes nowadays.
Cons: A competitor will develop an offline version of the same featured product during the time you finish your crypto code.
On protections that don't require network:
According to notes floated around it took two years to crack a popular application which used similar scheme as described in John's answer. (custom hardware dongle protection)
Another scheme which doesn't involve a dongle is "expansive protection". I coined this just now, but it works like this: There's an application which saves user data and for which the users can buy expansions and such from 3rd parties. When user loads the data or uses new expansion, the expansions and the saved data contains also code which performs checks. And of course these checks are also protected by checksum checks. It's not as secure on paper as the other scheme but in practise this application has been half-cracked all the time, so that it mostly functions as a trial despite being cracked as the cracks will always miss some checks and have to patch these expansions as well.
The key point is, while these can be cracked, if enough software vendors used such schemes, this would overwork the few people in the warescene who are willing to dedicate themselves to those. If you do the maths, the protections don't have to be even that great, as long as enough vendors used these custom protections that changed constantly, it would simply overwhelm the crackers and the warez scene would end then and there. *
The only reason this hasn't happened is because publishers buy a single protection that they use all over, making it a huge target just like Windows is target for malware, any protection used in more than single app is a bigger target. So everyone needs to be doing their own custom, unique multi-layered expansive protection. The amount of warez releases would drop to maybe dozen releases per year if it takes months to crack a single release by the very best crackers.
Now for some theorycrafting in marketing software:
If you believe that warez provides worthwhile marketing value, then that should be factored in the business plan. This could entail a very very (too) basic lite version that still cost few dollars to ensure it was cracked. Then you'd hook in the users with "limited time upgrade cheaply from the lite version" offers regularly and other upselling tactics. The lite version should really have at most one buy-worthy feature and otherwise be very crippled. The price should probably be <10 $. The full version should probably be twice as much as the upgrade price from the $10 lite pay-demo version. eg. If the full-version is $80, You'd offer upgrades from the lite version to full version for $40 or something that really seems like killer bargain. Of course you'd avoid revealing these bargains to purchasers who went direct for the $80 edition.
It would be critical that the full version shared no similarity in code to the lite version. You'd intend that the lite-version gets warezed and the full-version will either be time intensive to crack or have network dependency in functionality that will be hard to mimic locally. Crackers are probably more specialized in cracking than trying to code up/replicate parts of functionality that the application has on the web server.
* addendum: for apps/games the scene might end in such unlikely and theoretical circumstance, for other things like music/movies and in practise, I'd look at making it cheap for digital dl buyers to get additional collectible physical items or online-only value - many people are collectors of stuff (especially the pirates) and they could be enticed into buying if it gains something desirable enough over just a digital copy.
Beware though - There's something called "the law of rising expectations". Example from games: Ultima 4-6 standard box included a map made of cloth, and Skyrim Collectors edition has a map made of paper. Expectations had risen and some people aren't going to be happy with a paper map. You want to either keep quality of produce or service constant or manage expectations ahead of time. I believe this is critical when considering these value-add things as you want them to be desirably but not increasingly expensive to make and not turn into something that seems so worthless that it defeats the purpose.
This is one occasion where quality software is a bad thing, because if no one whats your software then they will not spend time trying to crack it, on the other hand things like Adobe's Master Collection CS3, were available just days after release.
So the moral of this story is if you don't want someone to steal your software there is one option: don't write anything worth stealing.
I think someone will come up with a dynamic AI way of defeating all the currently standard methods of copy protection; heck, I'd sure love to get paid to work on that problem. Once they get there then new methods will be developed, but it'll slow things down.
The second best way for society to stop theft of software, is to penalize it heavily, and enforce the penalties.
The best way is to reverse the moral decline, and thereby increase the level of integrity in society.
A lost cause if ever I heard one... of course that doesn't mean you shouldn't try.
Personally, I like Penny Arcade's take on it: "A Cyclical Argument With A Literal Strawman"alt text http://sonicloft.net/im/52