I currently believe cacert.pem is a bunch of keys that I can use to check that the site I'm talking to is in fact the site its claiming to be. As such, if I sent someone a program that was dependent on cacert.pem I can just send them a version on my computer and this has no security threat to me.
The only security threat would be for them and that is if I sent them a phony cacert.pem.
Is this correct and am I safe sending the version of cacert.pem on my computer to another potentially untrusted person?
EDIT:
AS Steffen pointed out, cacert.pem could refer to any file. I was referring to in particular the one that is found in the Requests Python package.
I don't know which cacert.pem file you are talking about, but/etc/ssl/cacert.pem on BSD or the /etc/ssl/certs folder on Linux contain just a public list of trusted certificate agencies, which are used to verify trust for SSL connections. There is no secret in these files and usually they are not even system specific (although one might add or remove CAs to manage own trust settings).
But again, I don't know what your cacert.pem file contains, because there is no inherent semantic with this file name. If it contains also private keys you should definitely not give it to others.
The only security threat would be for them and that is if I sent them a phony cacert.pem.
cacert.pem is a collection of Root CAs and Subordinate CAs used to certify a site or service.
The three threats here are:
You add your own CA, and then later MitM the connection
The wrong CA certifies the site or service, and an attacker then later MitM the connection
Your copy of cacert.pem is tampered in transit
(1) is less of a concern because it would require you to have a privileged network position, like on the same LAN or in the telecom infrastructure. You could add your own CA and the recipient would likely be no wiser.
(2) is a real problem. For example, we know Google is certified by Equifax Secure Certificate Authority. Equifax certifies a Subordinate CA called GeoTrust Global CA. And GeoTrust certifies a Google Subordinate CA called Google Internet Authority G2.
So the first problem with (2) is Diginotar and recently MSC Holdings claimed to certify Google properties, which we know is wrong. They could pull it off because of the collection of Roots and Subordinates.
The second problem with (2) is related to the first. Because you trust, say, Google Internet Authority G2, Google can mint certificates for any domain, and not just their properties. The problem here is its an unconstrained Subordinate CA, and it was done because it was too inconvenient.
(3) is simply an attack by a MitM. He can remove a needed certificate, which could result in a DoS. Or he could insert a CA, which leads back to (1). or he could corrupt the whole file.
Related
May the root store contain non-self-signed certificates, i.e. the issuer and subject are different?
If so, will certificate chain validations return “success” upon encountering a non-self-signed certificate in the root store, or will the validation continue in the root store until either a self-signed certificate (“success”) or none (“fail”) is encountered?
I suspect this behavior is implementation dependent, but I can’t find any reference.
A root as defined in RFC5280, Section 3.2:
(a) Internet Policy Registration Authority (IPRA): This
authority, operated under the auspices of the Internet
Society, acts as the root of the PEM certification hierarchy
at level 1. It issues certificates only for the next level
of authorities, PCAs. All certification paths start with the
IPRA.
Therefore, a "root store", even if it is a generic, non-specified, description (as President James K. Polk pointed out in his comment), should only contain root CA certificates, which means they have signed themselves.
If you do there might be unwanted side effects...
On Windows this breaks MutualTLS with Internet Information Services (IIS)
This is also not specified by the RFC but the issue is documented as Cause 2 here: https://learn.microsoft.com/en-us/troubleshoot/iis/http-403-forbidden-access-website
I want to ensure that client libraries (currently in Python, Ruby, PHP, Java, and .NET) are configured correctly and failing appropriately when SSL certificates are invalid. Shmatikov's paper, The Most Dangerous Code in the World:
Validating SSL Certificates in Non-Browser Software, reveals how confusing SSL validation is so I want to thoroughly test the possible failures.
Based on research a certificate is invalid if:
It is used before its activation date
It is used after its expiry date
It has been revoked
Certificate hostnames don't match the site hostname
Certificate chain does not contain a trusted certificate authority
Ideally, I think I would have one test case for each of the invalid cases. To that end I am currently testing an HTTP site accessed over HTTPS, which leads to a failure that I can verify in a test like so:
self.assertRaises(SSLHandshakeError, lambda: api.call_to_unmatched_hostname())
This is incomplete (only covering one case) and potentially wrong, so...
How can you test that non-browser software properly validates SSL certificates?
First off, you'll need a collection of SSL certificates, where each has just one thing wrong with it. You can generate these using the openssl command line tool. Of course, you can't sign them with a trusted root CA. You will need to use your own CA. To make this validate correctly, you'll need to install your CA certificate in the client libraries. You can do this in Java, for example, using the control panel.
Once you have the certificates, you can use the "openssl s_server" tool to serve an SSL socket using each one. I suggest you put one certificate on each port.
You now have to use the client library to connect to a port, and verify that you get the correct error message.
I know that Python by default does no certificate validation (look at the manual for httplib.HTTPSConnection). However, m2crypto does do validation. Java by default does do validation. I don't know about other languages.
Some other cases you could test:
1) Wildcard host names.
2) Certificate chaining. I know there was a bug in old browsers where if you had a certificate A signed by the root, A could then sign B, and B would appear valid. SSL is supposed to stop this by having flags on certificates, and A would not have the "can sign" flag. However, this was not verified in some old browsers.
Good luck! I'd be interested to hear how you get on.
Paul
Certificate hostnames don't match the site hostname
This is probably the easiest to check, and failure (to fail) there is certainly a good indication that something is wrong. Most certificates for well-known services only use host names for their identity, not IP addresses. If, instead of asking for https://www.google.com/, you ask for https://173.194.67.99/ (for example) and it works, there's something wrong.
For the other ones, you may want to generate your own test CA.
Certificate chain does not contain a trusted certificate authority
You can generate a test certificate using your test CA (or a self-signed certificate), but let the default system CA list be used for the verification. Your test client should fail to verify that certificate.
It is used before its activation date, It is used after its expiry date
You can generate test certificates using your test CA, with notBefore/notAfter dates that make the current date invalid. Then, use your test CA as a trusted CA for the verification: your test client should fail to validate the certificate because of the dates.
It has been revoked
This one is probably the hardest to set up, depending on how revocation is published. Again, generate some test certificates that you've revoked immediately, using your own test CA.
Some tools expect to be configured with a set of CRL files next to the set of trusted CAs. This requires some setup for the test itself, but very little online setup: this is probably the easiest. You can also set up a local online revocation repository, e.g. using CRL distribution points or OCSP.
PKI testing can be more complex than that more generally. A full test suite would require a fairly good understanding of the specifications (RFC 5280). Indeed, you may need to check the dates for all intermediate certificates, as well as various attributes for each certificate in the chain (e.g. key usage, basic constraints, ...).
In general, client libraries separate the verification process into two operations: verifying that the certificate is trusted (the PKI part) and verifying that it was issued to the entity you want to connect to (the host name verification part). This is certainly due to the fact these are specified in different documents (RFC 3280/5280 and RFC 2818/6125, respectively).
From a practical point of view, the first two points to check when using an SSL library are:
What happens when you connect to a known host, but with a different identifier for which the certificate isn't valid (such as its IP address instead of the host)?
What happens when you connect to a certificate that you know cannot be verified by any default set of trusted anchors (for example, a self-signed certificate or from your own CA).
Failure to connect/verify should happen in both cases. If it all works, short of implementing a full PKI test suite (which require a certain expertise), it's often the case that you need to check the documentation of that SSL library to see how these verifications can be turned on.
Bugs aside, a fair number of problems mentioned in this paper are due to the fact that some library implementations have made the assumption that it was up to their users to know what they were doing, whereas most of their users seem to have made the assumption that the library was doing the right thing by default. (In fact, even when the library is doing the right thing by default, there is certainly no shortage of programmers who just want to get rid of the error message, even if it makes their application insecure.)
I would seem fair to say that making sure the verification features are turned on would be sufficient in most cases.
As for the status of a few existing implementations:
Python: there was a change between Python 2.x and Python 3.x. The ssl module of Python 3.2 has a match_hostname method that Python 2.7 doesn't have. urllib.request.urlopen in Python 3.2 also has an option to configure CA files, which its Python 2.7 equivalent doesn't have. (This being said, if it's not set, verification won't occur. I'm not sure about the host name verification.)
Java: verification is turned on by default for both PKI and host name for HttpsUrlConnection, but not for the host name when using SSLSocket directly, unless you're using Java 7 and you've configure its SSLParameters using setEndpointIdentificationAlgorithm("HTTPS") (for example).
PHP: as far as I'm aware, fopen("https://.../") won't perform any verification at all.
Ok Advanced SSL gals and guys - I'll be adding a bounty to this after the two-day period as I think it's a complex subject that deserves a reward for anyone who thoughtfully answers.
Some of the assumptions here are simply that: assumptions, or more precisely hopeful guesses. Consider this a brain-teaser, simply saying 'This isn't possible' is missing the point.
Alternative and partial solutions are welcome, personal experience if you've done something 'similar'. I want to learn something from this even if my entire plan is flawed.
Here's the scenario:
I'm developing on an embedded Linux system and want its web server to be able to serve out-of-the-box, no-hassle SSL. Here's the design criteria I'm aiming for:
Must Haves:
I can't have the user add my homegrown CA certificate to their browser
I can't have the user add a statically generated (at mfg time) self-signed certificate to their browser
I can't have the user add a dynamically generated (at boot time) self-signed certificate to their browser.
I can't default to HTTP and have an enable/disable toggle for SSL. It must be SSL.
Both the embedded box and the web browser client may or may not have internet access so must be assumed to function correctly without internet access. The only root CAs we can rely on are the ones shipped with operating system or the browser. Lets pretend that that list is 'basically' the same across browsers and operating systems - i.e. we'll have a ~90% success rate if we rely on them.
I cannot use a fly-by-night operation i.e. 'Fast Eddie's SSL Certificate Clearing House -- with prices this low our servers MUST be hacked!'
Nice to Haves:
I don't want the user warned that the certificate's hostname doesn't match the hostname in the browser. I consider this a nice-to-have because it may be impossible.
Do not want:
I don't want to ship the same set of static keys for each box. Kind of implied by the 'can't' list, but I know the risk.
Yes Yes, I know..
I can and do provide a mechanism for the user to upload their own cert/key but I consider this 'advanced mode' and out of scope of this question. If the user is advanced enough to have their own internal CA or purchase keys then they're awesome and I love them.
Thinking Cap Time
My experience with SSL has been generating cert/keys to be signed by 'real' root, as well as stepping up my game a little bit with making my own internal CA, distributing internally 'self-signed' certs. I know you can chain certificates, but I'm not sure what the order of operations is. i.e. Does the browser 'walk up' the chain see a valid root CA and see that as a valid certificate - or do you need to have verification at every level?
I ran across the description of intermediate certificate authority which got me thinking about potential solutions. I may have gone from 'the simple solution' to 'nightmare mode', but would it be possible to:
Crazy Idea #1
Get an intermediate certificate authority cert signed by a 'real' CA. ( ICA-1 )
ROOT_CA -> ICA-1
This certificate would be used at manufacturing time to generate a unique passwordless sub-intermediate certificate authority pair per box.
ICA-1 -> ICA-2
Use ICA-2 to generate a unique server cert/key. The caveat here is, can you generate a key/pair for an IP (and not a DNS name?)? i.e. A potential use-case for this would be the user connects to the box initially via http, and then redirects the client to the SSL service using the IP in the redirect URL (so that the browser won't complain about mismatches). This could be the card that brings the house down. Since the SSL connection has to be established before any redirects can happen, I can see that also being a problem. But, if that all worked magically
Could I then use the ICA-2 to generate new cert/key pairs any time the box changes IP so that when the web server comes back up it's always got a 'valid' key chain.
ICA-2 -> SP-1
Ok, You're So Smart
Most likely, my convoluted solution won't work - but it'd be great if it did. Have you had a similar problem? What'd you do? What were the trade offs?
Basically, no, you can't do this the way you hope to.
You aren't an intermediate SSL authority, and you can't afford to become one. Even if you were, there's no way in hell you'd be allowed to distribute to consumers everything necessary to create new valid certificates for any domain, trusted by default in all browsers. If this were possible, the entire system would come tumbling down (not that it doesn't already have problems).
You can't generally get the public authorities to sign certificates issued to IP addresses, though there's nothing technically preventing it.
Keep in mind that if you're really distributing the private keys in anything but tamper-proof secured crypto modules, your devices aren't really secured by SSL. Anyone who has one of the devices can pull the private key (especially if it's passwordless) and do valid, signed, MITM attacks on all your devices. You discourage casual eavesdropping, but that's about it.
Your best option is probably to get and sign certificates for a valid internet subdomain, and then get the device to answer for that subdomain. If it's a network device in the outgoing path, you can probably do some routing magic to make it answer for the domain, similarly to how many walled-garden systems work. You could have something like "system432397652.example.com" for each system, and then generate a key for each box that corresponds to that subdomain. Have direct IP access redirect to the domain, and either have the box intercept the request, or do some DNS trickery on the internet so that the domain resolves to the correct internal IP for each client. Use a single-purpose host domain for that, don't share with your other business websites.
Paying more for certificates doesn't really make them any more or less legit. By the time a company has become a root CA, it's far from a fly-by-night operation. You should check and see if StartSSL is right for your needs, since they don't charge on a per-certificate basis.
When faced by an untrusted certificate, every single browser I know displays a blaring error like this:
Why is that?
This strongly discourages web developers to use an awesome technology like SSL out of fears that users will find the website extremely shady. Ilegitimate (ie: phishing) sites do fine on HTTP, so that can't be a concern.
Why do they make it look like such a big deal? Isn't having SSL even if untrusted better than not having it at all?
It looks like I am being misunderstood. I am taking issue with the fact that HTTP sites cannot be more secure than an HTTPS site, even if untrusted. HTTP doesn't do encryption or identification. Phishers can make their sites on HTTP and no warnings are shown. In good faith, I am at the very least encrypting traffic. How can that be a bad thing?
They do that because a SSL certificate isn't just meant to secure the communication over the wire. It is also a means to identify the source of the content that is being secured (secured content coming from a man in the middle attack via a fake cert isn't very helpful).
Unless you have a third party validate that you are who you say you are, there's no good reason to trust that your information (which is being sent over SSL) is any more secure than if you weren't using SSL in the first place.
SSL provides for secure communication between client and server by allowing mutual authentication, the use of digital signatures for integrity, and encryption for privacy.
(apache ssl docs)
Yep, I don't see anything about third party certificate authorities that all browsers should recognize as "legit." Of course, that's just the way the world is, so if you don't want people to see a scary page, you've got to get a cert signed by someone the browsers will recognize.
or
If you're just using SSL for a small group of individuals or for in-house stuff, you can have people install your root cert in their browser as a trusted cert. This would work fairly well on a lan, where a network admin could install it across the entire network.
It may sound awkward to suggest sending your cert to people to install, but if you think about it, what do you trust more: a cert that came with your browser because that authority paid their dues, or a cert sent to you personally by your server admin / account manager / inside contact?
Just for shits and giggles I thought I'd include the text displayed by the "Help me understand" link in the screenshot in the OP...
When you connect to a secure website, the server hosting that site presents your browser with something called a "certificate" to verify its identity. This certificate contains identity information, such as the address of the website, which is verified by a third party that your computer trusts. By checking that the address in the certificate matches the address of the website, it is possible to verify that you are securely communicating with the website you intended, and not a third party (such as an attacker on your network).
For a domain mismatch (for example trying to go to a subdomain on a non-wildcard cert), this paragraph follows:
In this case, the address listed in the certificate does not match the address of the website your browser tried to go to. One possible reason for this is that your communications are being intercepted by an attacker who is presenting a certificate for a different website, which would cause a mismatch. Another possible reason is that the server is set up to return the same certificate for multiple websites, including the one you are attempting to visit, even though that certificate is not valid for all of those websites. Chromium can say for sure that you reached , but cannot verify that that is the same site as foo.admin.example.com which you intended to reach. If you proceed, Chromium will not check for any further name mismatches. In general, it is best not to proceed past this point.
If the cert isn't signed by a trusted authority, these paragraphs follow instead:
In this case, the certificate has not been verified by a third party that your computer trusts. Anyone can create a certificate claiming to be whatever website they choose, which is why it must be verified by a trusted third party. Without that verification, the identity information in the certificate is meaningless. It is therefore not possible to verify that you are communicating with admin.example.com instead of an attacker who generated his own certificate claiming to be admin.example.com. You should not proceed past this point.
If, however, you work in an organization that generates its own certificates, and you are trying to connect to an internal website of that organization using such a certificate, you may be able to solve this problem securely. You can import your organization's root certificate as a "root certificate", and then certificates issued or verified by your organization will be trusted and you will not see this error next time you try to connect to an internal website. Contact your organization's help staff for assistance in adding a new root certificate to your computer.
Those last paragraphs make a pretty good answer to this question I think. ;)
The whole point of SSL is that you can verify that the site is who it says it is. If the certificate cannot be trusted, then it's highly likely that the site is not who it says it is.
An encrypted connection is really just a side-benefit in that respect (that is, you can encrypt the connection without the use of certificates).
People assume that https connections are secure, good enough for their credit card details and important passwords. A man-in-the-middle can intercept the SSL connection to your bank or paypal and provide you with their own self-signed or different certificate instead of the bank's real certificate. It's important to warn people loudly if such an attack might be taking place.
If an attacker uses a false certificate for the bank's domain, and gets it signed by some dodgy CA that does not check things properly, he may be able to intercept SSL traffic to your bank and you will be none the wiser, just a little poorer. Without the popup warning, there's no need for a dodgy CA, and internet banking and e-commerce would be totally unsafe.
Why is that?
Because most people don't read. They don't what what https means. A big error is MANDATORY to make people read it.
This strongly discourages web developers to use an awesome technology like SSL out of fears that users will find the website extremely shady.
No it doesn't. Do you have any evidence for that? That claim is ridiculous.
This strongly encourages developers and users to know whom they are dealing with.
"fears that users will find the website extremely shady"
What does this even mean? Do you mean "fears that lack of a certificate means that users will find the website extremely shady"?
That's not a "fear": that's the goal.
The goal is that "lack of a certificate means that users will find the website extremely shady" That's the purpose.
Judging from your comments, I can see that you're confused between what you think people are saying and what they are really saying.
Why do they make it look like such a big deal? Isn't having SSL even if untrusted better than not having it at all?
But why do they have to show the error? Sure, an "untrusted" cert can't be guaranteed to be more secure than no SSL, but it can't be less secure.
If you are solely interested in an encrypted connection, yes this is true. But SSL is designed for an additional goal: identification. Thus, certificates.
I am not talking about certs that don't match the domain (yes, that is pretty bad). I am talking about certs signed by authorities not in the browser's trusted CA's (eg: self-signed)
How can you trust the certificate if it is not trusted by anyone you trust?
Edit
The need to prevent man-in-the-middle attacks arises because you are trying to establish a privileged connection.
What you need to understand is that with plain HTTP, there is absolutely no promise of security, and anyone can read the contents passed over the connection. Therefore, you don't pass any sensitive information. There is no need for a warning because you are not transferring sensitive information.
When you use HTTPS, the browser assumes you will be transferring sensitive information, otherwise you would be using plain HTTP. Therefore, it makes a big fuss when it cannot verify the server's identity.
Why is that?
Because if there's a site that's pretending to be a legit site, you really want to know about it as a user!
Look, a secure connection to the attacker is no damn good at all, and every man and his dog can make a self-signed certificate. There's no inherent trust in a self-signed cert from anyone, except for the trust roots you've got installed in your browser. The default set of trust roots is picked (carefully!) by the browser maker with the aim that only CAs who only act in a way to secure trust will be trusted by the system, and this mostly works. You can add your own trust roots too, and if you're using a private CA for testing then you should.
This strongly discourages web developers to use an awesome technology like SSL out of fears that users will find the website extremely shady. Ilegitimate (ie: phishing) sites do fine on HTTP, so that can't be a concern.
What?! You can get a legit certificate for very little. You can set up your own trust root for free (plus some work). Anyone developing and moaning about this issue is just being lazy and/or over-cheap and I've no sympathy for such attitudes.
Ideally a browser would look for information that you want kept secure (such as things that look like credit card numbers) and throw that sort of warning up if there was an attempt to send that data over an insecure or improperly-secured channel. Alas, it's hard to know from just inspection whether data is private or not; just as there's no such thing as an EVIL bit, there's also no PRIVATE bit. (Maybe a pervasive metadata system could do it… Yeah, right. Forget it.) So they just do the best they can and flag up situations where it is extremely likely that there's a problem.
Why do they make it look like such a big deal? Isn't having SSL even if untrusted better than not having it at all?
What threat model are you dealing with?
Browser makers have focused on the case where anyone can synthesize an SSL certificate (because that's indeed the case) and DNS hacks are all too common; what the combination of these means is that you can't know that the IP address you've got for a host name corresponds to the legitimate owner of that domain, and anyone can claim to own that domain. Ah, but you instead trust a CA to at least check that they're issuing the certificate to the right person and that in turn is enough (plus a few other things) to make it possible to work out whether you're talking to the legitimate owner of the domain; it provides a basis for all the rest of the trust involved in a secure conversation. Hopefully the bank will have used other unblockable communications (e.g., a letter sent by post) to tell people to check that the identity of the site is right (EV certs help a little here) but that's still a bit of a band-aid given how unsuspicious some users are.
The problems with this come from CAs who don't apply proper checks (frankly, they ought to be kicked off the gravy train for failing their duty) and users who'll tell anyone anything. You can't stop them from deliberately posting their own CC# on a public message board run by some shady characters from Smolensk[1], no matter how stupid an idea that is…
[1] Not that there's anything wrong with that city. The point would be the same if you substituted with Tallahassee, Ballarat, Lagos, Chonqing, Bogota, Salerno, Durban, Mumbai, … There are scum all over.
Question for all the SSL experts out there:
We have an embedded device with a little web server on it, and we can install our own SSL self-signed certificates on it. The client is written in .NET (but that doesn't matter so much).
How can I authenticate the device in .NET? Is it enough to compare the fingerprint of the certificate against a known entry in the database?
My understanding is that the fingerprint is a hash of the whole certificate, including the public key. A device faking to be my device could of course send the same public certificate, but it couldn't know the private key, right?
Or do I have to build up my own chain of trust, create my own CA root certificate, sign the web server certificate and install that on the client?
What you propose is in principle ok. It is for example used during key signing parties. Here the participants usually just exchange their name and fingerprints of their public keys and make sure that the person at the party really is who he/she claims. Just verifying fingerprints is much easier than to verify a long public key.
Another example is the so called self certifying file system. Here again only hashes of public keys get exchanged over a secure channel. (I.e. these hashes are embedded in URLs.) In this scheme the public keys don't have to be sent securely. The receiver only has to check that the hash of the public keys matche the hashes embedded in the URLs. Of course the receiver also has to make sure that these URLs come from a trusted source.
This scheme and what you propose are simpler than using a CA. But there is a disadvantage. You have to make sure that your database with hashes is authentic. If your database is large then this will likeley be difficult. If you use CAs then you only have to ensure that the root keys are authentic. This usually simplifies the key management significantly and is of course one reason, why CA based schemes are more popular than e.g. the self certifying file system mentioned above.
In the same way you wouldn't and shouldn't consider two objects to be equal just because their hash codes matched, you shouldn't consider a certificate to be authentic just because its fingerprint appears in a list of "known certificate fingerprints".
Collisions are a fact of life with hash algorithms, even good ones, and you should guard against the possibility that a motivated attacker could craft a rogue certificate with a matching fingerprint hash. The only way to guard against that is to check the validity of the certificate itself, i.e. check the chain of trust as you're implying in your last statement.
Short:
Well in theory you then do exactly what a Certificate Authority does for you. So it should be fine.
Longer:
When a Certificate Authority signs your public-key/certificate/certificate request it doesn't sign the whole certificate data. But just the calculated hash value of the whole certificate data.
This keeps the signature small.
When you don't want to establish your own CA or use a commercial/free one -
by comparing the fingerprint with the one you trust you'll gain the second most trustworthy configuration. The most trustworthy solution would be by comparing the whole certificate, because also protects you from hash collision attacks.
As the other guys here stated you should make sure to use a secure/safe hashing algorithm. SHA-1 is no longer secure.
more detailed informations to this topic:
https://security.stackexchange.com/questions/6737
https://security.stackexchange.com/questions/14330