I've done a little googling but have been a bit overwhelmed by the amount of information. Until now, I've been considering asking for a valid md5 hash for every API call but I realized that it wouldn't be a difficult task to hijack such a system. Would you guys be kind enough to provide me with a few links that might help me in my search? Thanks.
First, consider OAuth. It's somewhat of a standard for web-based APIs nowadays.
Second, some other potential resources -
A couple of decent blog entries:
http://blog.sonoasystems.com/detail/dont_roll_your_own_api_security_recommendations1/
http://blog.sonoasystems.com/detail/more_api_security_choices_oauth_ssl_saml_and_rolling_your_own/
A previous question:
Good approach for a web API token scheme?
I'd like to add some clarifying information to this question. The "use OAuth" answer is correct, but also loaded (given the spec is quite long and people who aren't familiar with it typically want to kill themselves after seeing it).
I wrote up a story-style tutorial on how to go from no security to HMAC-based security when designing a secure REST API here:
http://www.thebuzzmedia.com/designing-a-secure-rest-api-without-oauth-authentication/
This ends up being basically what is known as "2-legged OAuth"; because OAuth was originally intended to verifying client applications, the flow is 3-parts involving the authenticating service, the user staring at the screen and the service that wants to use the client's credentials.
2-legged OAuth (and what I outline in depth in that article) is intended for service APIs to authenticate between each other. For example, this is the approach Amazon Web Services uses for all their API calls.
The gist is that with any request over HTTP you have to consider the attack vector where some malicious man-in-the-middle is recording and replaying or changing your requests.
For example, you issue a POST to /user/create with name 'bob', well the man-in-the-middle can issue a POST to /user/delete with name 'bob' just to be nasty.
The client and server need some way to trust each other and the only way that can happen is via public/private keys.
You can't just pass the public/private keys back and forth NOR can you simply provide a unique token signed with the private key (which is typically what most people do and think that makes them safe), while that will identify the original request coming from the real client, it still leaves the arguments to the comment open to change.
For example, if I send:
/chargeCC?user=bob&amt=100.00&key=kjDSLKjdasdmiUDSkjh
where the key is my public key signed by my private key only a man-in-the-middle can intercept this call, and re-submit it to the server with an "amt" value of "10000.00" instead.
The key is that you have to include ALL the parameters you send in the hash calculation, so when the server gets it, it re-vets all the values by recalculating the same hash on its side.
REMINDER: Only the client and server know the private key.
This style of verification is called an "HMAC"; it is a checksum verifying the contents of the request.
Because hash generation is SO touchy and must be done EXACTLY the same on both the client and server in order to get the same hash, there are super-strict rules on exactly how all the values should be combined.
For example, these two lines provides VERY different hashes when you try and sign them with SHA-1:
/chargeCC&user=bob&amt=100
/chargeCC&amt=100&user=bob
A lot of the OAuth spec is spent describing that exact method of combination in excruciating detail, using terminology like "natural byte ordering" and other non-human-readable garbage.
It is important though, because if you get that combination of values wrong, the client and server cannot correctly vet each other's requests.
You also can't take shortcuts and just concatonate everything into a huge String, Amazon tried this with AWS Signature Version 1 and it turned out wrong.
I hope all of that helps, feel free to ask questions if you are stuck.
Related
A little background: I am going to be constructing a webserver, likely the most up to date version of apache when I get around to it. It is going to be updated with sensory information from a makeshift security system I have.
As a counterpart, I am designing an app to go along with it, that will automatically contact the webserver and pull the sensory information about once every 1.5 minutes.
I want to have an authentication method so that the average Bob can't see this information, mostly due to the fact that there will be some command and control as part of the server as well.
The question: I feel like a simple username and password is the wrong way to go about this since it isn't dynamic and theoretically seeing the same credentials sent that frequent could be dangerous, so is there any other authentication method that could mitigate this?
The question pt. 2: Obviously I want an encrypted channel, will https stumble over itself if it tries to renegotiate every minute and a half?
I haven't begun this project yet much less chosen any language to write it in, meaning I am super open minded to suggestions, any help is greatly appreciated.
The question: I feel like a simple username and password is the wrong
way to go about this since it isn't dynamic and theoretically seeing
the same credentials sent that frequent could be dangerous, so is
there any other authentication method that could mitigate this?
You could use Google Sign-In to allow log on via a Google account.
Or you could implement two factor authentication with say Google Authenticator or via SMS to prove that the user logging in has more than one factor of authentication. These factors could be:
Something you know (e.g. password)
Something you have (e.g. phone that provides a One Time Password)
Edit: Having re-read your question - yes you are fine to authenticate with username and password (over HTTPS), however you should then store a session identifier client-side and simply send this in future rather than the username/password each time. This is more secure as it can be stored safely client-side, and if exposed the identifier can be easily revoked.
The question pt. 2: Obviously I want an encrypted channel, will https
stumble over itself if it tries to renegotiate every minute and a half?
Nope, this is what it is designed for. Browsers will keep open an HTTPS connection for a length of time. Additionally, they will use session resumption rather than executing a full HTTPS handshake in the case that a new connection needs to be established. Session resumption is much quicker than establishing a completely new session. See this article on the CloudFlare blog for more info.
We are planning to go for a security testing certificate. For that reason we are using Paros tool to test our system.
The system is written in GWT on front end and database connectivity is happening through Hibernate.
When we use this tool to test our application following behaviour is happening which needs to be restricted.
The tool is able to see the data which is passed to server. This is fine but when we make any changes in the data through tool it gets updated in the system on database end. This is a big security issue.
Can someone guide me in this?
If you're still looking for a solution to this problem, you could use request signing. The reason I didn't mention it earlier was because the only time I had seen request signing, there were certificates involved, and it was mostly using the Web Services Security Standard. The other time I recommended implementation of request signing was for a mobile application - its relatively easier to do there also, since you can use certificates that are on the device to perform the signing, and the server can verify this signature (essentially, a public key encryption mechanism).
As you mention in the comments, there are multiple aspects to it - one is to prevent XSRF, which is essentially including a nonce to ensure that an attacker cannot replay requests, or craft requests that might harm an authenticated user. This nonce will have to come from the server, since anything that you create using Javascript, the attacker can create also. This nonce will make sure that your request is time specific, and that it cannot be replayed at a later point of time.
However, a nonce isn't going to stop attacks where a user is in a hostile network, and an attacker is performing a MitM attack on all traffic. The attacker can still modify a request, and since the server has never seen that nonce before, it will accept the request as valid. To prevent this, you need to countermeasures in place - one, all traffic should go via SSL, and two, all requests must be signed so as to prevent tampering. The signature part is particularly hard, especially if you have to ensure that an attacker cannot perform the same signing. The examples I have seen of it involve certificate level authentication for the webapp, and using these certificates to then perform the signing - which might be too stringent a requirement for the application that you seem to be developing. Other methodologies involve using something that the user has/knows - maybe a token, password, secret answer, etc. - that cannot be replicated by an attacker, and using that information to sign requests.
Here's an example on how you can do this via PHP. I don't know if this mechanism can be adapted to do it for your purposes, though. OAuth might be another possible method, but since I've never seen an application do it that way, I am not very sure.
Sorry I don't have a specific methodology or examples of code for you to look at, but most implementations I've seen are only from a design standpoint, versus an actual code standpoint.
Technically two questions - but they are so heavily related I didn't want to split them up; but if the community feels I should, I will.
Following a recent question I am implementing SCRAM for a website login and web service API. Client environments will be .Net and Javascript (with Java likely in the future).
My first issue is basic: The protocol utilises a client and server key as key steps in the authentication process; and yet in order to be validated, both need to be known by both parties in advance since the protocol doesn't allow for exchange of these (to do so would result in a bit of a chicken and egg scenario). If you consider a Javascript client, for example, this means both keys are likely to be constants defined in the source - thus making them easy to fetch. So: why bother? Is it just to mitigate against 'Eve' where that 'Eve', for some reason, hasn't bothered to get the JS or client source code, which will necessarily be public!?
Secondly, like practically any other authentication mechanism it requires a client + server nonce.
Given that the authentication nonce, by definition, should never be used more than once (at least by the same user), this presumably means that a server must maintain a record of all nonce values used by all users forever. Unlike other data that we regularly archive off, such a table is only ever going to get bigger and queries against it likely to get slower and slower!
If that's correct, then it's technically unfeasible to implement this or almost any other authentication mechanism! Since I know that's plainly ridiculous; it must be common to define some additional scope that factors in a reasonable timescale as well.
As always with authentication and encryption; despite being a very experienced software developer I feel like I'm going back to school! What am I missing!?
both need to be known by both parties
in advance since the protocol doesn't
allow for exchange of these (to do so
would result in a bit of a chicken and
egg scenario).
Yes that's correct. Challenge response isn't a key-exchange protocol. It only norms, once client and server share a key, how to compute the same value from that key without transmitting in clear the key via network.
If you consider a Javascript client,
for example, this means both keys are
likely to be constants defined in the
source - thus making them easy to
fetch.
That's not a good idea. Alternatively client and server can agree on a key during a preliminary registration process.
Given that the authentication nonce,
by definition, should never be used
more than once (at least by the same
user), this presumably means that a
server must maintain a record of all
nonce values used by all users
forever.
NO. A new nonce should be generated for each new session using pseudo-random number generation. It's very improbable that you will get the same nonce twice, anyway It doesn't matter if a nonce it has already been used if the attacker don't know that .
Many users – myself included – would like the security of having everything they do on a web service encrypted. That is, they don't won't any one at the web service to be able to look at their: posts, info, tasks, etc...
This is also major complaint in this discussion of an otherwise cool service: http://news.ycombinator.com/item?id=1549115
Since this data needs to be recoverable, some sort of two-way encryption is required. But unless you're prompting the user for the encryption key on every request, this key will need to be stored on the server, and the point of encrypting the data is basically lost.
What is a way to securely encrypt user data without degrading the user experience (asking for some key on every request)?
-- UPDATE --
From #Borealid's answer, I've focused on two possibilities: challenge-response protocols, where no data (password included) is sent in the "clear", and non-challenge-response protocols, where data (password included) is sent in the "clear" (although over HTTPS).
Challenge-response protocols (specifically SRP: http://srp.stanford.edu/)
It seems that its implementation would need to rely on either a fully AJAX site or using web storage. This is so the browser can persist the challenge-response data during encryption and also the encryption key between different "pages". (I'm assuming after authentication is completed I would send them back the encrypted encryption key, which they would decrypt client-side to obtain the real encryption key.)
The problem is that I'm either:
fully AJAX, which I don't like because I love urls and don't won't a user to live exclusively on a single url, or
I have to store data encryption keys in web storage, which based on http://dev.w3.org/html5/webstorage/ will persist even after the browser is closed and could be a security vulnerability
In addition, as SRP takes more than one request ( http://srp.stanford.edu/design.html ), there needs to be some persistence on the server-side. This is just another difficulty.
Traditionally
If I'm ok transmitting passwords and data in the clear (although over HTTPS), then the client-side issues above are not present.
On registration, I'll generate a random unique encryption key for the user, and encrypt it using their password and a random salt.
In the database, I'll store the user's password hash and salt (through bcrypt), encrypted encryption key, encryption key salt, and encryption iv.
After an authentication, I'll also need to use their password to decrypt the encryption key so that they may view and enter new data. I store this encryption key only temporarily and delete it when they explicitly "log out".
The problems with this approach is that (like #Borealid points out) evil sysadmins can still look at your data when you are logged in.
I'm also not sure how to store the encryption keys when users are logged in. If they are in the same data store, a stolen database would reveal all data of those who were logged in at the time of theft.
Is there a better in-memory data store for storing these encryption keys (and challenge data during an SRP authentication)? Is this something Redis would be good for?
If the data need to be recoverable in the event of user error, you can't use something like a cookie (which could get deleted). And as you point out, server-side keys don't actually secure the user against malicious sysadmins; they only help with things like databases stolen offline.
However, if you're running a normal web service, you've already gotten pretty lucky - the user, in order to be unique and non-ephemeral, must be logged in. This means they go through some authentication step which proves their identity. In order to prove their identity, most web sites use a passed credential (a password).
So long as you don't use a challenge-response authentication protocol, which most web sites don't, you can use an encryption key derived from a combination of a server-side secret and the user's password. Store the encryption key only while the user is authenticated.
If you do this, the users are still vulnerable to sysadmins peeking while they're using the service (or stealing their passwords). You might want to go a step further. To go one up, don't send the password to the server at all. Instead, use a challenge-response protocol for authentication to your website, and encrypt the data with a derivative of the user's password via JavaScript before uploading anything.
This is foolproof security: if you try to steal the user's password, the user can see what you're doing because the code for the theft is right there in the page you sent them. Your web service never touches their data unencrypted. This is also no hindrance to the normal user experience. The user just enters their password to log in, as per normal.
This method is what is used by Lacie's storage cloud service. It's very well done.
Note: when I say "use foo to encrypt", I really mean "use foo to encrypt a secure symmetric key which is then used with a random salt to encrypt". Know your cryptography. I'm only talking about the secret, not the methodology.
None of those other solutions are going to maintain the feature set requested -- which specifically wants to preserve the user experience. If you look at the site referenced in the link, they email you a nightly past journal entry. You're not going to get that with JavaScript trickery per above because you don't have the browser to depend on. So basically this is all leading you down a path to a degraded user experience.
What you would want, or more precisely the best solution you're going to find in this space, is not so much what wuala does per above, but rather something like hush.com. The handling of user data needs to be done on the client side at all times -- this is generally accomplished via full client-side Java (like the Facebook photo uploader, etc), but HTML/JavaScript might get you there these days. JavaScript encryption is pretty poor, so you may be better off ignoring it.
OK, so now you've got client-side Java running a Journal entry encryption service. The next feature was to email past journal entries to users every night. Well, you're not going to get that in an unencrypted email obviously. This is where you're going to need to change the user experience one way or the other. The simplest solution is not to email the entry and instead to provide for instance a journal entry browser in the Java app that reminds them of some old entry once they get to the website based on a link in the daily email. A much more complex solution would be to use JavaScript encryption to decrypt the entry as an attachment inline in the email. This isn't rocket science but there is a fairly huge amount of trickery involved. This is the general path used by several web email encryption services such as IronPort. You can get a demo email by going to http://www.ironport.com/securedemo/.
As much as I'd love to see a properly encrypted version of all this, my final comment would be that journal entries are not state secrets. Given a solid privacy policy and good site security semantics, I'm sure 99% of your users will feel just fine about things. Doing all this right with true security will take an enormous amount of effort per above and at least some design/UE changes.
You should look into the MIT project CryptDB which supports querying an encrypted database using a subset of SQL. (see the forbes article, mefi thread, or Homomorphic encryption on wikipedia)
There is the Tahoe-LAFS project for cloud storage too, which conceivably could be leveraged into a fully anonymous social networking application, one day in the distant future.
If you want to perform computations on a server without even the server being able to see the data, you may be interested in knowing about fully homomorphic encryption. A fully homomorphic encryption scheme lets you perform arbitrary computations on encrypted data, even if you can't decrypt it. However, this is still a topic of research.
For now, I guess your best bet would be to encrypt all posts and assign meaningless (e.g. sequential) IDs to each one. For a more in-depth discussion of how to encrypt server-side data with today's technology, look up.
I want to create a portal website for log-in, news and user management. And another web site for a web app that the portal redirects to after login.
One of my goals is to be able to host the portal and web-app on different servers. The portal would transmit the user's id to the web-app, once the user had successfully logged in and been redirected to the web app. But I don't want people to be able to just bypass the login, or access other users accounts, by transmitting user ids straight to the web app.
My first thought is to transmit the user id encrypted as a post variable or query string value. Using some kind of public/private key scenario, and adding a DateTime stamp to key to make it vary everytime.
But I haven't done this kind of thing before, so I'm wondering if there aren't better ways to do this.
(I could potentially communicate via database, by having the portal store the user id with a key in a database and passing that key to the web app which uses it to get the user id from that database. But that seems crazy.)
Can anyone give a way to do this or advice? Or is this a bad idea all-together?
Thanks for your time.
Basically, you are asking for a single-sign-on solution. What you describe sounds a lot like SAML, although SAML is a bit more advanced ;-)
It depends on how secure you want this entire thing to be. Generating an encrypted token with embedded timestamp still leaves you open to spoofing - if somebody steals the token (i.e. through a network sniffing) he will be able to submit his own request with the stolen token. Depending on the time to live you will give your token this time can be limited, but a determined hacker will be able to do this. Besides you cannot make time to live to small - you will be rejecting valid requests.
Another approach is to generate "use once" tokens. This is 'bullet proof' in terms of spoofing, but it requires coordination among all the servers within the server farm servicing your app, so that if one of them processed the token the other ones would reject it.
To make it really secure for the failover scenarios, etc. it would require some additional steps, so it all boils down to how secure you need it to be and how much you want to invest in building it up
I suggest looking at SAML
PGP would work but it might get slow on a high-traffic site
One thing I've done in the past is used a shared secret method. Some token that only myself and the other website operator knows concatenated to something identifying the user (like their user name), then hash that with a checksum algorithm such as SHA256 (you can use MD5 or SHA1 which usually are more available but they are much easier to break)
The other end should do the same thing as above. Take the passed identifying information and checksum it. Compare that to the passed checksum, if they match the login is valid.
For added security you could also concat the date or some other rotating key. Helps to run SSL on both sides as well.
In general, the answer resides somewhere in SHA256 / MD5 / SHA1 plus shared secret based on human actually has to think. If there is money somewhere, we may assume there are no limits to what some persons will do - I ran with [ a person ] in High School for a few months to observe what those ilks will do in practice. After a few months, I learned not to be running with those kind. Tediously avoiding work, suddenly at 4 AM on Saturday Morning the level of effort and analytical functioning could only be described as "Expertise" ( note capitalization ) There has to be a solution else sites like Google and this one would not stand the chance of a dandelion in lightning bolt.
There is a study in the mathematical works of cryptography whereby an institution ( with reputable goals ) can issue information - digital cash - that can exist on the open wire but does not reveal any information. Who would break them? My experience with [ person ]
shows that it is a study in socialization, depends on who you want to run with. What's the defense against sniffers if the code is already available more easily just using a browser?
<form type="hidden" value="myreallysecretid">
vis a vis
<form type="hidden" value="weoi938389wiwdfu0789we394">
So which one is valuable against attack? Neither, if someone wants to snag some Snake Oil from you, maybe you get the 2:59 am phone call that begins: "I'm an investor, we sunk thousands into your website. I just got a call from our security pro ....." all you can do to prepare for that moment is use established, known tools like SHA - of which the 256 variety is the acknowledged "next thing" - and have trace controls such that the security pro can put in on insurance and bonding.
Let alone trying to find one who knows how those tools work, their first line of defense is not talking to you ... then they have their own literature - they will want you to use their tools.
Then you don't get to code anything.