REST Web Service authentication token implementation - security

I'm implementing a REST web service using C# which will be hosted on Azure as a cloud service. Since it is a REST service, it is stateless and therefore no cookies or session states.
The web service can only be accessed over HTTPS (Certificate provided by StartSSL.com).
Upon a user successfully logging into the service they will get a security token. This token will provide authentication in future communications.
The token will contain a timestamp, userid and ip address of the client.
All communication will only happen over HTTPS so I'm not concerned about the token being intercepted and used in replay attacks; the token will have an expiry anyway.
Since this is a public facing service I am however concerned that someone could register with the service, login and then modifying the token that they receive to access the accounts of other users.
I'm wondering how best to secure the content of the token and also verify that it hasn't been tampered with.
I plan on doing the following to secure the token:
The client successfully logs into the service and the service does:
Generate a random value and hash it with SHA256 1000 times.
Generate a one-time session key from private key + hashed random value.
Hash the session key with SHA256 1000 times and then use it to encrypt the token
Use private key to sign the encrypted token using RSA.
Sends the encrypted token + the signature + the hashed random value to the client in an unencrypted JSON package.
When the client calls a service it sends the encrypted token and signature in an unencrypted JSON package to the service. The service will
Recreate the session key from the private key + the hashed random value
Use the private key to verify the signature
Use the hashed session key to decrypt the token
Check that the token hasn't expired
Continue with the requested operation...
I don't really know anything about encryption so I have some questions:
Is this sufficient or is it overkill?
I read that to detect tampering I should include an HMAC with the token. Since I am signing with the private key, do I still need an HMAC?
Should I be using Rijndael instead of RSA?
If Rijndael is preferred, is the generated IV required for decrypted? i.e. can i throw it away or do I need to send it will the encrypted token? e.g. Encrypted Token + HMAC + IV + hashed random value.
Since all communication happens over HTTPS the unencrypted JSON package isn't really unencrypted until it reaches the client.
Also I may want to re-implement the service in PHP later so this all needs to be doable in PHP as well.
Thanks for your help

You are really over-thinking the token. Truthfully, the best token security relies on randomness, or more accurately unpredictability. The best tokens are completely random. You are right that a concern is that a user will modify his/her token and use it to access the accounts of others. This is a common attack known as "session stealing." This attack is nearly impossible when the tokens are randomly generated and expired on the server side. Using the user's information such as IP and/or a time stamp is bad practice because it improves predictability. I did an attack in college that successfully guessed active tokens that were based on server time stamps in microseconds. The author of the application thought microseconds would change fast enough that they'd be unpredictable, but that was not the case.
You should be aware that when users are behind proxy servers, the proxy will sometimes view their SSL requests in plain text (for security reasons, many proxies will perform deep packet inspection). For this reason it is good that you expire the sessions. If you didn't your users would be vulnerable to an attack such as this, and also possible XSS and CSRF.
RSA or Rijndael should be plenty sufficient, provided a reasonable key length. Also, you should use an HMAC with the token to prevent tampering, even if you're signing it. In theory it would be redundant, since you're signing with a private key. However, HMAC is very well tested, and your implementation of the signing mechanism could be flawed. For that reason it is better to use HMAC. You'd be surprised how many "roll your own" security implementations have flaws that lead them to compromise.
You sound pretty savvy on security. Keep up the good work! We need more security conscious devs in this world.
EDIT:
It is considered safe to include timestamps/user IDs in the token as long as they are encrypted with a strong symmetric secret key (like AES, Blowfish, etc) that only the server has and as long as the token includes a tamper-proof hash with it such as HMAC, which is encrypted with the secret key along with the user ID/timestamp. The hash guarantees integrity, and the encryption guarantees confidentiality.
If you don't include the HMAC (or other hash) in the encryption, then it is possible for users to tamper with the encrypted token and have it decrypt to something valid. I did an attack on a server in which the User ID and time stamp were encrypted and used as a token without a hash. By changing one random character in the string, I was able to change my user ID from something like 58762 to 58531. While I couldn't pick the "new" user ID, I was able to access someone else's account (this was in academia, as part of a course).
An alternative to this is to use a completely random token value, and map it on the server side to the stored User ID/time stamp (which stays on the server side and is thus outside of the clients control). This takes a little more memory and processing power, but is more secure. This is a decision you'll have to make on a case by case basis.
As for reusing/deriving keys from the IV and other keys, this is usually ok, provided that the keys are only valid for a short period of time. Mathematically it is unlikely someone can break them. It is possible however. If you want to go the paranoid route (which I usually do), generate all new keys randomly.

Related

Sessions vs. Token based authentication

I want to know which is more safe to implement for authentication and why?
Session based authentication OR Token based authentication?
I know sessions can be used for other things as well, but right now I am only interested about authentication.
Is it true that nothing is stored on server side if using tokens (not even in memory)? If yes, then how it identifies against expired tokens as that had also been signed using the same secret?
The question on Information Security linked in the comment above has a lot of relevant information. That being said, a few additional concerns raised in this question should be addressed:
Safety
Knowing nothing about the server implementation, both methods can be as secure. Session-based authentication mostly relies on the guessability of the session identifier (which, as described in the Information Security answer, it in itself a very simple token). If the session identifier is a monotonously incrementing numeric id, then it is not very secure, OTOH it could be an opaque cryptographically strong unique ID with a huge keyspace, making it very safe. You are probably going to use the session implementation offered by your server framework of choice, so you need to check that. After that, using session authentication, your server implementation needs to verify that the server stored session contains the relevant authorization (i.e. user account data, role, etc) - as a lot of server session frameworks will be default auto-generate empty sessions as needed, the fact that a session exists must not be relied upon as proof enough for a valid authentication and authorization.
For example, PHP's internal session ID generation uses a completely random 288 bits number (default setting) so it is considered safe, OTOH - by default it generates sessions automatically so the previous comment must be adhered to (or to disable automatic session creation and make sure the server only creates session as needed).
Also, if the session ID is passed using an HTTP URL query string, as was the default in days of old, then the session can be stolen quite easily and that makes the whole process insecure.
Token safety is mostly based around secure token generation: if the server generates tokens in a secure manner - i.e. non-guessable and verifiable - as demonstrated in the Information Security answer. A naive implementation (that I saw one time) might be to MD5 hash a known token, such as a user name, and that makes it very unsafe, even when salted. When using cryptographic tokens, safety is closely related to the strength of encryption which is determined by the algorithm used, length of the key, and - most importantly - how well the server key is secured: if the server key is hard-coded into the server implementation and that code is then open-sourced...
Storage
Whether the server needs to store anything generally depends on the implementation of the tokens.
A lot of implementations use the concept of an "API key" as "token authentication" and so often tokens are just some cryptographically secure ID to a database that records which "API keys" have been generated. This requires storage but has the advantage of a simpler implementation and - more importantly - the ability to revoke tokens.
The other approach is to have the token carry its own authenticity - this allows the server to essentially offload the storage of tokens to the client and use the client as the database - very much like how HTTP Cookies allow servers to offload some storage requirements to the client (often used for client's specific settings, like whether the user wants a light interface or a dark interface).
The two patterns used for this are demonstrated well in the Information Security answer: signing and encrypting.
Signing: The token is some simple encoding (like JSON or CSV) of the authenticator credentials (such as the username) and possibly the token's expiry time (if you want to have tokens expire - generally a good idea, if you can't revoke tokens), and then the server signs the generated text using a server secret and adds that to the token. When the client submits the token, the server can extract the clear text from the token, re-sign it and compare the new signature to the signature part in the submitted token - if they are identical, then the token is valid. After validation, you probably want to check the validated expiry date against the current time. The main disadvantage here is that care should be taken that the clear-text authentication details are strongly insufficient for an attacker to re-authenticate - otherwise, it harms the safety requirement. i.e. don't send the password as part of the token, or any other internal detail.
Encrypting: the token is generated by again encoding all the relevant authentication details and then encrypting the clear-text with a server secret and only submitting the encrypted result. If the encryption scheme can be trusted, the authentication details can include internal data - but care should be taken as having a large crypt-text available offline to an attacker is a larger attack surface than a small signature and weak encryption algorithms will be less resilient in this usage than in just signing.
In both methods, safety is closely related to the strength of the encryption/signing algorithm - a weak algorithm will allow an attacker to reverse engineer the server secret and generate new valid tokens without authentication.
A personal note
In my opinion, cryptographic token-based authentication tends to be less safe than session-based, as it relies on the (often single) developer doing everything right from design to implementation to deployment, while session-based authentication can leverage existing implementations to do most of the heavy lifting, where it is very easy to find high-quality, secure and massively used and tested session storage implementations. I would need a very compelling reason as to why session storage is unwanted before recommending using cryptographic tokens.
Always remember the no.1 rule of cryptographic security: never design your own single-use cryptographic measures.

If you can decode JWT, how are they secure?

If I get a JWT and I can decode the payload, how is that secure? Couldn't I just grab the token out of the header, decode and change the user information in the payload, and send it back with the same correct encoded secret?
I know they must be secure, but I just would really like to understand the technologies. What am I missing?
JWTs can be either signed, encrypted or both. If a token is signed, but not encrypted, everyone can read its contents, but when you don't know the private key, you can't change it. Otherwise, the receiver will notice that the signature won't match anymore.
Answer to your comment: I'm not sure if I understand your comment the right way. Just to be sure: do you know and understand digital signatures? I'll just briefly explain one variant (HMAC, which is symmetrical, but there are many others).
Let's assume Alice wants to send a JWT to Bob. They both know some shared secret. Mallory doesn't know that secret, but wants to interfere and change the JWT. To prevent that, Alice calculates Hash(payload + secret) and appends this as signature.
When receiving the message, Bob can also calculate Hash(payload + secret) to check whether the signature matches.
If however, Mallory changes something in the content, she isn't able to calculate the matching signature (which would be Hash(newContent + secret)). She doesn't know the secret and has no way of finding it out.
This means if she changes something, the signature won't match anymore, and Bob will simply not accept the JWT anymore.
Let's suppose, I send another person the message {"id":1} and sign it with Hash(content + secret). (+ is just concatenation here). I use the SHA256 Hash function, and the signature I get is: 330e7b0775561c6e95797d4dd306a150046e239986f0a1373230fda0235bda8c. Now it's your turn: play the role of Mallory and try to sign the message {"id":2}. You can't because you don't know which secret I used. If I suppose that the recipient knows the secret, he CAN calculate the signature of any message and check if it's correct.
You can go to jwt.io, paste your token and read the contents. This is jarring for a lot of people initially.
The short answer is that JWT doesn't concern itself with encryption. It cares about validation. That is to say, it can always get the answer for "Have the contents of this token been manipulated"? This means user manipulation of the JWT token is futile because the server will know and disregard the token. The server adds a signature based on the payload when issuing a token to the client. Later on it verifies the payload and matching signature.
The logical question is what is the motivation for not concerning itself with encrypted contents?
The simplest reason is because it assumes this is a solved problem for the most part. If dealing with a client like the web browser for example, you can store the JWT tokens in a cookie that is secure (is not transmitted via HTTP, only via HTTPS) and httpOnly (can't be read by Javascript) and talks to the server over an encrypted channel (HTTPS). Once you know you have a secure channel between the server and client you can securely exchange JWT or whatever else you want.
This keeps thing simple. A simple implementation makes adoption easier but it also lets each layer do what it does best (let HTTPS handle encryption).
JWT isn't meant to store sensitive data. Once the server receives the JWT token and validates it, it is free to lookup the user ID in its own database for additional information for that user (like permissions, postal address, etc). This keeps JWT small in size and avoids inadvertent information leakage because everyone knows not to keep sensitive data in JWT.
It's not too different from how cookies themselves work. Cookies often contain unencrypted payloads. If you are using HTTPS then everything is good. If you aren't then it's advisable to encrypt sensitive cookies themselves. Not doing so will mean that a man-in-the-middle attack is possible--a proxy server or ISP reads the cookies and then replays them later on pretending to be you. For similar reasons, JWT should always be exchanged over a secure layer like HTTPS.
Let's discuss from the very beginning:
JWT is a very modern, simple and secure approach which extends for Json Web Tokens. Json Web Tokens are a stateless solution for authentication. So there is no need to store any session state on the server, which of course is perfect for restful APIs.
Restful APIs should always be stateless, and the most widely used alternative to authentication with JWTs is to just store the user's log-in state on the server using sessions. But then of course does not follow the principle that says that restful APIs should be stateless and that's why solutions like JWT became popular and effective.
So now let's know how authentication actually works with Json Web Tokens. Assuming we already have a registered user in our database. So the user's client starts by making a post request with the username and the password, the application then checks if the user exists and if the password is correct, then the application will generate a unique Json Web Token for only that user.
The token is created using a secret string that is stored on a server. Next, the server then sends that JWT back to the client which will store it either in a cookie or in local storage.
Just like this, the user is authenticated and basically logged into our application without leaving any state on the server.
So the server does in fact not know which user is actually logged in, but of course, the user knows that he's logged in because he has a valid Json Web Token which is a bit like a passport to access protected parts of the application.
So again, just to make sure you got the idea. A user is logged in as soon as he gets back his unique valid Json Web Token which is not saved anywhere on the server. And so this process is therefore completely stateless.
Then, each time a user wants to access a protected route like his user profile data, for example. He sends his Json Web Token along with a request, so it's a bit like showing his passport to get access to that route.
Once the request hits the server, our app will then verify if the Json Web Token is actually valid and if the user is really who he says he is, well then the requested data will be sent to the client and if not, then there will be an error telling the user that he's not allowed to access that resource.
All this communication must happen over https, so secure encrypted Http in order to prevent that anyone can get access to passwords or Json Web Tokens. Only then we have a really secure system.
So a Json Web Token looks like left part of this screenshot which was taken from the JWT debugger at jwt.io. So essentially, it's an encoding string made up of three parts. The header, the payload and the signature Now the header is just some metadata about the token itself and the payload is the data that we can encode into the token, any data really that we want. So the more data we want to encode here the bigger the JWT. Anyway, these two parts are just plain text that will get encoded, but not encrypted.
So anyone will be able to decode them and to read them, we cannot store any sensitive data in here. But that's not a problem at all because in the third part, so in the signature, is where things really get interesting. The signature is created using the header, the payload, and the secret that is saved on the server.
And this whole process is then called signing the Json Web Token. The signing algorithm takes the header, the payload, and the secret to create a unique signature. So only this data plus the secret can create this signature, all right?
Then together with the header and the payload, these signature forms the JWT,
which then gets sent to the client.
Once the server receives a JWT to grant access to a protected route, it needs to verify it in order to determine if the user really is who he claims to be. In other words, it will verify if no one changed the header and the payload data of the token. So again, this verification step will check if no third party actually altered either the header or the payload of the Json Web Token.
So, how does this verification actually work? Well, it is actually quite straightforward. Once the JWT is received, the verification will take its header and payload, and together with the secret that is still saved on the server, basically create a test signature.
But the original signature that was generated when the JWT was first created is still in the token, right? And that's the key to this verification. Because now all we have to do is to compare the test signature with the original signature.
And if the test signature is the same as the original signature, then it means that the payload and the header have not been modified.
Because if they had been modified, then the test signature would have to be different. Therefore in this case where there has been no alteration of the data, we can then authenticate the user. And of course, if the two signatures
are actually different, well, then it means that someone tampered with the data.
Usually by trying to change the payload. But that third party manipulating the payload does of course not have access to the secret, so they cannot sign the JWT.
So the original signature will never correspond to the manipulated data.
And therefore, the verification will always fail in this case. And that's the key to making this whole system work. It's the magic that makes JWT so simple,
but also extremely powerful.
The contents in a json web token (JWT) are not inherently secure, but there is a built-in feature for verifying token authenticity. A JWT is three hashes separated by periods. The third is the signature. In a public/private key system, the issuer signs the token signature with a private key which can only be verified by its corresponding public key.
It is important to understand the distinction between issuer and verifier. The recipient of the token is responsible for verifying it.
There are two critical steps in using JWT securely in a web application: 1) send them over an encrypted channel, and 2) verify the signature immediately upon receiving it. The asymmetric nature of public key cryptography makes JWT signature verification possible. A public key verifies a JWT was signed by its matching private key. No other combination of keys can do this verification, thus preventing impersonation attempts. Follow these two steps and we can guarantee with mathematical certainty the authenticity of a JWT.
More reading: How does a public key verify a signature?
I would explain this with an example.
Say I borrowed $10 from you, then I gave you an IOU with my signature on it. I will pay you back whenever you or someone else bring this IOU back to me, I will check the signature to make sure that is mine.
I can't make sure you don't show the content of this IOU to anyone or even give it to a third person, all I care is that this IOU is signed by me, when someone shows this IOU to me and ask me to pay it.
The way how JWT works is quite the same, the server can only make sure that the token received was issued by itself.
You need other measures to make it secure, like encryption in transfer with HTTPS, making sure that the local storage storing the token is secured, setting up origins.
Ref - JWT Structure and Security
It is important to note that JWT are used for authorization and not authentication.
So a JWT will be created for you only after you have been authenticated by the server by may be specifying the credentials. Once JWT has been created for all future interactions with server JWT can be used. So JWT tells that server that this user has been authenticated, let him access the particular resource if he has the role.
Information in the payload of the JWT is visible to everyone. There can be a "Man in the Middle" attack and the contents of the JWT can be changed. So we should not pass any sensitive information like passwords in the payload. We can encrypt the payload data if we want to make it more secure. If Payload is tampered with server will recognize it.
So suppose a user has been authenticated and provided with a JWT. Generated JWT has a claim specifying role of Admin. Also the Signature is generated with
This JWT is now tampered with and suppose the
role is changed to Super Admin
Then when the server receives this token it will again generate the signature using the secret key(which only the server has) and the payload. It will not match the signature
in the JWT. So the server will know that the JWT has been tampered with.
Only JWT's privateKey, which is on your server will decrypt the encrypted JWT. Those who know the privateKey will be able to decrypt the encrypted JWT.
Hide the privateKey in a secure location in your server and never tell anyone the privateKey.
I am not a cryptography specialist and hence (I hope) my answer can help somebody who is neither.
There are two possible ways of using cryptography in programming:
Signing / verifying
Encryption / decryption
We use Signing when we want to ensure that data comes from a trusted source.
We use Encryption when we want to protect the data.
Signing / verifying uses asymmetrical algorithms i.e. we sign with one key (private) and the data receiver uses the other (public) key to verify.
A symmetric algorithm uses the same key to encrypt and decrypt data.
The encryption can be done using both symmetric and asymmetric algorithms.
relatively simple article on subject
The above is common knowledge below is my opinion.
When JWT is used for simple client-to-server identification there is no need for signing or asymmetric encryption. JWT can be encrypted with AES which is fast and supersecure. If the server can decrypt it, it means the server is the one who encrypted it.
Summary: non-encrypted JWT is not secure. Symmetric encryption can be used instead of signing in case no third party is involved.

OAuth 1.0a, 2-legged: how do you securely store clients' credentials (keys/secrets)?

Am I correct that OAuth 1.0a credentials need to be stored in plaintext (or in a way that can be retrieved as plaintext) on the server, at least when doing 2-legged authentication? Isn't this much less secure than using a username and salted+hashed password, assuming you're using HTTPS or other TLS? Is there a way to store those credentials in such a way that a security breach doesn't require every single one to be revoked?
In more detail: I'm implementing an API and want to secure it with OAuth 1.0a. There will possibly be many different API clients in the future, but the only one so far has no need for sensitive user data, so I'm planning to use "2-legged" OAuth.
As I understand it, this means I generate a consumer key and a shared secret for each API client. On every API request, the client provides both the consumer key, and a signature generated with the shared secret. The secret itself is not sent over the wire, and I definitely understand why this is better than sending a username and password directly.
However, as I understand it, both the consumer and the provider must explicitly store both the consumer key and the shared secret (correct me if I'm wrong), and this seems like a major security risk. If an attacker breached the provider's data store containing the consumer keys and shared secrets, every single API client would be compromised and the only way to re-secure the system would be to revoke every single key. This is in contrast to passwords, which are (ideally) never stored in a reversible fashion on the server. If you're salting and hashing your passwords, then an attacker would not be able to break into any accounts just by compromising your database.
All the research I've done seems to just gloss over this problem by saying "secure the credentials as you would with any sensitive data", but that seems ridiculous. Breaches do occur, and while they may expose sensitive data they shouldn't allow the attacker to impersonate the user, right?
You are correct. oAuth allows you however to login on the behalf of a user, so the target server (the one you access data from) needs to trust the token you present.
Password hashes are good when you are the receiver of the secret as keyed-in by the user (which, by the way, is what effectively what happens when oAuth presents the login/acceptance window to the user to generate afterwards the token). This is where the "plaintext" part happens (the user inputs his password in plaintext).
You need to have an equivalent mechanism so that the server recognizes you; what oAuth offers is the capacity to present something else than a password - a limited authorization form the use to login on his behalf. If this leaks then you need to invalidate it.
You could store these secrets in more or less elaborated ways, at the end of the day you still need to present the "plaintext" version t the server (that server, however, may use a hash to store it for checking purposes, as it just needs to verify that what you present in plain text, when hashed, corresponds to the hash they store)

How to store consumer secret for two-legged OAuth provider?

I'm implementing the provider side of a two-legged OAuth protocol for API authentication. We will provide the consumer with a consumer key and secret, which they will use to sign requests. The 2-legged OAuth is dictated by an interoperability standard, and thus a requirement.
The secret is sort of akin to a password, and I would never normally store a password as plain text (bCrypt or similar would be my normal choice). But because my provider needs access to the plain-text secret to verify the signature, it has to be either in some plain-text or reversible form.
I've considered the following options:
Store the secret as plain text
It's the most obvious solution, but if the database is compromised somehow, then all of the secrets will have to be changed. To me this solution is not ideal because it has all of the problems of storing a password in plain-text.
Apply reversible encryption (e.g. AES) with an encryption key stored elsewhere
This will provide some security, because if the database is compromised then the secrets will still be safe. But reversible encryption requires an encryption key, and the key has to be stored on the server. It means that if an attacker compromises the machine, then the encryption can be circumvented.
Is there something I haven't thought of?
Clarification Effectively it's using 2-legged Oauth as a single-signon system. The 'consumer' creates a request including the consumer key, a nonce, and several other parameters. The whole request is then signed by computing an HMAC-SHA1 with the consumer secret. When the request reaches our provider system, the process is repeated and if the signatures match then the request processing continues. We therefore need the plain-text secret to compute the HMAC-SHA1 signature on our side too. Unfortunately this mechanism is dictated by the industry-standard protocol that we need to comply with.
Take a look at this previous question. I'm not an expert on the topic, but I think you're missing part of the equation. In addition to the consumer key and secret, you'll be verifying the application that's sending the request (using an x509 certificate if you're using RSA-SHA1).
Are you sure that the provider needs the plain text password?
If this is the case then you simply can't have a 'customer secret'. As soon as the customer discloses this secret to someone else (including you) it fails to become a secret any longer.
Maybe if you explained more of what you are trying to do we could come up with a more elegant appraoch.

Can you spot a vulnerability in my authentication protocol?

Some time ago we needed a solution for Single Sign On authentication between multiple web services. At least at that time we considered OpenID protocol too complicated and we were not convinced about the Ruby on Rails plugins for it. Therefore we designed a protocol of our own instead of implementing an OpenID provider and OpenID consumers.
I have two questions:
Was it a bad thing not to create
our own OpenID provider and setup
our OpenID consumers accept only it?
Public login or registration are not
allowed and we wanted to keep
authentication simple.
Can you spot a crucial error or a vulnerability in the following design?
If you as a commune can approve this design, I will consider extracting this code into a Ruby on Rails plugin.
Please look at the flowchart and sequence diagram.
Details:
Authentication Provider ("AP"):
Central service which holds all data
about the users.
Only one "AP" exists in this setup.
It could be possible to have multiple "AP"s, but that should not be relevant in this context.
"AP" knows each "S" beforehand.
Authentication Client (Service "S"):
There exists several internal and external web services.
Each service knows "AP" and its public key beforehand.
Actor ("A"):
The end user who authenticates
herself with AP by a username and password
May request directly any URI of "S" or "AP" prior to her login
Connections between "A", "S" and "AP" are secured by HTTPS.
Authentication logic described briefly:
These are a description for the graphical flowchart and sequence diagram which were linked at the top of this post.
1) Auth Provider "AP"
"AP" makes a server-to-server HTTP POST request to "S" to get a nonce.
"AP" generates an authentication token.
Authentication token is an XML entity which includes:
an expiration date (2 minutes from now),
the previously requested nonce (to prevent replay),
identifying name of "S" (token for Service_1 is not good for Service_2),
information about the end user.
Authentication token is encrypted with AES256 and the encryption key and initialization vector are signed by AP's private RSA key.
Resulting strings ("data", "key" and "iv") are first Base64 encoded and then URL encoded to allow them be delivered in the URL query string.
End user "A" is HTTP-redirected to service "S" (HTTPS GET request).
2) Service "S"
Receives authentication token in URL parameters from user agent.
Decrypts authentication token with AP's pre-shared public key.
Accepts one authentication token only once (token includes a nonce which is valid only once).
Checks that identifying name in authentication token corresponds to service's name.
Checks that authentication token is not expired.
Remarks:
It is not a problem if somebody else can also decrypt the authentication token, because it contains no confidential information about the user. However, it is crucial that nobody else than AP is able to generate a valid authentication token. Therefore the RSA key pair is involved.
RSA private key is used only for signing the token, because it cannot encrypt data which is longer than the actual key length. Therefore AES is used for encryption.
Since the authentication token is delivered as an HTTP GET request, it will be stored e.g. in Apache's log file. Using a disposable nonce and an expiration date should minimize the possibility of a replay attack. POST request would need an HTML page with a form which is submitted automatically by Javascript, which is why GET is used.
Service "S" generates a nonce only in a server-to-server API request. Therefore unauthenticated generation requests should not pose a DoS-vulnerability.
You're confusing authentication ("I am who I say I am") and authorization/access control ("I am allowed to access this"). You can just implement OAuth, and then query a server over HTTPS with "is this OAuth identity allowed to access me?". You don't have to worry about replay attacks, since you're using HTTPS.
"Security is hard, so I'll design my own."
Authentication token is encrypted with AES256 and the encryption key and initialization vector are signed by AP's private RSA key.
AES-256 and AES-192 have weak key schedules. But you're not using it for confidentiality; you're using it as some sort of "integrity" check. It doesn't work: Attacker gets a "signed" authentication token. Attacker recovers the key and IV. Attacker encrypts a different authentication token with the same key and IV, and uses the same "signature".
What's wrong with hashing it and signing the hash? Also note that if you're going to use custom signing, you need to be careful about padding (IIRC PKCS-whatever adds at least 11 bytes).
EDIT: And if you're using a cipher where you should be using a hash/MAC, you really shouldn't be designing a security protocol!
Here are a few quick thoughts about question 1:
Designing a working security protocol is very hard, so on general principle I would favor using an existing one.
However, I appreciate that OpenID might not have been very established at the time. Also OpenID is still relatively new and might not have all of its limitations figured out yet.
Still, you'd be using OpenID in a restricted scenario where the big issue of OpenID (involvement of multiple actors) doesn't come into play. You'd only be using the “technical core” of OpenID, which is easier to understand.
Your requirements and the overview of your protocol remind me of Kerberos. I'm also tempted to push towards LDAP + single sign on, but I don't know what concrete solutions exist for that.
A point in favor of your protocol is that you've taken the time to describe it in detail. Just that puts you above than most self-made security protocol designers!
In short I find this protocol to be over engineered in the wrong places and ultimately vulnerable to attack.
So What is the vulnerability?
End user "A" is HTTP-redirected to service "S" (HTTPS GET request).
This is likely to be a violation of OWASP A9. At no point can a user's session ID be passed over an insecure channel such as http. Even if the session id hasn't been authenticated yet, an attacker is patient he can sniff the wire looking for session id's and then periodically check if they have been authenticated and then use them to access your system.
"Complexity is the worst enemy of security."
--Bruce Schneier
Authentication token is encrypted with
AES256 and the encryption key and
initialization vector are signed by
AP's private RSA key.
First of all RSA can be used to encrypt a message, so aes is unnecessary. HTTPS on the other hand is going to be more efficient and proven to be secure. I don't understand why you have to pass an authentication token to the client, when you already have a secure server-to-server communication channel. A server could just say "Hey someone has been redirected to me with this session id, what is his state information?". Its a matter of the weakest link in the chain, and your session id should be strong enough. This would require the session id to be sent as a GET or POST request by the client when the hand off occures which could open the door to Session Fixation. You could check the ip address before and after the handoff, sometimes the ip address of the client can change legitimately but the handoff is going to be a very narrow window in which this can happen, and most importantly it is stops Session Fixation altogether.
In general you should avoid re-inventing the wheal. Especially when it comes to security problems like this which have already been solved. Kerberos is beautiful especially if you need to tie in non-http authentication. Using LDAP for session management is another possibility.
Do you really just sign AES key and then send encrypted token, RSA signature of key and then key-iv in PLAINTEXT?
It's a fail. Attacker can use this key to decrypt a token, change it in any needed way and encrypt back. Your server will never find a difference.
If you want to check token integrity, just make a hash of it and sign this hash. This is what hashes used for. No need to use encryption here.

Resources