It is a multi-tenant serverless system.
The system has groups with permissions.
Users derive permissions based on the groups they are in.
If it makes a difference, we are using Cognito for authentication and it is a stateless application.
For example:
GET endpoint for sites (so sites that the logged-in user has access to based on the groups they are in)
GET endpoint for devices (so sites that the logged-in user has access to based on the groups they are in)
In REST APIs. "The idea is that the data returned by an endpoint should depend solely on the parameters passed meaning two different users should receive the same result for the identical request.
"
What should the REST URI look like to ensure the above-stated idea? Since the deciding factor for the list here is "groups" and thus effective permissions, I was thinking we could pass the groups a user in, in the URI in sorted order to leverage caching on GET endpoints as well, Is there a better way to do it?
In REST APIs. "The idea is that the data returned by an endpoint should depend solely on the parameters passed meaning two different users should receive the same result for the identical request. "
No this is not strictly true. It can be a desirable property, but absolutely not needed. In fact, if you build a proper hypermedia REST api, you would likely want to hide links/actions that the current user is not allowed to use.
Furthermore, a cache will never store responses and send to different users if an AUthorization header is present on the request.
Anyway, there could be other reasons to want this.. maybe it's a simpler design for your case, and there is a pretty reasonable solution.
What I'm inferring from your question is that you might have two endpoints:
/sites
/devices
They return different things depending on who's accessing. Instead of using those kind of routes, you could just do:
/user/1234/sites
/user/1234/devices
Now every user has their own separate 'sites' and 'devices' collection. The additional benefit is that if you ever want to let a user find the list of sites or devices from another user, the API is ready to support that.
The idea is that the data returned by an endpoint should depend solely
on the parameters passed
This is called the statelessness constraint, but if you check the parameters always include auth parameters because of this. The idea is keeping the session data on the client side, because managing sessions becomes a problem when you have several million users and multiple servers all around the world. Since the parameters include auth data, the response can depend on this data, so you can use here the exact same endpoints for users with different permissions.
As of the responses you might want to send back hyperlinks, which represent the available operations. The concept is the same here, if the user does not have permission for the actual operation, then they won't get a hyperlink for that operation and in theory they should never get a 403 status either, because you must follow the hyperlinks you got from the service instead of hardcoding URI templates into your client. So you have to handle less errors and junk requests, and another reason here that you can change your URI templates without breaking the clients. This is called hypermedia as the engine of application state, it is part of the uniform interface constraint.
I am currently developing a strategy for Dialogflow on https://passportjs.org.
From what I've learnt, Dialogflow doesn't authenticate users. So I'm thinking about making a strategy (for passportjs) that identify users from every plateform differently (analyse the originalRequest differently for each plateform).
For example, the Telegram originalRequest has this field:
originalRequest.data.message.from.id
The Telegram says this field is a:
"Unique identifier for this user or bot"
So I think it is safe to use it for authentication and identify every intent of my users fulfilled by my webhook.
I was wondering about the actions-on-google authentication and I found the field originalRequest.data.user.userId.
The documentation says:
"Users can reset this identifier, so don't store important user data keyed off this identifier, because once it's reset, that information is no longer accessible by the user."
So the only reason to not trust the userId is because it can be reset? At the end of the documentation it says:
User ID lifetime - User IDs are reset automatically after 30 days of inactivity or if users unlink their accounts on the device.
And:
"If a registered user's voice isn't recognized by the device or no registered voice exists, then a different ID is used that is unique for just that conversation."
How to differentiate users from one other? Can some IDs be recycled?
The best way to differentiate users from each other is to use the userId field, as you've determined. On the AoG platform, the userId is meant to be used somewhat like a web cookie can be used - if you see it again, you are assured that this is the same user that used it last time. But if you see a new one, you have to assume that you've never seen this user before, even if it means they deleted the cookie.
To be clear - most of the time, the UserId will remain the same and you can expect returning users to have the same ID. This won't be true in only three cases:
They have reset the ID for this Action. So they have deliberately chosen to start over.
They didn't use the Action for 30 days, in which case it makes sense to treat them as a brand new user anyway in most cases.
They were not recognized as a normal user of this device, so they are treated anonymously. (This is the equivalent of the clunky "Do not remember me on this machine" setting you see on websites, which forces a session cookie rather than a persistent cookie.)
The phrasing is poor in the documentation - I think it is meant to remind developers that the user is ultimately in charge of their privacy. And Google both forces you to do the same and adopt policies that do so.
IDs will not be recycled. In fact, they won't even be re-used between different Actions, even for the same Assistant account.
Summary: If you see the same UserId, you can trust it is the same user you saw before. If you see a new one, assume they are a new user.
If you want a more robust way to identify users, you might consider using Account Linking which puts you in control of the identifying token. But that has significant additional overhead.
Be careful when using other authentication methods - Google limits how you're allowed to use them as part of an Action, and expressly forbids them in some cases. See the General Policies for details.
I am doing sequence diagrams, but I have some issues in the design of these diagrams. So I was doing the diagram for Account Creation as follows:
So, first the user goes to the registration page and then enters his information. I also wanted to show what would happen if the information entered by the user in the registration page are incorrect: I added the verify input arrow from the registration page to the database.
I am not sure who should verify that the input is correct, is it correct to say it's the database that checks if the entered data are valid?
Moreover, is it correct to add the dashed arrow from the database to the registration page or should I go directly to the if condition?
What would you advice me to change?
1) ..who should verify..input...database..?
You can delegate the responsibility for data validation to whichever actor with whatever name. Database is good candidate, and many SQL databases have some kind of integrity constraint checking mechanism built-in
If configured, such database will validate data and reject them at the moment of insert/update attempt. So in your case if a SQL database would handle it the way they usually do, the VerificationResult would be returned from AddNewUser() message, not sooner
However, for real-life application the good candidates according to Open Web Application Security Project (OWASP) → Data Validation would be:
..Integrity checks must be included wherever data passes from a trusted to a less trusted boundary, such as from the application to the user's browser in a hidden field, or to a third party payment gateway, such as a transaction ID used internally upon return..
..Validation must be performed on every tier..
2) ..dashed arrow from the database to the registration page..directly to the if..?
I would extend the execution (activation) down as shown below, but it does not mean that your way is not correct
See also: uml-diagrams.org: UML Sequence Diagrams Examples
3) ..advice..to change?
I would review the use of synchronous/asynchronous messages. It seems that you use one type of arrow for everything
Recently I discovered how useful and easy parse.com is.
It really speeds up the development and gives you an off-the-shelf database to store all the data coming from your web/mobile app.
But how secure is it? From what I understand, you have to embed your app private key in the code, thus granting access to the data.
But what if someone is able to recover the key from your app? I tried it myself. It took me 5 minutes to find the private key from a standard APK, and there is also the possibility to build a web app with the private key hard-coded in your javascript source where pretty much anyone can see it.
The only way to secure the data I've found are ACLs (https://www.parse.com/docs/data), but this still means that anyone may be able to tamper with writable data.
Can anyone enlighten me, please?
As with any backend server, you have to guard against potentially malicious clients.
Parse has several levels of security to help you with that.
The first step is ACLs, as you said. You can also change permissions in the Data Browser to disable unauthorized clients from making new classes or adding rows or columns to existing classes.
If that level of security doesn't satisfy you, you can proxy your data access through Cloud Functions. This is like creating a virtual application server to provide a layer of access control between your clients and your backend data store.
I've taken the following approach in the case where I just needed to expose a small view of the user data to a web app.
a. Create a secondary object which contains a subset of the secure objects fields.
b. Using ACLs, make the secure object only accessible from an appropriate login
c. Make the secondary object public read
d. Write a trigger to keep the secondary object synchronised with updates to the primary.
I also use cloud functions most of the time but this technique is useful when you need some flexibility and may be simpler than cloud functions if the secondary object is a view over multiple secure objects.
What I did was the following.
Restrict read/write for public for all classes. The only way to access the class data would be through the cloud code.
Verify that the user is a logged in user using the parameter request.user ,and if the user session is null and if the object id is legit.
When the user is verified then I would allow the data to be retrieved using the master key.
Just keep a tight control on your Global Level Security options (client class creation, etc...), Class Level Security options (you can for instance, disable clients deleting _Installation entries. It's also common to disable user field creation for all classes.), and most important of all, look out for the ACLs.
Usually I use beforeSave triggers to make sure the ACLs are always correct. So, for instance, _User objects are where the recovery email is located. We don't want other users to be able to see each other's recovery emails, so all objects in the _User class must have read and write set to the user only (with public read false and public write false).
This way only the user itself can tamper with their own row. Other users won't even notice this row exists in your database.
One way to limit this further in some situations, is to use cloud functions. Let's say one user can send a message to another user. You may implement this as a new class Message, with the content of the message, and pointers to the user who sent the message and to the user who will receive the message.
Since the user who sent the message must be able to cancel it, and since the user who received the message must be able to receive it, both need to be able to read this row (so the ACL must have read permissions for both of them). However, we don't want either of them to tamper with the contents of the message.
So you have two alternatives: either you create a beforeSave trigger that checks if the modifications the users are trying to make to this row are valid before committing them, or you set the ACL of the message so that nobody has write permissions, and you create cloud functions that validates the user, and then modifies the message using the master key.
Point is, you have to make these considerations for every part of your application. As far as I know, there's no way around this.
Let's take a simple "Account Registration" example, here is the flow:
User visit the website
Click the "Register" button and fill out the form, click the "Save" button
MVC Controller: Validate UserName uniqueness by reading from ReadModel
RegisterCommand: Validate UserName uniqueness again (here is the question)
Of course, we can validate UserName uniqueness by reading from ReadModel in the MVC controller to improve performance and user experience. However, we still need to validate the uniqueness again in RegisterCommand, and obviously, we should NOT access ReadModel in Commands.
If we do not use Event Sourcing, we can query the domain model, so that's not a problem. But if we're using Event Sourcing, we are not able to query the domain model, so how can we validate UserName uniqueness in RegisterCommand?
Notice: User class has an Id property, and UserName is not the key property of the User class. We can only get the domain object by Id when using event sourcing.
BTW: In the requirement, if the entered UserName is already taken, the website should show the error message "Sorry, the user name XXX is not available" to the visitor. It's not acceptable to show a message, that says, "We are creating your account, please wait, we will send the registration result to you via Email later", to the visitor.
Any ideas? Many thanks!
[UPDATE]
A more complex example:
Requirement:
When placing an order, the system should check the client's ordering history, if he is a valuable client (if the client placed at least 10 orders per month in the last year, he is valuable), we make 10% off to the order.
Implementation:
We create PlaceOrderCommand, and in the command, we need to query the ordering history to see if the client is valuable. But how can we do that? We shouldn't access ReadModel in command! As Mikael said, we can use compensating commands in the account registration example, but if we also use that in this ordering example, it would be too complex, and the code might be too difficult to maintain.
If you validate the username using the read model before you send the command, we are talking about a race condition window of a couple of hundred milliseconds where a real race condition can happen, which in my system is not handled. It is just too unlikely to happen compared to the cost of dealing with it.
However, if you feel you must handle it for some reason or if you just feel you want to know how to master such a case, here is one way:
You shouldn't access the read model from the command handler nor the domain when using event sourcing. However, what you could do is to use a domain service that would listen to the UserRegistered event in which you access the read model again and check whether the username still isn't a duplicate. Of course you need to use the UserGuid here as well as your read model might have been updated with the user you just created. If there is a duplicate found, you have the chance of sending compensating commands such as changing the username and notifying the user that the username was taken.
That is one approach to the problem.
As you probably can see, it is not possible to do this in a synchronous request-response manner. To solve that, we are using SignalR to update the UI whenever there is something we want to push to the client (if they are still connected, that is). What we do is that we let the web client subscribe to events that contain information that is useful for the client to see immediately.
Update
For the more complex case:
I would say the order placement is less complex, since you can use the read model to find out if the client is valuable before you send the command. Actually, you could query that when you load the order form since you probably want to show the client that they'll get the 10% off before they place the order. Just add a discount to the PlaceOrderCommand and perhaps a reason for the discount, so that you can track why you are cutting profits.
But then again, if you really need to calculate the discount after the order was places for some reason, again use a domain service that would listen to OrderPlacedEvent and the "compensating" command in this case would probably be a DiscountOrderCommand or something. That command would affect the Order Aggregate root and the information could be propagated to your read models.
For the duplicate username case:
You could send a ChangeUsernameCommand as the compensating command from the domain service. Or even something more specific, that would describe the reason why the username changed which also could result in the creation of an event that the web client could subscribe to so that you can let the user see that the username was a duplicate.
In the domain service context I would say that you also have the possibility to use other means to notify the user, such like sending an email which could be useful since you cannot know if the user is still connected. Maybe that notification functionality could be initiated by the very same event that the web client is subscribing to.
When it comes to SignalR, I use a SignalR Hub that the users connects to when they load a certain form. I use the SignalR Group functionality which allows me to create a group which I name the value of the Guid I send in the command. This could be the userGuid in your case. Then I have Eventhandler that subscribe to events that could be useful for the client and when an event arrives I can invoke a javascript function on all clients in the SignalR Group (which in this case would be only the one client creating the duplicate username in your case). I know it sounds complex, but it really isn't. I had it all set up in an afternoon. There are great docs and examples on the SignalR Github page.
I think you are yet to have the mindset shift to eventual consistency and the nature of event sourcing. I had the same problem. Specifically I refused to accept that you should trust commands from the client that, using your example, say "Place this order with 10% discount" without the domain validating that the discount should go ahead. One thing that really hit home for me was something that Udi himself said to me (check the comments of the accepted answer).
Basically I came to realise that there is no reason not to trust the client; everything on the read side has been produced from the domain model, so there is no reason not to accept the commands. Whatever in the read side that says the customer qualifies for discount has been put there by the domain.
BTW: In the requirement, if the entered UserName is already taken, the website should show error message "Sorry, the user name XXX is not available" to the visitor. It's not acceptable to show a message, say, "We are creating your account, please wait, we will send the registration result to you via Email later", to the visitor.
If you are going to adopt event sourcing & eventual consistency, you will need to accept that sometimes it will not be possible to show error messages instantly after submitting a command. With the unique username example the chances of this happening are so slim (given that you check the read side before sending the command) its not worth worrying about too much, but a subsequent notification would need to be sent for this scenario, or perhaps ask them for a different username the next time they log on. The great thing about these scenarios is that it gets you thinking about business value & what's really important.
UPDATE : Oct 2015
Just wanted to add, that in actual fact, where public facing websites are concerned - indicating that an email is already taken is actually against security best practices. Instead, the registration should appear to have gone through successfully informing the user that a verification email has been sent, but in the case where the username exists, the email should inform them of this and prompt them to login or reset their password. Although this only works when using email addresses as the username, which I think is advisable for this reason.
There is nothing wrong with creating some immediately consistent read models (e.g. not over a distributed network) that get updated in the same transaction as the command.
Having read models be eventually consistent over a distributed network helps support scaling of the read model for heavy reading systems. But there's nothing to say you can't have a domain specific read model thats immediately consistent.
The immediately consistent read model is only ever used to check data before issuing a command, you should never use it for directly displaying read data to a user (i.e. from a GET web request or similar). Use eventually consistent, scaleable read models for that.
About uniqueness, I implemented the following:
A first command like "StartUserRegistration". UserAggregate would be created no matter if user is unique or not, but with a status of RegistrationRequested.
On "UserRegistrationStarted" an asynchronous message would be sent to a stateless service "UsernamesRegistry". would be something like "RegisterName".
Service would try to update (no queries, "tell don't ask") table which would include a unique constraint.
If successful, service would reply with another message (asynchronously), with a sort of authorization "UsernameRegistration", stating that username was successfully registered. You can include some requestId to keep track in case of concurrent competence (unlikely).
The issuer of the above message has now an authorization that the name was registered by itself so now can safely mark the UserRegistration aggregate as successful. Otherwise, mark as discarded.
Wrapping up:
This approach involves no queries.
User registration would be always created with no validation.
Process for confirmation would involve two asynchronous messages and one db insertion. The table is not part of a read model, but of a service.
Finally, one asynchronous command to confirm that User is valid.
At this point, a denormaliser could react to a UserRegistrationConfirmed event and create a read model for the user.
Like many others when implementing a event sourced based system we encountered the uniqueness problem.
At first I was a supporter of letting the client access the query side before sending a command in order to find out if a username is unique or not. But then I came to see that having a back-end that has zero validation on uniqueness is a bad idea. Why enforce anything at all when it's possible to post a command that would corrupt the system ? A back-end should validate all it's input else you're open for inconsistent data.
What we did was create an index table at the command side. For example, in the simple case of a username that needs to be unique, just create a user_name_index table containing the field(s) that need to be unique. Now the command side is able to query a username's uniqueness. After the command has been executed it's safe to store the new username in the index.
Something like that could also work for the Order discount problem.
The benefits are that your command back-end properly validates all input so no inconsistent data could be stored.
A downside might be that you need an extra query for each uniqueness constraint and you are enforcing extra complexity.
I think for such cases, we can use a mechanism like "advisory lock with expiration".
Sample execution:
Check username exists or not in eventually consistent read model
If not exists; by using a redis-couchbase like keyvalue storage or cache; try to push the username as key field with some expiration.
If successful; then raise userRegisteredEvent.
If either username exists in read model or cache storage, inform visitor that username has taken.
Even you can use an sql database; insert username as a primary key of some lock table; and then a scheduled job can handle expirations.
Have you considered using a "working" cache as sort of an RSVP? It's hard to explain because it works in a bit of a cycle, but basically, when a new username is "claimed" (that is, the command was issued to create it), you place the username in the cache with a short expiration (long enough to account for another request getting through the queue and denormalized into the read model). If it's one service instance, then in memory would probably work, otherwise centralize it with Redis or something.
Then while the next user is filling out the form (assuming there's a front end), you asynchronously check the read model for availability of the username and alert the user if it's already taken. When the command is submitted, you check the cache (not the read model) in order to validate the request before accepting the command (before returning 202); if the name is in the cache, don't accept the command, if it's not then you add it to the cache; if adding it fails (duplicate key because some other process beat you to it), then assume the name is taken -- then respond to the client appropriately. Between the two things, I don't think there'll be much opportunity for a collision.
If there's no front end, then you can skip the async look up or at least have your API provide the endpoint to look it up. You really shouldn't be allowing the client to speak directly to the command model anyway, and placing an API in front of it would allow you to have the API to act as a mediator between the command and read hosts.
It seems to me that perhaps the aggregate is wrong here.
In general terms, if you need to guarantee that value Z belonging to Y is unique within set X, then use X as the aggregate. X, after all, is where the invariant really exists (only one Z can be in X).
In other words, your invariant is that a username may only appear once within the scope of all of your application's users (or could be a different scope, such as within an Organization, etc.) If you have an aggregate "ApplicationUsers" and send the "RegisterUser" command to that, then you should be able to have what you need in order to ensure that the command is valid prior to storing the "UserRegistered" event. (And, of course, you can then use that event to create the projections you need in order to do things such as authenticate the user without having to load the entire "ApplicationUsers" aggregate.