In the application I'm building there are a lot of scenarios where I need to select a group of aggregates on which to perform a specific operation. For instance, I may have to mark a bunch of Reminder aggregates as expired if they meet the expiration policy (there is only one).
I have a ReminderExpirationPolicy domain service that is always applied before delivering reminders. This policy does something like:
reminderRepository.findRemindersToExpire().forEach(function (reminder) {
reminder.expire(clockService.currentDateTime());
});
The expiration policy is currently duplicated as it exists as a SQL predicate within the SqlReminderRepository.findRemindersToExpire method and also within the Reminder.expire aggregate's method.
The answer to the question may be strongly opiniated (although there should definitely be pros and cons - and perhaps a widely adopted practice), but should I simply trust that the Reminder.expire method will only get called as part of the ReminderExpirationPolicy process and trust that the repository implementation will return the correct set of reminders to expire or should I also protect the invariant within the Reminder aggregate itself?
NOTE: I am aware that modifying multiple aggregates in a single transaction is sub-optimal and hinders scalability, but it's the most pragmatic solution in my case.
should I simply trust that the Reminder.expire method will only get called as part of the ReminderExpirationPolicy process and trust that the repository implementation will return the correct set of reminders to expire or should I also protect the invariant within the Reminder aggregate itself?
Short answer: you are backwards. You must protect the invariant within the Reminder aggregate; using the policy as a query specification is optional.
The key thing to realize is that, in your scenario, using the policy as a query specification is really is optional. Eliding persistence concerns, you should be able to do this
repo.getAll () { a -> a.expire(policy) ; }
with the aggregate declining to change state when doing so would violate the business invariant.
In general, the reason that this distinction is important is that any data that you could get by querying the repository is stale -- there could be another thread running coincident with yours that updates the aggregate after your query has run but before your expire command runs, and if that coincident work were to change the aggregate in a way that the policy would no longer be satisfied, then your expire command would come along later and threaten to violate the invariant.
Since the aggregate has to protect itself against this sort of race condition anyway, checking the policy in the query is optional.
It's still a good idea, of course -- in the course of normal operations, you shouldn't be sending commands that you expect to fail.
What's really happening, if you squint a little bit, is that the expire command and the query are using the same policy, but where the command execution path is evaluating whether the writeModel state satisfies the policy, the query is evaluating whether the readModel state satisfies the policy. So it isn't really duplicated logic - we're using different arguments in each case.
However, where my assumptions are different than yours is that from as far as I can see (assuming optimistic locking), even if the data become stale after aggregates are loaded and that I do not enforce the expiration rule within the aggregate, the operation would still fail because of a concurrency conflict.
Yes, if you have assurance that the version of the aggregate that processes the command is the same as the version that was used to test the policy, then the concurrent write will protect you.
An additional consideration is that you are losing one of the benefits of encapsulation. If the policy check happens in the aggregate, then you are guaranteed that every code path which can expire the aggregate must also evaluate the policy. You don't get that guarantee if the aggregate is relying on the caller to check the policy (a variant on the "anemic domain" model).
Related
I'm new to DDD and I'm trying to model and implement a simple CRM system based on DDD, CQRS and event sourcing to get a feel for the paradigm. I have, however, run in to some difficulties that I'm not sure how to handle. I'm not sure if my difficulties stem from me not having modeled the domain properly or that I'm missing something else.
For a basic illustration of my problems, consider that my CRM system has the aggregate CustomerAggregate (which seems reasonble to me). The purpose of this aggregate is to make sure each customer is consistent and that its invarints hold up (name is required, social security number must be on the correkct format, etc.). So far, all is well.
When the system receives a command to create a new customer, however, it needs to make sure that the social security number of the new customer doesn't already exist (i.e. it must be unique across the system). This is, of cource, not an invariant that can be enforced by the CustomerAggregate aggregate since customers don't have any information regarding other customers.
One suggestion I've seen is to handle this kind of constraint in its own aggregate, e.g. SocialSecurityNumberUniqueAggregate. If the social security number is not already registered in the system, the SocialSecurityNumberUniqueAggregate publishes an event (e.g. SocialSecurityNumberOfNewCustomerWasUniqueEvent) which the CustomerAggregate subscribes to and publishes its own event in response to this (e.g. CustomerCreatedEvent). Does this make sense? How would the CustomerAggregate respond to, for example, a missing name or another hard constraint when responding to the SocialSecurityNumberOfNewCustomerWasUniqueEvent?
The search term you are looking for is set-validation.
Relational databases are really good at domain agnostic set validation, if you can fit the entire set into a single database.
But, that comes with a cost; designing your model that way restricts your options on what sorts of data storage you can use as your book of record, and it splits your "domain logic" into two different pieces.
Another common choice is to ignore the conflicts when you are running your domain logic (after all, what is the business value of this constraint?) but to instead monitor the persisted data looking for potential conflicts and escalate to a human being if there seems to be a problem.
You can combine the two (ex: check for possible duplicates via query when running the domain logic, and monitor the results later to mitigate against data races).
But if you need to maintain an invariant over a set, and you need that to be part of your write model (rather than separated out into your persistence layer), then you need to lock the entire set when making changes.
That could mean having a "registry of SSN assignments" that is an aggregate unto itself, and you have to start thinking about how much other customer data needs to be part of this aggregate, vs how much lives in a different aggregate accessible via a common identifier, with all of the possible complications that arise when your data set is controlled via different locks.
There's no rule that says all of the customer data needs to belong to a single "aggregate"; see Mauro Servienti's talk All Our Aggregates are Wrong. Trade offs abound.
One thing you want to be very cautious about in your modeling, is the risk of confusing data entry validation with domain logic. Unless you are writing domain models for the Social Security Administration, SSN assignments are not under your control. What your model has is a cached copy, and in this case potentially a corrupted copy.
Consider, for example, a data set that claims:
000-00-0000 is assigned to Alice
000-00-0000 is assigned to Bob
Clearly there's a conflict: both of those claims can't be true if the social security administration is maintaining unique assignments. But all else being equal, you can't tell which of these claims is correct. In particular, the suggestion that "the claim you happened to write down first must be the correct one" doesn't have a lot of logical support.
In cases like these, it often makes sense to hold off on an automated judgment, and instead kick the problem to a human being to deal with.
Although they are mechanically similar in a lot of ways, there are important differences between "the set of our identifier assignments should have no conflicts" and "the set of known third party identifier assignments should have no conflicts".
Do you also need to verify that the social security number (SSN) is really valid? Or are you just interested in verifying that no other customer aggregate with the same SSN can be created in your CRM system?
If the latter is the case I would suggest to have some CustomerService domain service which performs the whole SSN check by looking up the database (e.g. via a repository) and then creates the new customer aggregate (which again checks it's own invariants as you already mentioned). This whole process - the lookup of existing SSN and customer creation - needs to happen within one transaction to to ensure consistency. As I consider this domain logic a domain service is the perfect place for it. It does not hold data by itself but orchestrates the workflow which relates to business requirements - that no to customers with the same SSN must be created in our CRM.
If you also need to verify that the social security number is real you would also need to perform some call the another service I guess or keep some cached data of SSNs in your CRM. In this case you could additonally have some SocialSecurityNumberService domain service which is injected into the CustomerService. This would just be an interface in the domain layer but the implementation of this SocialSecurityNumberService interface would then reside in the infrastructure layer where the access to whatever resource required is implemented (be it a local cache you build in the background or some API call to another service).
Either way all your logic of creating the new customer would be in one place, the CustomerService domain service. Additional checks that go beyond the Customer aggregate boundaries would also be placed in this CustomerService.
Update
To also adhere to the nature of eventual consistency:
I guess as you go with event sourcing you and your business already accepted the eventual consistency nature. This also means entries with the same SSN could happen. I think you could have some background job which continually checks for duplicate entries and depending on the complexity of your business logic you might either be able to automatically correct the duplicates or you need human intervention to do it. It really depends how often this could really happen.
If a hard constraint is that this must NEVER happen maybe event sourcing is not the right way, at least for this part of your system...
Note: I also assume that command de-duplication is not the issue here but that you really have to deal with potentially different commands using the same SSN.
In CQRS and DDD, an aggregate is a transactional boundary. Hence I have been modeling commands always in such a way that each command always only ever addresses a single aggregate. Of course, technically, it would be possible to write a command handler that addresses multiple aggregates, but that would not be within a single transaction and hence would not be consistent.
If you actually have to address multiple aggregates, I usually go with a process manager, but this sometimes feels like pretty much overhead. In addition, from my understanding a process manager always only reacts to domain events, it is not directly addressed by commands. So you need to decide which aggregate to put the starting point to.
I have seen that some people solve this using so-called domain or application services, which can receive commands as well, and then work on multiple aggregates – but in this case the transactional nature of the process gets lost.
To give a simple example, to better illustrate the scenario:
A user shall join a group.
A user has a max number of groups.
A group has a max number of users.
Where to put the command that triggers the initial joining process, and what to call it? user.join(group) feels as right or wrong as group.welcome(user). I'd probably go for the first one, because this is closer to the ubiquitous language, but anyway…
If I had something above the aggregates, like the aforementioned services, then I could run something such as:
userManagement.addUserToGroup(user, group);
However, this addUserToGroup function would then need to call both commands, which in turn means it has to take care of both commands being processed – which is somewhat counterintuitive to having separate aggregates at all, and having aggregates as transactional boundaries.
What would be the correct way to model this?
It may be worth reviewing Greg Young on Eventual Consistency and Set Validation.
What is the business impact of having a failure
This is the key question we need to ask and it will drive our solution
in how to handle this issue as we have many choices of varying degrees
of difficulty.
And certainly Pat Helland on Memories, Guesses, and Apologies.
Short version: the two generals tell us that, if two pieces of information must be consistent, then we need to write both pieces of information in the same place. The "invariant" constrains our data model.
The invariant you describe is effectively a couple of set validation problems: the "membership" collection allows only so many members with user A, and only so many members with group B. And if you really are in a "we go out of business if those rules are violated" situation, then you cannot distribute the members of that set -- you have to lock the entire set when you modify it to ensure that the rule is not broken and that first writer wins.
An element that requires some care in your modeling: is the domain model the authority for membership? or is the "real world" responsible for membership and the domain model is just caching that information for later use? You want to be very careful about trying to enforce an invariant on the real world.
There's a risk that you end up over constraining the order in which information is accepted by the model.
Essentially what you have is many to many relationships between users and groups with restrictions on both sides:
Restriction on the number of groups a user can join
Restriction on the number of users a group can have
VoiceOfUnreason already gave a great answer, so I'll share one way I've solved similar problems and go straight to the model and implementation in case you have to ensure that these constraints are enforced at all costs. If you don't have to, do not make the model and implementation that complex.
Ensuring consistency with such constraints on both Group and User entities will be difficult in a single operation because of the concurrency of the operations.
You can model this by adding a collection of RegisteredUsers to a Group or vice versa, adding a collection of JoinedGroups to a User, and enforce the constraint on one side, but enforcing it on the other side is still an issue.
What you can do is introduce another concept in your domain. The concept of a
"Slot" in a Group. "Slots" are limited by the max number of Slots for a Group.
Then a User will issue a JoinGroupRequest that can be Accepted or Rejected.
A Slot can be either Taken or Reserved. Then you can introduce the concept of SlotReservation. The process of joining a User to a Group will be:
Issue a JoinGroupRequest from a User
Try to Reserve a Slot enforcing the MaxUsersPerGroup constraint.
Acquire a Slot or Reject the SlotReservation of a User enforcing the MaxGroupsPerUser constraint.
Accept or Reject the JoinGroupRequest depending on the outcome of the SlotReservation
If the SlotReservation is Rejected, another User will be able to use this Slot later.
For the implementation, you can add SlotReservation Queue Per Group to ensure that once a Slot is free after a Rejected SlotReservation, the next User that wants to join the Group will be able to.
For the implementation, you can add a collection of Slots to a Group, or you can make Slot an aggregate in its own right.
You can use a Saga for this process. The Saga will be triggered when a JoinGroupRequest is made by a User.
Essentially, this operation becomes a Tentative Operation.
For more details take a look and the Accountability Pattern and Life beyond distributed transactions an apostate's opinion and Life beyond distributed transactions an apostate's implementation.
Suppose I have an aggregate root Tenant and an aggregate root Organization. Multiples Organizations can be linked to a single Tenant. Tenant only has the Id of the Organizations in it's aggregate.
Suppose I have the following invariant in the Organization aggregate: Organization can only have one subscription for a specific product type.
Suppose I have the following invariant in the Tenant aggregate: only one subscription for a product type must exists across all Organizations related to a Tenant.
How can we enforce those invariants using the one aggregate per transaction rule?
When adding a subscription to an Organization, we can easily validate the first invariant, and fire a domain event to update (eventual consistency) the Tenant, but what happens if the invariant is violated in the Tenant aggregate?
Does it imply to fire another domain event to rollback what happens in the Organization aggregate? Seems tricky in the case a response had been sent to a UI after the first aggregate had been modified successfully.
Or is the real approach here is to use a domain service to validate the invariants of both aggregates before initiating the update? If so, do we place the invariants/rules inside the domain service directly or do we place kind of boolean validation methods on aggregates to keep the logic there?
UPDATE
What if the UI must prevent the user from saving in the UI if one invariants is violated? In this case we are not even trying to update an aggregate.
One thing you might want to consider is the possibility of a missing concept in your domain. You might want to explore the possibility of your scenario having something as a Subscription Plan concept, which by itself is an aggregate and enforces all of these rules you're currently trying to put inside the Tenant/Organization aggregates.
When facing such scenarios I tend to think to myself "what would an organization do if there was no system at all facilitating this operation". In your case, if there were multiple people from the same tenant, each responsible for an organization... how would they synchronize their subscriptions to comply with the invariants?
In such an exercise, you will probably reach some of the scenarios already explored:
Have a gathering event (such as a conference call) to make sure no redundant subscriptions are being made: that's the Domain Service path.
Each make their own subscriptions and they notify each other, eventually charging back redundant ones: that's the Event + Rollback path.
They might compromise and keep a shared ledger where they can check how subscriptions are going corporation wide and the ledger is the authority in such decisions: that's the missing aggregate path.
You will probably reach other options if you stress the issue enough.
How can we enforce those invariants using the one aggregate per transaction rule?
There are a few different answers.
One is to abandon the "rule" - limiting yourself to one aggregate per transaction isn't important. What really matters is that all of the objects in the unit of work are stored together, so that the transaction is an all or nothing event.
BEGIN TRANSACTION
UPDATE ORGANIZATION
UPDATE TENANT
COMMIT
A challenge in this design is that the aggregates no longer describe atomic units of storage - the fact that this organization and this tenant need to be stored in the same shard is implicit, rather than explicit.
Another is to redesign your aggregates - boundaries are hard, and its often the case that our first choice of boundaries are wrong. Udi Dahan, in his talk Finding Service Boundaries, observed that (as an example) the domain behaviors associated with a book title usually have little or nothing to do with the book price; they are two separate things that have a relation to a common thing, but they have no rules in common. So they could be treated as part of separate aggregates.
So you can redesign your Organization/Tenant boundaries to more correctly capture the relations between them. Thus, all of the relations that we need to correctly evaluate this rule are in a single aggregate, and therefore necessarily stored together.
The third possibility is to accept that these two aggregates are independent of each other, and the "invariant" is more like a guideline than an actual rule. The two aggregates act like participants in a protocol, and we design into the protocol not only the happy path, but also the failure modes.
The simple forms of these protocols, where we have reversible actions to unwind from a problem, are called sagas. Caitie McCaffrey gave a well received talk on this in 2015, or you could read Clemens Vasters or Bernd Rücker; Garcia-Molina and Salem introduced the term in their study of long lived transactions.
Process Managers are another common term for this idea of a coordinated protocol, where you might have a more complicated graph of states than commit/rollback.
The first idea that came to my mind is to have a property of the organization called "tenantHasSubscription" that property can be updated with domain events. Once you have this property you can enforce the invariant in the organization aggregate.
If you want to be 100% sure that the invariant is never violated, all the commands SubscribeToProduct(TenantId, OrganizationId) have to be managed by the same aggregate (maybe the Tenant), that has internally all the values to check the invariant.
Otherwise to do your operation you will always have to query for an "external" value (from the aggregate point of view), this will introduce "latency" in the operation that open a window for inconsistency.
If you query a db to have values, can it happen that when the result is on the wire, somebody else is updating it, because the db doesn't wait you consumed your read to allow others to modify it, so your aggregate will use stale data to check invariants.
Obviously this is an extremism, this doesn't mean that it is for sure dangerous, but you have to calculate the probability of a failure to happen, how can you be warned when it happen, and how to solve it (automatically by the program, or maybe a manual intervention, depending on the situation).
I am currently working on my first bigger DDD application. For now it works pretty well, but we are stuck with an issue since the early days that I cannot stop thinking about:
In some of our aggreagtes we keep references to another aggregate-root that is pretty essential for the whole application (based on their IDs, so there are no hard references - also the deletion is based on events/eventual consistency). Now when we create a new Entity "Entity1" we send a new CreateEntity1Command that contains the ID of the referenced aggregate-root.
Now how can I check if this referenced ID is a valid one? Right now we check it by reading from the other aggregate (without modifying anything there) - but this approach somehow feels dirty. I would like to just "trust" the commands, because the ID cannot be entered manually but must be selected. The problem is, that our application is a web-application and it is not really safe to trust the user input you get there (even though it is not accessibly by the public).
Did I overlook any possible solutions for this problems or should I just ignore the feeling that there needs to be a better solution?
Verifying that another referenced Aggregate exists is not the responsibility of an Aggregate. It would break the Single responsibility principle. When the CreateEntity1Command arrive at the Aggregate, it should be considered that the other referenced Aggregate is in a valid state, i.e. it exists.
Being outside the Aggregate's boundary, this checking is eventually consistent. This means that, even if it initially passes, it could become invalid after that (i.e. it is deleted, unpublished or any other invalid domain state). You need to ensure that:
the command is rejected, if the referenced Aggregate does not yet exists. You do this checking in the Application service that is responsible for the Use case, before dispatching the command to the Aggregate, using a Domain service.
if the referenced Aggregate enters an invalid state afterwards, the corrects actions are taken. You should do this inside a Saga/Process manager. If CQRS is used, you subscribe to the relevant events; if not, you use a cron. What is the correct action it depends on your domain but the main idea is that it should be modeled as a process.
So, long story short, the responsibilty of an Aggregate does not extend beyond its consistency boundary.
P.S. resist the temptation to inject services (Domain or not) into Aggregates (throught constructor or method arguments).
Direct Aggregate-to-Aggregate interaction is an anti-pattern in DDD. An aggregate A should not directly send a command or query to an aggregate B. Aggregates are strict consistency boundaries.
I can think of 2 solutions to your problem: Let's say you have 2 aggregate roots (AR) - A and B. Each AR has got a bunch of command handlers where each command raises 1 or more events. Your command handler in A depends on some data in B.
You can subscribe to the events raised by B and maintain the state of B in A. You can subscribe only to the events which dictate the validity.
You can have a completely independent service (S) coordinating between A and B. Instead of directly sending your request to A, send your request to S which would be responsible for a query from B (to check for validity of referenced ID) and then forward request to A. This is sometimes called a Process Manager (PM).
For Example in your case when you are creating a new Entity "Entity1", send this request to a PM whose job would be to validate if the data in your request is valid and then route your request to the aggregate responsible for creating "Entity1". Send a new CreateEntity1Command that contains the ID of the referenced aggregate-root to this PM which uses ID of the referenced AR to make sure it's valid and if it's valid then only it would pass your request forward.
Useful Links: http://microservices.io/patterns/data/saga.html
Did I overlook any possible solutions for this problems
You did. "Domain Services" give you a possible loop hole to play in.
Aggregates are consistency boundaries; their behaviors are constrained by
The current state of the aggregate
The arguments that they are passed.
If an aggregate needs to interact with something outside of its boundary, then you pass to the aggregate root a domain service to encapsulate that interaction. The aggregate, at its own discretion, can invoke methods provided by the domain service to achieve work.
Often, the domain service is just a wrapper around an application or infrastructure service. For instance, if the aggregate needed to know if some external data were available, then you could pass in a domain service that would support that query, checking against some cache of data.
But - here's the trick: you need to stay aware of the fact that data from outside of the aggregate boundary is necessarily stale. There might be another process changing the data even as you are querying a stale copy.
The problem is, that our application is a web-application and it is not really safe to trust the user input you get there (even though it is not accessibly by the public).
That's true, but it's not typically a domain problem. For instance, we might specify that an endpoint in our API requires a JSON representation of some command message -- but that doesn't mean that the domain model is responsible for taking a raw byte array and creating a DOM for it. The application layer would have that responsibility; the aggregate's responsibility is the domain concerns.
It can take some careful thinking to distinguish where the boundary between the different concerns is. Is this sequence of bytes a valid identifier for an aggregate? is clearly an application concerns. Is the other aggregate in a state that permits some behavior? is clearly a domain concern. Does the aggregate exist at all...? could go either way.
In CQRS + ES and DDD, is it a good thing to have small read model in aggregate to get data from other aggregate or bounded context?
For example, in order validation (In Order aggregate), there is a business rules which validate order only if customer is not flagged. The flag information is put in read model (specific to the aggregate) via synchronous domain events.
What do you think about this ?
is it a good thing to have small read model in aggregate to get data from other aggregate or bounded context?
It's not ideal. Aggregates, due to their nature, are not good at enforcing consistency that involves state outside of themselves.
What this usually means is that the business is going to need some way to respond when two aggregates produce an unacceptable state.
You also have the option of checking for the flag before you run the placeOrder command on the aggregate. That check for the flag could be done in the command handler, or in the client -- basically, you have was of "validating" that the command should succeed before passing it to the aggregate.
That said, if it were critical to try to consult the read model while processing the command, a way to do it would be to use a "domain service"; you pass a service provider to the aggregate as part of the command, and let the interface abstract away the fact that running the query requires looking outside of the aggregate.
That gives you some of the decoupling you need to keep the aggregate testable.
It's doable, but not in the form of a read model, rather a Value Object in the Aggregate (since we're on the Write side).
If you already have a CustomerId in Order, you just have to compose a VO with it and a Flagged member.
Of course, this remains prone to all the problems of cross-aggregate communication since the data originates from Customer. Order has to be kept in sync with the flagged status of its Customer, which can require quite a bit of work.
In any case, you should probably first determine with your domain expert whether immediate consistency is an absolute requirement (in which case you have to somehow wrap Customer + Order in a transaction) or if you can afford a small delay in Flagged freshness when enforcing that invariant.
If the latter, you can choose between duplicating Flagged in the Order aggregate or the first option given by #VoiceOfUnreason - the main difference being probably that if the data is in the aggregate, you'll get it for free at the Domain level should you need it in multiple occasions, instead of duplicating the check in multiple use cases/command handlers at the application level.