consider the following scenario:
we have several aggregates that share a common trait (for example: they belong to the same user)
the events of each aggregate are used to materialize its own status.
what if a higher level status is required. that status should represent the overall status of all the user aggregates.
consider that there is a special business logic that defines the materialization of the higher level status (for example: if more than x aggregates are in error state, a higher level error status is required on the customer-aggregated state)
this requirement involves materializing over events of multiple aggregates. furthermore, the higher level state is considered as a fact in the sense that it is fully determined by the lower level events. For that reason, it seems that the usage of Saga is inappropriate here.
on the other hand, it is required to issue events when a transition occurs on the higher level state (for example: when more than x aggregates are reporting an error, issue an event that can drive further business logic). so generating events based on other events also seems inappropriate.
What would be the recommended approach here? I can see how a Saga can listen to the lower level aggregate events and transform some of them into commands that go to a higher level entity aggregate, that in turn generates its own events but while taking that route I could not escape the feeling that this event => command => event flow is a ceremony rather than a real design requirement.
thanks for your insights
Jonathan
Udi Dahan once wrote
Here’s the strongest indication I can give you to know that you’re doing CQRS correctly: Your aggregate roots are sagas.
In this case: "it is required to issue events when a transition occurs on the higher level state" is the big clue that there is a "higher level aggregate" that is observing the lower level events and applying your business logic to determine if the higher level state has changed.
This higher level aggregate might have an interface like
void handle(LowerLevelEvent e);
So the lower level aggregates output their events, and then the plumbing carries a copy of that event to the input of the higher level aggregate, and the higher level aggregate outputs the higher level events.
Note that there's no magic here -- the higher level aggregate isn't using all events to decide the higher level transitions, it's only using the ones it currently knows about.
The entire flow looks something like
the lower level aggregate broadcasts event:{id:123} to everybody interested
the plumbing observes event:{id:123}, and in response dispatches a handle(event:{id:123}) command to the higher level aggregate
the higher level aggregate applies its own business logic, broadcasting its own events if appropriate
There no low level and high level.
That's an important point; the low level and high level labels above are really just reference points. As far as an aggregate is concerned, there is just itself, and everything else -- there's no hierarchy of boundaries.
Among other things, this means that you can test the "higher level" aggregate's behaviors without the lower level aggregates even existing - you just send it messages and verify that it reacts correctly.
When it comes to dealing with processes there are typically two approaches.
Choreography
This relies on an implicit process by having each aggregate take part in a process and the events typically carry all required information. For me this breaks down when part of the process could be split into parallel bits. Having any human-based interaction also does not fit into this too well. There is also no view into the state of the process.
Orchestration
This is where we explicitly track a process and where process managers (I prefer this term above a "saga"). This may be why some folks regard all aggregate roots as process managers but the same aggregate root may partake in different processes which means that if we do regard an AR as a process manager it is going to have to be aware of which process it is part of.
I prefer orchestration since you can view the current state of the process. To this end I tend to make my process managers first-class citizens in that they are also an aggregate root but focus only on process management.
In a bounded context world I will have my process management (higher level) act as a type of integration layer since a process may involve more than one (lower level) bounded context.
For instance, a CustomerInteraction, or a Journal, or EMail may be part of many processes.
On my lower level bounded context level I'd have my message handler respond to commands an I'd publish events. I would not typically react to events on that level.
On the higher level process management bounded context I react to both commands that typically kick off a process and to events that move a process along.
I think that you should focus more on your business than on the implementation design. Is there any business process that needs that information? Then go ahead and implement it.
There no low level and high level. There are just different points of view of the same complex system, different slices that we must do in order to understand it. It's like we can see only 2d but the world is 3d. You have to choose a slice in the reality that is best for you in some circumstance.
In your case you choose to have strong consistency on each Aggregate that use the events to maintain a state use for command validation; they use their own events to have a perspective of the world. But you could have a different view of the world, as real as of that of the aggregates. You could have an Saga/Process manager that listen to the relevant events and based on some business requirements generates some commands that further change the system's state. That is OK, it's how complex systems work. You just have to have clear boundaries around these processes and use bounded contexts and context maps to keep them separated.
In conclusion, go ahead and create Sagas around any business process that you identify.
Related
I'm aware of the general rule that only a single aggregate should be modified per transaction, mostly for concurrency and transactional consistency issues, as far as I'm aware.
I have a use case where I want to create multiple aggregates in a single transaction: a RestaurantManager, a Restaurant, and a Menu. They seem like a single aggregate because their life-cycles begin and end together: it doesn't make sense within the domain to create a RestaurantManager without a Restaurant, or vice versa; the same goes for a Restaurant and a Menu. Further, if the Restaurant or the RestaurantManager is deleted (unregistered), they should all be deleted together.
However, I've split them into separate aggregates because, once created, they are updated separately, maintain their own invariants, and I don't want to load them all into memory just to update one property on the Restaurant, for example.
The only thing that ties them together is their life-cycle.
My question is whether this represents a case where it is okay to go against the "rule" that each transaction should only operate on a single aggregate.
I'd also like to know if I should enforce their shared life-cycle in the domain model by having each aggregate root hold the identifier of the aggregate root it depends on, i.e. by having Restaurant require a MenuId as a constructor parameter, and likewise for Menu and RestaurantId, so that neither can be created without the other. However, this still wouldn't enforce that they should be saved together by the application service anyway, since it could create them all in memory, then only save the Menu, for example.
Your requirement is a pretty normal use case in DDD, IMHO. There are always multiple aggregates working in tandem to support the application, and they are interlinked in their lifecycles. But the modeling concepts still stand true. Let me attempt to explain what your model would look like with the help of a few DDD rules:
Aggregates are transaction boundaries
Aggregates ensure that no business invariants are broken at any point. This means that if you have multiple aggregates strung together as part of one transaction, you have to load all of them into memory for the validation.
This is especially a problem when your application is data-rich and stores data in a database cluster - partitioned, distributed (think Mongo or Elasticsearch). You will have the problem of loaded up data from potentially different clusters as part of a single transaction.
Aggregates are loaded in entirety
Aggregates and their associated data objects are loaded in entirety into memory. This means that unnecessary objects (say the restaurant's schedule for the upcoming month, for example) for the transaction may be loaded into memory. By itself, this is not a problem. But when multiple aggregates get together, the amount of data loaded into memory needs to be considered.
Aggregates refer to each other by their unique identifiers
This one is straightforward and means that each aggregate stores its referenced aggregates by their identifiers instead of enclosing the other aggregate's data within it.
State changes across Aggregates are handled through Domain Events
In cases where you want a state change in one aggregate to have side-effects on other aggregates, you publish a domain event, and a subscriber handles the change on other aggregates in the background. This is how you would want to handle your requirement for cascade deletes.
By following these rules, you are essentially zooming in one single aggregate at a time and ensuring that the complexity remains low. When you string up multiple aggregates, though it is clear and understandable on day 1, eventually, the application tends towards becoming a big ball of mud, as dependencies and invariants start crisscrossing each other.
"only a single aggregate should be modified per transaction"
Contention at creation doesn't matter as much. You can create many ARs in a single transaction without problem because the only other operation that could conflict is another duplicate creation process.
Another reason to avoid involving many ARs in a single transaction is coupling between modules though, but you could always keep things loosely coupled using synchronously dispatched domain events.
As for the deletion, it's probably less problematic to make it eventually consistent. Does it really matter that Restaurant is closed while RestaurantManager remains registered for a short period of time?
The fact you are asking this question tells me your system is not distributed? If your system is running with a single DB server and used by a few people it may be that eventual consistency make things more complex for scalability you don't actually need.
Start simple and refactor as needed, but crossing AR boundaries is not something that should be done consistently or else your boundaries are clearly wrong.
Furthermore, if you want to communicate that a RestaurantManager can't be spawned from nowhere and associated with an invalid RestaurantId by mistake you may want to look at your ubiquitous language for guidance.
e.g.
"A RestaurantManager is registered for a given Restaurant": not sure it truly aligns with your UL, but it's just for the sake of the example.
RestaurantManager manager = restaurant.registerManager(...);
This obviously increases coupling and could affect performance, but it aligns well with the UL and makes it more difficult to misuse the model. Also note that with a single DB, you could enforce referential integrity which takes cares of these uninteresting referential constraints.
As pointed out by #plalx, contention doesn't matter as much when creating aggregates in terms of transactions, since they don't yet exist so can't be involved in contention.
As for enforcing the mutual life cycle of multiple aggregates in the domain, I've come to think that this is the responsibility of the application layer (i.e. an application service, or use case).
Maybe my thinking is closer to Clean or Hexagonal architecture, but I don't think it's possible or even sensible to try and push every single business rule down into the "domain model". The point of the domain model for me is to partition the problem domain into small chunks (aggregates), which encapsulate common business data/operations that change together, but it's the application layer's responsibility to use these aggregates properly in order to achieve the business' end goal (which is the application as a whole), including mediating operations between the aggregates and controlling their life cycles.
As such, I think this stuff belongs in an application service. That being said, frequently updating multiple aggregates in each use case could be a sign of incorrect domain boundaries.
I`m reading the book PATTERNS, PRINCIPLES, AND PRACTICES OF DOMAIN-DRIVEN DESIGN, written by Scott Millett with Nike Tune. In the chapter 19, Aggregates, he states:
Sometimes it is actually good practice to modify multiple aggregates within a transaction. But it’s
important to understand why the guidelines exist in the first place so that you can be aware of the
consequences of ignoring them.
When the cost of eventual consistency is too high, it’s acceptable to consider modifying two objects in the same transaction. Exceptional circumstances will usually be when the business tells you that the customer experience will be too unsatisfactory.
To summarize, saving one aggregate per transaction is the default approach. But you should
collaborate with the business, assess the technical complexity of each use case, and consciously ignore
the guideline if there is a worthwhile advantage, such as a better user experience.
I face to a case in my project when user request a operation to my app and this operation affects two aggregate, and there are rules that must be verified by the two aggregates for the operation takes place successfully.
it is something like "Allocating a cell for a detainee":
the user makes the request
the Detainee (AR1) is fetched from database and receives a command: detainee.AllocateTo(cellId);
3 the Cell (AR2) is fetched and receive a command: cell.Allocate(detaineeId);
Both steps 2 and 3 could throw an exception, depending on the detainee's status and cell capacity. But abstract it.
Using eventual consistency, if step 2 is executed successfully, emiting the event DetaineeAllocated, but step 3 fails (will run in another transaction, inside an event handler), the state of aggregates will be inconsistent, and worse, the operation seemed to be executed successfully for the user.
I know that there are cases like "when the user makes a purchase over $ 100, its type must be changed to VIP" that can be implemented using eventual consistency, but the case I mentioned does not seem to be one.
Do you think that this is a special case that the book mentions?
Each aggregate must not have an invalid state (internal state), but that does not imply aggregates have to be consistent with one another (external, or system state).
Given the context of your question, the answer could be either yes or no.
The Case for No
The external state can become eventually consistent, which may be acceptable to your product owner. In this case you design ways to detect the inconsistency and deal with it (e.g. by retrying operations, issuing compensating transactions, etc.)
The Case for Yes
In your orchestration layer, go ahead and update the aggregates in a transaction. You might choose to do this because it's "easy" and "right", or you might choose to do this because your product owner says the inconsistency can't be tolerated for whatever reason.
Another Case for No
There's another way out for saying this is not a special case, not a reason for more than one transaction. That way out requires a change to your model. Consider removing the mutual dependency between your detainee and the cell, and instead introducing another aggregate, CellAssignment, which represents a moment-interval (a temporal relationship) that can be constructed and saved in a single transaction. In this case, your detainee and the cell don't change.
"the state of aggregates will be inconsistent"
Well, it shouldn't be inconsistent forever or that wouldn't be eventual consistency. You would normally discuss with business experts to establish an acceptable consistency timeframe.
Should something go wrong an event will be raised which should trigger compensating actions and perhaps a notification to a human stating something went wrong after-all.
Another approach could be to introduce a process manager which is responsible to carry out the business process by triggering commands and listening to events, until completion or timeout. The ARs are often designed to allow small incremental steps
towards consistency. For instance, there could be a command to reserve cell space first rather than directly allocating the detainee. The UI could always poll the state of the process to know when it's complete if necessary.
Eventual consistency obviously comes at a cost. If you have a single DB in a monolith that doesn't need extreme scalability you could very well favor to modify both ARs in a single transaction until that becomes a problem.
Eventual consistency is often sold as less costly that strong consistency, but I believe that's mostly for distributed systems where you'd have to deal with XA transactions.
Do you think that this is a special case that the book mentions?
No.
What I suspect you have here is a modeling error.
From your description, it sounds like you are dealing with something like a CellAssignment, and the invariant that you are trying to maintain is to ensure that there are no conflicts among active cell assignments.
That suggests to me that you are missing some sort of aggregate - something like a seating chart? - that keeps track of all of the active assignments and conflicts.
How can you know? One way is to graph your aggregates; create a node for each piece of information you need to save, and join nodes with lines if there is a rule that requires locking both nodes. If you find yourself with disconnected graphs, or two graphs that only connect at the root id, then it's a good bet that separating some information into a new graph will improve your modeling.
All Our Aggregates Are Wrong, by Mauro Servienti, would be a good talk to review.
Following the Vaughn Vernon recommendation, to achieve a high level of decoupling and single responsibility, just one aggregate should be changed per transaction.
In the chapter 8 of the Red Book Vaughn Vernon demonstrated how two aggregates can "talk" to each other with domain events. In the chapter 13 how different aggregates in two different bounded context can "talk" to each other with notifications.
My question is, why should I deal with these situations differently once both of them happen in different transaction? If is it just one or multiple bounded contexts the possible problems wouldn't be the same?
For example, if the application crashes between two domain events in the same bounded context I'll end up with inconsistency as with two bounded contexts.
It seems that the safest way to deal with two aggregates "talking" to each other asynchronously is to have a transitional status in it, persist the events before send them (to avoid lose events), have idempotent operations when possible and deduplicate the event in the receiving side when it's not possible to execute the operation in an idempotent way.
I see two aspects to consider in your question:
The DDD aspect: Event types and what you do with them
A technical aspect: how to implement it reliably
Regarding the types of Events what I would say is that events that stay within the boundaries of a bounded context (often called Domain Events) normally carry a lot of information. Potentially a big part of the state of the Aggregate. If you use CQRS, they are used to create the Read Model. Events that cross the BC boundaries are sometimes called Integration Events and they should carry as little data as possible (potentially, only global IDs, like CustomerId, OrderId). The reason is that every extra property that you add is extra coupling between the publisher BC and the subscriber BCs, which is what you want to minimize.
I would say that it's this distinction between the types of Events which might lead to have different technical solutions, but I agree with you that it doesn't have to be this way if you find a solution that works well for both cases.
The solution you propose is correct. It looks very similar to the Outbox feature of NServiceBus, which basically takes care of all this for you.
Another approach that I've used, if your message broker supports it, is what Azure Service Bus calls Send Via. With this feature, you can publish events Via your own queue but the send will be committed transactionally with the removal of the incoming message from the queue. This means that if for some reason the message that you are processing is not deleted from the queue successfully (DB update exception, broker unavailable, etc) and therefore it will be retried, you know for sure that the events won't be sent and you can safely publish them again during the retry. This makes making idempotent operations simpler and avoids publishing ghost messages.
When discussing how to decide whether transactional or eventual consistency should be used in Part II of Vaughn Vernon's Effective Aggregate Design, he states
When examining the use case (or story), ask whether it's the job of
the user executing the use case to make the data consistent. If it is,
try to make it transactionally consistent, but only by adhering to the
other rules of aggregate. If it is another user's job, or the job of
the system, allow it to be eventually consistent.
I don't follow. Does anyone have a good example of applying this rule of thumb?
Here's how I get it :
MovePiece() on ChessBoard aggregate => user's responsibility. The action should all take place in one transaction contained within the ChessBoard boundary.
DecideGameOver() on ChessGame aggregate => system's responsibility.
Some handler subscribes to ChessBoard's PieceMoved events and
delegates to the ChessGame aggregate for it to decide if the game is
over. We can tolerate a delay between the final move being made on
the Board and when the Game aggregate is updated - it's eventual
consistency.
It's not a hard and fast rule though, more of an indicator generalized from observation of dozens of systems.
Suppose you've got two aggregates in your bounded context which have some constraints amongst each other. Using DDD these inter aggregate constraints can't be enforced in the same transaction i.e. the aggregate boundaries are transactional boundaries.
Would you consider using what in the Microsoft CQRS journey is called a "process manager" to coordinate two aggregates in the same bounded context or is a process manager only used to coordinate between two bounded contexts? What would the equivalent of a process manager that coordinates two or more aggregate roots within the same bounded context be?
An aggregate root defines a bounded context by default, albeit a lower level one (btw the lowest level bounded context you can find is an object, any object). The process manager is the name they used instead of a saga, proabably you can come up with other names too, it doesn't matter, they all have the same purpose.
And yes, I would consider using a saga to achieve eventual consistency. In fact, I think this is the best way and this is exactly what I'm doing in my own apps. Anyway, I'm using a message driven architecture (yes, in a local, non-distributed application) and I have automatically saga support via the service bus (my own, not released yet).
What is important when dealing with eventual consistency is to ensure idempotency everywhere. That is the aggregate roots should reject a duplicate operation and of course the event handler should be able to cope with the fact that the same event can be published more than once. However, be aware that you can't guarantee 100% idempotency but you can get very close to.