Is it possible to add all data in one object in the collection,
for example :
{
id:134....,
fullname : "jack...",
email : "amsdk#gm....",
messages :[
all messages here
Perhaps up to fifty thousand items
],
orders : [
all orders here
],.........
}
Does this cause any overload on the server?
The Mongo docs have helpful guidance for selecting a data model. The nested model you describe above is known as the Embedded Data Model:
In general, embedding provides better performance for read operations,
as well as the ability to request and retrieve related data in a
single database operation. Embedded data models make it possible to
update related data in a single atomic write operation.
Embedding documents simplifies getting data out of the database. The flip-side is that it can be challenging to add and update documents with a complex document hierarchy.
Mongo calls a traditional PK/FK structure the Normalized Data Model
In general, use normalized data models:
when embedding would result in duplication of data but would not provide sufficient read performance advantages to outweigh the implications of the duplication.
to represent more complex many-to-many relationships.
to model large hierarchical data sets.
The Aggregation Pipeline can be scary, which may make an Embedded Data Model tempting. If you feel that way, check out this free course on the MongoDB Aggregation Pipeline. A Normalized Data Model may be more appealing if you can confidently retrieve related data across collections.
Test read/write queries with a small data set for each model. You'll gain clarity about the benefits and challenges of each approach. For my current project, I found a Normalized Data Model simplified the many write operations required. Ultimately, separating concerns like orders and users into distinct collections proved simpler, especially once I understood how to use $lookup to merge data from related collections.
Related
I'm having trouble getting my head around how to use the repository pattern with a more complex object model. Say I have two aggregate roots Student and Class. Each student may be enrolled in any number of classes. Access to this data would therefore be through the respective repositories StudentRepository and ClassRepository.
Now on my front end say I want to create a student details page that shows the information about the student, and a list of classes they are enrolled in. I would first have to get the Student from StudentRepository and then their Classes from ClassRepository. This makes sense.
Where I get lost is when the domain model becomes more realistic/complex. Say students have a major that is associated with a department, and classes are associated with a course, room, and instructors. Rooms are associated with a building. Course are associated with a department etc.. etc..
I could easily see wanting to show information from all these entities on the student details page. But then I would have to make a number of calls to separate repositories per each class the student is enrolled in. So now what could have been a couple queries to the database has increased massively. This doesn't seem right.
I understand the ClassRepository should only be responsible for updating classes, and not anything in other aggregate roots. But does it violate DDD if the values ClassRepository returns contains information from other related aggregate roots? In most cases this would only need to be a partial summary of those related entities (building name, course name, course number, instructor name, instructor email etc..).
But then I would have to make a number of calls to separate repositories per each class the student is enrolled in. So now what could have been a couple queries to the database has increased massively. This doesn't seem right.
Yup.
But does it violate DDD if the values ClassRepository returns contains information from other related aggregate roots?
Nobody cares about "violate DDD". What we care about is: do you still get the benefits of the repository pattern if you start pulling in data from other aggregates?
Probably not - part of the point of "aggregates" is that when writing the business code you don't have to worry to much about how storage is implemented... but if you start mixing locked data and unlocked data, your abstraction starts leaking into the domain code.
However: if you are trying to support reporting, or some other effectively read only function, you don't necessarily need the domain model at all -- it might make sense to just query your data store and present a representation of the answer.
This substitution isn't necessarily "free" -- the accuracy of the information will depend in part on how closely your stored information matches your in memory information (ie, how often are you writing information into your storage).
This is basically the core idea of CQRS: reads and writes are different, so maybe we should separate the two, so that they each can be optimized without interfering with the correctness of the other.
Can DDD repositories return data from other aggregate roots?
Short answer: No. If that happened, that would not be a DDD repository for a DDD aggregate (that said, nobody will go after you if you do it).
Long answer: Your problem is that you are trying to use tools made to safely modify data (aggregates and repositories) to solve a problem reading data for presentation purposes. An aggregate is a consistency boundary. Its goal is to implement a process and encapsulate the data required for that process. The repository's goal is to read and atomically update a single aggregate. It is not meant to implement queries needed for data presentation to users.
Also, note that the model you present is not a model based on aggregates. If you break that model into aggregates you'll have multiple clusters of entities without "lines" between them. For example, a Student aggregate might have a collection of ClassEnrollments and a Class aggregate a collection of Atendees (that's just an example, note that modeling many to many relationships with aggregates can be a bit tricky). You'll have one repository for each aggregate, which will fully load the aggregate when executing an operation and transactionally update the full aggregate.
Now to your actual question: how do you implement queries for data presentation that require data from multiple aggregates? well, you have multiple options:
As you say, do multiple round trips using your existing repositories. Load a student and from the list of ClassEnrollments, load the classes that you need.
Use CQRS "lite". Aggregates and respositories will only be used for update operations and for query operations implement Queries, which won't use repositories, but access the DB directly, therefore you can join tables from multiple aggregates (Student->Enrollments->Atendees->Classes)
Use "full" CQRS. Create read models optimised for your queries based on the data from your aggregates.
My preferred approach is to use CQRS lite and only create a dedicated read model when it's really needed.
Assume read model ProductCatalogueItem is built from aggregates/write-models, stored separately from write-models, and contains each product available for selling, and has following properties:
basics: product_code, name, price, number_of_available_stock,
documentation: short_description, description,...
product characteristics: weight, length, depth, width, color,...
And, there are two views:
product list containing list/table/grid of available product offers, and the view needs only following basic properties: product_code, name, price, number_of_available_stock,
product details showing all the properties - basics, documentation, product characteristics.
Naturally, there come two ViewModels in mind:
ProductCatalogueListItem containing only basic properties,
ProductCatalogueItemDetails containing all the properties.
Now,.. there two options (I can see).
ViewModels are 1:1 representation of ReadModels
Therefore the are two read models, not one, ProductCatalogueListItem and ProductCatalogueItemDetails. And, the read service will have two methods:
List<ProductCatalogueListItem> searchProducts(FilteringOptions),
ProductCatalogueItemDetails getProductDetails(product_code).
And, controllers return these models directly (or, mapped to dto for transport layer).
The issue here is filtering,.. should read service perform search query on a different read model, than is returned from the method call? Because, ProductCatalogueListItem doesn't have enough information to perform filtering.
ViewModels are another project of ReadModels
The read service will have two methods:
List<ProductCatalogueItem> searchProducts(FilteringOptions),
ProductCatalogueItem getProduct(product_code).
And, the mapping from ReadModels to ViewModels is done by upper layer (probably controller).
There is no issue with filtering,... But, there is another issue, that more data leave domain layer, than is actually needed. And, controllers would grow with more logic. As there might be different controllers for different transport technologies, then mapping code would probably get duplicated in those controllers.
Which approach to organize responsibilities is correct according to DDD/CQRS, or completely something else?
The point is:
should I build two read models, and search using one, then return other?
should I build single read model, which is used, and then mapped to limited view to contain only base information for view?
First of all, you do a wrong assertion:
...read model ProductCatalogueItem is built from aggregates/write-models...
Read model doesn't know of aggregates or anything about write model, you build the read model directly from the database, returning the data needed by the UI.
So, the view model is the read model, and it doesn't touch the write model. That's the reason why CQRS exists: for having a different model, the read model, to optimize the queries for returning the data needed by the client.
Update
I will try to explain myself better:
CQRS is simply splitting one object into two, based on the method types. There are two method types: command (any method that mutates state) and query (any method that returns a value). That's all.
When you apply this pattern to the service boundary of an application, you have a write service and a read service, and so you can scale differently the command and query handling, and you can have also two models.
But CQRS is not having two databases, is not messaging, is not eventual consistency, is not updating read model from write model, is not event sourcing. You can do CQRS wihtout them. I say this because I've seen some misconceptions in your assertions.
That said, the design of the read model is done according to what information the user wants to see in the UI, i.e., the read model is the view model, you have no mapping between them, they both are the same model. You can read about it in the references (3) and (6) bellow. I think this answer to your whole question. What I don't understand is the filtering issue.
Some good references
(1) http://codebetter.com/gregyoung/2010/02/16/cqrs-task-based-uis-event-sourcing-agh/
(2) http://www.cqrs.nu/Faq/command-query-responsibility-segregation
(3) "Implementing Domain Driven Design" book, by Vaughn Vernon. Chapter 4: Architecture, "Command-Query Responsibility Segregation, or CQRS" section
(4) https://kalele.io/really-simple-cqrs/
(5) https://martinfowler.com/bliki/CQRS.html
(6) http://udidahan.com/2009/12/09/clarified-cqrs/
As you already built your read model using data which arrived from one or more services, your problem is now in another space(perhaps MVC) rather in CQRS.
Now assume your read model is a db object and ProductCatalogueListItem and ProductCatalogueItemDetails are 2 view models. When you have a request to serve list of products you will make a query in your read db from read model (ProductCatalog table). May be you make queries for additional filters using additional where clauses. Now where do you put your mapping activities in your code after fetching db objects? Its a personal choice. You don't have to do it on uupper llayer aat aall. When I use dapper I fetch db objects using view models inside generic. So I can directly return result from my service method whose return type would be IEnumerable.
For a detail view I would use the same db object. I know CQRS suggests to have different read models for different views. But question yourself - do you really need another db object for detail view? You will need only an id to get all columns where in the first case you needed some selected columns. So I would design your case with a mixture of your 2 above mentioned methods - have 2 service methods returning 2 different objects but instead of having a 1:1 read model to view model have a single read db object and build 2 different view models from it.
How do you solve a situation when you have multiple representations of same object, depending on a view?
For example, lets say you have a book store. Within a book store, you have 2 main representations of Books:
In Lists (search results, browse by category, author, etc...): This is a compact representation that might have some aggregates like for example NumberOfAuthors and NumberOfRwviews. Each Author and Review are entities themselves saved in db.
DetailsView: here you wouldn't have aggregates but real values for each Author, as Book has a property AuthorsList.
Case 2 is clear, you get all from DB and show it. But how to solve case 1. if you want to reduce number of connections and payload to/from DB? So, if you don't want to get all actual Authors and Reviews from DB but just 2 ints for count for each of them.
Full normalized solution would be 2, but 1 seems to require either some denormalization or create 2 different entities: BookDetails and BookCompact within Business Layer.
Important: I am not talking about View DTOs, but actually getting data from DB which doesn't fit into Business Layer Book class.
For me it sounds like multiple Query Models (QM).
I used DDD with CQRS/ES style, so aggregate roots are producing events based on commands being passed in. To those events multiple QMs are subscribed. So I create multiple "views" based on requirements.
The ES (event-sourcing) has huge power - I can introduce another QMs later by replaying stored events.
Sounds like managing a lot of similar, or even duplicate data, but it has sense for me.
QMs can and are optimized to contain just enough data/structure/indexes for given purpose. This is the way out of "shared data model". I see the huge evil in "RDMS" one for all approach. You will always get lost in complexity of managing shared model - like you do.
I had a very good result with the following design:
domain package contains #Entity classes which contain all necessary data which are stored in database
dto package which contains view/views of entity which will be returned from service
Dto should have constructor which takes entity as parameter. To copy data easier you can use BeanUtils.copyProperties(domainClass, dtoClass);
By doing this you are sharing only minimal amount of information and it is returned in object which does not have any functionality.
On SQLAlchemy's documentation page the author starts with a philosophy,
SQL databases behave less like object collections the more size and
performance start to matter; object collections behave less like
tables and rows the more abstraction starts to matter.
I'm scratching my head trying to understand the idea behind these two sentences, but failed. Could someone give an example illustrate the idea here? Thanks.
When you are creating an application using an Object Oriented language and a SQL database, you are simultaneously working with two very different conceptual models for storing information:
The relational model says how to store data in tables and rows and how to link elements through keys and joins.
The object model establishes a way to store entities with attributes in memory (usually) and how to set links between them using pointers or references.
So, let's say that you have an User entity that is linked to addresses and other users in your application. Those entities will need to be stored in the form several tables in the database (users table, addresses table and a many to many table for associating users to users, for instance). At the same time, if your code uses object oriented constructs, users and addresses will exist in memory in the form of objects with references between them, pointing to objects of the same or different kind.
The thing is, moving information between those two different worlds is much much more difficult than it looks at first:
You might associate one object with one row in a table, but that is not always possible and sometimes a single object must be associated to multiple rows in different tables.
Inheritance and polymorphic behavior are particularly difficult to map to a relational model.
Traversing objects and querying the database are vastly different actions.
Performance factors to take into account in an object model and a relational model are completely different.
And those are just a few examples. ORMs such as SQLAlchemy are essentially translators that convert information from one world into the other and back.
What I think that Mike Bayer was trying to convey is: the more you adapt your entity information to the object model (lots of inheritance, polymorphism, traversal of objects, ...), the farther it will resemble the natural structure in a relational model and the more performance concessions you will be making. And the other way around: the more you design your tables to perform well and be optimized for your queries, the less they will adapt to a natural structure of objects.
Martin Fowler has a nice write-up about the need of this translation in this article: ORM Hate (from which I took the above image).
Edit: further clarification on the abstraction vs performance issue
At the end, I think that the bottom line of that SQLAlchemy presentation text is: many ORMs hide the relational side of the relational-object oriented translation to make things easier. With them you only have to worry about the object oriented side, and the library is in charge of taking away the burden of dealing with the database. You get persistence for your objects without having to deal with SQL. However, they incur in a performance penalty in doing so, because the details of working with the database are abstracted away and you have no control over them. And those details are essential when you have to optimize performance. SQLAlchemy takes the opposite approach. It hides nothign of the relational side, you are in control of how SQL is generated and when and when not use joins, subqueries and other SQL constructs. That makes it a much more complex library to learn, but at the same time you are in control of the whole relational-object oriented translation process.
My domain consists of Products, Departments, Classes, Manufacturers, DailySales, HourlySales.
I have a ProductRepository which facilitates storing/retrieving products from storage.
I have a DepartmentAndClass repository which facilitates storing/retrieving of departments and classes, as well as adding and removing products from those departments and classes.
I also have a DailySales repository which I use to retrieve statistics about daily sales from multiple groupings. ie..
DailySales.GetSalesByDepartment(dateTime)
DailySales.GetSalesByClass(dateTime)
DailySales.GetSalesByHour(dateTime)
Is it correct to have these sales tracking methods in their own repository like this? Am I on the right track?
Since domains are so dependent on context some answers are harder than others. I would, however, place statistics on the Query side of things. You probably do not want to be calculating those stats on the fly as you will be placing some heavy processing on your database. Typically the stats should be denormalized for quick access where only filtering is required.
You may want to take a look at CQRS if you haven't done so.
Although most queries return an object or a collection of objects, it also fits within the concept to return some types of summary calculations, such as an object count, or a sum of a numerical attribute that was intended by the model to be tallied.
Eric Evans - Domain-Driven Design
This might be considered a read model. Are these daily sales objects being used in any domain model behaviour? Does any business logic depend on them? If not, it might be a good idea to separate this out into a distinct read model - at which point you're taking your first steps into CQRS.