Related
I'm working on a project and was trying to create data models for it. We've a usecase where a user can host an event and add members to it.
class Event {
event_name: String
}
class User {
username: String
}
I wanted to know which of the following way to store the event members in the Event class.
//v1
class Event {
event_name: String,
event_members: Array<String> // List of usernames
}
//v2
class Event {
event_name: String,
event_members: Array<User> // List of user objects
}
By using v2, I feel I'll be able to move the logic to get user information, from DB, from client side to my server.
Latency is also something that I'm considering. If I go with v1, then I need to make multiple calls to the server to fetch all the information about event members, resulting in increase in wait time. Whereas, in v2, the response payload is increasing which might impact our network calls.
I wanted to know which will be a better way to store my model among the two and if there's a different and more efficient way then please let me know.
There is no singular "better" data model here. When modeling data in a NoSQL database, it always depends on your use-cases. As you add more use-cases to your app, you'll expand and modify the data model to fit your needs.
That said, I typically store both directions of a many-to-many relationship, so both v1 and v2. This allows fast lookup of the related items in both directions, at the cost of some extra storage - so a typical time-vs-space trade-off.
But as said: there is no singular best data model, and when you're just getting started I typically focus on getting a simple model working quickly, and on securing access to that data.
For a good introduction to the topic in general, I recommend reading NoSQL data modeling, and for Firestore specifically watch Todd's excellent Getting to know Cloud Firestore series.
Assume read model ProductCatalogueItem is built from aggregates/write-models, stored separately from write-models, and contains each product available for selling, and has following properties:
basics: product_code, name, price, number_of_available_stock,
documentation: short_description, description,...
product characteristics: weight, length, depth, width, color,...
And, there are two views:
product list containing list/table/grid of available product offers, and the view needs only following basic properties: product_code, name, price, number_of_available_stock,
product details showing all the properties - basics, documentation, product characteristics.
Naturally, there come two ViewModels in mind:
ProductCatalogueListItem containing only basic properties,
ProductCatalogueItemDetails containing all the properties.
Now,.. there two options (I can see).
ViewModels are 1:1 representation of ReadModels
Therefore the are two read models, not one, ProductCatalogueListItem and ProductCatalogueItemDetails. And, the read service will have two methods:
List<ProductCatalogueListItem> searchProducts(FilteringOptions),
ProductCatalogueItemDetails getProductDetails(product_code).
And, controllers return these models directly (or, mapped to dto for transport layer).
The issue here is filtering,.. should read service perform search query on a different read model, than is returned from the method call? Because, ProductCatalogueListItem doesn't have enough information to perform filtering.
ViewModels are another project of ReadModels
The read service will have two methods:
List<ProductCatalogueItem> searchProducts(FilteringOptions),
ProductCatalogueItem getProduct(product_code).
And, the mapping from ReadModels to ViewModels is done by upper layer (probably controller).
There is no issue with filtering,... But, there is another issue, that more data leave domain layer, than is actually needed. And, controllers would grow with more logic. As there might be different controllers for different transport technologies, then mapping code would probably get duplicated in those controllers.
Which approach to organize responsibilities is correct according to DDD/CQRS, or completely something else?
The point is:
should I build two read models, and search using one, then return other?
should I build single read model, which is used, and then mapped to limited view to contain only base information for view?
First of all, you do a wrong assertion:
...read model ProductCatalogueItem is built from aggregates/write-models...
Read model doesn't know of aggregates or anything about write model, you build the read model directly from the database, returning the data needed by the UI.
So, the view model is the read model, and it doesn't touch the write model. That's the reason why CQRS exists: for having a different model, the read model, to optimize the queries for returning the data needed by the client.
Update
I will try to explain myself better:
CQRS is simply splitting one object into two, based on the method types. There are two method types: command (any method that mutates state) and query (any method that returns a value). That's all.
When you apply this pattern to the service boundary of an application, you have a write service and a read service, and so you can scale differently the command and query handling, and you can have also two models.
But CQRS is not having two databases, is not messaging, is not eventual consistency, is not updating read model from write model, is not event sourcing. You can do CQRS wihtout them. I say this because I've seen some misconceptions in your assertions.
That said, the design of the read model is done according to what information the user wants to see in the UI, i.e., the read model is the view model, you have no mapping between them, they both are the same model. You can read about it in the references (3) and (6) bellow. I think this answer to your whole question. What I don't understand is the filtering issue.
Some good references
(1) http://codebetter.com/gregyoung/2010/02/16/cqrs-task-based-uis-event-sourcing-agh/
(2) http://www.cqrs.nu/Faq/command-query-responsibility-segregation
(3) "Implementing Domain Driven Design" book, by Vaughn Vernon. Chapter 4: Architecture, "Command-Query Responsibility Segregation, or CQRS" section
(4) https://kalele.io/really-simple-cqrs/
(5) https://martinfowler.com/bliki/CQRS.html
(6) http://udidahan.com/2009/12/09/clarified-cqrs/
As you already built your read model using data which arrived from one or more services, your problem is now in another space(perhaps MVC) rather in CQRS.
Now assume your read model is a db object and ProductCatalogueListItem and ProductCatalogueItemDetails are 2 view models. When you have a request to serve list of products you will make a query in your read db from read model (ProductCatalog table). May be you make queries for additional filters using additional where clauses. Now where do you put your mapping activities in your code after fetching db objects? Its a personal choice. You don't have to do it on uupper llayer aat aall. When I use dapper I fetch db objects using view models inside generic. So I can directly return result from my service method whose return type would be IEnumerable.
For a detail view I would use the same db object. I know CQRS suggests to have different read models for different views. But question yourself - do you really need another db object for detail view? You will need only an id to get all columns where in the first case you needed some selected columns. So I would design your case with a mixture of your 2 above mentioned methods - have 2 service methods returning 2 different objects but instead of having a 1:1 read model to view model have a single read db object and build 2 different view models from it.
The NestJS documentation showcases how to add DTOs to use in Controllers to validate request objects by using class-validator package. DTOs described there are TypeScript classes. Now, while controllers deal with DTOs(TS Classes), NestJS providers (or services), on the other hand, makes use of TypeScript interfaces. These DTOs and interfaces are pretty much of the same shape.
Now, I am seeing duplication of shape definition here. And wondering if the interfaces are needed at all?
Can we not make DTOs source of truth for the shape and validations? One of the approaches we were considering (to make the DTO source of truth) was, to have an openapi generator take the DTOs as input and generate openapi definition and from there another codegen can generate a set of typescript interfaces to be consumed by NestJS itself and which can be shared with another set of consumer applications like Angular too.
Have any people come across a similar problem? What do you think of the above? Feedback appreciated.
According to the Nestjs docs:
But first (if you use TypeScript), we need to determine the DTO (Data Transfer
Object) schema. A DTO is an object that defines how the data will be
sent over the network. We could determine the DTO schema by using
TypeScript interfaces, or by simple classes. Interestingly, we
recommend using classes here. Why? Classes are part of the JavaScript
ES6 standard, and therefore they are preserved as real entities in the
compiled JavaScript. On the other hand, since TypeScript interfaces
are removed during the transpilation, Nest can't refer to them at
runtime. This is important because features such as Pipes enable
additional possibilities when they have access to the metatype of the
variable at runtime.
I'm no expert but I'm not using DTO's at all. I really couldn't see a use for them. In each module I have a service, module, entity, and controller.
I would like to explain the concept of DTO with the simplest example possible for your better understanding.
DTO stands for Data Transfer Object. Now DTO's are used to reduce code duplication. It simply defines a schema which are passed in the parameters of functions to make it easy to get the required data from them. Here is an example of a DTO
export class AuthCredentialsDto {
email: string;
password: string;
}
Now if we make a method to check whether the password is correct or not
async password_check(usercredentials: AuthCredentialsDTO)
{
//Destructuring in JAVASCRIPT
const {email} = usercredentials;
//Database Logic to find the email
return user;
}
Now if we didn't make use of the DTO, then the code would have looked like
async password_check(email: string, password: string)
{
//Database Logic to find the email
return user;
}
also the point is that this is just one function, in a framework, Multiple function call multiple other functions which requires passing the parameters again and again. Just consider that a function requires 10 parameters. you would have to pass them multiple times. although it is possible to work without a DTO but it is not a development friendly practice. Once you get used to DTO you would love to use them as they save a lot of extra code and effort.
Regards
To extend #Victor's answer regarding the DTO concept and its role, I'd like to point that interfaces allow us to set a contract which represents something meaningful in our app. We can then implement and/or extend this contract in other places where needed as well e.g. entity definition for database objects - DAOs, data transfer objects - DTOs, and business models definitions notably.
Also interfaces for DTOs can be shared across a backend and a front-end so that both projects can avoid code duplicate and differences between objects exchanged for ease of development and maintainability.
one thing that dto provides more than interface is. with dto and class validator you can make the validations quickly at request level. But when it comes to interface you cannot add class validator to it. dtos are class in general. that means you have more to do with that than a interface.
DTO has a little bit different mission. This is an additional abstraction for data transfer connection by the network between FE and BE layers of your application and at the same time, DTO gives a description of your data like it doing Interface. But the main mission is a distinct place for data connection and due to this you could implement many helpful things in your DTO layer it could be some transformation for DTO fields values, validation for them, checking with regular expression, etc. So you have a convenient place for attendant changes for data just on early receiving or sending to FE side
TLDR
The answer to your question is yes, you could use them for shape if you wanted to, but it might be unnecessary in some situations.
DTOs are a great solution for when you need to enforce some shape on your data(specially on the nestjs ecosystem where it interacts a lot with class-validator) or transform it somehow. Examples of that would be when you're recieving data from your client or from another service. In this case the DTOs are the way to go for setting contracts.
However when you're sending data for example, between two layers of the same application -- for instance between your controller and your usecase or between your usecase and your repository -- you might want to use an interface there since you know your data is comming in the correct format in this scenarios.
One key difference to understand is that the interface serves as a development tool, it keeps you for making mistakes like passing an object lacking a certain required property between two classes, while the DTO affects the application itself, it's an object that's going to be instantiated at runtime and might be used for validation and data transformation purposes, that`s the idea, but of course it also has the capacities of an interface.
There might be exceptions to this rule of thumb depending on the architecture you're going for. For example, on a project I'm working on it's very common to have the contract between domain and presentation layers equal to the contract between frontend and the API. On this project, to avoid having to create two similar contracts, I`ve chosen to use a DTO to set the contract between the presentation and domain layers. Now in most cases I just extend the DTO when setting the contracts between API and clients.
Reason for using of DTO and Interface in NestJS
Basically in rest API we have to types of operation, One is Input and Another is Output. which is Request and Response
During response we doesn't need to validate return value. We just need to pass data based on interface
But in request we need to validate body
for example you want to create a user. Then the request body might be something like this
const body = {
name: "Test Name",
email: "test#gmail.com",
phone: "0393939",
age: 25
}
so during request we need to validate email, phone number or password is matched regex etc.
so in DTO we can do all validation
Here is one of my DTO example
import { IsEmail, IsNotEmpty, IsString, MinLength } from 'class-validator';
export class RegisterUserRequest {
#IsString()
#IsNotEmpty()
name: string;
#IsEmail()
#IsNotEmpty()
email: string;
#IsNotEmpty()
#MinLength(6)
password: string;
}
export class LoginUserRequest {
#IsEmail()
#IsNotEmpty()
email: string;
#IsNotEmpty()
#MinLength(6)
password: string;
}
And here is the interface example
import { UserRole } from './user.schema';
export interface UserType {
_id?: string;
email?: string;
name?: string;
role: UserRole;
createdAt?: Date;
updatedAt?: Date;
}
Hope you understand.
Thanks
I read through all of the answers, and none really seem to answer your question. I believe that yes, although DTOs and interfaces serve different purposes, I don't see a reason why you need to create an additional interface for typing purposes.
Happy to be proven wrong here, but none of the answers really address the OP's point, which is that the DTO serves a validation purpose, but you also get the typing for free.
In the nestjs-query packages there are two types of DTOs referenced. Read DTO - The DTO returned from queries and certain mutations, the read DTO does not typically define validation and is used as the basis for querying and filtering. Input DTOs - The DTO used when creating or updating records.
Basically you can validate request input without Dto. But imagination, you have to work with body payload, route params, query params or even header values. Without Dto you have to put your validation code inside each controller's methods to handle the request.
With Class Validation and Class Transformer, you can use decorator to do that. Your mission is defining your Dto Classes and add the validation annotations for each property.
You can find out the details here How to validate request input in nestjs and How to use pipe in nestjs
For example, if the Dto you created for the incoming request needs to be checked for the incoming data, the best way to do this is to create the Dto as a class. Because after typescript is compiled, they continue to exist in your javascript code. In this way, you can add various validations. For example "IsNotEmpy", "IsString" etc. If the data doesn't need to be validated, you can create Dto using interface. So here, rather than a single correct or correct method, it's about what you need.
BTW, even despite on DTO is a Java convention it can't solve the problem of Generic fields, e.g.:
#Get(url/${variable})
#Reponse({
[$variable: string]: $value
})
TS Interfaces can solve this issue only, but you cant describe it in DTO
And to show it you will pass some hardcoded example
class ResponseDto {
#ApiProperty({
...
example: [...]
})
[$variable]: SomeTSInterface[]
}
#Reponse({ status: 200, type: ResponseDto })
DTOs represent the structure of data transferred over the network it is meant to be for a specific use case whereas interfaces are more generalized specificity helps with better readability and optimizations. Interfaces don't exist after transpiling but nest accommodates dtos to be useful even after the transpilation face.
I think the NestJs documentation answered this precisely:
A DTO is an object that defines how the data will be sent over the network. We could determine the DTO schema by using TypeScript interfaces, or by simple classes. Interestingly, we recommend using classes here. Why? Classes are part of the JavaScript ES6 standard, and therefore they are preserved as real entities in the compiled JavaScript. On the other hand, since TypeScript interfaces are removed during the transpilation, Nest can't refer to them at runtime. This is important because features such as Pipes enable additional possibilities when they have access to the metatype of the variable at runtime.
Link to this pragraph: https://docs.nestjs.com/controllers#request-payloads
In my opinion,
DTO = Data Transfer Object. Dtos are like interfaces but their whole goal is to transfer data and validate it. They are mostly used in routers/controllers.
You can simplify your API body and query validation logic by using them. For instance, you have a AuthDto which automatically maps the user email and password to an object dto to enforce validations.
Where as the interface is just to declare how your response or a particular data model will be.
I'm not an expert but I do not understand why we use Dto
When we can use the schema model - what is the need for Dto and additional objects
I am redesigning my NodeJS application because I want to use the Rich Domain Model concept. Currently I am using Anemic Domain Model and this is not scaling well, I just see 'ifs' everywhere.
I have read a bunch of blog posts and DDD related blogs, but there is something that I simply cannot understand... How do we handle Persistence properly.
To start, I would like to describe the layers that I have defined and their purpose:
Persistence Model
Defines the Table Models. Defines the Table name, Columns, Keys and Relations
I am using Sequelize as ORM, so the Models defined with Sequelize are considered my Persistence Model
Domain Model
Entities and Behaviors. Objects that correspond to the abstractions created as part of the Business Domain
I have created several classes and the best thing here is that I can benefit from hierarchy to solve all problems (without loads of ifs yay).
Data Access Object (DAO)
Responsible for the Data management and conversion of entries of the Persistence Model to entities of the Domain Model. All persistence related activities belong to this layer
In my case DAOs work on top of the Sequelize models created on the Persistence Model, however, I am serializing the records returned on Database Interactions in different objects based on their properties. Eg.: If I have a Table with a column called 'UserType' that contains two values [ADMIN,USER], when I select entries on this table, I would serialize the return according to the User Type, so a User with Type: ADMIN would be an instance of the AdminUser class where a User with type: USER would simply be a DefaultUser...
Service Layer
Responsible for all Generic Business Logic, such as Utilities and other Services that are not part of the behavior of any of the Domain Objects
Client Layer
Any Consumer class that plays around with the Objects and is responsible in triggering the Persistence
Now the confusion starts when I implement the Client Layer...
Let's say I am implementing a new REST API:
POST: .../api/CreateOrderForUser/
{
items: [{
productId: 1,
quantity: 4
},{
productId: 3,
quantity: 2
}]
}
On my handler function I would have something like:
function(oReq){
var oRequestBody = oReq.body;
var oCurrentUser = oReq.user; //This is already a Domain Object
var aOrderItems = oRequestBody.map(function(mOrderData){
return new OrderItem(mOrderData); //Constructor sets the properties internally
});
var oOrder = new Order({
items: aOrderItems
});
oCurrentUser.addOrder(oOrder);
// So far so good... But how do I persist whatever
// happened above? Should I call each DAO for each entity
// created? Like, first create the Order, then create the
// Items, then update the User?
}
One way I found to make it work is to merge the Persistence Model and the Domain Model, which means that oCurrentUser.addOrder(...) would execute the business logic required and would call the OrderDAO to persist the Order along with the Items in the end. The bad thing about this is that now the addOrder also have to handle transactions, because I don't want to add the order without the items, or update the User without the Order.
So, what I am missing here?
Aggregates.
This is the missing piece on the story.
In your example, there would likely not be a separate table for the order items (and no relations, no foreign keys...). Items here seem to be values (describing an entity, ie: "45 USD"), and not entities (things that change in time and we track, ie: A bank account). So you would not directly persist OrderItems but instead, persist only the Order (with the items in it).
The piece of code I would expect to find in place of your comment could look like orderRepository.save(oOrder);. Additionally, I would expect the user to be a weak reference (by id only) in the order, and not orders contained in a user as your oCurrentUser.addOrder(oOrder); code suggests.
Moreover, the layers you describe make sense, but in your example you mix delivery concerns (concepts like request, response...) with domain concepts (adding items to a new order), I would suggest that you take a look at established patterns to keep these concerns decoupled, such as Hexagonal Architecture. This is especially important for unit testing, as your "client code" will likely be the test instead of the handler function. The retrieve/create - do something - save code would normally be a function in an Application Service describing your use case.
Vaughn Vernon's "Implementing Domain-Driven Design" is a good book on DDD that would definitely shed more light on the topic.
In a demo project I am setting up as a proof of concept I am finding myself with a lot of duplicated DTOs and fields. For instance considering 1 root object representing an item or inventory, I would have the following classes and properties
CreateItem [ Code, Description, Weight ]
Entity on aggregate root [ Code, Description, Weight ]
ItemCreated event [ Code, Description, Weight ]
Item read model [ Code, Description, Weight ]
Query request object [ Code, Description, Weight, Page, PageSize ]
Response DTO [ Code, Description, Weight ]
and so on
All these objects are a result of separating my app into the traditional Domain, Application, Presentation layers.
How are you managing all this duplication? Tools like AutoMapper and such help to convert between them, but if I wanted to add a new property to Item that would be used everywhere I would have to update all these models.
Because the domain model may not be exactly the same as the application read model, I understand the need for separate definitions, however this can very quickly become a maintenance nightmare.
Charles,
An approach that might help you get rid of some bits of duplication could be the following:
do not add public properties on the entity. As you are using event sourcing, the internal state of the entity will be restored by replaying the events associated with the entity in question. And nobody from outside the entity should know how the reconstructed state of the entity is represented - a series of private fields, an array of string objects, whatever. After all, you can choose to have no internal representation of its state, but implement all the behavior (all the methods) by simply replaying the events each and every time :) So, at one extreme, you may have 1 single field on the entity = a collection of events.
Item read model and Response DTO... Make them the same thing! As you are using cqrs, that is, you have already segregated the read model from the write model, there is no need to make a lasagna out of the read-side of the application. It is OK to have a minimum number of abstraction layers on the read-side. The read model is already behavior-less, it is data only. It's a DTO that gets constructed from events, gets persisted into a data storage with a (more or less) denormalized schema and, upon user request, is retrieved from the storage and presented on the UI or so. It's pure data that gets transferred from one place to another. Though, if, for some reason, you have to return to the end user (human or machine) some other data in addition to the data from the read model object, apply basic object composition. Compose two or more behavior-less objects into one such object. And send the latter on the wire.
speaking of composition, you may even define a (value) type such as ItemDescriptor [Code, Description, Weight] in a shared library and use it when defining the CreateItem command, the ItemCreated event, the read model, the query request object or so. If you are using a language that supports mixins, then mix that ItemDescriptor in :) Else, basic composition may be applied.
Furthermore, the "maintenance nightmare" can be ameliorated to some extent if packaging by feature would be used instead of packaging by layer.
Also, perhaps this post might help a bit.