I have a question about the integration events used in a microservice / CQRS architecture.
The payload of the event can only have references to aggregates or can it have more information?
If only reference ids can be sent, the only viable solution is to bring the rest of the information with some type of call but the origin would have to implement an endpoint and the services would end up more coupled.
ex. when a user is created and the event is raised.
UserCreated {
userId
name
lastname
document
...
}
Is this correct?
If only reference ids can be sent,
Why would only that be allowed? I have worked with a system which was using micro-services, CQRS and DDD(similar like yours) and we did not have such restrictions. Like in most cases it is: "What works best for your application/business domain". Do not follow any rule blindly. This is perfectly fine to put other information in the events Payload as well.
the only viable solution is to bring the rest of the information with
some type of call but the origin would have to implement an endpoint
and the services would end up more coupled.
This is fine in some cases as well but this brings you to the situation to have additional call's after the event has been processed. I would not do this unless you have a really heavy model/models and it would affect your performance. For example if you have an event executed and based on userId you would need to load a collection of related objects/models for some reason. I had one similar case where I had to load a collection of other objects based on some action on user like event UserCreated. Of course in this case you don't want to send all that data in one Event payload. Instead you send only the id of the user and later call a Get api from the other service to get and save that data to your micro-service.
UserCreated
{
userId
name
lastname
document
... }
Is this correct?
Yes this is fine :)
What you could do instead:
Depending of your business scenario you could publish the information with multiple events with Stages and in different States.
Lets say from UI you have some Wizard-like screen with multiple steps of creation. You could publish
event: UserCreatedDraft with some initial data from 1st Wizard page
event: UserPersonalDataCreated with only part of the object related to private data
event: UserPaymentDataCreated with only the payment data created
UserCreatedFinal with the last step
Of this is just an example for some specific scenario which depends on your use case and your Business requirements. This is just to give you an Idea what you could do in some cases.
Summary:
As you can see there are multiple ways how you can work with these kind of systems. Keep in mind that following the rules is good but in some cases you need to do what is the best based on your business scenario and what works for some application might not be the best solution for your. Do what is most efficient for your system. Working with micro-services we need to deal with latency and async operations anyways so saving some performance on other parts of the system is always good.
Related
I am trying to understand how to implement this in Event sourcing model / DDD.
Assume a distributed application in which user submits an application for something, say Job/Loan. So the application raises an UserApplied Event.
There are few micro services like credit service, criminal record service.. they consume this UserApplied event do some validation, responds with CriminalCheckPassed, CreditCheckPassed ... etc. Assume there are 5 checks to be done. In future we might also add more checks like this.
The app consume these events and take some decision. That is - only if they are all validated successfully app can approve the user application by changing the status to UserApproved. Any of the validations failed, them it would be UserDeclined. Something like that.
It sounds simple. But I am banging my head how to implement that correctly?
This is my event store
I have a materialized view
If we have to update the materialized view/aggregate whenever we receive an event, app needs 5 different events to take decision. Till then it will be pending. Even when I receive the 5th event, the materialized view does not know how many events it has received before. I will end up querying entire event-store.
Another approach is - adding these columns in the materialized view. So that we know if we have received all these events. It will work. But looks super ugly.
My question is - how to use the aggregation properly in this case?
If I understand correctly, the validation is a part of domain logic (it is something that has to be made sure that it passes). Here, there are some external services like Credit Service and Criminal Record Service.
First, I would model User as an Entity and an Aggregate Root of itself. Then, I will model Job Application as another Entity and another Aggregate Root of itself. Now there are 2 aggregates, with the relationship: User can have many Job Applications.
Now, you need to validate some things before you create a Job Application instance. This validation requires some knowledge from other services. This can be solved by creating a domain service, say JobApplicationCreationService which sole responsibility is to create new instance of Job Application. Then, you would want to inject those external services here. Inside the service, do the validation using the services you injected, then if all validations pass, return a new Job Application instance. This Aggregate instance will have fulfilled your validation rules/domain logic.
Events here is not suitable for validation, rather it is used to synchronize states between Aggregates using eventual consistency. When Events are published and being processed, you want to make sure that the Aggregate that produces the events is already in a consistent state (in this case, the Job Application aggregate).
Here is my personal rule of thumb: Try to create an Aggregate from static factory method to contain the creation logic. If the creation requires something outside of the boundary of the Aggregate itself, refactor it to a Domain Service.
Well, if CriminalCheckPassed are domain events, then they need to somehow mutate the domains state, so you need to store it within your domain (which will be restored when you load your domain entity), say a private readonly List<RequiredCheck> RequiredChecks and check these on recieving of any of the responsible events, then decide.
If it's not a domain event and is not persisted with the aggregate root, then have a process manager (aka Saga) (i.e. UserApprovalProcessmaanger) collect these external events and process/persist them and once all of them are collected fire off an UserApproved / UserDeclined event which is processed by the domain model/aggregate root
I have product service, category service, promotions service, search service.
When User want to add product. CreateProductRequest come to product service. Request includes product data and datas of other services like categoryId,uncalculated price , too. After product is added. I need to send other servie datas. Category service needs productId and CategoryId. Promotions service needs productId and price.
After creat eproduct transaction commited;
1) I put all data in ProductCreatedEvent that includes saved productId, categoryId, uncalculated price etc. Every service get what it needs from event and save to own db. I publish event with RabbitMQ
2) Send via seperated commands to services.I send commands with RabbitMQ
And What If there is no category that id come with event and Category services didn't save. But Product saved at product Services ?
or what do you suggest ?
To answer the question, it's important to keep in mind the difference between a command and event. A command is a request to do something. An event is a record of something that has happened. One key difference is that a command can be rejected.
When looking at your use case, publishing events to other services makes the most sense. The product has been created and you are notifying the other bounded contexts that care about the change. If you issue a command, you are telling other bounded context to make a change that may or may not fail.
That said, you each bounded context may receive the event and produce a command within their own context to update aggregates managed within. As such, the difference is subtle between these two:
- Issue a command to each bounded context
- Issue an event to each bounded context and they can then trigger a command as needed
But given the above, the notification of the creation of the product should not fail. It has happened already. From there, each context can decide what to do about it.
We have a bot that will be used by different customers and depending on their database, sector of activity we're gonna have different answers from the bot and inputs from users. Intents etc will be the same for now we don't plan to make a custom bot for each customer.
What would be the best way to separate data per customer within Chatbase?
I'm not sure if we should use
A new API key for each customer (Do we have a limitation then?)
Differentiate them by the platform filter (seems to not be appropriated)
Differentiate them by the version filter (same it would feel a bit weird to me)
Using Custom Event, not sure how though
Example, in Dialogflow we pass the customer name/id as a context parameter.
Thank you for your question. You listed the two workarounds I would suggest, I will detail the pros/cons:
New API Key for each customer: Could become unwieldy to have to change bots everytime you want to look at a different users' metrics. You should also create a general api (bot) where you send all messages in order to get the aggregate metrics. This would mean making two api calls per message.
Differentiate by version filter: This would be the preferred method, however it could lengthen load times for your reports as your number of users grows. The advantage would be that all of your metrics are in one place, and they will be aggregated while only having to send one api call per message.
I'm having a problem understanding how basic communication between microservices should be made and I haven't been able to find a good solution or standard way to do this in the other questions. Let's use this basic example.
I have an invoice service that return invoices, every invoice will contain information(ids) about the user and the products. If I have a view in which I need to render the invoices for a specific user, I just make a simple request.
let url = "http://my-domain.com/api/v2/invoices"
let params = {userId:1}
request(url,params,(e,r)=>{
const results = r // An array of 1000 invoices for the user 1
});
Now, for this specific view I will need to make another request to get all the details for each product on each invoice.
results.map((invoice)=>{
invoice.items.map((itemId)=>{
const url=`http://my-domain.com/api/v2/products/${itemId}`
request(url,(e,r)=>{
const product = r
//Do something else.....
});
});
});
I know the code example is not perfect but you can see that this will generate a huge number of requests(at least 1000) to the product service and just for 1 user, now imagine if I have 1000 users making this kind of requests.
What is the right way to get the information off all the products without having to make this number of requests in order to avoid performance issues?.
I found some workarounds for this kind of scenarios such as:
Create an API endpoint that accepts a list of IDs in order to make a single request.
Duplicate the information from the Product service within the invoice service and find a way to keep them in sync.
In a microservices architecture are these the right ways to deal with this kind of issues? For me, they look like simple workarounds.
Edit #1: Based on Remus Rusanu response.
As per Remus recommendation, I decided to isolate my services and describe them a little bit better.
As shown in the image above the microservices are now isolated(in specific the Billing-service) and they now are the owners of the data. By using this structure I ensure that Billing-service is able to work even if there are async jobs or even if the other two services are down.
If I need to create a new invoice, I can call the other two microservices(Users, Inventory) synchronously and then update the data on the "cache" tables(Users, Inventory) in my billing service.
Is it also good to assume these "cache" tables are read-only? I assume they are since only the user/inventory services should be able to modify this information to preserve isolation and authority over the information.
You need to isolate the services as so they do not share state/data. The design in your question is a single macroservice split into 3 correlated storage silos. Case in point, you cannot interpret a result form the 'Invoicing' service w/o correlating the data with the 'Products' response(s).
Isolated microservices mean they own their data and they can operate independently. An invoice is complete as returned from the 'Invoices' service. It contains the product names, the customer name, every information on the invoice. All the data came from its own storage. A separate microservice could be 'Inventory', that operates all the product inventories, current stock etc. It would also have its own data, in its own storage. A 'product' can exist in both storage mediums, and there once was logical link between them (when the invoice was created), but the link is severed now. The 'Inventory' microservice can change its products (eg. remove one, add new SKUs etc) w/o affecting the existing Invoices (this is not only a microservice isolation requirement, is also a basic accounting requirement). I'm not going to enter here into details of what is a product 'identity' in real life.
If you find yourself asking questions like you're asking it likely means you do not have microservices. You should think at your microservice boundaries while considering what happens if you replace all communication with async queued based requests (a response can come 6 days later): If the semantics break, the boundary is probably wrong. If the semantics hold, is the right track.
It all depends on the resilience requirements that you have. Do you want your microservice to function when the other microservices are down or not?
The first solution that you presented is the less resilient: if any of the Users or Products microservices goes down, the Invoice microservice would also go down. Is this what you want? On the other hand, this architecture is the simplest. A variation of this architecture is to let the client make the join requests; this leads to a chatty conversation but it has the advantage that the client could replace the missing information with default information when the other microservices are down.
The second solution offers the biggest possible resilience but it's more complex. Having an event-driven architecture helps a lot in this case. In this architecture the microservices act as swimming lanes. A failure in one of the microservices does not propagate to other microservices.
Let's assume there are two domain entities:
UserImages with methods addNewImage(), removeImage($imageId), getImages($from, $count).
UserProfile with fields name, age, mainImageId, etc.
Following functionality is desired inside the domain: when application layer calls UserImages -> addNewImage(), UserProfile -> mainImageId is set automatically in case it was empty.
So, what is the best way and best place to implement an in-domain over-entity business logic? Domain events with observing services, referencing special services from the entities, or somewhat else?
I create all the entities using a some kind of factory, i.e.
$userImages = Domain::userImages($userId); // getting an instance of UserImages
$newImageId = $userImages -> addNewImage(); // adding a new image
I also should mention that I will have a lot of logic like described above in my project.
Thank you very much for help!
First off, as specified in your other question, UserImages is not itself an entity. Instead, there is likely an entity called UserImage which refers to a user's image, singular. A UserImageRepository can then provide access to all images associated with a user, using pagination where necessary.
Now, the use case is:
When a new user image is added, if a user's profile image is not set,
set it to the added image.
The term when indicates a domain event. In this case, the event is UserImageAdded. The bold if and the subsequent set represent the desired behavior in response to the event.
To implement this use case, an event must be published when a user image is added and a handler implementing the desired behavior must be subscribed to this event. This linked domain events article provides a sample implementation in C#, however the concept can be easily ported to PHP. Overall, the required components are:
An application service which handles the adding of user images. This service will publish the event.
A domain event publisher which is called by the application service and allows decoupling of publishers from subscribers.
A domain event handler which handles the UserImageAdded event and invokes the desired behavior.
User sounds like Aggregate for me (it's difficult to give you an opinion without context :) ), but it can solve your problem.
If Profil and Images are differents bounded context, may be you can use Domain events (it's helpful for cross bounded context communication)