I'm experimenting with event sourcing / cqrs pattern using serverless architecture in Azure.
I've chosen Cosmos DB document database for Event Store and Azure Event Grid for dispachting events to denormalizers.
How do I achieve that events are reliably delivered to Event Grid exactly once, when the event is stored in Cosmos DB? I mean, if delivery to Event Grid fails, it shouldn't be stored in the Event Store, should it?
Look into Cosmos Db Change Feed. Built in event raiser/queue for each change in db. You can register one or many listeners/handlers. I.e. Azure functions.
This might be exactly what you are asking for.
Some suggest you can go directly to cosmos db and attach eventgrid at the backside of changefeed.
You cannot but you shouldn't do it anyway. Maybe there are some very complicated methods using distributed transactions but they are not scalable. You cannot atomically store and publish events because you are writing to two different persistences, with different transactional boundaries. You can have a synchronous CQRS monolith, but only if you are using the same technology for the events persistence and readmodels persistence.
In CQRS the application is split in Write/Command and Read/Query sides (this long video may help). You are trying to unify the two parts into a single one, a downgrade if you will. Instead you should treat them separately, with different models (see Domain driven design).
The Write side should not depend on the outcome from the Read side. This means, that after the Event store persist the events, the Write side is done. Also, the Write side should contain all the data it needs to do its job, the emitting of events based on the business rules.
If you have different technologies in the Write and Read part then your Read side should be decoupled from the Write side, that is, it should run in a separate thread/process.
One way to do this is to have a thread/process that listens to appends to the Event store, fetch new events then publish them to the Event Grid. If this process fails or is restarted, it should resume from where it left off. I don't know if CosmosDB supports this but MongoDB (also a document database) has the rslog that you can tail to get the new events, in a few milliseconds.
Related
So I was watching this video Event Sourcing You are doing it wrong by David Schmitz at the 15:17 he was talking about eventual consistency in event-sourcing. At first, I was like oh I got it so this is why CQRS is helpful with Event-sourcing because we can validate this things through command model before publishing an event right? but after I did a few research I was wrong. So I wonder why command model even exists since we can just retrieve the request body (suppose it's http request) put some business logic and then publish event.
With Event Sourcing we store events. A model exists in your application to support applying your business logic before deciding to save new event(s). To be able to make that decision, it must be possible to consistently read/write events to materialize your model before applying your business logic on it when a command is processed.
You need to store the events consistently to be able to make decisions further on. If you only publish your events to other parts of the system your model cannot use them in a consistent way.
The publishing of events to other systems is something that potentially can happen as a side-effect to use these events to also create read-models/projections or to react to them in other ways.
It does not have to be a push/publish though. It is perfectly valid to have a pull-based solution where downstream systems poll for events.
For example, in Serialized we store events in Aggregates. Feeds are used to provide a poll-based (eventually-consistent) downstream view of these events, but there is no publishing at all.
I have a large message submitted to a rest service, it could be 100k or 50mb. I need to process it asynchronously, and it looks like the transactional outbox event pattern is suitable for my needs.
Effectively, my service would commit the data to a database, along with an event record in a single transaction. A process would poll for events, and push the event to a message queue. THe event contains a reference to the data in the db, usually through a unique identifier of some sort. The consumer of the queue would query the data from the database, do whatever it needs to do and then remove the event record from the database.
This pattern is well documented. Here and here are two places.
I have a reasonable understanding of how one could implement this on-premis, in a .net/sql server environment that we are familiar with. In azure what would this look like? are there other ways I can transitionally write to the database and a queue that do not require the outbox pattern, or following the outbox pattern, what would be the mechanism that polls for events in the db, and what would provide the queue service?
Usually if you want to use the transactional outbox event pattern in azure you can use a logic app or an azure function to get events in the db and send them to the queue.
Doing that would be great if you use cosmos change feed so that your architecture is also reactive and will perform well with less resource consumption.
To avoid this pattern well....you should find a queue in azure that is able to be in transaction with your db and for what I know is not possible at least that you don ' t use a 3rd part queue .
I'm working on a project where DynamoDB is being used as database and every use case of the application is triggered by a message published after an item has been created/updated in DB. Currently the code follows this approach:
repository.save(entity);
messagePublisher.publish(event);
Udi Dahan has a video called Reliable Messaging Without Distributed Transactions where he talks about a solution to situations where a system can fail right after saving to DB but before publishing the message as messages are not part of a transaction. But in his solution I think he assumes using a SQL database as the process involves saving, as part of the transaction, the correlationId of the message being processed, the entity modification and the messages that are to be published. Using a NoSQL DB I cannot think of a clean way to store the information about the messages.
A solution would be using DynamoDB streams and subscribe to the events published either using a Lambda or another service to transformed them into domain-specific events. My problem with this is that I wouldn't be able to send the messages from the domain logic, the logic would be spread across the service processing the message and the Lambda/service reacting over changes and the solution would be platform-specific.
Is there any other way to handle this?
I can't say a specific solution based on DynamoDB since I've not used this engine ever. But I've built an event driven system on top of MongoDB so I can share my learnings you might find useful for your case.
You can have different approaches:
1) Based on an event sourcing approach you can just save the events/messages your use case produce within a transaction. In Mongo when you are just inserting/appending new items to the same collection you can ensure atomicity. Anyway, if the engine does not provide that capability the query operation is so centralized that you are reducing the possibility of an error at minimum.
Once all the events are stored, you can then consume them and project them to a given state and then persist the updated state in another transaction.
Here you have to deal with eventual consistency as data will be stale in your read model until you have projected the events.
2) Another approach is applying the UnitOfWork pattern where you cache all the query operations (insert/update/delete) to save both events and the state. Once your use case finishes, you execute all the cached queries against the database (flush). This way although the operations are not atomic you are again centralizing them quite enough to minimize errors.
Of course the best is to use an ACID database if you require that capability and any other approach will be a workaround to get close to it.
About publishing the events I don't know if you mean they are published to a messaging transportation mechanism such as rabbitmq, Kafka, etc. But that must be a background process where you fetch the events from the DB and publishes them in order to break the 2 phase commit within the same transaction.
I want to use event sourcing and CQRS, and so I need projections (I hope I use the proper term) to update my query databases. How can I handle database errors?
For example one of my query cache databases is not available, but I already updated the others. So the not-available database won't be in snyc with the others when it comes back to business. How will it know that it have to run for instance the last 10 domain events from the event storage? I guess I have to store information about the current state of the databases, but what if that database state storage fails? Any ideas, best practices how to solve this kind of problems?
In either case, you must tell your messaging bus that the processing failed and it should redeliver the event later, in the hope that the database will be back online then. This is essentially why we are using message bus systems with an "at least once"-delivery guarantee.
For transactional query databases, you should also rollback the transaction, of course. If your query database(s) do not support transactions, you must make sure on the application side that updates are idempotent - i.e., if your event arrives on the next delivery attempt, your projection code and/or database must be designed such that the repeated processing of the event does not harm the state of the database. This is sometimes trivial to achieve (e.g., when the event leads to a changed person's name in the projection), but often not-so-trivial (e.g., when the projection simply increments view counts). But this is what you pay for when you are using non-transactional databases.
We recently started developing apps using concepts from both DDD and CQRS (no Event Sourcing). I am still somewhat confused as where exactly to call the repository to persist my Aggregate Roots.
Do I do it inside Command Handlers or do I do it in the Event Handlers?
You do this inside your command handler. You save your domain objects as normal. Even if you are not using event sourcing as a means of persisting your domain entities, you will still need to fire events that your query service will subscribe to. The event handlers on the read side will then update de-normalised tables tailor made for the UI screens. So basically you have two sets of data access code: one for the domain, one for the query service (read side). Its less work if you make use of event sourcing to persist your domain entities...