Here, it is described, that delta lake uses Optimistic concurrency control by reading the current state, wirte all changes and validating if there are conflicts (which might end up in throwing an exception).
Here the isolation levels are described and as far as I understood, no table locks exist: so are the isolation levels just for describing when write operations fail? So as soon as I do a non-serializable operation I get an exception?
Related
What mechanisms does Delta Lake use to ensure the atomicity, consistency, isolation, and durability of transactions initiated by user operations on a DeltaTable?
0. the DeltaLog
Deltalog = Delta Lake's transaction log.
The deltalog is a collection of ordered json files. It acts as a single source of truth giving to users access to the last version of a DeltaTable's state.
1. Atomicity
Delta Lake breaks down every operation performed by an user into commits, themselves composed of actions.
A commit is recorded in the deltalog only once each of its actions has successfully completed (else it is reverted and restarted or an error is thrown), ensuring its atomicity.
2. Consistency
The consistency of a DeltaTable is guaranteed by their strong schema checking.
3. Isolation
Concurrency of commits is managed to ensure their isolation. An optimistic concurrency control is applied:
When a commit execution starts, the thread snapshots the current deltalog.
When the commit actions have completed, the thread checks if the Deltalog has been updated by another one in the meantime:
If not it records the commit in the deltalog
Else it updates its DeltaTable view and attempts again to register the commit, after a step of reprocessing if needed.
4. Durability
Commits containing actions that mutate the DeltaTable's data need to finish their writes/deletions on underlying Parquet files (stored on the filesystem) to be considered as successfully completed, making them durable.
Further readings:
Diving Into Delta Lake: Unpacking The Transaction Log
ACID properties
From the Microsoft documentation I don't fully understand whether or not CosmosDB using session consistency garantuees no out of order writes. The following quote makes it seem like it has the same garantuees as cosistent prefix:
The reads are guaranteed to honor the consistent-prefix (assuming a single “writer” session), ...
Although from the baseball example further down the page it seems like a reader could get a completely random order back, similar to eventual consistency. From other sources online I also can't find a definitive anwser, apart from the images shown on the Azure Portal that seem to implicitly suggest the same order as the writer.
(I'm from the Cosmos DB team)
A given client using session consistency will see its own writes in order, but see other clients' writes with eventual consistency (assuming using a different session token).
We're gonna update the docs to make that more clear. New text will read something like this:
Session: Within a single client session reads are guaranteed to honor the consistent-prefix (assuming a single “writer” session), monotonic reads, monotonic writes, read-your-writes, and write-follows-reads guarantees. Clients outside of the session performing writes will see eventual consistency.
Per my research, i think session consistency level can't guarantee the clients always read the value in the order.
My evidence is from this link:
When the consistency level is set to bounded staleness, Cosmos DB
guarantees that the clients always read the value of a previous
write, with a lag bounded by the staleness window.
When the consistency level is set to strong, the staleness window is
equivalent to zero, and the clients are guaranteed to read the latest
committed value of the write operation.
For the remaining three consistency levels, the staleness window is
largely dependent on your workload. For example, if there are no
write operations on the database, a read operation with eventual,
session, or consistent prefix consistency levels is likely to yield
the same results as a read operation with strong consistency level.
As above said,the staleness window is dependent on your actual workload if you choose Session Consistency Level. So,if you do concern about the read order, i suggest you using bounded staleness or even Strong Consistency Level.
Query 1: Event data from device is stored in Cassandra table. Obviously this is time series data. If we need to store how older dated events (if cached in device due to some issue) at current time, are we going to get performance issue? If yes, what is the solution to avoid that?
Query 2: Is it good practice to write the event into Cassandra table as soon as the event comes in? Or shall we queue it for sometime to write multiple events in one go if that improves Cassandra write performance significantly?
Q1: this all depends on the table design. Usually this shouldn't be an issue, but this may depend on your access patterns & compaction strategy. If you have table structure, please share it.
Q2: Individual writes shouldn't be a problem, but it really depends on your requirements for throughput. If you'll write several data points that belong to the same partition key you potentially may use unlogged batches, and in this case Cassandra will perform only one write for several inserts that are in this batch. Please read this document.
What is the mechanism to achieve Row-level locking in Cassandra ? What I want to do is to allow only one process to modify a given row at any given time.
Cassandra does not provide locking. It does provide lightweight transactions, which can replace locking in some cases. Also note that operations on a single row are atomic, so locking a row is not necessarily to ensure a read or write of a row provides consistent field values for that row.
Cassandra does not provide locking because of a fundamental constraint on all kinds of distributed data store: a distributed data store can not ensure consistency while also providing performance and availability, but it can provide two of those three properties. The design of Cassandra chooses not to provide consistency, so it can provide high performance and availability. Locking is about consistency. So Cassandra could not provide locking without sacrificing performance and/or availability. As that would go against the design goals of Cassandra, it is a good bet that Cassandra will never have locking functionality.
Do you really need locking? You might be surprised that it is not as often necessary as you might think, if you have an RDBMS background. If you do need it, you must choose a different data-store, which is designed to provide consistency at the cost of either performance or availability (or both).
From:
Invariants, which are consistency rules that must be maintained
whenver data changes, will involve relationships between members of
the AGGREGATE. Any rule that spans AGGREGATES will not be expected to
be up to date at all times. Through event processing, batch
processing, or other update mechanisms, other dependencies can be
resolved within some specified time. But the invariants applied within
an AGGREGATE will be enforced with the completion of each transaction.
a) I interpret this as saying that rule designed to maintain consistency between several Aggregates, doesn't have to be enforced at the time one of these Aggregates is saving its changes to some persistence storage, but instead can be enforced at some later time when this Aggregate already completes its transaction with persistence storage?
b) But why is such behavior tolerated, since it will lead to inconsistent/corrupt data?
Thank you
a). Correct. An aggregate can be defined as a consistency boundary. Things outside of that boundary, such as other aggregates, are those entities which can be acceptably updated as part of a subsequent transaction. This is eventual consistency.
b). Often times, temporarily stale data is acceptable from the business perspective and should be discussed explicitly. Also, inconsistency actually manifests in places typically regarded as consistent. For instance, as soon as you request data from some service and receive it, the data has the potential to be inconsistent unless there is an ambient transaction. If the requested resource is remote, the transaction will have to be distributed and thus subject to all of the pitfals of 2PC.
Take a look at Udi Dahan's article on Event-Driven Architecture which discussed eventual consistency from the business perspective.