As I understand a row in a cassandra table is a Set of Key-value pairs (corresponding to each column)
I notice a strange issue during insert, values are not persisted in couple of columns, though I am fairly confident it has values before insert.
It happens sporadically and succeeds if we retry later. We are suspecting some kind of race condition or db connection drop etc.
Is it possible that only a subset of keys gets saved in a row of cassandra table ? Does cassandra guarantee all or nothing during save (row level consistency)
Cassandra Version : 2.1.8
Datastax cassandra-driver-core : 3.1.0
On the row level the concurrency guarantees are described pretty much in this answer.
Cassandra row level isolation
As far as your problem goes. First check if it's really cassandra with dropped mutations
nodetool tpstats
If you see dropped mutations, it's likely you are running underpowered setup and you simply have to put more hardware to the problem you are facing.
There isn't really more from your question that I can tell. Just as a precaution, please go into your code and check that you are actually creating a new bound statement every time and that you are not reusing the created bound statement instance. Once a client had this issue that the inserts were lost under mysterious circumstances and that was it. Hope this helps you, if not please give some code that you have.
There are consistency levels for read and writes in Cassandra.
It looks like you are using consistency level one, so your reads/writes are not consistent. Try to use quorum for both reads and writes and see if the problem resolves.
If this doesn't help, please provide example query, cluster size, rf factor.
Related
Consider cluster have 3 nodes with Keyspaces A having replicas=3 and consistency is quorom
When 2 nodes goes down, This would throw exception that consistency level doesn't met. Does 3rd node acts as a coordinator node and save the hint for some period and replay it once they are available?
Whether hints are managed even after consistency condition doesn't met?
The write will not be tried at all because there are not enough replicas available to satisfy the required consistency.
Even if the last surviving node gets picked as the coordinator, it will not store a hint. This is explained in a bit more detail in Hinted Handoff: repair during write path. We also have a video extracted from the DS201 Cassandra Foundations course at the DataStax Academy explaining the key concepts that might be of interest to you. Cheers!
No, the hint is not saved because the CL is not met. Hints are not included in the CL, so since you have 2 nodes out of 3 down, the quorum CL is not met and the driver will receive an UnavailableException from the coordinator.
I'm creating a sync program to periodically copy our Cassandra data into another database. The database I'm copying from only gets INSERTs - data is never UPDATEd or DELETEd. I would like to address Cassandra's eventual consistency model in two ways:
1 - Each sync scan overlaps the last by a certain time span. For example, if the scan happens every hour, then each scan looks an hour and a half backwards. The data contains a unique key, so reading the same record in more than one scan is not an issue.
2 - I use a Consistency level of ALL to ensure that I'm scanning all of the nodes on the cluster for the data.
Is ALL the best Consistency for this situation? I just need to see a record on any node, I don't care if it appears on any other nodes. But I don't want to miss any INSERTed records either. But I also don't want to experience timeouts or performance issues because Cassandra is waiting for multiple nodes to see that record.
To complicate this a bit more, this Cassandra network is made up of 6 clusters in different geographic locations. I am only querying one. My assumption is that the overlap mentioned in #1 will eventually catch up records that exist on other clusters.
The query I'm doing is like this:
SELECT ... FROM transactions WHERE userid=:userid AND transactiondate>:(lastscan-overlap)
Where userid is the partioning key and transactiondate is a clustering column. The list of userId's is sourced elsewhere.
I use a Consistency level of All to ensure that I'm scanning all of the nodes on the cluster for the data
So consistency ALL has more to do with the number of data replicas read than it does with the number of nodes contacted. If you have a replication factor (RF) of 3 and query a single row at ALL, then Cassandra will hash your partition key to figure out the three nodes responsible for that row, contact all 3 nodes, and wait for all 3 to respond.
I just need to see a record on one node
So I think you'd be fine with LOCAL_ONE, in this regard.
The only possible advantage of using ALL, is that it actually does help to enforce data consistency by triggering a read repair 100% of the time. So if eventual consistency is a concern, that's a "plus." But *_ONE is definitely faster.
The CL documentation talks a lot about 'stale data', but I am interested in 'new data'
In your case, I don't see stale data as a possibility, so you should be ok there. The issue that you would face instead, is in the event that one or more replicas failed during the write operation, querying at LOCAL_ONE may or may not get you the only replica that actually exists. So your data wouldn't be stale vs. new, it'd be exists vs. does not exist. One point I talk about in the linked answer, is that perhaps writing at a higher consistency level and reading at LOCAL_ONE might work for your use case.
A few years ago, I wrote an answer about the different consistency levels, which you might find helpful in this case:
If lower consistency level is good then why we need to have a higher consistency(QUORUM,ALL) level in Cassandra?
Sorry if this is an existing question, but any of the existing ones resolved my problem..
I've installed Cassandra single noded. I don't have a large application right now, but I think this can be the case soon, and I will need more and more nodes..
Well, I'm saving data from a stream to Cassandra, and this were going well, but suddently, when I tried to read data, I've started to receive this error:
"Not enough replica available for query at consistency ONE (1 required but only 0 alive)"
My keyspace was built using simplestrategy with replication_factor = 1. Im saving data separated by a field called "catchId", so most of my queries are like: "select * from data where catchId='xxx'". catchId is a partition key.
I'm using the cassandra-driver-core version 3.0.0-rc1.
The thing is that I don't have that much of data rigth now, and I'm thinking if it will be better to use a RDBMS for now, and migrate to Cassandra only when I have a better infrastructure.
Thanks :)
It seems that your node is unable to respond when you try to make your read (in general this error appears for more than one node). If you do not have lots of data, it's very strange, so this is probably a bad design choice. This can emanate from several things, so you have to make a few investigations.
study your logs ! In particular the system.log
you can change your read_request_timeout_in_ms parameter in cassandra.yaml. Although it's not agood idea in production, it will say you if it's just temporary problem (your request succeed after a little time) or a bigger problem
study your CPU and memory behavior when you are doing requests
if you are very motivated, you can install opscenter which will you give more valuable informations
How and how many write requests are you doing ? It can overwhelm cassandra (even if it's designed for). I recommend to make async requests to avoid problems.
I have 3 similar cassandra tables. Table A1, A2, A3.
All have same columns, but different partition keys.
Data is inserted in all three tables at same time through sequential inserts using Mapper Library (cassandra-driver-mapping-2.1.8.jar)
However, there has been inconsistency in few columns.
E.g. Sometimes A1.colX and A2.colX are same but A3.colX is having old value(not updated) and rest all columns in these three tables have exactly same value.
Another time A1.colY and A3.colY may have same value but A2.colY is having old value(not updated) and rest all columns in these three tables have exactly same value.
I am using Mapper Manager to save the input data in Cassandra.
Is it a known problem with mapper manager or something wrong in my approach?
Sample code:
public void insertInTables(String inputString){
.
.
ClassNameA1 classObjectA1=new Gson().fromJson(inputString, ClassNameA1.class);
ClassNameA2 classObjectA2=new Gson().fromJson(inputString, ClassNameA2.class);
ClassNameA3 classObjectA3=new Gson().fromJson(inputString, ClassNameA3.class);
MappingManager manager = new MappingManager(session);
Mapper<ClassNameA1> mapperA1 = manager.mapper(ClassNameA1.class);
Mapper<ClassNameA2> mapperA2 = manager.mapper(ClassNameA2.class);
Mapper<ClassNameA3> mapperA3 = manager.mapper(ClassNameA3.class);
mapperA1.save(classObjectA1);
mapperA2.save(classObjectA2);
mapperA3.save(classObjectA3);
.
.
}
It might happen as Cassandra is an eventual consistency store, not strong one. Typical reasons for similar behaviour I've witnessed in my experience are:
issues with read/write consistency level. If you have RF=3, but write data with CL=ONE, some nodes might fail to replicate your value on write for some reason (like network/hw glitch). Then if you read with CL=QUORUM (or ONE), the quorum may decide to show you the old column value because the new one not propagated to all nodes correctly. So make sure you're writing with CL=ALL/QUORUM and reading with CL=QUORUM.
issues with hinted hand-off (which is used to protect you from previous issue). Once I've observed a strange behaviour when the first column read was stale/inconsistent (in 1% of all queries), but the second (or third) one shown the correct column value all the time. So try to re-read your inconsistent column multiple times (and think about possible hw/net failures before).
internal database errors got due to hardware failure or the Cassandra itself.
Most of the issues described above are possible to fix with nodetool repair. You can do a full repair and see if this helps.
Read-your-own-writes consistency is great improvement from the so called eventual consistency: if I change my profile picture I don't care if others see the change a minute later, but it looks weird if after a page reload I still see the old one.
Can this be achieved in Cassandra without having to do a full read-check on more than one node?
Using ConsistencyLevel.QUORUM is fine while reading an unspecified data and n>1 nodes are actually being read. However when client reads from the same node as he writes in (and actually using the same connection) it can be wasteful - some databases will in this case always ensure that the previously written (my) data are returned, and not some older one. Using ConsistencyLevel.ONE does not ensure this and assuming it leads to race conditions. Some test showed this: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/per-connection-quot-read-after-my-write-quot-consistency-td6018377.html
My hypothetical setup for this scenario is 2 nodes, replication factor 2, read level 1, write level 1. This leads to eventual consistency, but I want read-your-own-writes consistency on reads.
Using 3 nodes, RF=3, RL=quorum and WL=quorum in my opinion leads to wasteful read request if I being consistent only on "my" data is enough.
// seo: also known as: session consistency, read-after-my-write consistency
Good question.
We've had http://issues.apache.org/jira/browse/CASSANDRA-876 open for a while to add this, but nobody's bothered finishing it because
CL.ONE is just fine for a LOT of workloads without any extra gymnastics
Reads are so fast anyway that doing the extra one is not a big deal (and in fact Read Repair, which is on by default, means all the nodes get checked anyway, so the difference between CL.ONE and higher is really more about availability than performance)
That said, if you're motivated to help, ask on the ticket and I'll be happy to point you in the right direction.
I've been following Cassandra development for a little while and I haven't seen a feature like this mentioned.
That said, if you only have 2 nodes with a replication factor of 2, I would question whether Cassandra is the best solution. You are going to end up with the entire data set on each node, so a more traditional replicated SQL setup might be simpler and more widely tested. Cassandra is very promising but it is still only version 0.8.2 and problems are regularly reported on the mailing list.
The other way to solve the 'see my own updates' problem would be to cache the results somewhere closer to the client, whether in the web server, the application layer, or using something like memcached.