I am trying to truncate a table in scylla which is a three node cluster. But immediately data is loading back, some times truncate working fine, which means I can able to delete the data not table schema using the following command, but some times data is loading back.
truncate table students ;
I tried to explore scylla and Cassandra documentations, they mentioned some thing because of tombstone filter this ghost replication happening, but my cluster tombstone grace period is default. So it should delete the data before tombstone expires. Anyone please help me why it is happening and what is the solution for this not to happen.
If truncate returned success (IOW, didn't fail or caused any exception), then your data should be gone. One explanation for what happened in your case is that the operation wasn't successful in some of the nodes. Truncate in this case should have returned a failure.
If it didn't return any failure and the data is still there, this is likely a malfunction and you should report that to the Scylla team in their bug tracker
Related
My application was working fine but today suddenly i keep getting this error from cassandra DB
com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency LOCAL_ONE (timeout while waiting for repair of inconsistent replica)
i read few articles but all of them were about (2 responses were required but only 1 replica responded) which not the this exception is about.
can someone explain what could be the reason how if possible how to resolve it??
So there are a few reasons that this message can suddenly start happening.
Querying from a new keyspace, which has the wrong data center name specified. If the data center name is incorrect, then Cassandra won't know how to route the query.
Querying too many rows. You haven't specified what your query is, which is important in diagnosing query timeouts. If you're using ALLOW FILTERING and/or a WHERE clause that is too liberal or not based on a partition key, then this can happen as the data set grows.
Querying a large partition. If your cluster has a "hot" partition or a partition that is subject to unbound growth, eventually that partition will get so big that it cannot be read.
I know this question is a bit old, but if you can edit your question with your query and table definition, that will help.
I want to understand how the truncate command works in Cassandra (version 3.9) to be able to know what would happen in the following scenario:
I have about 100GB of data on a table in production on a table that needs to be truncated.
I want to truncate this table, but at the same time there will be a few hundred requests per second that will be making inserts at the same time.
I am trying to understand, theoretically how would this play out.
Would the truncate try to acquire some sort of a lock on the table before it can proceed? and possibly stop the insert requests or itself be timed out?
Or would the truncate go through in sequence as the request came in and following insert requests would create the additional rows and I would end up with a small number of rows remaining after the truncate.
I am just trying to reclaim space, so I am not particularly concerned if a small amount of data remains from the insert requests run after the truncate command.
I am just trying to understand if you'd expect this to complete successfully or it would fail / time-out.
I will try to run a similar scenario on a smaller cluster, but I'm not sure if that will be a good substitute to understand the actual behavior. Any inputs will be helpful.
Truncate sends a message to all the nodes with a request to delete all the SSTables at the moment of execution, you will have information only of those upserts received after the truncate was issued.
In the Datastax documentation it is stated that this is done with JMX, but looking at the comments of this answer, this is done with CQL and the messaging service.
If you are trying to reclaim disk space, please note that a snapshot will be created with the truncate if auto_snapshot is set to true (true is the default value), so you will need to remove the snapshot after the execution of the command. Also, note that truncate will require to have all the nodes to be up and healthy to be able to complete.
I tried this for myself. On a 2 node Cassandra cluster I Made inserts at about 160 requests per second in the background and ran a truncate query on the same table that had about 200,000 records.
The table got truncated and the inserts continued without an error.
The new rows inserted after the truncate showed on the DB.
As I understand a row in a cassandra table is a Set of Key-value pairs (corresponding to each column)
I notice a strange issue during insert, values are not persisted in couple of columns, though I am fairly confident it has values before insert.
It happens sporadically and succeeds if we retry later. We are suspecting some kind of race condition or db connection drop etc.
Is it possible that only a subset of keys gets saved in a row of cassandra table ? Does cassandra guarantee all or nothing during save (row level consistency)
Cassandra Version : 2.1.8
Datastax cassandra-driver-core : 3.1.0
On the row level the concurrency guarantees are described pretty much in this answer.
Cassandra row level isolation
As far as your problem goes. First check if it's really cassandra with dropped mutations
nodetool tpstats
If you see dropped mutations, it's likely you are running underpowered setup and you simply have to put more hardware to the problem you are facing.
There isn't really more from your question that I can tell. Just as a precaution, please go into your code and check that you are actually creating a new bound statement every time and that you are not reusing the created bound statement instance. Once a client had this issue that the inserts were lost under mysterious circumstances and that was it. Hope this helps you, if not please give some code that you have.
There are consistency levels for read and writes in Cassandra.
It looks like you are using consistency level one, so your reads/writes are not consistent. Try to use quorum for both reads and writes and see if the problem resolves.
If this doesn't help, please provide example query, cluster size, rf factor.
We have a use case where we need to re-create a table every day with current data in Cassandra. For this should we use drop table or truncate table, which would be efficient? We do not want the data to be backed up etc?
Thanks
Ankur
I think for almost all cases Truncate is a safer operation than a drop recreate. There have been several issues with dropping/recreating in the past with ghost data, schema disagreement, ect... Although there have been a number of fixes to try to make drop/recreate more stable, if its an operation you are performing every day Truncate should be much cheaper and more stable.
Drop table drops the table and all data. Truncate clears all data in the table, and by default creates a snapshot of the data (but not the schema). Efficiency wise, they're close - though truncate will create the snapshot. You can disable this by setting auto_snapshot to false in cassandra yaml config, but it is server wide. If it's not too much trouble, I'd drop and recreate table - but I've seen issues if you don't wait a while after drop before recreating.
Source : https://support.datastax.com/hc/en-us/articles/204226339-FAQ-How-to-drop-and-recreate-a-table-in-Cassandra-versions-older-than-2-1
NOTE: By default, snapshots are created when tables are dropped or truncated. This will need to be cleaned out manually to reclaim disk space.
Tested manually as well.
Truncate will keep the schema though, drop will not.
Beware!
From datastax documentation: https://docs.datastax.com/en/archived/cql/3.3/cql/cql_reference/cqlTruncate.html
Note: TRUNCATE sends a JMX command to all nodes, telling them to delete SSTables that hold the data from the specified table. If any of these nodes is down or doesn't respond, the command fails and outputs a message like the following:
truncate cycling.user_activity;
Unable to complete request: one or more nodes were unavailable.
Unfortunately, there is nothing on the documentation saying if DROP behaves differently
I have a 3 nodes cluster, replicate_factor is 3 also. Consistency level is Write quorum, Read quorum.
Traffic has three major steps
Create:
Rowkey: xxxx
Column: status=new, requests="xxxxx"
Update:
Rowkey: xxxx
Column: status=executing, requests="xxxxx"
Delete:
Rowkey: xxxx
When one node down, it can work according to consistency configuration, and the final status is all requests are finished and deleted.
So if running cassandra client to list the result (also set consistency quorum). It shows empty (only rowkey left)， which is correct.
But if we start the dead node, the hinted handoff model will write back the data to this node. So there are lots of create, update, delete.
I don't know due to GC or compaction, the delete records on other two nodes seems not work, and if using cassandra client to list the data (also consistency quorum), the deleted row show again with column value. Due to the recovery node replay the history again.
And if using client to check the data several times, you can find the data is changed, seems hinted handoff replay operation, the deleted data show up and then disappear.
Is there a way to have this procedure invisible from external, until the hinted handoff finished?
What I want is final status synchronization, the temporary status is out of date and also incorrect, should never been seen from external.
Is it due to row delete instead of column delete? Or compaction?
After check the log and configuration, I found it caused by two reason.
GC grace seconds
I using hector client to connect cassandra, and the default value of GC grace seconds for each column family is Zero! So when hinted handoff replay the temporary value, the tombstone on other two node is deleted by compaction. And then client will get the temporary value.
Secondary index
Even after fix the first problem, I can still get temporary result from cassandra client. And I use the command like "get my_cf where column_one='value' " to query the data, then the temporary value show again. But when I using the raw key to query the record again, it disappeared.
And from client, we always using row key to get the data, and in this way, I didn't get the temporary value.
So it seems the secondary index is not restricted by the consistency configuration.
And when I change GC grace seconds to 10 days. our problem solved, but it is still a strange behavior when using index query.