I am struggling with a problem where a straightforward combination of a java app (that prepares sstables using the CQLSSTableWriter api) in combination with sstableloader fails to insert all rows.
The only suspect message I see during the creation of the sstables is
[Reference-Reaper:1] ERROR o.a.cassandra.utils.concurrent.Ref - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State#4651731d) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy#1452490935:Memory#[7f89dc05adc0..7f89dc05dfc0)
was not released before the reference was garbage collected
The sstableloader does not list anything suspect. After the load completes the number of rows does not match.
I checked key uniqueness and that does not seem to be the issue.
Anyone any thoughts on how to go about fixing this?
Many thanks indeed!
Peter
Related
I'm running OS Cassandra 3.11.9 with Datastax Java Driver 3.8.0. I have a Cassandra keyspace that has multiple tables functioning as lookup tables / search indices. Whenever I receive a new POST request to my endpoint, I parse the object and insert it in the corresponding Cassandra table. I also put inserts to each corresponding lookup table. (10-20 per object)
When ingesting a lot of data into the system, I've been running into WriteTimeoutExceptions in the driver.
I tried to serialize the insert requests into the lookup tables by introducing Apache Camel and putting all the Statements into a queue that the Session could work off of, but it did not help.
With Camel, since the exceptions are now happening in the Camel thread, the test continues to run, instead of failing on the first exception. Eventually, the test seems to crash Cassandra. (Nothing in the Cassandra logs though)
I also tried to turn off my lookup tables and instead insert into the main table 15x per object (to simulate a similar number of writes as if I had the lookup tables on). This test passed with no exception, which makes me think the large number of tables is the problem.
Is a large number (2k+) of Cassandra tables a code smell? Should we rearchitect or just throw more resources at it? Nothing indicative has shown in the logs, mostly just some status about the number of tables etc - no exceptions)
Can the Datastax Java Driver be used multithreaded like this? It says it is threadsafe.
There is a direct effect of the high number of tables onto the performance - see this doc (the whole series is good source of information), and this blog post for more details. Basically, with ~1000 tables, you get ~20-25% degradation of performance.
That's could be a reason, not completely direct, but related. For each table, Cassandra needs to allocate memory, have a part for it in the memtable, keep information about it, etc. This specific problem could come from the blocked memtable flushes, or something like. Check the nodetool tpstats and nodetool tablestats for blocked or pending memtable flushes. It's better to setup some continuous monitoring solution, such as, metrics collector for Apache Cassandra, and and for period of time watch for the important metrics that include that information as well.
I am trying to truncate a table in scylla which is a three node cluster. But immediately data is loading back, some times truncate working fine, which means I can able to delete the data not table schema using the following command, but some times data is loading back.
truncate table students ;
I tried to explore scylla and Cassandra documentations, they mentioned some thing because of tombstone filter this ghost replication happening, but my cluster tombstone grace period is default. So it should delete the data before tombstone expires. Anyone please help me why it is happening and what is the solution for this not to happen.
If truncate returned success (IOW, didn't fail or caused any exception), then your data should be gone. One explanation for what happened in your case is that the operation wasn't successful in some of the nodes. Truncate in this case should have returned a failure.
If it didn't return any failure and the data is still there, this is likely a malfunction and you should report that to the Scylla team in their bug tracker
I'm running a Cassandra 3.9 cluster, and today I noticed some NULL values in some generated reports.
I opened up cqlsh and after some queries I noticed that null values are appearing all over the data, apparently in random columns.
Replication factor is 3.
I've started a nodetool repair on the cluster but it hasn't finished yet.
My question is: I searched for this behavior and could not find it anywhere. Apparently the random appearance of NULL values in columns is not a common problem.
Does anyone know what's going on? This kind of data corruption seems pretty serious. Thanks in advance for any ideas.
ADDED Details:
Happens on columns that are frequently updated with toTimestamp(now()) which never returns NULL, so it's not about null data going in.
Happens on immutable columns that are only inserted once and never changed. (But other columns on the table are frequently updated.)
Do updates cause this like deletions do? Seems kinda serious to me, to wake up to a bunch of NULL values.
I also know specifically some of the data that has been lost, three entries I've already identified are for important entries which are missing. These have not been deleted for sure - there is no deletion on one specific table which is full of NULL everywhere.
I am the sole admin and nobody ran any nodetool commands overnight, 100% sure.
UPDATE
nodetool repair has been running for 6+ hours now and it fully recovered the data on one varchar column "item description".
It's a Cassandra issue and no, there were no deletions at all. And like I said functions which never return null had null in them(toTimestamp(now())).
UPDATE 2
So nodetool repair finished overnight but the NULLs were still there in the morning.
So I went node by node stopping and restarting them and voilà, the NULLs are gone and there was no data loss.
This is a major league bug if you ask me. I don't have the resources now to go after it, but if anyone else faces this here's the simple "fix":
Run nodetool repair -dcpar to fix all nodes in the datacenter.
Restart node by node.
I faced a similar issue some months ago. It's explained quite good in the following blog. (This is not written by me).
The null values actually have been caused by updates in this case.
http://datanerds.io/post/cassandra-no-row-consistency/
Mmmh... I think that if this was a Cassandra bug it would already be reported. So I smell code bug in your application, but you didn't post any code, so this will remain only a (wild) guess until you provide some code (i'd like to have a look at the update code).
You don't delete data, nor use TTL. It may seem there are no other ways to create NULL values, but there's one more tricky one: failing at binding, that is explictly binding to NULL. It may seem strange, but it happens...
Since
...null values are appearing all over the data...
I'd expect to catch this very fast enabling some debugging or assert code on the values before issuing any updates.
check the update query if it updates only the columns necessary, or it does it through Java beans which includes the list of all columns in the table. This would explain the NULL updates for other columns which weren't desired to be updated.
I created a columnstore table in memsql and populated it with around 10 million records after which I started running several update scenarios. I noticed that the size of the data in /var/lib/memsql/leaf-3307/data/columns keeps increasing constantly and nothing there seems to be deleted. Initially the size of that folder is a couple hundred Mb but it quickly jumps to a couple of Gb after some full table updates. The "Columnstore Disk Usage" reported by memsql-ops also increases but at a very slow pace (far from what I see on disk).
This makes me think that data is never actually deleted from disk. The documentation states that running the OPTIMIZE commands should compact the row segment groups and that deleted rows would be removed:
Delete - Deleting a row in a columnstore index causes the row to be marked as deleted in the segment meta data leaving the data in place within the row segment. Segments which only contain deleted rows are removed, and the optimization process covered below will compact segments that require optimization.
Running the OPTIMIZE command didn't help. I also tried truncating the table and even dropping it but nothing helped. The data in the columns folder is still there. The only way I could find of cleaning that up is to DROP the entire database.
This doesn't seem like the desired behavior and I can't find any documentation justifying it. Can anybody explain why this is happening, if it should happen or point me to some relevant documentation?
Thanks in advance
MemSQL will keep around columnstore_window_size bytes of deleted columnstore data on disk per partition database. This is part of the implementation of columnstore replication (it keeps some old files around in case slaves are behind). If you lower the value of that system variable you'll see the disk usage drop. If your not using redundancy 2 there is no harm in lowering it.
In our production environment we are using CQLSStablWriter to create SSTables for future bulk load Sample Gist. After the process of sstable creation. Once we boot cassandra up, we were getting following exceptions. All our CFs are counter column families.
Versions Tried 2.1.5 and 2.0.8
Exception 1 and Exception 2
When looking up the code, we found that in ColumnFamily.java, cellInternal is of type BufferedCounterUpdateCell, whose diff operation is not supported. As well reconcile method is called on a wrong type too.
Just to confirm the issue, we did use the sstable2json utility to inspect the data. And as we guessed
Cells of a properly dumped sstable had valid counters
(":1:c05:desktop/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/","00010000db5ad3b00d4711e5b52dab7bf928868d00000000000000590000000000000059",1433835586985,"c",-9223372036854775808),
(":1:c05:*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/","00010000db5ad3b00d4711e5b52dab7bf928868d00000000000000590000000000000059",1433835586985,"c",-9223372036854775808)
whereas the faulty ones had buffered counters and hence the sstable2json failed
("*:0:c01:*/direct/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/","0000000000000001",1433924262793),
("*:0:c01:*/*/singapore/*/*/*/*/*/*/*/*/*/*/*/*/*/*/*/","0000000000000002",1433924262793),
Basically sstable2json doesn't have support for dumping BufferedCounterUpdateCells, Hence it assumes such cells to be of normal type and dumps them.
It is evident from error logs and sstable2json output, that instead of dumping CounterColumns, CQLSSTableWriter dumped Counter Update Columns counter types with a different mask, which resulted in error while cassandra tried to load SSTables with such columns up.
We see this issue happening always when SSTable get created via CQLSSTableWriter.
While going through any issues reported on the same note, we stalled on this. Something related to switching CFs for writing. We guess problem could be on the same lines. Any inputs are welcome.
----Update
On further debugging we figured out that CQLSSTableWriter fails to convert CounterUpdateColumns to CounterColumns as it is usually done in all other cases row mutations. Might need a patch
Counters are not supported on CQLSStableWriter. See https://issues.apache.org/jira/browse/CASSANDRA-10258 for more information.