We use Cassandra 2.1.5 in a small dev environment (2 DCs, 3 nodes each).
We don't have much space on dev machines and almost every day face disk space errors. Main culprit is system_traces keyspace :
.../system_traces]# du -sh
8.1G .
I tried to turn tracing off in cqlsh:
cqlsh> tracing off;
Tracing is not enabled.
I tried nodetool settraceprobability 0 - but still tables are getting populated.
I can't delete tables and keyspace:
cqlsh> drop keyspace system_traces;
Unauthorized: code=2100 [Unauthorized] message="Cannot DROP <keyspace system_traces>"
The only working solution is "truncate system_traces.sessions; truncate system_traces.events;" - but those tables are getting full of rows quite soon.
How do I disable it once and for all?
Theres a chance the trace probability was enabled too, You can disable it via nodetool at runtime:
nodetool settraceprobability 0
but would have to be done on each node. You can truncate the events/sessions tables
cqlsh> truncate system_traces.events;
cqlsh> truncate system_traces.sessions;
but may want to then clear snapshots if truncate triggered one.
nodetool clearsnapshot system_traces
You must still have tracing enabled somewhere. Try to look at a few rows, that might give you a hint as to what generates these traces. For instance, when I trace a CQL query manually, the query string appears in sessions.parameters.
nodetool settraceprobability 1 helped me to start tracing and getting data in those system_traces tables like sessions and events. Earlier it was not showing any data for me. I did same step on all nodes. So that make it clear that nodetool settraceprobability 0 should stop the logging if we do it on all tables.
Related
My OpsCenter give me 'Failed' result on Tombstone count performance service. I read this paper and find that may be the insertion of NULL value is the casual.
So I try to fix this problem using the following procedures:
Set the NULL column of table channels and articles to ''. And for checking reason, there is no any insertings to these two tables.
Set gc_grace_seconds to 0 using commands:
alter table channels with gc_grace_seconds = 0
alter table articles with gc_grace_seconds = 0
Truncate bestpractice_results table in OpsCenter keyspace.
Restart agents and OpsCenter using commands:
service datastax-agent restart
service opscenterd restart
But, when OpsCenter run routine performance check (every 1 minute), the following 'Failed' information appeared again. And the number of tombstones is not changed (i.e., 23552 and 1374)
And I have the question:
How to remove these tombstones when there is no any insertion operations on two tables ?
Do I need repair the cluster ?
OpsCenter Version: 6.0.3 Cassandra Version: 2.1.15.1423 DataStax Enterprise Version: 4.8.10
With Cassandra 3.10+, use
nodetool garbagecollect keyspace_name table_name
Check https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsGarbageCollect.html
Please go through below link to get complete info about Delete and Tombstone.. It may be helpful for you.
http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html
I noticed an increase in the number of open files on my cassandra cluster and went to check the health of it. Nodetool status reported only 300gb in use per node of the 3TB each has allocated.
Shortly there after i began to see HEAP OOM errors showing up in the cassandra logs.
These nodes had been running for 3-4 months no issue, but had a series of test data populate and then dropped from them.
After checking the harddrives via the df command i was able to determine they were all between 90-100% filled in a jboded scenario.
edit: further investigation shows that the remaining files are in the 'snapshot' subfolder and the data subfolder itself has no db tables.
My question is, has anyone seen this? Why did compaction not free these tombstones? Is this a bug?
Snapshots aren't tombstones - they are a backup of your data.
As Highstead says you can drop any unused snapshots via the clearsnapshot command.
You can disable the automatic snapshot facility via the cassandra.yaml
https://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html#reference_ds_qfg_n1r_1k__auto_snapshot
Also check if you may have non-default true for snapshot_before_compaction
Snapshots occur over the lifetime of the cassandra cluster. These snapshots are not captured in a nodetool status but still occupy space. In this case the snapshots consuming all the space were created when a table was dropped.
To retrieve a list of current snapshots use the command nodetool listsnapshots
This feature can be disabled through editing /etc/cassandra/cassandra-env.sh and setting auto_snapshot to false. Alternatively these snapshots can be purged via the command nodetool clearsnapshot <name>.
I created a keyspace and a table(columnfamily) within it.
Let's say "ks.cf"
After entering few hundred thousand rows in the columnfamily cf, I saw the disk usage using df -h.
Then, I dropped the keyspace using the command DROP KEYSPACE ks from cqlsh.
After dropping also, the disk usage remains the same. I also did nodetool compact, but no luck.
Can anyone help me out in configuring these things so that disk usage gets freed up after deleting the data/rows ?
Ran into this problem recently. After dropping a table a snapshot is made. This snapshot will allow you to roll this back if this was not intended. If you do however want that harddrive space back you need to run:
nodetool -h localhost -p 7199 clearsnapshot
on the appropriate nodes. Additionally you can turn snapshots off with auto_snapshot: false in your cassandra.yml.
edit: spelling/grammar
If you are just trying to delete rows, then you need to let the deletion step go through the usual delete cycle(delete_row->tombstone_creation->compaction_actually_deletes_the_row).
Now if you completely want to get rid of your keyspace, check your cassandra data folder(it should be specified in your yaml file). In my case it is "/mnt/cassandra/data/". In this folder there is a subfolder for each keyspace(i.e. ks ). You can just completely delete the folder related to your keyspace.
If you want to keep the folder around, it is good to know that cassandra creates a snapshot of your keyspace before dropping it. Basically a backup of all of your data. You can just go into 'ks' folder, and find the snapshots subdirectory. Go into the snapshots subdirectory and delete the snapshot related to your keyspace drop.
Cassandra does not clear snapshots automatically when your are dropping a table or keyspace. if you enabled auto_snapshot in the cassandra.yaml then every time when you drop a table or keyspace Cassandra will capture a snapshot of that table. This snapshot will help you to rollback this table data if this was not done by mistake. If you do clear those table data from disk then you need to run below clearsnapshot command to free space.
nodetool -u XXXX -pw XXXXX clearsnapshot -t snapshotname
You can disable this auto_snapshot feature any time in cassandra.yaml.
nodetool cleanup removes all data from disk that is not needed there any more, i.e. data that the node is not responsible for. (clearsnapshot will clear all snapshots, that may be not what you want.)
The nodetool command can be used to clean up all unused (i.e. previously dropped) tables snapshots in one go (here issued inside a running bitnami/cassandra:4.0 docker container):
$ nodetool --username <redacted> --password <redacted> clearsnapshot --all
Requested clearing snapshot(s) for [all keyspaces] with [all snapshots]
Evidence: space used by old tables snapshots in the dicts keyspace:
a) before the cleanup:
$ sudo du -sch /home/<host_user>/cassandra_data/cassandra/data/data/<keyspace_name>/
134G /home/<redacted>/cassandra_data/cassandra/data/data/dicts/
134G total
b) after the cleanup:
$ sudo du -sch /home/<host_user>/cassandra_data/cassandra/data/data/<keyspace_name>/
4.0K /home/<redacted>/cassandra_data/cassandra/data/data/dicts/
4.0K total
Note: the accepted answer missed the --all switch (and the need to log in), but it still deserves to be upvoted.
I have a Cassandra cluster with three nodes, two of which are up. They are all in the same DC. When my Java application goes to write to the cluster, I get an error in my application that seems to be caused by some problem with Cassandra:
Caused by: com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency ONE (1 required but only 0 alive)
at com.datastax.driver.core.exceptions.UnavailableException.copy(UnavailableException.java:79)
The part that doesn't make sense is that "1 required but only 0 alive" statement. There are two nodes up, which means that one should be "alive" for replication.
Or am I misunderstanding the error message?
Thanks.
You are likely getting this error because the Replication Factor of the keyspace the table you are querying belongs to has a Replication Factor of one, is that correct?
If the partition you are reading / updating does not have enough available replicas (nodes with that data) to meet the consistency level, you will get this error.
If you want to be able to handle more than 1 node being unavailable, what you could do is look into altering your keyspace to set a higher replication factor, preferably three in this case, and then running a nodetool repair on each node to get all of your data on all nodes. With this change, you would be able to survive the loss of 2 nodes to read at a consistency level of one.
This cassandra parameters calculator is a good reference for understanding the considerations of node count, replication factor, and consistency levels.
I hit this today because the datacenter field is case sensitive. If your dc is 'somedc01' this isn't going to work:
replication =
{
'class': 'NetworkTopologyStrategy',
'SOMEDC01': '3' # <-- BOOM!
}
AND durable_writes = true;
Anyway, it's not that intuitive, hope this helps.
in my case, I got a message 0 available, but cassandra was up and cqlsh worked correctly, the problem was accessing from java: query was for a complete table, and some records were not accesible (all nodes containing them down). From cqlsh, select * from table works, only shows accesible records. So, the solution is to recover down nodes, and maybe to change replication factors with:
ALTER KEYSPACE ....
nodetool repair -all
then nodetool status to see changes and cluster structure
For me it was that my endpoint_snitch was still set to SimpleSnitch instead of something like GossipingPropertyFileSnitch. This was preventing the multi-DC cluster from connecting properly and manifesting in the error above.
I have a eight node cassandra setup. I am saving data with 3 days TTL. But the data is useless after I take a summary (using my java script, count of things etc). I want to delete all the data in a table. I can stop cassandra for sometime to do the deletion. So the data is removed from all nodes.
Should I run truncate and nodetool repair afterwards or should I flush first then delete. Whats the proper way to do it.
You can drop the tables or truncate them... but keep in mind that Cassandra will snapshot your tables by default, so you'll also need to run nodetool clearsnapshot on all of your nodes afterwards. There is no need to stop Cassandra while you do this delete.
I don't know that there is a right way per se... but when I do when I need to clear a table is, first, I run truncate on the table using cqlsh. Then I run nodetool clearsnapshot on my nodes using pssh (https://code.google.com/p/parallel-ssh/).
Hope this helps