Through ops-center and nodetool cfstats i was able to find that one of the partitions of a keyspace table is 560 Mb, but couldn't find out which partition is that. How can we trace which partition of the table is that big ??
The fastest possible way is to look for messages in the log about compacting large partitions. Sort of a cheat, but it often works.
Short of that, you'll need to dump the sstables to json, and then inspect the json. There are a number of people who have written tools for this online - https://github.com/BrianGallew/cassandra_tools is one example.
Related
command I ran was
nodetool tablehistograms <keyspace> <table>
The bug was
No SSTables exists, unable to calculate 'Partition Size' and 'Cell Count' percentiles
I tried to calculate partition size for better selections on partition keys, but nodetool command did not work fine as the partition size is not provided with this error
SSTables are immutable as far as concerned, and I do not know if I should (and how to) create SSTables based on existed ones?
Experts, please come solve this problem, really appreciate it.
Best
How exact do you need to be when measuring the partition sizes?
For a quick estimate, 'nodetool tablestats <keyspace.table>' will give you the min, max and avg partition size.
If a more accurate measurement is needed, you could download and use DSBulk and run the count option to pull the largest n partitions for a table, which will also print the key, for example:
dsbulk count --stats.modes partitions --stats.numPartitions <n> -k myKeyspace -t myTable
There are no histograms available for the command to report if there are no SSTables on disk.
The nodetool tablehistograms command collects the metrics from the SSTables but if there are none stored on disk then there is nothing for the command to report.
Make sure that the table contains data in the data/ directory then try again. Cheers!
👉 Please support the Apache Cassandra community by hovering over the cassandra tag above and click on Watch tag. 🙏 Thanks!
What is the best tool to find the number of rows in each Cassandra partition? I have a big partition and I want to know how much records are there in that partition
nodetool tablehistograms <keyspace> <table> will give you the distribution of the cells and sizes of thee partition for the table. But that does not give you for sure that partition. To get the specific one you must use count(*) on a select query that specifies the partition key in where clause. A very large partition and that can fail though.
sstablemetadata after 4.0 is based off the describe command in sstable-tools. It will give you the partitions largest in size, and largest in number of rows, and the partitions with most tombstones if you provide the -s to scan the sstable. These can be used against 3.0 and 3.11 sstables. I think 2.1 sstables are not able to be processed though.
...
Partitions: 22515
Rows: 13579337
Tombstones: 0
Cells: 13579337
Widest Partitions:
[12345] 999999
[99049] 62664
[99007] 60437
[99017] 59728
[99010] 59555
Largest Partitions:
[12345] 189888705
[99049] 2965017
[99007] 2860391
[99017] 2826094
[99010] 2818038
...
above example has parititon key an int, it will print out key like:
Widest Partitions:
[frodo] 1
Largest Partitions:
[frodo] 104
You can find the total number of partitions available for a table with nodetool command. ./nodetool cfstats <keyspace>.<table>.
If you know the partition key, you can fire a select count(*) for the partition to get no. of the records in that partition. It's possible that query can timeout for count queries on big partitions set cqlsh request-timeout before executing the query.
To understand how to calculate the physical partition size, go through the Datastax DS220: Data Modeling partition size
Instaclustr has a tool to find the partition size. However, this does not show the number of records in each partition:
https://github.com/instaclustr/cassandra-sstable-tools
As mentioned above either use inbuilt node tool, which could be find within Cassandra Folder extracted from jar , and run nodetool inside terminal .
nodetool toppartitions
Additionally you can also use online tool such as : https://www.cqlguru.io/ , but this need some prior information as vaerage number of rows per partition, average number of text in varchar and all . But this tool is good for rough estimation.
I am recently working in spark and came across few queries which I still couldn't resolve.
Let's say i have a dataset of 100GB and my ram size of the cluster is
16 GB.
Now, I know in case of simply reading the file and saving it in the HDFS will work as Spark will do it for each partition. What will happen when I perform sorting or aggregation transformation on 100GB data? How will it process 100GB in memory since we need entire data in case of sorting?
I have gone through below link but this only tells us what spark do in case of persisting, what I am looking is Spark aggregations or sorting on dataset greater than ram size.
Spark RDD - is partition(s) always in RAM?
Any help is appreciated.
There are 2 things you might want to know.
Once Spark reaches the memory limit, it will start spilling data to
disk. Please check this Spark faq and also there are severals
question from SO talking about the same, for example, this one.
There is an algorihtm called external sort that allows you to sort datasets which do not fit in memory. Essentially, you divide the large dataset by chunks which actually fit in memory, sort each chunk and write each chunk to disk. Finally, merge every sorted chunk in order to get the whole dataset sorted. Spark supports external sorting as you can see here and here is the implementation.
Answering your question, you do not really need that your data fit in memory in order to sort it, as I explained to you before. Now, I would encourage you to think about an algorithm for data aggregation dividing the data by chunks, just like external sort does.
There are multiple things you need to consider. Because you have 16RAM and 100GB data set, it will be good idea to keep persistence in DISK. It maybe difficult as when aggregating if data set has high cardinality. If the cardinality is low you will be better of to do aggregate at each RDD before merging into whole dataset. Also remember to make sure that each partition in RDD is less than memory (default value 0.4*container_size)
We have some large partition in cassandra and I would like to see what caused the large partition. Is there a tool to get the partition data out of cassandra and analyze it ? Right now cqlsh query is timing out even if select single row from the partition.
It can depend on why you get the timeout but theres a some of options.
increase column_index_size_in_kb in your cassandra.yaml to something like 1024 and rebuild the sstables. This works around the object allocation issues a wide partition index introduces.
increasing heap size.
increase read_request_timeout_in_ms
increase key cache size (nodetool setcachecapacity 1000 0 0) then make the read. Watch the read stage until down to zero then try request again. This is hard unless cluster is essentially unused. The read continues even after the timeout, once it finishes reading the index it will cache it so the following read will skip that part (generally the worst) which will speed up significantly
pull raw data from sstable with sstabledump or sstabletools
I'm working on designing Cassandra column family.
I met with a situation of higher GC while SELECTing, after loading a higher density of data. That is, amount of data in a partition increased. Also for low density data, it works fine.
I want to know how Cassandra does the SELECT query (with both partition and cluster key specified)?
Is the whole set of data in a partition is loaded into memory while we execute SELECT?
Will large number of partition keys affect performance?
Cassandra does not load the entire partition into memory, but it does load IndexInfo objects which help Cassandra find the relevant CQL rows within the partition. These are short lived java objects which can create quite a bit of heap pressure (GC pauses) - this is a design issue that will be addressed in CASSANDRA-9754 (known as Birch, a b-tree implementation of the index data structure).
Until cassandra-4.0 is released, you should target 100MB for your max partition size, and break larger partitions into smaller pieces.