[Question posted by a user on YugabyteDB Community Slack]
I have one question regarding the number of tablets for a table.
I am using YSQL API, my cluster is having 3 nodes with an RF of 3 and each node is having 16 cores.
I haven't specified the number of shards per table using SPLIT INTO N TABLETS syntax so I guess the number of tablets will be decided by the cores a node has, based on documentation it will be 8 shards per table per node.
In this case, the total shards for a table should be 24=(8 x 3)
We have RF=3 as well, so will that mean the total shards after replication will be 72? (24 x 3)
I am confused here, as I have seen only 24 shards in the tserver tablets UI where it's mentioned that 8 shards are the leaders out of 24. Seeing this it seems the 24 shards contain the replicated ones as well.
Please correct my understanding here. I am using YugabyteDB 2.12, latest stable
Output of - cat /proc/cpuinfo | grep 'core id'
This is for one table, at the left, it shows 24 shards, 8 are leaders and the rest are followers:
Your numbers are correct. 24 tablets is per tserver, 72 is for the 3 node cluster. 8 leaders per tserver, 24 leaders for the 3 node cluster.
And your screenshots are from one tserver endpoint (:9000/tables) The master endpoint (localhost:7000/table?id=000030af000030008000000000004200) will show you all tablets and their peers
Related
[Question posted by a user on YugabyteDB Community Slack]
I have one question regarding the number of tablets for a table.
I am using YSQL API, my cluster is having 3 nodes with an RF of 3 and each node has 16 cores.
I haven't specified the number of shards per table using SPLIT INTO N TABLETS syntax so I guess the number of tablets will be decided by the cores a node has, based on documentation it will be 8 shards per table per node.
In this case, the total shards for a table should be 24(8 x 3)
We have RF=3 as well, so will that mean the total shards after replication will be 72? (24 x 3)
I am confused here, as I have seen only 24 shards in the tserver tablets UI where it's mentioned that 8 shards are the leaders out of 24. Seeing this it seems the 24 shards contain the replicated ones as well.
Please correct my understanding here.
I am using 2.12, latest stable.
More questions related to the same topic, if ysql_num_shards_per_tserver=8, then:
If we create a cluster with 4 nodes with RF 3 , then the total tablets/shards will be 8 x 4 = 32 (without peers)? and 32 x 3 = 96 (including peers) ?
Also, suppose if we add one more node in an existing cluster with 3 nodes, then after node addition, a new 8 tablets/shards will be created for the new node? and then tablets/shards will be rebalanced ? or new tablets/shards are not created and just the rebalancing of existing ones will happen?
num_shards_per_tserver is 8
This is for one table, at the left it shows 24 shards, 8 are leaders and rest are followers
In this case, the total shards for a table should be 24(8 x 3),
No, what is called shard in ysql_num_shards_per_tserver is the number of tablets. Each tablet/shard has 3 tablet peers (one leader and two followers). So, with ysql_num_shards_per_tserver=8 it is expected that you see 8 leaders on a server, and 16 followers which are the tablet peers of the tablets having their followers in the two other servers.
And your screenshots are from one tserver endpoint (:9000/tables) The master endpoint (localhost:7000/table?id=000030af000030008000000000004200) will show you all tablets and their peers
The master shows the tablets (you have 24) and where are their leader and followers (1 leader 2 followers) so 24*4=the number of tablet peers in total. The tserver shows the tablet peers within this server
If we create a cluster with 4 nodes with RF 3 , then the total tablets/shards will be 8 x 4 = 32 (without peers)? and 32 x 3 = 96 (including peers) ?
Yes, it will be 96 total.
Also, suppose if we add one more node in an existing cluster with 3 nodes, then after node addition, a new 8 tablets/shards will be created for the new node?
No new tablets will be created.
or new tablets/shards are not created and just the rebalancing of existing ones will happen?
Yes, only rebalancing. Note that you can use auto-splitting to get new tablets created on a few threshold (> size and < number per node)
As described in this example each tserver started with 12 tablet as we set number of shards to 4.
And when we added a new node the number of tablet per tserver became 9. it seems the total number of tablet, which is 36, will not increase.
My question is:
How many node could we add while we have 36 total tablet(in this example)?
And Is it possible to increase shards count in a running universe to be able to add more node?
How many node could we add while we have 36 total tablet(in this example)?
In this example, you can expand to 12 nodes (each node would end up with 1 leader and 2 followers).
Reasoning: There are 36 total tablets for this table and the replication factor is 3. So there will 12 tablet leaders and 24 tablet followers. Leaders are responsible for handling writes and reads (unless you're doing follower reads, lets assume that is not the case). If you go to 12 nodes, each node would at least have one leader and be doing some work.
Ideally, you should create enough tablets upfront so that you end up with 4 tablets per node eventually.
And Is it possible to increase shards count in a running universe to be able to add more node?
This is currently not possible, but being worked on and getting close to the finish. expected to be released in Q1 2020. If you are interested in this feature, please subscribe to this GitHub issue for updates.
Until that is ready, as a workaround, you can split the table into sufficient number of tablets.
We are running 2 nodes in a cluster - replication factor 1.
After writing a burst of data, we see the following via node tool status.
Node 1 - load 22G (owns 48.2)
Node 2 - load 17G (owns 51.8)
As the payload size per record is exactly equal - what could lead to a node showing higher load despite lower ownership?
Nodetool status uses the Owns column to indicate the effective percentage of the token range owned by the nodes. While GB is Size of your records
Dont see anything wrong here. Your data is almost evenly distributed around your two nodes which is exactly what you want for perfekt performance.
I've 5 node Cass cluster and each one currently owning 1 TB of data. When I tried adding another node, it almost took 15+ Hours to bring to 'UN' State.
Is there a way to make it fast?
Cassandra version: 3.0.13
Environment : AWS , m4.2xlarge machines.
1 TB is a lot of data per node. Since you have a 5 node cluster and you're adding a new node that node will take 0,833 TB data that has to be streamed from all nodes. That is the equivalent of 6,67 Tbit, or 6990507 Mbit. Cassandra has a default value for stream_throughput_outbound_megabits_per_sec of 200. 6990507รท200 = 34952,535 seconds = 9,7 hours to transfer all data. Since you're probably running other traffic at the same time etc. this could very well take 15 hours.
Solution: Change the stream_throughput_outbound_megabits_per_sec on all nodes to a higher value.
Note: Don't forget to run nodetool cleanup after the node has joined the cluster.
we have 8 cassandra nodes in 2 DC configuration and we have around 250k record insertion per second and our ttl is 1 year , Recently we upgraded to 3.0.14 . Our main problem is slow read performance .
to improve this we increased compaction period from 1 hour to 1 week .( Full compaction still needs to run ) But now we are thinking to install new cassandra nodes and divide data to 2 pieces ( 6 months period each ) and new casandra nodes can be responsible only querying first 6 months .Is this possible to move data like this ? is the procedure to follow ? thanks