Let's imagine we have a server with 10 disk storage and i gonna create a yugabyte cluster with replication factor of 5 (and 20 tablet sharding). I have two option to do that:
Create 5 node and give each tserver 2 disk to use with 100 tablet per node?(which i'm not sure can a tserver use 2 disk to balance tablets between them).
Create 10 node and give each tserver 1 disk to use with 50 tablet per node.
which one recommended if there is any difference?
Either of those options would work. A yb-tserver can utilize multiple disks, it will spray the data for the tablets it hosts across multiple disks and utilize them.
That said, there are other factors matter, outlining some of them below:
The most important consideration is the size of the machines (number of vCPU's / CPU cores) in the two cases. We recommend nodes which are at least 8 or 16 cores to achieve optimal performance. In your setup, assuming that in option #2 you would use smaller machines with half the number of vCPUs (so that the aggregate vCPUs across cluster remains the same in both cases) - please pick which ever setup gets you to at least 8 cores. If both cases are less than 8 vCPUs, then option #1 is better since it has more cores.
Assuming both options satisfy the above point, a second consideration is the impact of a failure. If you have more nodes, the impact of a failure is not as high on the cluster compared to fewer nodes - so from this perspective, option #1 is better. Of course, the reality is a bit more nuanced - factors such as is it a multi-zone setup vs single zone, etc will affect this decision.
Hope that helps.
Related
I am using a Cassandra cluster with 4 nodes. 2 nodes have much more resources than the other two in terms of CPU cores and RAM.
Right now I am using the DCAwareRoundRobin load balancing policy. I believe in this policy, all nodes receive the same number of requests. Due to this, 2 smaller nodes are NOT performing well. This is resulting in high IO and CPU usage on smaller nodes.
I want to distribute the traffic from the Java application to the Cassandra cluster in the ratio of available resources.
For example: small node 1 - 20%, small node 2 - 20%, large node 1 - 30%, large node 2 - 30% of queries.
Need you suggestion about any method or approach I can use to distribute the traffic in the manner.
I understand that I can use LatencyAwarePolicy [1]. I am worried that when a node is taken out of query plan due to a threshold breach, the remaining nodes might see the ripple effect.
[1] https://docs.datastax.com/en/drivers/java/2.2/com/datastax/driver/core/policies/LatencyAwarePolicy.html
In short, the approach you have is an anti-pattern and should be avoided at all costs.
TL;DR: Make the nodes the same.
During the request processing, The query will reach not a single node but all the nodes responsible for the partition you are accessing, based on the Replication Factor, doesn't matter whatever Consistency Level you use. If you have RF=3, 3 of your 4 nodes will be reached for each write or read request. You do not have and should not have control over request distribution, "the medicine will be worse than the sickness".
I wish to estimate how many cassandra storage nodes I would need to serve a specific number of reads per second.
My node specs are 32 cores, 256GB ram, 10Gbps NIC, 10x 6TB HDDs. Obviously SSDs would be much more preferrable, but these are not available in this instance.
I have around 5x10^11 values of 1kB each = 500TB of values to serve, at a rate of 100,000 read requests per second. The distribution of these requests is completely even, ie, ram capacity caching will have no effect.
If we assume that each HDD can sustain ~100 IOps, then I could expect that I need at least ~ 100 nodes to serve this read load - correct?
I also estimate that I would need at least ~ 20 machines for the total storage with a replication factor of 2, plus overhead.
It's a really broad question - you need to try to test your machines with tools, like, NoSQLBench that was specially built for such tasks.
Typical recommendation is to store ~1Tb of data per Cassandra node (including replication). You need to take into account other factors, like, how long it will take to replace the node in the cluster, or add new one - the speed of streaming is directly proportional to size of data on disk.
HDDs are really not recommended if you want to have low latency answers. I have a client with ~150Tb spread over ~30 machines with HDDs, with a lot of writes although, and read latencies regularly are going above 0.5 second, and higher. You need to take into account that Cassandra requires random access to data, and head of HDD simply couldn't move so fast to serve requests.
We started using Cassandra years ago with four nodes. All the nodes have same instance size, M5.xlarge which is vCPU:4 and Memory:16GB. By time we increased disk space for each node up to 300 GB each.
If we want to add a new node/nodes with different and higher hardware specs such as vCPU 16, memory 64GB (1 TB disk), does it cause problem across the nodes? It may sound silly but IOPS will be different, latency may be lower, even the data may be more up-to-date in higher spec node(maybe it chokes other nodes?). Since the traffic is distributed evenly the way new node handle the I/O will be different than other nodes.
Thank you.
Yes, it will cause problems, especially if you're using consistency level that is higher than LOCAL_ONE or ONE... The latencies when reading or writing data will be different, and even if your new nodes will answer fast, they will need to answers from smaller nodes, etc.
If you're using AWS (or other cloud provider) - why not upgrade the nodes in-place? If you're using EBS, just remount them to bigger nodes, but keep the instance types the same for all nodes.
We have a 30+ node Cassandra cluster (3.11.2) in 4 data centers. One of the centers consists of 8 nodes in Azure running on Standard DS12 v2 (4cpu, 28gb) nodes with a 500GB premium SSD drive. All in the same data center (central US).
We are seeing a dramatic CPU imbalance in the node activity when pushed to the max. We have a keyspace with about 200 million records, and we're running a process to check and refresh the records if necessary from another data stream.
What's happening, is we have 4 nodes that are running at 70-90% CPU compared to 15-25% of the other 4. The measurement of the CPU is being done in the nodes themselves, because Azure's own metrics is broken and never represents what is actually happening.
Digging into a pair of nodes (one low CPU and one high) the difference is the iowait% of the two. The data in the keyspace is balanced (within reason - they are all within 5% of another in record count and size). It looks like the number of reads is balanced, and even the read latency as reported by Cassandra is similar.
When I do an iostat compare of the nodes, the high CPU node is reporting a much higher (by 50 to 100%) rKB/s numbers... which is likely leading to the difference in iowait% time.
These nodes are 100% configured the same, running the same version of everything (OS, libraries, everything) that I can think to look. I cannot figure out why some nodes are deciding to do more disk reads that the others, resulting in the cluster as a whole slowing down.
Anybody have any suggestions on where I can look for differences?
The only thing that is a pattern, is the nodes that are slower are the 4 nodes that were added later in our expansion. We started with 4 nodes for a while and added 4 more when we needed space. All the appropriate repairs and other tasks required with node additions were done - the fact that the records and physical size of the data files on disk being equal should attest to that.
When we shut down our refresh process, the all the nodes settle down to a even 5% or less CPU across the board. No compaction or any other maintenance is happening that would indicate something different.
plz help... :)
Our final solution for this - to fix ONLY the unbalanced problem was to cleanup, full repair and compact. At that point the nodes are relatively equally used. We suspect expanding the cluster (adding nodes) may have left elements of data on the older nodes that were not compacted out based on regular compaction events.
We are still working to try to solve the load issue; but now at least all the nodes are feeling the same CPU crunch.
I have 6 cassandra nodes in two datacenters with 16GB of memory and 1TB HD drive.
Now I am adding 3 more nodes with 32GB of memory. will these machines will cause overhead for existing machines ( May be in token distribution )? if so please suggest how to configure these machine to avoid those problems.
Thanks in advance.
The "balance" between nodes is best regulated using vnodes. If you recall (if you don't, you should read about it), the ring that Cassandra nodes form is actually consisted out of virtual nodes (vnodes). Each node in the ring has a certain portion of vnodes, which is set up in the Cassandra configuration on each node. Based on that number of vnodes, or rather the proportion between them, the amount of data going to those nodes is calculated. The configuration you are looking for is num_tokens. If you have similarly powerful machines, than an equal vnode number is available. The default is 256.
When adding a new, more powerful machine, you should assign a greater number of vnodes to it. How much? I think it's hard to tell. It's unwise to give it twice more, only be looking at the RAM, since those nodes will have twice as many data than the others. Than you might expect more IO operations on them (remember, you still have the same HDD) and CPU utilization (and the same CPU). You might want to take a look at this answer also.