[Question posted by a user on YugabyteDB Community Slack]
I've been googling a bit and it seems that, although it is not recommended to change the RF of an already running cluster, is it possible?
We support changing RF at the database layer at the very least. One way this can be accomplished is by using the modify_placement_info command in yb-admin [see yb-admin doc]. For example, if you have 3 nodes, one in cloud-c1, region-r1, and zone-z1, one in c2.r2.z2, and one in c3.r3.z3, you can use the below command to increase the RF to 3.
bin/yb-admin --master_addresses {list of master IPs:Ports} modify_placement_info c1.r1.z1,c2.r2.z2,c3.r3.z3 3
Another way to control RF is to use tablespaces https://docs.yugabyte.com/preview/explore/ysql-language-features/going-beyond-sql/tablespaces/
Related
I'm trying to add a new node into existing cluster and allocate_tokens_for_local_replication_factor is 2 for the current nodes but the default value from newly installed Cassandra is 3. I was tried to find information about this configuration but can't find a clear description how this configuration is working.
The allocate_tokens_for_local_replication_factor works in much the same way as allocate_tokens_for_keyspace where it triggers an algorithm that attempts to choose tokens such that the load (data density) is balanced or optimised for the nodes in the local data centre.
The main difference is that allocate_tokens_for_local_replication_factor optimises the algorithm for a defined replication factor of 3 (default) instead of the replication factor for a given keyspace (allocate_tokens_for_keyspace).
allocate_tokens_for_keyspace was added in Cassandra 3.0 (CASSANDRA-7032) to improve token allocation for clusters configured with virtual nodes. However, it suffers from the problem where the replication factor of a keyspace cannot be used when adding a new DC since the keyspace is not replicated to the new DC yet.
Cassandra 4.0 solved this problem by allowing operators to specify the replication factor to be used for optimising the token allocation algorithm with allocate_tokens_for_local_replication_factor (CASSANDRA-15260).
In your case where the existing nodes have:
allocate_tokens_for_local_replication_factor: 2
a previous operator would have configured it that way because the application keyspace(s) had a replication factor of 2.
As a side note, thanks for bringing this to our attention. I have logged CASSANDRA-17984 so we could improve the docs. Cheers!
[Question posted by a user on YugabyteDB Community Slack]
I have a cluster that I run a compaction on a very large table once a week. However the last weekend, one node was unhealthy and the compaction did not run on that node.
Is there any way to run the compaction just on that node?
My understanding is that the compaction through yb-admin will run against all nodes that have tablets associated with that table?
Is there a way to run just against one node ?
I also assume there will be less work to do on the nodes that it ran successfully on over the weekend, as most of the data should already have been compacted?
Yes, there is yb-ts-cli compact_all_tablets command that compacts all tablets of a yb-tserver.
And there’s also yb-ts-cli compact_tablet <tablet_id> that compacts a single tablet in a yb-tserver.
My understanding is that the compaction through yb-admin will run
against all nodes that have tablets associated with that table?
Yes, it will run on all tablets of that table.
I also assume there will be less work to do on the nodes that it ran
successfully on over the weekend, as most of the data should already
have been compacted?
The compaction is forced, therefore it will be redone on the servers where it already ran. It will be less work, but still a lot because it will rewrite the whole tablet into 1 sst file on disk, thus not recommended.
You can use yb-admin list_tablets and filter by the ones that have the leader-IP of the node you are interested in to get all the tablets ids that you need to compact for that specific table. And then you run yb-ts-cli compact_tablet on each tablet.
[Question posted by a user on YugabyteDB Community Slack]
I set the variables in sql:
SET default_transaction_read_only = TRUE
SET yb_read_from_followers=true;
SET yb_follower_read_staleness_ms=30000;
But yet my read query ran on master nodes instead. A screenshot:
The behavior does not seem consistent though. On some queries the read directly from the table seems to be happening on the read node (verified by looking at iostats on the nodes) ... but some queries that use an index seem to go to the main cluster. I’m using yugabyte-2.8.1.0-b37
That version doesn’t support reading from replicas when an index is being used. A fix has been included in https://github.com/yugabyte/yugabyte-db/commit/2ba51212d874bb3e68a676b59f7f69635223892c and will be included in the next minor release 2.8.2.
I want to test my cluster a little, how data replicates, etc.
I have a cassandra cluster formed by 5 machines ( centos 7 & cassie 3.4 on them).
Are there anywhere tables already created for testing that I can import in my db in some keyspace?
If yes, please be kind enough and explain me how to import them into a keyspace and where from to take them.
You can use Cassandra-stress. This is great to create data for your style of table and also has some default tables.
http://docs.datastax.com/en/cassandra_win/3.0/cassandra/tools/toolsCStress.html
I highly recommend it.
Actually , it is a lot of data in internet that can be used for testing
e.g.
https://www.quora.com/Where-can-I-find-large-datasets-open-to-the-public
http://bigdata-madesimple.com/70-websites-to-get-large-data-repositories-for-free/
Cassandra provide with tool cqlsh for executing CQL command as COPY for importing CSV data to database.
P.S.But pay attention on the fact that cqlsh has some restriction related to timeout. That is why it would be better to use some cassandra connector to make this process more effective.
Here is a production cassandra cluster with one datacenter of 3 hosts. The version is 1.0.7. I want to upgrade from 1.0.7 to 2.1.8 and then add another cassandra data center with 3 hosts of version 2.1.8.
I have experimented on test cluster and can upgrade the cluster without any ERRORS. But I still worry about is there any data loss or modified. So I want to design a quick method to validate the following 2 points.
Are there any data losses or damages when the cluster upgraded from 1.0.7 to 2.1.8?
I add an extra data center in the cluster and alter the keyspace strategy to NETWORKTOPOLOGYSTRATEGY with 2 replicas each data center. How to validate 2 data centers holding the same replicas?
There are about 10G rows in the current clusters. It is tedious to match the rows. Are there any better way to validate the points above? Or I can just trust the cassandra itself.
I'm not sure it's really practical (or necessary) in most cases to check every row of data.
I'd probably do some before and after checks of things like this:
Spot check some selected subset of rows. If some of them are correct, likely all of them are.
Compare the data sizes before and after the upgrade to make sure they are in the same ballpark.
Monitor the upgrade process for errors (which you're already doing).
Run full repairs on the nodes after the upgrade and see if there is an unusual amount of data movement suggesting some nodes were not fully populated.