I'm trying datastax enterprise 4.8.4 on ubuntu. Installation is OK following the instructions. I can start a cassandra node with the command sudo service dse start. Then I want to try restart as a solr node on the same machine. So I edit the /etc/default/dse and set SOLR_ENABLED=1. Now I tried to start cassandra again, it failed.
#ubuntu:~$ sudo service dse start
* Starting DSE daemon dse
DSE daemon starting with Solr enabled (edit /etc/default/dse to disable)
[ OK ]
#ubuntu:~$ sudo service dse status
* dse is running
#ubuntu:~$ sudo service dse status
* dse is not running
I checked the /var/log/cassandra/system.log. Here's the error message:
ERROR [main] 2016-01-28 16:44:02,767 CassandraDaemon.java:294 - Fatal exception during initialization
org.apache.cassandra.exceptions.ConfigurationException: Cannot start
node if snitch's data center (Solr) differs from previous data center
(Cassandra). Please fix the snitch configuration, decommission and
rebootstrap this node or use the flag -Dcassandra.ignore_dc=true.
The question is how could I fix it? or where do I add -Dcassandra.ignore_dc=true?
As of 2.1.13 you are not allowed to change the data center of an existing node. This is to protect users from making a mistake in production since changing data center (DC) names in a live system will result in downtime.
The DSE simple snitch assigns nodes to logical data centers based on workload type. I.E. solr nodes will be added to a solr DC, Cassandra nodes will be added to a Cassandra data center. If you change the workload type using DSE simple snitch you change the DC name which is not permitted by default.
If this is a test environment and you know what you're doing (you are willing to incur the breif downtime while all the nodes in your DC are changed to a new DC) you can submit the following jvm flag in your cassandra-env.sh
-Dcassandra.ignore_rack=true and/or
-Dcassandra.ignore_dc=true.
**make sure your topology after the DC / name change is still the same, otherwise replication can get messed up and you can end up with a lot of streaming + CPU utilization + io + stale reads **
Below are the steps you can take to solve the problem -
Error : - CassandraDaemon.java:709 - Cannot start node if snitch's data center (Cassandra) differs from previous data center (SearchGraphAnalytics).
Solution : -
Change the network topology to GossipingPropertyFileSnitch from SimpleSnitch in Cassandra.yaml file , placed at /usr/share/dse/resource/cassandra/conf ( if installed as no service)
Create a properties file with datacenter and rack information if not exists .
Path : /usr/share/dse/resource/cassandra/conf
filename: cassandra-rackdc.properties
And add the below entries -
--indicate the rack and dc for this node dc=SearchGraphAnalytics // the previous name of your DC before upgrading rack=RAC1
If Gremlin server Error persists then run the graph using below parameters -
This error may com when DataStax Enterprise was installed from a tarball or from the DataStax Installer with the No Services option.
Start the dse using the below parameters : -
/usr/share/dse$ bin/cassandra -g -s -k
Related
After cassandra install via sudo apt install -y cassandra, cassandra is running, but I have not yet configured cassandra.yaml and cassandra-topology.properties. Once I configure them cassandra won't start because Cannot start node if snitch's data center (0) differs from previous data center (datacenter1)
Assuming that you don't care about the data stored, you can fix that by deleting everything in your data_file_directories. If you're not setting that, you'll find it at $CASSANDRA_HOME/data/data.
Basically, the cluster metadata gets written in the system keyspace, which uses the local replication strategy (system is unique on each node). At start time, Cassandra verifies the stored metadata vs. the config properties being passed. When you change something about the node, Cassandra will throw an error on specific properties like cluster_name, dc, rack (and possibly a few others) when they don't match what's on disk.
tl;dr;
You probably only need to delete only the data for the system keyspace.
But another option, would be to uncomment and set data_file_directories. Then the new node's system metadata would be written there, and the node would start.
We want to deploy DSE cluster of 3 nodes, where each node is Analytics running Spark.
We want to use vnodes in cassandra, because it enables much more even data distribution and easier adding of the nodes. We deploy DSE on AWS, using one of the available AMI images.
Although DSE by default deploys Cassandra cluster using single token nodes, we have to manually change cassandra.yaml file on all the nodes.
According to datastax documentation, I should:
uncomment num_tokens field (I left 256 default value)
leave initial_token field unassigned
After that, when I do nodetool status command, I see that my cluster still uses single token mode.
According to this, I should restart nodes in the cluster, so that changes take effect.
But after nodes are restarted both thru OPS center or AWS console, I get errors, nodes are in unresponsive state, and I cannot use nodetool command on my nodes, with error:
Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused'.
Is there something that I am doing wrong?
How to enable vnodes on DSE when deployed using AMI image?
Thank you
We are running 6 node cluster with:
HADOOP_ENABLED=0
SOLR_ENABLED=0
SPARK_ENABLED=0
CFS_ENABLED=0
Now, we would like to add Spark to all of them. It seems like "adding" is not the right term because this would not fail. Anyways, the steps we've done:
1. drained one of the nodes
2. changed /etc/default/dse to SPARK_ENABLED=1 and HADOOP_ENABLED=0
3. sudo service dse restart
And got the following in the log:
ERROR [main] 2016-05-17 11:51:12,739 CassandraDaemon.java:294 - Fatal exception during initialization
org.apache.cassandra.exceptions.ConfigurationException: Cannot start node if snitch's data center (Analytics) differs from previous data center (Cassandra). Please fix the snitch configuration, decommission and rebootstrap this node or use the flag -Dcassandra.ignore_dc=true.
There are two related questions that have been already answered:
Unable to start solr aspect of DSE search
Two node DSE spark cluster error setting up second node. Why?
Unfortunately, clearing the data on the node is not an option - why would I do that? I need the data to be intact.
Using "-Dcassandra.ignore_rack=true -Dcassandra.ignore_dc=true" is a bit scary in production. I don't understand why DSE wants to create another DC and why can't it just use the existing one?
I know that according to datastax's doc one should partition the load using different DC for different workloads. In our case we just want to run SPARK jobs on the same nodes that Cassandra is running using the same DC.
Is that possible?
Thanks!
The other answers are correct. The issue here is trying to warn you that you have previously identified this node as being in another DC. This means that it probably doesn't have the right data for any key-spaces with Network Topology Strategy. For example if you had a NTS keyspace which had only one replica in "Cassandra" and changed the DC to "Analytics" you could inadvertently lose all of the data.
This warning and the accompanying flag are telling you that you are doing something that you should not be doing in a production cluster.
The real solution to this is to explicitly name your dc's using GossipingFileSnitch and not rely on SimpleSnitch which names based on the DSE workload.
In this case, switch to GPFS and set the DC name to Cassandra.
I am using Cassandra 2.0 and cluster has been setup with 3 nodes. Nodetool status and ring showing all the three nodes. I have specified tokens for all the nodes.
I followed the below steps to change the configuration in one node:
1) sudo service cassandra stop
2) updated cassandra.yaml (to update thrift_framed_transport_size_in_mb)
3) sudo srevice cassandra start
The specific not started successfully and system.log shows below exception:
org.apache.cassandra.exceptions.ConfigurationException: Cannot change
the number of tokens from 1 to 256
What is best mechanism to restart the node without losing the existing data in the node or cluster ?
Switching from Non-Vnodes to Vnodes has been a slightly tricky proposition for C* and the mechanism for previously performing this switch (shuffle) is slightly notorious for instability.
The easiest way forward is to start fresh nodes (in a new datacenter) with vnodes enabled and to transfer data to those nodes via repair.
I was also getting this error while I was trying to change the number of tokens from 1 to 256. To solve this I tried the following:
Scenario:
I have 4 node DSE (4.6.1) cassandra cluster. Let say their FQDNs are: d0.cass.org, d1.cass.org, d2.cass.org, d3.cass.org. Here, the nodes d0.cass.org and d1.cass.org are the seed providers. My aim is to enable nodes by changing the num_token attribute in the cassandra.yaml file.
Procedure to be followed for each node (one at a time):
Run nodetool decommission on one node: nodetool decommission
Kil the cassandra process on the decommissioned node. Find the process id for dse cassandra using ps ax | grep dse and kill <pid>
Once the decommissioning of the node is successful, go to one of the remaining nodes and check the status of the cassandra cluster using nodetool status. The decommissioned node should not appear in the list.
Go to one of the active seed_providers and type nodetool rebuild
On the decommissioned node, open the cassandra.yaml file and uncomment the num_tokens: 256. Save and close the file. If this node was originally seed provider, make sure that it's ip-address is removed from the seeds: lists from cassandra.yaml file. If this is not done, the stale information about the cluster topology it has will hinder with the new topology which is being provided by the new seed node. On successful start, it can be added again in the seed list.
Restart the remaining cluster either using the corresponding option in opscenter or manually stopping cassandra on each node and starting it again.
Finally, start cassandra on it using dse cassandra command.
This should work.
I am trying to connect Apache Cassandra nodes into a ring. They are not Datastax versions, but Cassandra 1.2.8 from the Apache website. When trying to add one as the seed of the other I get following exception:
Unable to find compaction strategy class 'com.datastax.bdp.hadoop.cfs.compaction.CFSCompactionStrategy'
Before that I change the "listen_address" and "rpc_address" to local IP address of each node. Next step I add one IP as a seed to another node. The nodes start up, an exception is printed but both nodes run fine until restart. After restarting either node the exception is printed and nodes do not run.
This is very strange - I do not have any DSE components.
Did you previously use any DSE components? If you did and are using the same data directory on any of your nodes, it may find old column families that were created with this compaction strategy. If you have no data you want in the data directories on all your nodes, you should clear them by stopping all nodes, deleting the directories, then starting the nodes.
Or if you have any DSE nodes still up, they may be joining the new cluster and propagating their schema, so creating column families with this compaction strategy. You can find out by looking in the logs and seeing which nodes try to connect. If any aren't from your 1.2.8 ring then this is probably the cause.
That error means you either had a DSE Analytics node in your ring at some point, or you restored your schema from someplace that had an Analytics node.
I would check if you have the folder /etc/dse/ on your VM, that would mean DSE was installed there.
To just wipe the node and start from scratch schema wise, you can stop the node, remove the /system/schema_* folders, then start the node. When it starts it will have no schema. Re-create any keyspaces/column families you had before, and they will get read from disk.