In my Cassandra cluster of 30 nodes, having datacenter 1, replication factor 3.
We are facing issues when tried to load data into cassandra.
Error from log:
ERROR [pool-1-thread-8] 2022-09-23 16:44:29,430 CassandraLoader.java (line 392) Hector error while inserting
me.prettyprint.hector.api.exceptions.HInvalidRequestException: InvalidRequestException(why:Keyspace 'huawei' does not exist)
WARN [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] 2022-09-23 16:44:31,498 CassandraHostRetryService.java (line 217) Downed dmbas-r410-26(10.173.226.43):9160 host still appears to be down: Unable to open transport to dmbas-r410-26(10.173.226.43):9160 , java.net.NoRouteToHostException: No route to host
WARN [pool-1-thread-10] 2022-09-23 16:44:39,458 HConnectionManager.java (line 303) Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException(acknowledged_by:1)
Can someone suggest how to fix these warnings and errors.
Thanks,
Related
I have 2 node Cassandra cluster with datastax-agent up and running(one seed node) also nodetool status showing healthy.
In the 3 node I have opscenter install and the UI is loading fine with a blank screen, when I saw the var log it's complaining 'NO cassandra connection available for hostlist with a invalid unsupported version'(paster the log details message below). Any help is highly appreciated
2017-02-25 06:33:06+0000 [CLUSTER_NAME] ERROR: Control connection failed to connect, shutting down Cluster: ('Unable to connect to any servers', {'SEED-IP': })
2017-02-25 06:33:06+0000 [CLUSTER_NAME] WARN: No cassandra connection available for hostlist ['SEED-IP'] . Retrying.
I was using an old version of opscenter which was not matching the version of dse. Here is the map for your refference.
Reference: datastax doc
I am facing issue when I try to connect opscenter-agent to opscenter . In UI when we put any node ip it's shows Error creating cluster: Timeout while adding cluster. Please check the log for details on the problem.
In opscenter log I getting below error
2016-10-18 09:35:12+0000 [MW_Dev_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.164.120.116:61621/cluster/topology?node_ip=10.184.27.237 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2016-10-18 09:35:12+0000 [MW_Dev_Cluster] WARN: HTTP request http://10.185.183.5:61621/cluster/topology?node_ip=10.112.73.35 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
In opscenter agent conf. file address.yaml only below entry is there
stomp_interface: "10.166.71.92"
Im tring t build up an Hawkular Server in Windows 7 - unfortunately this server works with a Cassandra DB - ive installed the newest version and in the starting process of Hawkular I got following error:
14:57:20,102 ERROR [org.hawkular.alerts.bus.log] (Thread-195 (ActiveMQ-client-global-threads-205387390)) HAWKALERT210009: Error accesing to DefinitionsService. Description: [java.lang.RuntimeException: Cassandra session is null]
14:57:19,843 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [18] Retrying connecting to Cassandra cluster in [2]s...
14:57:19,839 ERROR [org.hawkular.alerts.engine.log] (EE-ManagedExecutorService-default-Thread-10) HAWKALERT220009: Definitions Service error in [Triggers]. Msg: [java.lang.RuntimeException: Cassandra session is null]
14:57:20,354 ERROR [org.hawkular.alerts.bus.log] (Thread-190 (ActiveMQ-client-global-threads-205387390)) HAWKALERT210009: Error accesing to DefinitionsService. Description: [com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for qu
ery failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.InvalidQueryException: unconfigured table schema_keyspaces))]
14:57:22,104 INFO [org.hawkular.inventory.impl.tinkerpop] (ServerService Thread Pool -- 111) HAWKINV001000: Using graph provider: org.hawkular.inventory.impl.tinkerpop.provider.TitanProvider
14:57:22,360 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
14:57:22,396 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.d
atastax.driver.core.exceptions.InvalidQueryException: unconfigured table schema_keyspaces))
14:57:22,397 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200004: [19] Retrying connecting to Cassandra cluster in [3]s...
14:57:23,102 WARN [org.hawkular.inventory.cdi] (ServerService Thread Pool -- 111) HAWKINV003501: Inventory backend failed to initialize in an attempt 10 of 15 with message: Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.cassandra.th
rift.CassandraThriftStoreManager.
14:57:25,398 INFO [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
14:57:25,438 WARN [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect to Cassandra cluster - assuming its not up yet: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.d
atastax.driver.core.exceptions.InvalidQueryException: unconfigured table schema_keyspaces))
Cassandra DB is online and I can connect to localhost:9160 with the Cassandra CQL Shell - but not the hawkular server - have I forgotten something?
From the log, the port Hawkular is trying to use is 9042. Make sure of this setting in your cassandra.yaml
start_native_transport: true
native_transport_port: 9042
The port you are using (9160) is thrift port.
I can only take a stab in the dark with such limited information.
However you said you connected to localhost:9160. In the log it mentions "127.0.0.1:9042".
I have problems to get a existing Cassandra node to join the cluster again after reboot (on a new virtual machine instance).
I had a running Cassandra cluster with 4 nodes all in state "up and normal" according to nodetool status. The nodes are running on virtual machines in Azure. I changed the instance type of the virtual machine for 10.0.0.6, which returned in a reboot of this machine. The machine stayed on 10.0.0.6.
After the reboot I am unable to start Cassandra again. I am getting this exception:
INFO 22:39:07 Handshaking version with /10.0.0.4
INFO 22:39:07 Node /10.0.0.6 is now part of the cluster
INFO 22:39:07 Node /10.0.0.5 is now part of the cluster
INFO 22:39:07 Handshaking version with cassandraprd001/10.0.0.6
INFO 22:39:07 Node /10.0.0.9 is now part of the cluster
INFO 22:39:07 Handshaking version with /10.0.0.5
INFO 22:39:07 Node /10.0.0.4 is now part of the cluster
INFO 22:39:07 InetAddress /10.0.0.6 is now UP
INFO 22:39:07 Handshaking version with /10.0.0.9
INFO 22:39:07 InetAddress /10.0.0.4 is now UP
INFO 22:39:07 InetAddress /10.0.0.9 is now UP
INFO 22:39:07 InetAddress /10.0.0.5 is now UP
ERROR 22:39:08 Exception encountered during startup
java.lang.RuntimeException: A node with address cassandraprd001/10.0.0.6 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:455) ~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:667) ~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:615) ~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:509) ~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338) [apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:457) [apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:546) [apache-cassandra-2.1.0.jar:2.1.0]
java.lang.RuntimeException: A node with address cassandraprd001/10.0.0.6 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:455)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:667)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:615)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:509)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:457)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:546)
Exception encountered during startup: A node with address cassandraprd001/10.0.0.6 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
INFO 22:39:08 Announcing shutdown
I am using Cassandra 2.1.0. I am not replaying a dead node - I am just trying to get the old node up and running again. According to nodetool status (on the other nodes) all nodes are "up and normal" except 10.0.0.6 which is "down and normal".
How do I get this node up and running again?
First, on another node, use
nodetool status
the results show you list of nodes in the cluster. Find your node with ip which fail to start, get its id and fill to command:
nodetool removenode <node_id>
then start cassandra.
Best,
Quick answer, if the node's ip is 10.200.10.200
add this
JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=10.200.10.200"
to the end of your
cassandra-env.sh
Don't forget to remove it once your done.
You can look this blog, http://blog.alteroot.org/articles/2014-03-12/replace-a-dead-node-in-cassandra.html.
It works for me, this is a bug for Cassandra. If your node's host_id changed, but use old IP, will throw this exception.
If you use Cassandra 2.x.x, you should modify cassandra/conf/cassandra-env.sh.
Finally, don't forget to REMOVE modifications on cassandra-env.sh after the complete bootstrap!
I have a cluster running on datastax-cassandra 1.2.5, it works fine, because of vnodes adn leveled compaction strategy issue i tried promoting it to 1.2.6.
So upgrading involved -
1 - stopping all the nodes
2 - deleting 1.2.5 rpm
3 - installing 1.2.6 rpm
4 - fixing cassandra.yaml
5 - starting cassandra.
Problem Statement - The problem now is that all the nodes are up and running, but not in one cluster. They all are operating in their own cluster even though the seeds in yaml points to the original seed.
nodetool status also just shows the one node (the node on which we are on)
system log shows one error
ERROR [WRITE-/10.93.3.46] 2013-10-21 19:43:29,101 CassandraDaemon.java (line 192)
Exception in thread Thread[WRITE-/10.10.10.10,5,main]
java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:79)
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:66)
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:351)
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:143)
**** 10.10.10.10 is the seed ip
Any help on how to pass through it
Try to set the internode_compression to none. It will disable compression between nodes, which is failing because snappy cannot initialize
internode_compression: none