Please help me how I can stop Cassandra update automatically?
At present when I install Cassandra through opscentre I get an error.
Unable to restart DSE service. See /var/log/cassandra/system.log and /var/log/cassandra/output.log on the target node for details.
system.log
ERROR [main] 2018-03-28 07:58:26,123 CassandraDaemon.java:705 - Exception encountered during startup
java.lang.AbstractMethodError: org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150) ~[na:1.8.0_162]
at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135) ~[na:1.8.0_162]
at javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405) ~[na:1.8.0_162]
at org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:143) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:471) ~[dse-core-5.1.0.jar:5.1.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:599) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at com.datastax.bdp.DseModule.main(DseModule.java:93) [dse-core-5.1.0.jar:5.1.0]
But when I update my OS with the command **yum update**, it updates DSE 5.1.0 to 5.1.7. After updating dse start. I am not able to understand why its happening?
On other node dse 5.1 working perfectly.
OS: centos 7
Related
I have installed a DSE 6.0 cassandra cluster using LCM| opscenter 6.5 and node is up is running . During LCM cluster install, it installed datastax agent as well .
But the agent is not connecting to DSE and opscenter is not showing any details about the node.Later I tried with tarball install of datastax agent but that too is showing same issue. Please see below agent.log and screenshots.
WARN [async-dispatch-2] 2018-07-24 09:23:19,915 JMX marked as down, restarting JMX components.
ERROR [async-dispatch-2] 2018-07-24 09:23:19,916 Error starting DynamicEnvrionmentComponent.
java.io.IOException: Process failed: bash -c /tmp/opsc_3882111672138551416/dense.sh
Exit val: 126
Output:
bash: /tmp/opsc_3882111672138551416/dense.sh: Permission denied
at opsagent.proc$handle_proc_results.invokeStatic(proc.clj:61)
at opsagent.proc$handle_proc_results.invoke(proc.clj:51)
at opsagent.proc$run_proc.invokeStatic(proc.clj:84)
at opsagent.proc$run_proc.doInvoke(proc.clj:65)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at opsagent.environment.utils$package_config_paths.invokeStatic(utils.clj:161)
at opsagent.environment.utils$package_config_paths.invoke(utils.clj:141)
at opsagent.environment.utils$all_config_paths.invokeStatic(utils.clj:197)
at opsagent.environment.utils$all_config_paths.doInvoke(utils.clj:190)
at clojure.lang.RestFn.invoke(RestFn.java:805)
at opsagent.environment.dynamic$dynamic_env_state.invokeStatic(dynamic.clj:162)
at opsagent.environment.dynamic$dynamic_env_state.invoke(dynamic.clj:148)
at clojure.lang.AFn.applyToHelper(AFn.java:171)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:652)
at clojure.core$partial$fn__4765.doInvoke(core.clj:2534)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at opsagent.jmx$create_jmx_pool_with_config$wrapper__5941.doInvoke(jmx.clj:239)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at opsagent.environment.dynamic$add_dynamic_state.invokeStatic(dynamic.clj:276)
at opsagent.environment.dynamic$add_dynamic_state.invoke(dynamic.clj:264)
at opsagent.environment.dynamic.DynamicEnvironmentComponent.start(dynamic.clj:299)
at com.stuartsierra.component$fn__2593$G__2587__2595.invoke(component.clj:4)
at com.stuartsierra.component$fn__2593$G__2586__2598.invoke(component.clj:4)
at clojure.lang.Var.invoke(Var.java:379)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.core$apply.invokeStatic(core.clj:648)
at clojure.core$apply.invoke(core.clj:641)
at com.stuartsierra.component$try_action.invokeStatic(component.clj:116)
at com.stuartsierra.component$try_action.invoke(component.clj:115)
at clojure.lang.Var.invoke(Var.java:401)
at opsagent.config_service$update_system$fn__22445.invoke(config_service.clj:223)
at clojure.lang.ArraySeq.reduce(ArraySeq.java:114)
at clojure.core$reduce.invokeStatic(core.clj:6544)
at clojure.core$reduce.invoke(core.clj:6527)
at opsagent.config_service$update_system.invokeStatic(config_service.clj:217)
at opsagent.config_service$update_system.doInvoke(config_service.clj:213)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at opsagent.config_service$start_system_BANG_.invokeStatic(config_service.clj:243)
at opsagent.config_service$start_system_BANG_.invoke(config_service.clj:236)
at opsagent.config_service$fn__22551$fn__22552$state_machine__4942__auto____22553$fn__22555.invoke(config_service.clj:266)
at opsagent.config_service$fn__22551$fn__22552$state_machine__4942__auto____22553.invoke(config_service.clj:266)
at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973)
at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975)
at clojure.core.async.impl.ioc_macros$take_BANG_$fn__4958.invoke(ioc_macros.clj:986)
at clojure.core.async.impl.channels.ManyToManyChannel$fn__707$fn__708.invoke(channels.clj:95)
at clojure.lang.AFn.run(AFn.java:22)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
INFO [async-dispatch-2] 2018-07-24 09:23:19,917 Starting JMXComponent
please note "/tmp/opsc_3882111672138551416/dense.sh: Permission denied" in your logs.
You probably don't have permissions to create anything under /tmp/
You can try fix the permissions or to reconfigure your temporary directory with -Djava.io.tmpdir in datastax-agent-env.sh:
JVM_OPTS="$JVM_OPTS -Xmx128M -Djava.io.tmpdir=/other/temp/directory"
You can find it here: /usr/share/datastax-agent/bin/
In version 6, there is datastax-agent instead of datastax-agent-env.sh
Remember to add this line at the beggining of datastax-agent file
After using Adding an existing cluster to add DSE(6.0.1) cluster to OpsCenter, and select install agent automatically after all agents installed I have something like below picture in OpsCenter(version 6.5):
As you see MONITORED DSE is down.
When see log of agent see below error :
INFO [async-dispatch-1] 2018-07-02 15:37:30,962 Starting system.
INFO [async-dispatch-1] 2018-07-02 15:37:30,966 Starting DynamicEnvironmentComponent
WARN [async-dispatch-1] 2018-07-02 15:37:30,987 Exception while processing JMX data: java.lang.NullPointerException
ERROR [async-dispatch-1] 2018-07-02 15:37:30,988 Error starting DynamicEnvironmentComponent.
java.lang.NullPointerException
at clojure.java.io$as_relative_path.invokeStatic(io.clj:414)
at clojure.java.io$file.invokeStatic(io.clj:426)
at clojure.java.io$file.invoke(io.clj:418)
at opsagent.environment.collection$cassandra_yaml_location__GT_install_location.invokeStatic(collection.clj:128)
at opsagent.environment.collection$cassandra_yaml_location__GT_install_location.invoke(collection.clj:119)
at opsagent.environment.dynamic$dynamic_env_state.invokeStatic(dynamic.clj:151)
at opsagent.environment.dynamic$dynamic_env_state.invoke(dynamic.clj:148)
at clojure.lang.AFn.applyToHelper(AFn.java:171)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:652)
at clojure.core$partial$fn__4765.doInvoke(core.clj:2534)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at opsagent.jmx$create_jmx_pool_with_config$wrapper__5941.doInvoke(jmx.clj:239)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at opsagent.environment.dynamic$add_dynamic_state.invokeStatic(dynamic.clj:276)
at opsagent.environment.dynamic$add_dynamic_state.invoke(dynamic.clj:264)
at opsagent.environment.dynamic.DynamicEnvironmentComponent.start(dynamic.clj:299)
at com.stuartsierra.component$fn__2593$G__2587__2595.invoke(component.clj:4)
at com.stuartsierra.component$fn__2593$G__2586__2598.invoke(component.clj:4)
at clojure.lang.Var.invoke(Var.java:379)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.core$apply.invokeStatic(core.clj:648)
at clojure.core$apply.invoke(core.clj:641)
at com.stuartsierra.component$try_action.invokeStatic(component.clj:116)
at com.stuartsierra.component$try_action.invoke(component.clj:115)
at clojure.lang.Var.invoke(Var.java:401)
at opsagent.config_service$update_system$fn__22445.invoke(config_service.clj:223)
at clojure.lang.ArraySeq.reduce(ArraySeq.java:114)
at clojure.core$reduce.invokeStatic(core.clj:6544)
at clojure.core$reduce.invoke(core.clj:6527)
at opsagent.config_service$update_system.invokeStatic(config_service.clj:217)
at opsagent.config_service$update_system.doInvoke(config_service.clj:213)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at opsagent.config_service$start_system_BANG_.invokeStatic(config_service.clj:243)
at opsagent.config_service$start_system_BANG_.invoke(config_service.clj:236)
at opsagent.config_service$fn__22551$fn__22552$state_machine__4942__auto____22553$fn__22555.invoke(config_service.clj:266)
at opsagent.config_service$fn__22551$fn__22552$state_machine__4942__auto____22553.invoke(config_service.clj:266)
at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973)
at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975)
at clojure.core.async$ioc_alts_BANG_$fn__5131.invoke(async.clj:384)
at clojure.core.async$do_alts$fn__5085$fn__5088.invoke(async.clj:253)
at clojure.core.async.impl.channels.ManyToManyChannel$fn__754.invoke(channels.clj:265)
at clojure.lang.AFn.run(AFn.java:22)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO [async-dispatch-1] 2018-07-02 15:37:30,991 Finished starting system.
what's problem? Does anyone have an idea? How can I fix it?
I finally solved it, this error occurs when OpsCenter can't find config of DSE in another word OpsCenter agent can't find DSE when you installing DSE via tarball, you should not install DSE in /etc/dse or /etc/* because when you install DSE by package it uses this directory and when you use it in tarball installation become confused, it's bug of DSE installation if you change installation directory to /opt or another, the problem is solved.
I tried upgrading Cassandra from 2.2.1 to 3.0.0 but Cassandra doesn't start:
ERROR [main] 2015-11-30 15:44:50,164 CassandraDaemon.java:702 - Exception encountered during startup
java.lang.NullPointerException: null
at org.apache.cassandra.io.util.FileUtils.delete(FileUtils.java:374) ~[apache-cassandra-3.0.0.jar:3.0.0]
at org.apache.cassandra.db.SystemKeyspace.migrateDataDirs(SystemKeyspace.java:1341) ~[apache-cassandra-3.0.0.jar:3.0.0]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:180) [apache-cassandra-3.0.0.jar:3.0.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:561) [apache-cassandra-3.0.0.jar:3.0.0]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689) [apache-cassandra-3.0.0.jar:3.0.0]
Anyone else faces this problem?
I also raised an issue here:
https://issues.apache.org/jira/browse/CASSANDRA-10788
Here is TRACE level logging:
TRACE [MemtablePostFlush:1] 2015-12-01 16:47:52,675 ColumnFamilyStore.java:868 - forceFlush requested but everything is clean in schema_columns
TRACE [main] 2015-12-01 16:47:52,675 ColumnFamilyStore.java:1563 - Snapshot for Keyspace(name='system') keyspace data file /data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/la-46-big-Data.db created in /data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/snapshots/1448984872341-upgrade-2.2.1-3.0.0
TRACE [main] 2015-12-01 16:47:52,676 ColumnFamilyStore.java:1563 - Snapshot for Keyspace(name='system') keyspace data file /data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/la-45-big-Data.db created in /data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/snapshots/1448984872341-upgrade-2.2.1-3.0.0
TRACE [main] 2015-12-01 16:47:52,676 ColumnFamilyStore.java:1563 - Snapshot for Keyspace(name='system') keyspace data file /data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/la-47-big-Data.db created in /data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/snapshots/1448984872341-upgrade-2.2.1-3.0.0
TRACE [main] 2015-12-01 16:47:52,676 SystemKeyspace.java:1327 - Checking directory /data for old files
ERROR [main] 2015-12-01 16:47:52,751 CassandraDaemon.java:702 - Exception encountered during startup
java.lang.NullPointerException: null
at org.apache.cassandra.io.util.FileUtils.delete(FileUtils.java:374) ~[apache-cassandra-3.0.0.jar:3.0.0]
at org.apache.cassandra.db.SystemKeyspace.migrateDataDirs(SystemKeyspace.java:1341) ~[apache-cassandra-3.0.0.jar:3.0.0]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:180) [apache-cassandra-3.0.0.jar:3.0.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:561) [apache-cassandra-3.0.0.jar:3.0.0]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689) [apache-cassandra-3.0.0.jar:3.0.0]
It looks like the code is expected legacy files under /data directory but it can't find any and call FileUtils.delete(null).
One thing that may impact your migration is the 2.1.x to 3.0 conversion is not recommended by Datastax.
Cassandra 3.0.x restrictions
Upgrade from Cassandra 2.1 versions greater or equal to 2.1.9 or from
Cassandra 2.2 versions greater or equal to 2.2.2.
*Emphasis added
The null pointer could be happening due to non-existent files or a duplicated delete call.
I would try upgrading to at least 2.2.2 before the 3.0 migration.
I was able to fix the problem by using this patch:
https://github.com/stef1927/cassandra/commit/1c464adf097d323320ce11db6daf05e1a31c62b6
More details:
https://issues.apache.org/jira/browse/CASSANDRA-10788
And thanks, I'll see if upgrading from 2.2.1 to 2.2.2 first helps.
I am trying to install CDH5 (cloudera) on a single node(64-bit machine) running ubuntu 14.04 LTS. I have successfully installed hdfs (namenode and datanode are running fine). After configuring all the properties for deploying yarn, when I try to start resource manager or node manager it gives gives following errors:
ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Returning, interrupted : java.lang.InterruptedException
2015-06-29 14:02:44,265 ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
2015-06-29 14:02:44,264 INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer thread interrupted
2015-06-29 14:02:44,265 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to standby state
2015-06-29 14:02:44,265 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:278)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:983) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1083)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1215)
Caused by: java.io.IOException: Problem in starting http server. Server handlers failed
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:841)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:273)
... 4 more
Any kind of help will be appreciated.
I have problems to get a existing Cassandra node to join the cluster again after reboot (on a new virtual machine instance).
I had a running Cassandra cluster with 4 nodes all in state "up and normal" according to nodetool status. The nodes are running on virtual machines in Azure. I changed the instance type of the virtual machine for 10.0.0.6, which returned in a reboot of this machine. The machine stayed on 10.0.0.6.
After the reboot I am unable to start Cassandra again. I am getting this exception:
INFO 22:39:07 Handshaking version with /10.0.0.4
INFO 22:39:07 Node /10.0.0.6 is now part of the cluster
INFO 22:39:07 Node /10.0.0.5 is now part of the cluster
INFO 22:39:07 Handshaking version with cassandraprd001/10.0.0.6
INFO 22:39:07 Node /10.0.0.9 is now part of the cluster
INFO 22:39:07 Handshaking version with /10.0.0.5
INFO 22:39:07 Node /10.0.0.4 is now part of the cluster
INFO 22:39:07 InetAddress /10.0.0.6 is now UP
INFO 22:39:07 Handshaking version with /10.0.0.9
INFO 22:39:07 InetAddress /10.0.0.4 is now UP
INFO 22:39:07 InetAddress /10.0.0.9 is now UP
INFO 22:39:07 InetAddress /10.0.0.5 is now UP
ERROR 22:39:08 Exception encountered during startup
java.lang.RuntimeException: A node with address cassandraprd001/10.0.0.6 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:455) ~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:667) ~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:615) ~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:509) ~[apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338) [apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:457) [apache-cassandra-2.1.0.jar:2.1.0]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:546) [apache-cassandra-2.1.0.jar:2.1.0]
java.lang.RuntimeException: A node with address cassandraprd001/10.0.0.6 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:455)
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:667)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:615)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:509)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:457)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:546)
Exception encountered during startup: A node with address cassandraprd001/10.0.0.6 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
INFO 22:39:08 Announcing shutdown
I am using Cassandra 2.1.0. I am not replaying a dead node - I am just trying to get the old node up and running again. According to nodetool status (on the other nodes) all nodes are "up and normal" except 10.0.0.6 which is "down and normal".
How do I get this node up and running again?
First, on another node, use
nodetool status
the results show you list of nodes in the cluster. Find your node with ip which fail to start, get its id and fill to command:
nodetool removenode <node_id>
then start cassandra.
Best,
Quick answer, if the node's ip is 10.200.10.200
add this
JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=10.200.10.200"
to the end of your
cassandra-env.sh
Don't forget to remove it once your done.
You can look this blog, http://blog.alteroot.org/articles/2014-03-12/replace-a-dead-node-in-cassandra.html.
It works for me, this is a bug for Cassandra. If your node's host_id changed, but use old IP, will throw this exception.
If you use Cassandra 2.x.x, you should modify cassandra/conf/cassandra-env.sh.
Finally, don't forget to REMOVE modifications on cassandra-env.sh after the complete bootstrap!