DataStax agent 6.5 not connecting to DSE cluster - cassandra

I have installed a DSE 6.0 cassandra cluster using LCM| opscenter 6.5 and node is up is running . During LCM cluster install, it installed datastax agent as well .
But the agent is not connecting to DSE and opscenter is not showing any details about the node.Later I tried with tarball install of datastax agent but that too is showing same issue. Please see below agent.log and screenshots.
WARN [async-dispatch-2] 2018-07-24 09:23:19,915 JMX marked as down, restarting JMX components.
ERROR [async-dispatch-2] 2018-07-24 09:23:19,916 Error starting DynamicEnvrionmentComponent.
java.io.IOException: Process failed: bash -c /tmp/opsc_3882111672138551416/dense.sh
Exit val: 126
Output:
bash: /tmp/opsc_3882111672138551416/dense.sh: Permission denied
at opsagent.proc$handle_proc_results.invokeStatic(proc.clj:61)
at opsagent.proc$handle_proc_results.invoke(proc.clj:51)
at opsagent.proc$run_proc.invokeStatic(proc.clj:84)
at opsagent.proc$run_proc.doInvoke(proc.clj:65)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at opsagent.environment.utils$package_config_paths.invokeStatic(utils.clj:161)
at opsagent.environment.utils$package_config_paths.invoke(utils.clj:141)
at opsagent.environment.utils$all_config_paths.invokeStatic(utils.clj:197)
at opsagent.environment.utils$all_config_paths.doInvoke(utils.clj:190)
at clojure.lang.RestFn.invoke(RestFn.java:805)
at opsagent.environment.dynamic$dynamic_env_state.invokeStatic(dynamic.clj:162)
at opsagent.environment.dynamic$dynamic_env_state.invoke(dynamic.clj:148)
at clojure.lang.AFn.applyToHelper(AFn.java:171)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:652)
at clojure.core$partial$fn__4765.doInvoke(core.clj:2534)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at opsagent.jmx$create_jmx_pool_with_config$wrapper__5941.doInvoke(jmx.clj:239)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at opsagent.environment.dynamic$add_dynamic_state.invokeStatic(dynamic.clj:276)
at opsagent.environment.dynamic$add_dynamic_state.invoke(dynamic.clj:264)
at opsagent.environment.dynamic.DynamicEnvironmentComponent.start(dynamic.clj:299)
at com.stuartsierra.component$fn__2593$G__2587__2595.invoke(component.clj:4)
at com.stuartsierra.component$fn__2593$G__2586__2598.invoke(component.clj:4)
at clojure.lang.Var.invoke(Var.java:379)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.core$apply.invokeStatic(core.clj:648)
at clojure.core$apply.invoke(core.clj:641)
at com.stuartsierra.component$try_action.invokeStatic(component.clj:116)
at com.stuartsierra.component$try_action.invoke(component.clj:115)
at clojure.lang.Var.invoke(Var.java:401)
at opsagent.config_service$update_system$fn__22445.invoke(config_service.clj:223)
at clojure.lang.ArraySeq.reduce(ArraySeq.java:114)
at clojure.core$reduce.invokeStatic(core.clj:6544)
at clojure.core$reduce.invoke(core.clj:6527)
at opsagent.config_service$update_system.invokeStatic(config_service.clj:217)
at opsagent.config_service$update_system.doInvoke(config_service.clj:213)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at opsagent.config_service$start_system_BANG_.invokeStatic(config_service.clj:243)
at opsagent.config_service$start_system_BANG_.invoke(config_service.clj:236)
at opsagent.config_service$fn__22551$fn__22552$state_machine__4942__auto____22553$fn__22555.invoke(config_service.clj:266)
at opsagent.config_service$fn__22551$fn__22552$state_machine__4942__auto____22553.invoke(config_service.clj:266)
at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973)
at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975)
at clojure.core.async.impl.ioc_macros$take_BANG_$fn__4958.invoke(ioc_macros.clj:986)
at clojure.core.async.impl.channels.ManyToManyChannel$fn__707$fn__708.invoke(channels.clj:95)
at clojure.lang.AFn.run(AFn.java:22)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
INFO [async-dispatch-2] 2018-07-24 09:23:19,917 Starting JMXComponent

please note "/tmp/opsc_3882111672138551416/dense.sh: Permission denied" in your logs.
You probably don't have permissions to create anything under /tmp/
You can try fix the permissions or to reconfigure your temporary directory with -Djava.io.tmpdir in datastax-agent-env.sh:
JVM_OPTS="$JVM_OPTS -Xmx128M -Djava.io.tmpdir=/other/temp/directory"

You can find it here: /usr/share/datastax-agent/bin/
In version 6, there is datastax-agent instead of datastax-agent-env.sh
Remember to add this line at the beggining of datastax-agent file

Related

Apache Pulsar Zookeeper: Unable to access datadir, exiting abnormally

I am using these steps to use apache pulsar on docker: https://github.com/streamnative/tgip/blob/master/episodes/001/demo.md
I was able to use these steps before to install and use pulsar but for some reason now when am creating a directory, it is going to write protected and pulsar zookeeper container is exiting with following logs as soon as it is created:
ERROR org.apache.zookeeper.server.ZooKeeperServerMain - Unable to access datadir, exiting abnormally
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Unable to create data directory data/zookeeper/version-2
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:136) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:137) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:112) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:67) [org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:140) [org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:90) [org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
Unable to access datadir, exiting abnormally
23:36:15.223 [main] INFO org.apache.zookeeper.audit.ZKAuditProvider - ZooKeeper audit is disabled.
23:36:15.226 [main] ERROR org.apache.zookeeper.util.ServiceUtils - Exiting JVM with code 3
23:36:15.196 [PurgeTask] ERROR org.apache.zookeeper.server.DatadirCleanupManager - Error occurred while purging.
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Unable to create data directory data/zookeeper/version-2
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:136) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:80) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:141) [org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at java.util.TimerThread.mainLoop(Timer.java:556) [?:?]
at java.util.TimerThread.run(Timer.java:506) [?:?]
23:36:15.229 [PurgeTask] INFO org.apache.zookeeper.server.DatadirCleanupManager - Purge task completed
I have made sure that SELinux is disabled and tried changing permission using chmod 777 data/ and every other step available to resolve this but still unable to find any. Please help me with the possible resolution.

Get "Error starting DynamicEnvironmentComponent." in staring OpsCenter 6.5 agents.DSE 6.0.1

After using Adding an existing cluster to add DSE(6.0.1) cluster to OpsCenter, and select install agent automatically after all agents installed I have something like below picture in OpsCenter(version 6.5):
As you see MONITORED DSE is down.
When see log of agent see below error :
INFO [async-dispatch-1] 2018-07-02 15:37:30,962 Starting system.
INFO [async-dispatch-1] 2018-07-02 15:37:30,966 Starting DynamicEnvironmentComponent
WARN [async-dispatch-1] 2018-07-02 15:37:30,987 Exception while processing JMX data: java.lang.NullPointerException
ERROR [async-dispatch-1] 2018-07-02 15:37:30,988 Error starting DynamicEnvironmentComponent.
java.lang.NullPointerException
at clojure.java.io$as_relative_path.invokeStatic(io.clj:414)
at clojure.java.io$file.invokeStatic(io.clj:426)
at clojure.java.io$file.invoke(io.clj:418)
at opsagent.environment.collection$cassandra_yaml_location__GT_install_location.invokeStatic(collection.clj:128)
at opsagent.environment.collection$cassandra_yaml_location__GT_install_location.invoke(collection.clj:119)
at opsagent.environment.dynamic$dynamic_env_state.invokeStatic(dynamic.clj:151)
at opsagent.environment.dynamic$dynamic_env_state.invoke(dynamic.clj:148)
at clojure.lang.AFn.applyToHelper(AFn.java:171)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:652)
at clojure.core$partial$fn__4765.doInvoke(core.clj:2534)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at opsagent.jmx$create_jmx_pool_with_config$wrapper__5941.doInvoke(jmx.clj:239)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at opsagent.environment.dynamic$add_dynamic_state.invokeStatic(dynamic.clj:276)
at opsagent.environment.dynamic$add_dynamic_state.invoke(dynamic.clj:264)
at opsagent.environment.dynamic.DynamicEnvironmentComponent.start(dynamic.clj:299)
at com.stuartsierra.component$fn__2593$G__2587__2595.invoke(component.clj:4)
at com.stuartsierra.component$fn__2593$G__2586__2598.invoke(component.clj:4)
at clojure.lang.Var.invoke(Var.java:379)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.core$apply.invokeStatic(core.clj:648)
at clojure.core$apply.invoke(core.clj:641)
at com.stuartsierra.component$try_action.invokeStatic(component.clj:116)
at com.stuartsierra.component$try_action.invoke(component.clj:115)
at clojure.lang.Var.invoke(Var.java:401)
at opsagent.config_service$update_system$fn__22445.invoke(config_service.clj:223)
at clojure.lang.ArraySeq.reduce(ArraySeq.java:114)
at clojure.core$reduce.invokeStatic(core.clj:6544)
at clojure.core$reduce.invoke(core.clj:6527)
at opsagent.config_service$update_system.invokeStatic(config_service.clj:217)
at opsagent.config_service$update_system.doInvoke(config_service.clj:213)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at opsagent.config_service$start_system_BANG_.invokeStatic(config_service.clj:243)
at opsagent.config_service$start_system_BANG_.invoke(config_service.clj:236)
at opsagent.config_service$fn__22551$fn__22552$state_machine__4942__auto____22553$fn__22555.invoke(config_service.clj:266)
at opsagent.config_service$fn__22551$fn__22552$state_machine__4942__auto____22553.invoke(config_service.clj:266)
at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973)
at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975)
at clojure.core.async$ioc_alts_BANG_$fn__5131.invoke(async.clj:384)
at clojure.core.async$do_alts$fn__5085$fn__5088.invoke(async.clj:253)
at clojure.core.async.impl.channels.ManyToManyChannel$fn__754.invoke(channels.clj:265)
at clojure.lang.AFn.run(AFn.java:22)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO [async-dispatch-1] 2018-07-02 15:37:30,991 Finished starting system.
what's problem? Does anyone have an idea? How can I fix it?
I finally solved it, this error occurs when OpsCenter can't find config of DSE in another word OpsCenter agent can't find DSE when you installing DSE via tarball, you should not install DSE in /etc/dse or /etc/* because when you install DSE by package it uses this directory and when you use it in tarball installation become confused, it's bug of DSE installation if you change installation directory to /opt or another, the problem is solved.

Stop Cassandra update automatically

Please help me how I can stop Cassandra update automatically?
At present when I install Cassandra through opscentre I get an error.
Unable to restart DSE service. See /var/log/cassandra/system.log and /var/log/cassandra/output.log on the target node for details.
system.log
ERROR [main] 2018-03-28 07:58:26,123 CassandraDaemon.java:705 - Exception encountered during startup
java.lang.AbstractMethodError: org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150) ~[na:1.8.0_162]
at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135) ~[na:1.8.0_162]
at javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405) ~[na:1.8.0_162]
at org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:143) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:471) ~[dse-core-5.1.0.jar:5.1.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:599) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at com.datastax.bdp.DseModule.main(DseModule.java:93) [dse-core-5.1.0.jar:5.1.0]
But when I update my OS with the command **yum update**, it updates DSE 5.1.0 to 5.1.7. After updating dse start. I am not able to understand why its happening?
On other node dse 5.1 working perfectly.
OS: centos 7

NEO4J local server does not start

I am running Linux in VirtualBox and am having an issue that I did not encounter on my machine with Linux as the primary OS.
When launching the neo4j service through sudo ./neo4j start in /opt/neo4j-community-2.3.1/bin I get a timeout with the message Failed to start within 120 seconds. Neo4j Server may have failed to start, please check the logs
my log from /opt/neo4j-community-2.3.1/data/graph.db/messages.log says:
http://pastebin.com/wUA715QQ
and data/log/console.log says:
2016-01-06 02:07:03.404+0100 INFO Successfully started database
2016-01-06 02:07:03.603+0100 INFO Successfully stopped database
2016-01-06 02:07:03.604+0100 INFO Successfully shutdown Neo4j Server
2016-01-06 02:07:03.608+0100 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.security.auth.FileUserRepository#9ab182' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.security.auth.FileUserRepository#9ab182' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.security.auth.FileUserRepository#9ab182' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:67)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:234)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:97)
at org.neo4j.server.CommunityBootstrapper.start(CommunityBootstrapper.java:48)
at org.neo4j.server.CommunityBootstrapper.main(CommunityBootstrapper.java:35)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.security.auth.FileUserRepository#9ab182' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:462)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:194)
... 3 more
Caused by: java.nio.file.AccessDeniedException: /opt/neo4j-community-2.3.1/data/dbms/auth
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.Files.readAllBytes(Files.java:3152)
at org.neo4j.server.security.auth.FileUserRepository.loadUsersFromFile(FileUserRepository.java:208)
at org.neo4j.server.security.auth.FileUserRepository.start(FileUserRepository.java:73)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 5 more
Any idea why the server won't start?
Check the permissions on /opt/neo4j-community-2.3.1/data/dbms/auth
See the line that says:
Caused by: java.nio.file.AccessDeniedException: /opt/neo4j-community-2.3.1/data/dbms/auth

Dataxtax Agent error

While adding existing cluster in OpsCenter I receive an error:
ERROR: Agent for XXX.XXX.XXX.XXX was unable to complete operation (http://XXX.XXX.XXX.XXX:61621/snapshots/pit/properties?): java.lang.IllegalArgumentException: No implementation of method: :make-reader of protocol: #'clojure.java.io/IOFactory found for class: nil
On agent there is an error:
java.lang.IllegalArgumentException: No implementation of method: :make-reader of protocol: #'clojure.java.io/IOFactory found for class: nil
at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:541)
at clojure.java.io$fn__8551$G__8546__8558.invoke(io.clj:73)
at clojure.java.io$reader.doInvoke(io.clj:106)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at clojure.lang.AFn.applyToHelper(AFn.java:161)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.core$apply.invoke(core.clj:619)
at clojure.core$slurp.doInvoke(core.clj:6278)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at opsagent.backups.pit$read_properties.invoke(pit.clj:68)
at opsagent.backups.pit$enabled_QMARK_.invoke(pit.clj:106)
at clojure.core$eval37.invoke(NO_SOURCE_FILE:107)
at clojure.lang.Compiler.eval(Compiler.java:6619)
at clojure.lang.Compiler.eval(Compiler.java:6609)
at clojure.lang.Compiler.eval(Compiler.java:6582)
at clojure.core$eval.invoke(core.clj:2852)
at opsagent.opsagent$post_interface_startup.doInvoke(opsagent.clj:102)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at opsagent.conf$handle_new_conf.invoke(conf.clj:198)
at opsagent.messaging$message_callback$fn__12316.invoke(messaging.clj:52)
at opsagent.messaging.proxy$java.lang.Object$StompConnection$Listener$7f16bc72.onMessage(Unknown Source)
at org.jgroups.client.StompConnection.notifyListeners(StompConnection.java:324)
at org.jgroups.client.StompConnection.run(StompConnection.java:274)
at java.lang.Thread.run(Thread.java:745)
And cluster creation failed. Also i get this error during startup. I tried reinstall agent but in won't help
DataStax Agent version: 5.1.0
OpsCenter version 5.1.0
root#node1:~# java -version
java version "1.7.0_75"
OpenJDK Runtime Environment (IcedTea 2.5.4) (7u75-2.5.4-1~deb7u1)
OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)
root#node1:~#
Content of address.yaml
stomp_interface: "YYY.YYY.YYY.YYY"
Content of opscenterd.conf
[webserver]
port = 8888
interface = 0.0.0.0
use_ssl = false
[logging]
level = INFO
<cluster name>.conf is absent, because cluster not added
The problem the agent is having is finding your installation of DSE on that node. When it can't find DSE it can't get the archiving properties file to update and errors out.
This error message is unfortunately terribly unhelpful. I've created a ticket to fix the error message (it's unfortunately private, but you can use this ticket number when discussing the issue with DataStax: OPSC-4826)
For a work around, try setting cassandra_install_location in your address.yaml file on that node. After adjusting address.yaml please bounce the agent and you can retry that operation.
You can find a document listing this and more address.yaml config items here: http://www.datastax.com/documentation/opscenter/5.1/opsc/configure/agentAddressConfiguration.html
I think the issue will be with your Java installation. I believe you'll need Oracle Java, not OpenJDK.
This worked for me:
ubuntu:~$ sudo add-apt-repository ppa:webupd8team/java
ubuntu:~$ sudo apt-get update && sudo apt-get install oracle-java7-installer oracle-java7-set-default

Resources