I would like to know how OpsCenter communicates with its Agents and Cassandra Nodes.
Does it use Thrift? Is JMX required?
I'll base my answer on the latest released version of OpsCenter (1.3).
The main OpsCenter process can communicate with the agents in two ways. It can query the agents over an http rest api that each agent exposes. It uses this to ask the agent basic things about the cassandra node and also to have the agent send jmx commands to the cassandra process.
The other way is using the STOMP protocol. (http://stomp.github.com//) Agents send messages over STOMP to a message queue in OpsCenter. These generally contain details about the cassandra node and metric information.
Hope that helps.
Related
This question was migrated from Stack Overflow because it can be answered on Database Administrators Stack Exchange.
Migrated 26 days ago.
The previous DBA created and connected a cluster (4 nodes) to DSE OpsCenter. It's almost perfect (nodes are connected and visible) except for the STOMP service.
In the OpsCenter -> Nodes -> Agents there are question marks in the STOMP service column.
What should I check first?
The firewall seems to be ok, like in the other 3 clusters, which work perfectly fine.
The DataStax Enterprise (DSE) node agents communicate with the OpsCenter server over STOMP. The yellow question mark in the Agents Status View in the OpsCenter UI indicates that OpsCenter doesn't know the status of the agent because OpsCenter can't talk to it.
You stated that "the firewall seems to be ok" but you didn't provide details of what you think is "ok". You need to specifically check that there is connectivity between OpsCenter and the agents.
The opscenterd daemon listens for TCP traffic from the agents on stomp_port which by default is port 61620. The agents listen for TCP traffic from the OpsCenter server on agents_api_port which by default is port 61621 (see OpsCenter ports reference for details).
Once you've confirmed that bi-directional network traffic is working between OpsCenter and the agents, you need to check the following logs for clues:
/var/log/opscenter/opscenterd.log on the OpsCenter server
/var/log/datastax-agent/agent.log on the nodes
The errors in the logs will give you clues as to why OpsCenter cannot connect to the agents.
And as a friendly reminder that if you need assistance with troubleshooting the problem, you should log a ticket with DataStax Support and one of their engineers will help you. Cheers!
👉 Please support the Apache Cassandra community by hovering over the cassandra tag then click on the Watch tag button. 🙏 Thanks!
What is the best way to monitor if cassandra nodes are up? Due to security reasons JMX and nodetool is out of question. I have cluster metrics monitoring via Rest Api, but I understand that even if a node goes Rest Api will only report on a whole cluster.
Well, I have integrated a system where I can monitor all the metrics regarding to my cluster of all nodes. This seems like complicated but pretty simple to integrate. You will need the following components to build up a monitoring system for cassandra:
jolokia jar
telegraf
influxdb
grafana
I'm writing a short procedure, how it works.
Step 1: copy jolokia jvm jar to install_dir/apache-cassandra-version/lib/ , jolokia jvm agent can be downloaded from anywhere in google.
Step 2: add the following line to install_dir/apache-cassandra-version/conf/cassandra-env.sh
JVM_OPTS="$JVM_OPTS -javaagent:<here_goes_the_path_of_your_jolokia_jar>"
Step 3: install telegraf on each node and configure the metrics you want to monitor. and start telegraf service.
Step 4: install grafana and configure your ip, port, protocol. grafana will give you a dashboard to look after your nodes and start grafana service. Your metrics will be able get visibility here.
Step 5: install influxdb on another server from where you want to store your metrics data which will come through telegraf agent.
Step 6: browse the ip you have mentioned, where you have launched your grafana through browser and add data source ip (influxdb ip), then customize your dashboard.
image source: https://blog.pythian.com/monitoring-cassandra-grafana-influx-db/
This is not for monitoring but only for node state.
Cassandra CQL driver provides info if a particular node is UP or DOWN with Host.StateListener Interface. This info is used by driver to mark a node UP or Down. Thus it could be used if node is down or up if JMX is not accessible.
Java Doc API : https://docs.datastax.com/en/drivers/java/3.3/
I came up with a script which listens for DN nodes in the cluster and reports it to our monitoring setup which is integrated with pagerduty.
The script runs on one of our nodes and executes nodetool status every minute and reports for all down nodes.
Here is the script https://gist.github.com/johri21/87d4d549d05c3e2162af7929058a00d1
[1]:
I am trying to get to know Cassandra cfstats information from all the machines using JMX. This can be done using OpsCenter, but I do not want to use it. I started building my own utility. For now, my java program connects to JMX and fetching cfstats information such as estimateKeys, No of SSTables ..etc.
My requirement is, This is a java jar file, will run from one Cassandra node and should be able to connect to all the machines and fetch cfstats using their respective JMX per node.
I am planning to use java driver for this, as java driver will be able to connects all the machines in the cluster using system.peers coumnFamily. Once java driver connect to the machines, I will form the service:jmx:rmi using respective hostname and JMX port(7199). Then I will be able to connect to NodeProbe using this information.
My question is, after connecting to the another node using java driver, will I be able to retain state there and after forming service:jmx:rmi url, will this url really connects to the current node JMX and pull cfstats information from the current node. Because JMX host name it will take from the Cassandra-env.sh file. Can some one please help me in this.
Does this idea works or is there another best way to achieve this?
It's possible to use JMX remotely, but that's not the easiest thing to do.
But if you are writing your tool - maybe it's worth to check out a different connection. E.g. you can easily convert JMX calls to HTTP using Jolokia
I have install 3 arangodb servers. But i have always the same listening port 8529 no 8530 for coordinator so i cannot create a cluster.
tcp 0 0 0.0.0.0:8529 0.0.0.0:* LISTEN 13142/arangod
So when i try to create a cluster via the web interface, i have the following error
ERROR bootstrapping DB servers failed: Could not connect to 'tcp://10.0.0.18:8530' 'connect() failed with #111 - Connection refused'
How can i start and/or configure the corrdinator to have a listen on my servers?
Regards
Dispatcher based clusters
Please note that dispatcher based setups as you asked are intended for evaluation purposes only.
To start a cluster from the dispatcher webfrontend you need to configure all nodes to start the arangod daemon in dispatcher mode:
[cluster]
disable-dispatcher-kickstarter = no
disable-dispatcher-frontend = no
To start a cluster on a single machine you only need to install ArangoDB and reconfigure it once; it will then use the same installation to start the dispatcher and dbserver nodes.
One should know that the initial cluster startup may take a while.
Another side note is that authentication is not supported in this scenario, so you may need to turn it off.
You should now find the log output of the dbserver and coordinator instances under /var/log/arangodb/cluster/ so you can get the actual informations of what went wrong.
Script based cloud install clusters
A better way to get a cluster running in the cloud may be to use one of the scripts we prepared for Digital Ocean, Google Compute Engine, AWS or Azure.
ArangoDB Clusters based on Mesosphere DCOS
The currently recommended way of running an ArangoDB cluster is to use Mesosphere DCOS, as Max describes in these slides using some example configurations.
ArangoDB is an official Mesosphere partner and we offer an official DCOS subcommand to manage an ArangoDB Cluster on Mesosphere DCOS.
Mesosphere adds additional services on top of Mesos and eases management of the Mesos cluster via the dcos-cli.
If you want to use a raw Apache Mesos Cluster, you can use the Mesos framework directly to schedule the neccessary tasks to create ArangoDB cluster.
Meanwhile there is a better article about Running ArangoDB on DC/OS.
The usual method for installing the opscenter agent whether via the opscenter server or manually is to install it on the cassandra node server.
Is the agent configuration flexible enough that it can run anywhere and monitor cassandra remotely? Are there complexities that may not be readily apparent?
tl;dr - No
While there are several config options that would let the agent monitor certain JMX metrics remotely, you would lose critical metrics and functionality such as disk and CPU metrics, and the ability to stop/start/configure the Cassandra process. These all require the agent to be running locally.