I am using java -jar hawtio-app-2.16.1.jar --port 1080
I have some activemq and karaf (with camel) on the same machine.
I saw on hawtio console > connect > discover
No agents discovered.
I think it is possible to configure hawtio to discover all jolokia agents on my machine.
But I can't find this in the documentation.
So all my activeMQ agents are on localhost:1xx80/api/jolokia and all karaf agents are on localhost:2xx80/jolokia
How to configure hawtio to discover them automatically?
Related
This question was migrated from Stack Overflow because it can be answered on Database Administrators Stack Exchange.
Migrated 26 days ago.
The previous DBA created and connected a cluster (4 nodes) to DSE OpsCenter. It's almost perfect (nodes are connected and visible) except for the STOMP service.
In the OpsCenter -> Nodes -> Agents there are question marks in the STOMP service column.
What should I check first?
The firewall seems to be ok, like in the other 3 clusters, which work perfectly fine.
The DataStax Enterprise (DSE) node agents communicate with the OpsCenter server over STOMP. The yellow question mark in the Agents Status View in the OpsCenter UI indicates that OpsCenter doesn't know the status of the agent because OpsCenter can't talk to it.
You stated that "the firewall seems to be ok" but you didn't provide details of what you think is "ok". You need to specifically check that there is connectivity between OpsCenter and the agents.
The opscenterd daemon listens for TCP traffic from the agents on stomp_port which by default is port 61620. The agents listen for TCP traffic from the OpsCenter server on agents_api_port which by default is port 61621 (see OpsCenter ports reference for details).
Once you've confirmed that bi-directional network traffic is working between OpsCenter and the agents, you need to check the following logs for clues:
/var/log/opscenter/opscenterd.log on the OpsCenter server
/var/log/datastax-agent/agent.log on the nodes
The errors in the logs will give you clues as to why OpsCenter cannot connect to the agents.
And as a friendly reminder that if you need assistance with troubleshooting the problem, you should log a ticket with DataStax Support and one of their engineers will help you. Cheers!
👉 Please support the Apache Cassandra community by hovering over the cassandra tag then click on the Watch tag button. 🙏 Thanks!
What is the best way to monitor if cassandra nodes are up? Due to security reasons JMX and nodetool is out of question. I have cluster metrics monitoring via Rest Api, but I understand that even if a node goes Rest Api will only report on a whole cluster.
Well, I have integrated a system where I can monitor all the metrics regarding to my cluster of all nodes. This seems like complicated but pretty simple to integrate. You will need the following components to build up a monitoring system for cassandra:
jolokia jar
telegraf
influxdb
grafana
I'm writing a short procedure, how it works.
Step 1: copy jolokia jvm jar to install_dir/apache-cassandra-version/lib/ , jolokia jvm agent can be downloaded from anywhere in google.
Step 2: add the following line to install_dir/apache-cassandra-version/conf/cassandra-env.sh
JVM_OPTS="$JVM_OPTS -javaagent:<here_goes_the_path_of_your_jolokia_jar>"
Step 3: install telegraf on each node and configure the metrics you want to monitor. and start telegraf service.
Step 4: install grafana and configure your ip, port, protocol. grafana will give you a dashboard to look after your nodes and start grafana service. Your metrics will be able get visibility here.
Step 5: install influxdb on another server from where you want to store your metrics data which will come through telegraf agent.
Step 6: browse the ip you have mentioned, where you have launched your grafana through browser and add data source ip (influxdb ip), then customize your dashboard.
image source: https://blog.pythian.com/monitoring-cassandra-grafana-influx-db/
This is not for monitoring but only for node state.
Cassandra CQL driver provides info if a particular node is UP or DOWN with Host.StateListener Interface. This info is used by driver to mark a node UP or Down. Thus it could be used if node is down or up if JMX is not accessible.
Java Doc API : https://docs.datastax.com/en/drivers/java/3.3/
I came up with a script which listens for DN nodes in the cluster and reports it to our monitoring setup which is integrated with pagerduty.
The script runs on one of our nodes and executes nodetool status every minute and reports for all down nodes.
Here is the script https://gist.github.com/johri21/87d4d549d05c3e2162af7929058a00d1
[1]:
OS: Cent OS 6.4
ISSUE:
Installed gmond, gmetad and gweb on a server. Installed spark worker in the same server.
configured metrics.properties in $SPARK_HOME/conf/metrics.properties as below...
CONFIGURATION (metrics.properties in spark):
org.apache.spark.metrics.sink.GangliaSink
host localhost
port 8649
period 10
unit seconds
ttl 1
mode multicast
We are not able to see any metrics in ganglia web.
Please do the needful.
-pradeep samudrala
In the first place, those are just indications of the default settings of Ganglia. You should not uncomment that. Taken from the metrics section from the Spark web page (spark page):
To install the GangliaSink you’ll need to perform a custom build of Spark. Note that by embedding this library you will include LGPL-licensed code in your Spark package. For sbt users, set the SPARK_GANGLIA_LGPL environment variable before building. For Maven users, enable the -Pspark-ganglia-lgpl profile. In addition to modifying the cluster’s Spark build user applications will need to link to the spark-ganglia-lgpl artifact.
I am new to Cassandra.
I'm trying to deploy a test environment.
Win server 2012 (192.168.128.71) -> seed node
Win server 2008 (192.168.128.70) -> simple node
Win server 2008 (192.168.128.69) -> simple node
On all nodes, I installed the same version Cassandra (2.0.9 from Datastax).
Disabled windows firewall.
The cluster Ring formed. But on each node I see
Test Cluster (Cassandra 2.0.9) 1 of 3 agents connected
Node does not see the Remote Agent. On each PC, the agent service is running.
In file datastax_opscenter_agent-stderr, I see the following line
log4j:ERROR Could not read configuration file [log4j.properties].
log4j:ERROR Ignoring configuration file [log4j.properties].
Please tell me the possible cause how can I diagnose.
Thanks in advance!
The problem is that you have OpsCenter server running on all machines in the cluster. Agents connect to the local OpsCenter servers, so when you open the UI for one of them, you only see one agent connected.
To fix this, stop the server processes (DataStax_OpsCenter_Community) on all machines except for one, and add stomp_interface: <server-ip> to the address.yaml for the agents on all machines, then restart the agents.
I have made few entries in mapred-site.xml, to pick these changes i need to restart TT and JT running at my cluster nodes.
Is there any i can restart them using Cloud Era manager web services from command line.
So I can automate those steps any time changed made configuration files for hadoop it will restart TT and JT..
Since version 4.0, Cloudera Manager exposes its functionality through an HTTP API which allows you to do the operations through "curl" from the shell. The API is available in both the Free Edition and the Enterprise Edition.
Their repository hosts a set of client-side utilities for communicating with the Cloudera Manager API. You can find more on the documentation page.