This question was migrated from Stack Overflow because it can be answered on Database Administrators Stack Exchange.
Migrated 26 days ago.
The previous DBA created and connected a cluster (4 nodes) to DSE OpsCenter. It's almost perfect (nodes are connected and visible) except for the STOMP service.
In the OpsCenter -> Nodes -> Agents there are question marks in the STOMP service column.
What should I check first?
The firewall seems to be ok, like in the other 3 clusters, which work perfectly fine.
The DataStax Enterprise (DSE) node agents communicate with the OpsCenter server over STOMP. The yellow question mark in the Agents Status View in the OpsCenter UI indicates that OpsCenter doesn't know the status of the agent because OpsCenter can't talk to it.
You stated that "the firewall seems to be ok" but you didn't provide details of what you think is "ok". You need to specifically check that there is connectivity between OpsCenter and the agents.
The opscenterd daemon listens for TCP traffic from the agents on stomp_port which by default is port 61620. The agents listen for TCP traffic from the OpsCenter server on agents_api_port which by default is port 61621 (see OpsCenter ports reference for details).
Once you've confirmed that bi-directional network traffic is working between OpsCenter and the agents, you need to check the following logs for clues:
/var/log/opscenter/opscenterd.log on the OpsCenter server
/var/log/datastax-agent/agent.log on the nodes
The errors in the logs will give you clues as to why OpsCenter cannot connect to the agents.
And as a friendly reminder that if you need assistance with troubleshooting the problem, you should log a ticket with DataStax Support and one of their engineers will help you. Cheers!
👉 Please support the Apache Cassandra community by hovering over the cassandra tag then click on the Watch tag button. 🙏 Thanks!
Related
How to setup a fail-over node for Cassandra Opscenter. The Opscenter data is stored on Opscenter node itself. So to setup a failover node i need to setup an Opscenter different from current Opscenter and sync Opscenter data and config files between Opscenters.
The stomp_interface on nodes in the cluster are pointed towards Opscenter_1 how will it change automatically to Opscenter_2 when failover occurs??
There are steps on the datastax documentation that have details for doing this. At a minimum:
Mirror the configuration directories stored on the OpsCenter primary to the OpsCenter backup using the method you prefer.
On the backup OpsCenter in the failover directory, create a primary_opscenter_location configuration file that indicates the IP address of the primary OpsCenter daemon to monitor
The stomp_interface setting on the agents gets changed (address.yaml file updated as well) when failover occurs. This is why the documentations recommend making sure there is no 3rd party configuration management on it.
3 things :
If you have firewall on, allow the corresponding ports to communicate (61620,61621,9160,9042,7199)
always verify IF the cassandra is up and running, so agent can actually connect to something.
stop the agent, check again the address.yaml, restart the agent.
I've been working with Storm topologies and Cassandra databases for relatively short period of time. I recently realized that my development environment's spec is not strong enough for my testing, so I deployed a 3-node Cassandra cluster on Google Cloud instance. Now I'd like to let Storm topology (hosted on a separate box) to insert into Cassandra. Obviously, this feature is not enabled by default, and I'd like to have a guideline of how to, securely, open Cassandra for database queries from different IP in production scenario. ( I suspect that Google protects its instances with a firewall as well?)
Following Carlos Rojas's directions in THIS LINK, I could open the ports to access Cassandra from outside the network computer. Also, you can open ports in your firewall using this line :
gcutil addfirewall cassandra-rule --allowed="tcp:9042,tcp:9160" --network="default" --description="Allow external Cassandra Thrift/CQL connections" from THIS LINK
I've got a two-node Amazon Web Services cassandra cluster created using "DataStax Auto-Clustering AMI 2.5.1-pv". OpsCenter is running on node 0, as is the datastax-agent, but they don't seem to be completely connected. OpsCenter says 0 of 0 agents connected and the connection icon next to "New Cluster" is blinking red.
OpsCenter screenshot: http://i.stack.imgur.com/Z6Tnx.png
a nodetool status would really help.
check the agent logs on either of the nodes when the starts, if you don't see any errors, try "adding a new cluster" then "manage existing cluster" and add the seed IPs of your two nodes, opscenter will try to update the agents if needed.
BTW: upgrade to opscenter 5.1, there are a lot of bug already fixed
Is there any detail step-by-step document to address the multi-node cassandra installation in Windows? I read some documents/blogs and tried on Window7 workstations/Windows2008 servers but not be able to establish connection from the 2nd node to the 1st node.
When I was setting up my first cluster on windows I found this blogpost to be excellent. It covers many aspects of the setup including:
Firewall / Networking issues.
Running Cassandra as a service.
Monitoring and maintenance.
If you want to create a complete setup with using just cassandra have a look at this blog.
But to setup a multi-node cluster, you basically need to have the correct ports open on your servers. When it comes to configuration you are basically going to have identical cassandra.yaml configs accross all your nodes, with the same seeds list, and the only two fields need to be changed are the listen_address and possibly rpc_address (although you could just listen an all interfaces for the rpc_address by setting it to:
rpc_address: 0.0.0.0
I would like to know how OpsCenter communicates with its Agents and Cassandra Nodes.
Does it use Thrift? Is JMX required?
I'll base my answer on the latest released version of OpsCenter (1.3).
The main OpsCenter process can communicate with the agents in two ways. It can query the agents over an http rest api that each agent exposes. It uses this to ask the agent basic things about the cassandra node and also to have the agent send jmx commands to the cassandra process.
The other way is using the STOMP protocol. (http://stomp.github.com//) Agents send messages over STOMP to a message queue in OpsCenter. These generally contain details about the cassandra node and metric information.
Hope that helps.