How Cassandra behaves when the contact node is dead? I mean to say, Cassandra has the ring structure of "n" nodes, If client is going to access that first node but it is dead. The first node is specified in the java client.
I could not understand the failure handling. Could any one help me?
If you only have one node, or only specify one node, and that node is down then the client won't be able to connect (obviously); but usually a client library, like Hector, will allow you to specify a group of nodes and maintain a connection pool, connecting to which ever node is available.
The Hector documentation goes in to a little more detail, but the simplest way to specify multiple nodes is by passing a comma-separated list of hosts in the CassandraHostConfigurator when creating a cluster:
String hosts = "node1.example.com:9160, node2.example.com:9160, node3.example.com:9160";
Cluster cluster = HFactory.getOrCreateCluster(CLUSTER_NAME, new CassandraHostConfigurator(hosts));
It depends on the client, different client do different things to deal with this, but with most drivers you can supply several contact points.
Astyanax uses token discovery to keep track of various nodes in the cluster. Full docs here.
The host supplier associates the connection pool with a dynamic host registry. The connection pool will frequently poll this supplier for the current list of hosts and update its internal host connection pools to account for new or removed hosts.
This is configured during when you are setting up your context:
AstyanaxContext<Keyspace> context = new AstyanaxContext.Builder()
...
.withAstyanaxConfiguration(new AstyanaxConfigurationImpl()
.setDiscoveryType(NodeDiscoveryType.RING_DESCRIBE)
.setConnectionPoolType(ConnectionPoolType.TOKEN_AWARE)
)
...
context.start();
The DataStax java driver handles failure well as long as you've supplied multiple contact points. If you have 3 nodes with a SimpleStrategy for replication and replication factor of 3 (aka all data is present on all 3 nodes) than you can query as long as one of the 3 nodes is alive.
Cluster cluster = Cluster.builder()
.addContactPoint("127.0.0.1")
.addContactPoint("127.0.0.2")
.addContactPoint("127.0.0.3")
.build();
Session session = cluster.connect();
Related
I am trying to write some synchronization code for a java app that runs on each of the cassandra servers in our cluster (so each server has 1 cassandra instance + our app). For this I wanted to make a method that will return the 'local' cassandra node, using the java driver.
Every process creates a cqlSession using the local address as contactPoint. The driver will figure out the rest of the cluster from that. But my assumption was that the local address would be its 'primary' node, at least for requesting things from the system.local table. This seems not so, when trying to run the code.
Is there a way in the Java driver to determine which of the x nodes the process its running on?
I tried this code:
public static Node getLocalNode(CqlSession cqlSession) {
Metadata metadata = cqlSession.getMetadata();
Map<UUID, Node> allNodes = metadata.getNodes();
Row row = cqlSession.execute("SELECT host_id FROM system.local").one();
UUID localUUID = row.getUuid("host_id");
Node localNode = null;
for (Node node : allNodes.values()) {
if (node.getHostId().equals(localUUID)) {
localNode = node;
break;
}
}
return localNode;
}
But it seems to return random nodes - which makes sense if it just sends the query to one of the nodes in the cluster. I was hoping to find a way without providing hardcoded configuration to determine what node the app is running on.
my assumption was that the local address would be its 'primary' node, at least for requesting things from the system.local table. This seems not so, when trying to run the code.
Correct. When running a query where token range ownership cannot be determined, a coordinator is "selected." There is a random component to that selection. But it does take things like network distance and resource utilization into account.
I'm going to advise reading the driver documentation on Load Balancing. This does a great job of explaining how the load balancing policies work with the newer drivers (>= 4.10).
In that doc you will find that query routing plans:
are different for each query, in order to balance the load across the cluster;
only contain nodes that are known to be able to process queries, i.e. neither ignored nor down;
favor local nodes over remote ones.
As far as being able to tell which apps are connected to which nodes, try using the execution information returned by the result set. You should be able to get the coordinator's endpoint and hostId that way.
ResultSet rs = session.execute("select host_id from system.local");
Row row = rs.one();
System.out.println(row.getUuid("host_id"));
System.out.println();
System.out.println(rs.getExecutionInfo().getCoordinator());
Output:
9788de64-08ee-4ab6-86a6-fdf387a9e4a2
Node(endPoint=/127.0.0.1:9042, hostId=9788de64-08ee-4ab6-86a6-fdf387a9e4a2, hashCode=2625653a)
You are correct. The Java driver connects to random nodes by design.
The Cassandra drivers (including the Java driver) are configured with a load-balancing policy (LBP) which determine which nodes the driver contacts and in which order when it runs a query against the cluster.
In your case, you didn't configure a load-balancing policy so it defaults to the DefaultLoadBalancingPolicy. The default policy calculates a query plan (list of nodes to contact) for every single query so each plan is different across queries.
The default policy gets a list of available nodes (down or unresponsive nodes are not included in the query plan) that will "prioritise" query replicas (replicas which own the data) in the local DC over non-replicas meaning replicas will be contacted as coordinators before other nodes. If there are 2 or more replicas available, they are ordered based on "healthiest" first. Also, the list in the query plan are shuffled around for randomness so the driver avoids contacting the same node(s) all the time.
Hopefully this clarifies why your app doesn't always hit the "local" node. For more details on how it works, see Load balancing with the Java driver.
I gather from your post that you want to circumvent the built-in load-balancing behaviour of the driver. It seems like you have a very edge case that I haven't come across and I'm not sure what outcome you're after. If you tell us what problem you are trying to solve, we might be able to provide a better answer. Cheers!
I'm a beginner in Cassandra. I want to understand the two nodes(the streaming node and joining node) how to work when a new node joins an existing cluster. Can they provide normal services to the outside?
If the service is provided normally. I assumed the joining node is nodeA, and the node where the fetching data is nodeB. That means nodeA fetch data from nodeB. Assume that the data range C is transmitted from the nodeB to the nodeA, at which time new data falling into the range C is inserted into the cluster. Is the new data written to nodeA or nodeB?
I'm using datastax community edition of cassandra, version 3.11.3.
thanks!
Sun your question is bit confusing .. but what I make of it is , You want to understand the process to adding new node to existing cluster.
Adding a new node to existing cluster requires cassandra.yaml properties for new node identification and communications.
Set the following properties in the cassandra.yaml and, depending on the snitch, the cassandra-topology.properties or cassandra-rackdc.properties configuration files:
auto_bootstrap - This property is not listed in the default cassandra.yaml configuration file, but it might have been added and set to false by other operations. If it is not defined in cassandra.yaml, Cassandra uses true as a default value. For this operation, search for this property in the cassandra.yaml file. If it is present, set it to true or delete it..
cluster_name - The name of the cluster the new node is joining.
listen_address/broadcast_address - Can usually be left blank. Otherwise, use IP address or host name that other Cassandra nodes use to connect to the new node.
endpoint_snitch - The snitch Cassandra uses for locating nodes and routing requests.
num_tokens - The number of vnodes to assign to the node. If the hardware capabilities vary among the nodes in your cluster, you can assign a proportional number of vnodes to the larger machines.
seeds - Determines which nodes the new node contacts to learn about the cluster and establish the gossip process. Make sure that the -seeds list includes the address of at least one node in the existing cluster.
When a new node joins a cluster using topology defined, Seed nodes starts the gossip with the new node by the time it do not communicate with the client directly. Once the gossip completes the new node is ready to take the actual data load.
Hope this helps in understanding the process.
There is already a question on the difference between Hazelcast Instance and Hazelcast client.
And it is mentioned that
HazelcastInstance = HazelcastClient + AnotherFeatures
So is it right to say client just reads and writes to the cluster formed without getting involved in the cluster? i.e. client does not store the data?
This is important to know since we can configure JVM memory as per the usage. The instances forming the cluster will be allocated more than the ones that are just connecting as a client.
It is a little bit more complicated than that. The Hazelcast Lite Member is a full-blown cluster member, without getting partitions assigned. That said, it doesn't store any data but otherwise behaves like a normal member.
Clients on the other side are simple proxies that have to forward everything to one cluster member to get any operation done. You can imagine a Hazelcast client to be something like a JDBC client, that has just enough code to connect to the cluster and redirect requests / retrieve responses.
I might be misunderstanding something here, as it's not clear to me how I should connect to a Cassandra cluster. I have a Cassandra 1.2.1 cluster of 5 nodes managed by Priam, on AWS. i would like to use Astyanax to connect to this cluster by using a code similar to the code bellow:
conPool = new ConnectionPoolConfigurationImpl(getConecPoolName()) .setMaxConnsPerHost(CONNECTION_POOL_SIZE_PER_HOST).setSeeds(MY_IP_SEEDS)
.setMaxOperationsPerConnection(100) // 10000
What should I use as MY_IP_SEEDS? Should I use the IPs of all my nodes split by comma? Or should I use the IP of just 1 machine (the seed machine)? If I use the ip of just one machine, I am worried about overloading this machine with too many requests.
I know Priam has the "get_seeds" REST api (https://github.com/Netflix/Priam/wiki/REST-API) that for each node returns a list of IPs and I also know there is one seed per RAC. However, I am not sure what would happen if the seed node gets down... I would need to connect to others when trying to make new connections, right?
Seed nodes are only for finding the way into the cluster on node startup - no overload problems.
Of course one of the nodes must be reachable and up in the cluster to get the new one up and running.
So the best way is to update the seed list from Priam before starting the node. Priam should be behind an automatically updated DNS entry.
If you're highest availability you should regularly store the current list of seeds from Priam and store them in a mirrored fashion just as you store your puppet or chef config to be able to get nodes up even when Priam isn't reachable.
I'm a little confused about Cassandra seed nodes and how clients are meant to connect to the cluster. I can't seem to find this bit of information in the documentation.
Do the clients only contain a list of the seed node and each node delegates a new host for the client to connect to? Are seed nodes only really for node to node discovery, rather than a special node for clients?
Should each client use a small sample of random nodes in the DC to connect to?
Or, should each client use all the nodes in the DC?
Answering my own question:
Seeds
From the FAQ:
Seeds are used during startup to discover the cluster.
Also from the DataStax documentation on "Gossip":
The seed node designation has no purpose other than bootstrapping the gossip process
for new nodes joining the cluster. Seed nodes are not a single
point of failure, nor do they have any other special purpose in
cluster operations beyond the bootstrapping of nodes.
From these details it seems that a seed is nothing special to clients.
Clients
From the DataStax documentation on client requests:
All nodes in Cassandra are peers. A client read or write request can
go to any node in the cluster. When a client connects to a node and
issues a read or write request, that node serves as the coordinator
for that particular client operation.
The job of the coordinator is to act as a proxy between the client
application and the nodes (or replicas) that own the data being
requested. The coordinator determines which nodes in the ring should
get the request based on the cluster configured partitioner and
replica placement strategy.
I gather that the pool of nodes that a client connects to can just be a handful of (random?) nodes in the DC to allow for potential failures.
seed nodes serve two purposes.
they act as a place for new nodes to announce themselves to a cluster. so, without at least one live seed node, no new nodes can join the cluster because they have no idea how to contact non-seed nodes to get the cluster status.
seed nodes act as gossip hot spots. since nodes gossip more often with seeds than non-seeds, the seeds tend to have more current information, and therefore the whole cluster has more current information. this is the reason you should not make all nodes seeds. similarly, this is also why all nodes in a given data center should have the same list of seed nodes in their cassandra.yaml file. typically, 3 seed nodes per data center is ideal.
the cassandra client contact points simply provide the cluster topology to the client, after which the client may connect to any node in the cluster. as such, they are similar to seed nodes and it makes sense to use the same nodes for both seeds and client contacts. however, you can safely configure as many cassandra client contact points as you like. the only other consideration is that the first node a client contacts sets its data center affinity, so you may wish to order your contact points to prefer a given data center.
for more details about contact points see this question: Cassandra Java driver: how many contact points is reasonable?
Your answer is right. The only thing I would add is that it's recommended to use the same seed list (i.e. in your cassandra.yaml) across the cluster, as a "best practices" sort of thing. Helps gossip traffic propagate in nice, regular rates, since seeds are treated (very minimally) differently by the gossip code (see https://cwiki.apache.org/confluence/display/CASSANDRA2/ArchitectureGossip).