I have a 16 node cassandra cluster and I am inserting 50.000 rows, pretty much in parallel, from an external tool(which is installed in every node) in every cassandra node with Simba Cassandra JDBC Driver. While the insertion takes place, sometimes/rarely, I get the following error on (mostly/usually) two of the nodes:
Execute failed: [Simba]CassandraJDBCDriver Error setting/closing connection: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.simba.cassandra.shaded.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded))).
java.sql.SQLException: [Simba]CassandraJDBCDriver Error setting/closing connection: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.simba.cassandra.shaded.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded))).
Caused by: com.simba.cassandra.shaded.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.simba.cassandra.shaded.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded)))
The weird thing is that it is a readtimeout exception, while I am just trying to insert. I have not changed any read_time_out or other parameters in the .yaml file, so they are the default. This means that if I try to count(*) on something from cqlsh I also get a readtimeout exception.
ReadTimeout: Error from server: code=1200 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}
I do not know if these two are related though! Any ideas on what might be going on and how to avoid the first error "All host(s) tried for query failed"??
Related
I have a single node Cassandra cluster which hits to ReadTimeOutException and I observe the following logs in server which seems weird to me,
ERROR [SharedPool-Worker-91] 2018-05-29 12:09:53,023 ErrorMessage.java:338 - Unexpected exception during request
java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 1 responses.
at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:489) ~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:269) ~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:66) ~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) ~[apache-cassandra-3.0.8.jar:3.0.8]
It says Operation timed out - received only 1 responses., In Single node why Its saying expecting more than one response? Could some explain pls.
NOTE : I have enabled different strategy for this system_auth keyspace
CREATE KEYSPACE system_auth WITH replication = {'class': 'NetworkTopologyStrategy', 'datacenterproc': '1'} AND durable_writes = true;
and set consistency level as LOCAL_QUORUM
Cassandra server version : 3.0.8
Will this be a reason?
GCs longer than the timeout (like the 9 second GCs you posted in other questions) can cause internal auth timeouts. Most likely it received the response, had the GC, then registered as timeout.
I am not able to connect to Cassandra cluster using this code:
public static boolean tableCreate() {
// Query
String query = "CREATE KEYSPACE store WITH replication "
+ "= {'class':'SimpleStrategy', 'replication_factor':1};";
// creating Cluster object
Cluster cluster = Cluster.builder().addContactPoint("127.0.0.1").withPort(9042).build();
// Creating Session object
Session session = cluster.connect("tutorialspoint");
// Executing the query
session.execute(query);
// using the KeySpaceq
session.execute("USE store");
System.out.println("Keyspace created with store name");
return true;
}
It is giving me this error:
Exception in thread "main" com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1 (null))
What is my mistake in the code above?
Cassandra is running on my Local Windows 10 64bit and I also disabled the firewall.
You may need to check and possibly update the version of datastax driver that you are using. I faced exactly same error (ie same error message while connecting) and after upgrading driver 'datastax' version the problem went away and I could connect to DB.
Similar Issue: Unable to connect to Cassandra cluster running on local host
I am using the latest version of cassandra 3.0.2 and the latest version of datastax cassandra core java driver which is 3.0.0. The settings in cassandra.yaml remains unchanged. I have not changed that file. so whatever the default settings are they remain the same. I keep hearing that rpc_address should be 0.0.0.0 but by default it is localhost and the broadcast_rpc_address by default is commmented out however its value is 1.2.3.4 (again by default it is commented out). I have not changed any of the default settings so the cassandra.yaml file remains unchanged. I also don't understand why we need to set rpc_address and all that after all I hear that latest version of cassandra had moved away from rpc?!
Here is the snippet of the code
Cluster cassandra = Cluster.builder().addContactPoint("localhost").withPort(9042).build();
ListenableFuture<Session> session = cassandra.connectAsync("demo1");
......
Here is the error that I get when I turn on the DEBUG FLAG
com.datastax.driver.NEW_NODE_DELAY_SECONDS is undefined, using default value 1
com.datastax.driver.NON_BLOCKING_EXECUTOR_SIZE is undefined, using default value 8
com.datastax.driver.NOTIF_LOCK_TIMEOUT_SECONDS is undefined, using default value 60
Starting new cluster with contact points [localhost/127.0.0.1:9042, localhost/0:0:0:0:0:0:0:1:9042]
log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
com.datastax.driver.FORCE_NIO is undefined, using default value false
Did not find Netty's native epoll transport in the classpath, defaulting to NIO.
[localhost/127.0.0.1:9042] preparing to open 1 new connections, total = 1
com.datastax.driver.DISABLE_COALESCING is undefined, using default value false
Connection[localhost/127.0.0.1:9042-1, inFlight=0, closed=false] Connection established, initializing transport
[localhost/127.0.0.1:9042] Connection[localhost/127.0.0.1:9042-1, inFlight=0, closed=false] Transport initialized, connection ready
[Control connection] Refreshing node list and token map
You listed localhost/0:0:0:0:0:0:0:1:9042 in your contact points, but it wasn't found in the control host's system.peers at startup
[Control connection] Refreshing schema
[Control connection] Refreshing node list and token map
[Control connection] established to localhost/127.0.0.1:9042
Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
New Cassandra host localhost/127.0.0.1:9042 added
[localhost/127.0.0.1:9042] preparing to open 1 new connections, total = 2
Connection[localhost/127.0.0.1:9042-2, inFlight=0, closed=false] Connection established, initializing transport
[localhost/127.0.0.1:9042] Connection[localhost/127.0.0.1:9042-2, inFlight=0, closed=false] Transport initialized, connection ready
Created connection pool to host localhost/127.0.0.1:9042 (1 connections needed, 1 successfully opened)
Added connection pool for localhost/127.0.0.1:9042
Preparing query SELECT "hash" AS col1,"body" AS col2 FROM demo1.ents WHERE "hash"=?;
[1795897147-1] Doing retry 1 for query com.datastax.driver.core.Statement$1#19be093f at consistency null
[1795897147-1] Error querying localhost/127.0.0.1:9042 : com.datastax.driver.core.exceptions.OperationTimedOutException: [localhost/127.0.0.1] Timed out waiting for server response
error message -> All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.OperationTimedOutException: [localhost/127.0.0.1] Timed out waiting for server response))
error cause -> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.OperationTimedOutException: [localhost/127.0.0.1] Timed out waiting for server response))
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.OperationTimedOutException: [localhost/127.0.0.1] Timed out waiting for server response))
at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)
at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:37)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:98)
at com.datastax.driver.mapping.Mapper.getPreparedQuery(Mapper.java:118)
at com.datastax.driver.mapping.Mapper.getPreparedQuery(Mapper.java:129)
at com.datastax.driver.mapping.Mapper.getQuery(Mapper.java:333)
at com.datastax.driver.mapping.Mapper.getQuery(Mapper.java:325)
at com.datastax.driver.mapping.Mapper.getAsync(Mapper.java:388)
Here is where the exeption happens in cassandra driver
stmt = session().prepare(queryString); //Mapper.java
public PreparedStatement prepare(String query) { //AbstractSession.java
try {
return Uninterruptibles.getUninterruptibly(prepareAsync(query));
} catch (ExecutionException e) {
throw DriverThrowables.propagateCause(e);
}
}
I am using cassandra node js driver and i am getting following error:
error: Database error found %s . On selectAllJobs() call
{ name: 'ResponseError',
message: 'Operation timed out - received only 0 responses.',
info: 'Represents an error message from the server',
code: 4608,
consistencies: 1,
received: 0,
blockFor: 1,
isDataPresent: 0,
query: 'SELECT * FROM cron_tasks WHERE type =? AND starts < ? ALLOW FILTERING ;' }
This error occurred when i ported to new instance of AWS. Earlier, everything went fine.
Cassandra version:
[cqlsh 4.1.1 | Cassandra 2.0.12 | CQL spec 3.1.1 | Thrift protocol 19.39.0]
Read_timeout error means that the coordinator of the query does not know whether the request succeeded or failed, so all it can tell the client is that the request timed out.
In your case, it means that the coordinator of the query sent the request internally to the replica but the replica didn't respond in time.
You can enable query tracing and execute in cqlsh to understand why it is happening.
You can read more about how Cassandra deals with replica failure.
While creating db using cassandra cli, I am getting the following error. any idea what I am doing wrong here ? More details are listed below:
Command - create keyspace pcpro;
output - org.apache.thrift.transport.TTransportException
When I execute the same command again, exception changes to
output - org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe
FYI, I am using cassandra 2.0.1
Thank you.
If you restart cassandra, you have to also restart the CLI to reset the connection it is using to communicate with cassandra otherwise you get the broken pipe error or connection reset error:
// start cassandra
[default#unknown] create keyspace pcpro;
5d344e5d-635e-3745-a1a6-d82ef68bdf28
// reset cassandra
[default#unknown] create keyspace pcpro2;
org.apache.thrift.transport.TTransportException:
java.net.SocketException: Connection reset
// try the query a second time
[default#unknown] create keyspace pcpro2;
org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe