I keep getting this exception under small load.
com.datastax.driver.core.exceptions.NoHostAvailableException: All
host(s) tried for query failed (tried: /127.0.0.1:9042
(com.datastax.driver.core.exceptions.BusyPoolException: [/127.0.0.1]
Pool is busy (no available connection and the queue has reached its
max size 256)))
is there an option to check number of open connections ?
The driver provides a bunch of metrics providing you do not set withoutMetrics on the cluster builder. You can check the value attribute of the cluster1-metrics:name=open-connections mbean*.
Which version of Cassandra and the Java driver your running can make a big difference. With a recent version of C* and the Java driver it can have a lot more concurrent requests per connection than say a 2.0 version of the java driver.
You can use the PoolingOptions object to set the number of connections per host or the max queue size and pass it to your cluster builder.
* Note that the domain cluster1-metrics is generated by clusterName + "-metrics" so if on the Cluster builder you set withClusterName it would change domain accordingly. It will also auto increament the cluster1 to cluster2 etc if you create multiple Cluster objects in your jvm.
Related
I am new to Janusgraph. We have janusgraph setup with cassandra as backend.
We are using ConfiguredGraphFactory to dynamically create graphs at runtime. But when trying to open the created graph using ConfiguredGraphFactory.open("graphName") getting the below error
com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out after PT2S
at com.datastax.oss.driver.api.core.DriverTimeoutException.copy(DriverTimeoutException.java:34)
at com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:53)
at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:30)
at com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)
at com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute(SyncCqlSession.java:54)
We are using single cassandra node and not a cluster. If we are not using ConfiguredGraphFactory we are able to connect to cassandra & it is not a network/wrong port issue.
Any Help would be appreciated.
JanusGraph uses the Java driver to connect to Cassandra. This error comes from the driver and indicates that the nodes didn't respond:
com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out after PT2S
The DriverTimeoutException is different from a read or write timeout. It gets thrown when a request from the driver timed out because it didn't get a response from the Cassandra nodes after 2 seconds (PT2S).
You'll need to check if there's a network route between the JanusGraph server and the Cassandra nodes. One thing to check for is that a firewall is not blocking access to the CQL client port on the C* nodes (default is 9042). Cheers!
I am using cassandra according to the following struct:
21 nodes , AWS EC2 i3.2xlarge , version 3.11.4 .
The application is opening about 5000 connection per node (so its 100k connections per cluster) using the datastax java connection driver.
Application is using autoscale and frequently opens/close connections.
Number of connections to open at once by app servers can reach up to 500 per node (opens simultaneously on all nodes at once - so its 10k connections opens at the same time across the cluster)
This cause spikes of load on cassandra and cause reads and writes latency.
I have noticed each time connections opens/close there are high number of reads from system_auth.roles and system_auth.role_permissions.
How can I prevent the load and resolve this issue ?
You need to modify your application to work with as small number of connections as possible. You need to have following in mind:
Create Cluster/Session object, once at start and keep it. Initialization of session is very expensive operation, it adds a load to Cassandra, and to your application as well
you may increase the number of the simultaneous requests per connection, instead of opening new connections. Protocol allows to have up to 32k requests per connection. Although, if you have too many requests in-flight, then it's a sign that your Cassandra doesn't keep with workload and can't answer fast enough. See documentation on connection pooling
I'm trying to tune Cassandra because I keep getting this error:
com.datastax.driver.core.exceptions.OperationTimedOutException: Timed out waiting for server response
But I'm not sure whether I should increase the number of connections I have per host or increase the number of requests per connection in the datastax driver.
In general, what is the best way to determine if adding more connections is better or adding more requests per connection?
I am getting the following issue:
2015-05-06 00:00:24,368 648892 INFO [STDOUT] (ajp-0.0.0.0-8109-1:)
00:00:24,365 ERROR [DispatcherServlet]
com.datastax.driver.core.exceptions.NoHostAvailableException: All
host(s) tried for query failed (tried:
/:9042
(com.datastax.driver.core.exceptions.DriverException: Timeout during
read), /:9042
(com.datastax.driver.core.exceptions.DriverException: Timeout during
read)) com.datastax.driver.core.exceptions.NoHostAvailableException:
All host(s) tried for query failed (tried:
/:9042
(com.datastax.driver.core.exceptions.DriverException: Timeout during
read), /:9042
(com.datastax.driver.core.exceptions.DriverException: Timeout during
read)) at
com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:65)
at
com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:256)
The Cluster configuration is 2 Nodes with RF = 1. Running a query from an application which uses secondary index. The query is running fine on the TEST cluster but it is giving the above exception on STAGE cluster.
I was able to reproduce the issue from my local environment and added .withSocketOptions(socket.setReadTimeoutMillis(80000)) in the code which resolved the issue. But if the number of rows increase, I have to increase the above value. There is no error present in the Cassandra system logs and the application is able to connect to the cluster but it is timing out only for the STAGE cluster. Any idea why this might be happening? Configurations for both clusters is the same.
This is caused by the change detailed in JAVA-425 where a query that hits a read timeout marks a host in an unavailable state. This behavior has been reverted in 2.0.10 which was released Monday and will be included in 2.1.6 when it is released in the coming weeks. The new behavior is if a query hits the configured read timeout, the host will remain available instead of being in an unavailable state until the driver can reconnect.
Could you try again at java-driver 2.0.10 and let me know if the problem persists? If you are on driver version 2.1.X you may need to continue using an increasingly higher timeout until 2.1.6 is released.
As far as the issue only existing on your STAGE cluster, the performance of your query is going to be very dependent and your dataset and environment, especially with secondary indexes (which in general are only recommended for specific use cases). I'd take a look at When to use an index to determine if Secondary Indexes are right for your use case.
I am doing load testing on Cassandra By using JMeter.
After systematically increasing the load, I can see that more than 58000 Active connection has been established by the driver with different node of cassandra.
I have started with 500 and added up 500 more after 10 iteration. like this i reached upto 2500 request. where it is failing. And Throwing NoHOSTAvailableExecption. I thought that may be cassandra is down. But when i have tried to send request to cassnadra by using DataStax driver . Running in a different System it is working fine. So now My question is that
When I am increasing the load on DataStax java driver it is Opening
more connection instead of using existing connection. Why it is not
using the existing connection?
By default the driver should only have connections based on the number of nodes in the cluster (1 connection per node I believe). This makes me think that your issue is with your Jmeter code and not the Java driver.
In normal operation using the native protocol, the java driver sends multiple requests along each connection simultaneously so there is no need to open multiple connections to the same server. There is some work around upping the limit of simultaneous requests.
The connection was increasing Due to calling creating multiple session for each request. Now it is working Fine.
builder = new Cluster.Builder().
addContactPoints("192.168.114.42");
builder.withPoolingOptions(new PoolingOptions().setCoreConnectionsPerHost(
HostDistance.LOCAL, new PoolingOptions().getMaxConnectionsPerHost(HostDistance.LOCAL)));
cluster = builder
.withRetryPolicy(DowngradingConsistencyRetryPolicy.INSTANCE)
.withReconnectionPolicy(new ConstantReconnectionPolicy(100L))
.build();
session = cluster.connect("demodb");
Now Driver is maintain 17-26 number of connection irrespective of number of transaction.