Cassandra exception while live schema change - cassandra

Question

For our project we need live schema changes. For these purposes we created test. It adds and deletes tables in cassandra. I'm using 10 threads with datastax driver. But I am getting this Exception in some threads.
Exception in thread "Thread-7"
com.datastax.driver.core.exceptions.NoHostAvailableException: All
host(s) tried for query failed (tried: localhost/127.0.0.1
(com.datastax.driver.core.exceptions.DriverException: Timeout during
read))
Cassandra is alive. I can still execute queries throw cqlsh.
Threads does'n fail. Thread can add tables, then suddenly log this exception and again add tables. Exception can appear several times in one thread.
Each thread has its own session.
Looking to the cassandra.yaml:
concurrent_writes and reads = 32
increasing of MAX_HEAP_SIZE and HEAP_NEWSIZE in cassandra-env.sh doesn't help me.
Cassandra version 2.0.7
Could someone suggest how to deal with it?
public class AddTableThread extends Thread {
private Cluster cluster;
private Session session;
private final String KEYSPACE = "cas";
private final String HOST = "localhost";
private final int PORT = 9042;
private boolean isRunning = true;
// counter for the added tables
private int i = 0;
// Number of the tables that thread has to add
private int fin = -2;
public AddTableThread() {
connect();
}
public AddTableThread setTablesCount(int f) {
this.fin = f;
System.out.println(this.getName() + " : must add " + fin + " tables");
return this;
}
#Override
public void run() {
System.out.println(this.getName() + " : starting thread \n");
while (isRunning) {
String name = createName();
try {
session.execute(createTableQuery(name));
System.out.println(this.getName() + " : table is created "
+ name);
i++;
} catch (com.datastax.driver.core.exceptions.NoHostAvailableException e) {
System.out
.println("\n\n" + this.getName() + "!!!!FAILED!!!!!!");
System.out.println(this.getName() + " : added " + i + " tables");
System.out.println(this.getName() + " : table " + name
+ " wasn't added");
}
try {
Thread.sleep((new Random().nextInt(5000)));
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
if (i == fin) {
isRunning = false;
}
}
close();
log();
}
// creates query that creates table in keyspace with name tableName
private String createTableQuery(String tableName) {
return "CREATE TABLE cas." + tableName
+ "(id timeuuid PRIMARY KEY, text varchar, value int);";
}
public void finish() {
isRunning = false;
}
private void log() {
System.out.println("\n" + this.getName() + " : finish");
System.out.println(this.getName() + " : added " + i + " tables");
System.out.println("_____________________\n");
}
// generates unique table name
private String createName() {
return "cf_" + UUIDGen.getTimeUUID().toString().replace("-", "_");
}
private void connect() {
SocketOptions so = new SocketOptions();
so.setConnectTimeoutMillis(20000);
so.setReadTimeoutMillis(20000);
cluster = Cluster.builder().withSocketOptions(so).addContactPoint(HOST)
.withPort(PORT).build();
session = cluster.connect(KEYSPACE);
}
private void close() {
session.close();
cluster.close();
}
}

Answer 1

Creating takes time. So try to increment:
read_request_timeout_in_ms: 5000
and
request_timeout_in_ms: 10000
And try to uncomment
rpc_max_threads: 2048
Experiment with these parameters.