I am attempting to add a field to a user defined type in cassandra 2.1.2, using the nodejs driver from datastax. I added the field using ALTER TYPE in cqlsh. When I attempt to add a row containing the udt with a value for the new field, it gets inserted with null value, instead of value I supplied. I strongly suspect this has to do with the way the cluster is caching the prepared statement. Because I recall reading that the prepared statements are indexed by a hash of the query, I tried changing some whitespace in the query to see if it helped.This actually seemed to work, but only once. subsequent inserts result in error:
message: 'Operation timed out - received only 0 responses.',
info: 'Represents an error message from the server',
code: 4352,
consistencies: 10,
received: 0,
blockFor: 1,
writeType: 'SIMPLE',
coordinator: '127.0.0.1:9042',
and it would seem the new rows are not added.. until I restart cassandra, at which point not only do the inserts that I thought had failed show up, but subsequent ones work fine. This is very disconcerting, but fortunately I have only done this in test instances. I do need to make this change in production however, and restarting the cluster to add a single field is not really an option. Is there a better way to get the cluster to evict the cached prepared statement?
I strongly suspect this has to do with the way the cluster is caching the prepared statement.
Put Cassandra log in DEBUG mode to be sure the prepared statement cache is the root cause. If it is, create an JIRA so the dev team can fix it...
Optionally you can also enable tracing to see what is going on server-side
To enable tracing in cqlsh, just type TRACING ON
To enable tracing with the Java driver, just call enableTracing() on the statement object
Related
Got the following info log:
Query SELECT column1, column2 FROM table_name WHERE productId IN ('column1_value') is not prepared on /<example-ip-address>:9042, preparing before retrying executing. Seeing this message a few times is fine, but seeing it a lot may be source of performance problems
Any suggestions?
This happens when you send bound query with prepared query ID that is unknown for given node. Node may not "know" this query because of several reasons:
Node crashed and lost cache of prepared statements
Prepared statement was evicted from cache (you should see corresponding message server side). This may happen if somebody prepares too many queries
There were options set to not prepare queries on all nodes (see documentation)
You need to check these hypothesis before continue. I think that it could happen because of 2nd item - this may happen if people prepare "literal" queries that don't have bind markers.
When I first started with Cassandra, I used thrift.
After creating a table, I had to wait to make sure the table was created on all nodes. Without a wait, a node may not be ready and the table would not be usable (many errors would be emitted if the query was sent to "the wrong node".)
In order to "synchronized", I would wait until describe_schema_versions() returned exactly one version of the schema. That meant it was frozen on all nodes.
There does not seem to be an equivalent in CQL. Does that mean that the synchronization problem of older versions (with Thrift) was fixed? Or did I miss something in my searches?
In CQL since Cassandra 1.1 when you do a schema change your coordinator node does all of the heavy lifting to ensure that the schema is "synchronized". And if there's a node down during the schema change, before it comes up again it makes sure it has the latest schema version from the other nodes.More Detailed Information about the change from the old way here. Here's another related stackoverflow question about how to resolve schema conflicts.
The modern DataStax drivers automatically check and wait for schema agreement after you execute a schema-altering statement. Cassandra returns a special response for schema-altering statements that allows the drivers to do this automatically. Once the drivers gets that response, it (blockingly) queries the system.peers table of the node the query was executed against to check the reported schema versions of other nodes. It will continue to execute this query in a loop until either schema agreement is reached or a timeout is hit.
In the Python driver, if a timeout is hit without reaching schema agreement, the ResponseFuture object (that's returned from execute_async()) will have its is_schema_agreed property set to False. I'm not sure what the drivers for other languages do, but there's probably something similar available.
To summarize, the checks that you used to have to do with Thrift drivers are now automatic, and probably more robust.
I'm having the following issue when trying to alter cassandra:
I'm altering the table straight forward:
ALTER TABLE posts ADD is_black BOOLEAN;
on a single-node environment, both under EC2 server and on localhost everything work perfect - select, delete and so on.
When I'm altering on a cluster with 3 nodes - stuff are getting massy.
When I perform
select().all().from(tableName).where..
I'm getting the following exception:
java.lang.IllegalArgumentException: is_black is not a column defined in this metadata
at com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
at com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
at com.datastax.driver.core.ArrayBackedRow.getIndexOf(ArrayBackedRow.java:69)
at com.datastax.driver.core.AbstractGettableData.getString(AbstractGettableData.java:137)
Apparently I'm not the only one who's having this behaviour:
reference
p.s - drop creating the keyspace is not a possibility for me since I cannot delete the data contained in the table.
The bug was resolved :-)
I issue was that DataStax maintains in memory cache that contains the configuration of each node, this cache wasn't update when I alter the table since I used cqlsh instead of their SDK.
After restarting all the node, the in memory cache was dropped and the bug was resolved.
After adding a pair of columns in schema, I want to select them via select *. Instead select * returns old set of columns and none new.
By documentation recommendation, I use {prepare: true} to smooth JavaScript floats and Cassandra ints/bigints difference (I don't really need the prepared statement here really, it is just to resolve this ResponseError : Expected 4 or 0 byte int issue and I also don't want to bother myself with query hints).
So on first execution of select * I had 3 columns. After this, I added 2 columns to schema. select * still returns 3 columns if is used with {prepare: true} and 5 columns if used without it.
I want to have a way to reliably refresh this cache or make cassandra driver prepare statements on each app start.
I don't consider restarting database cluster a reliable way.
This is actually an issue in Cassandra that was fixed in 2.1.3 (CASSANDRA-7910). The problem is that on schema update, the prepared statements are not evicted from the cache on the Cassandra side. If you are running a version less than 2.1.3 (which is likely since 2.1.3 was released last week), there really isn't a way to work around this unless you create another separate prepared statement that is slightly different (like extra spaces or something to cause a separate unique statement).
When running with 2.1.3 and changing the table schema, C* will properly evict the relevant prepared statements from the cache, and when the driver sends another query using that statement, Cassandra will respond with an 'UNPREPARED' message, which should provoke the nodejs driver to reprepare the query and resend the request for you.
On the Node.js driver, you can programatically clear the prepared statement metadata:
client.metadata.clearPrepared();
I'm seeing some very strange effects after modifying column metadata in a columnfamily after executing the following CQL query: ALTER TABLE keyspace_name.table_name ADD column_name cql_type;
I have a cluster of 4 nodes on two data centers (Cassandra version 2.0.9). I also have two application servers talking to the Cassandra cluster via the datastax java driver (version 2.0.4).
After executing this kind of query I see no abnormal behaviour whatsoever (no exceptions detected at all), however long I wait. But once I restart my application on one of the servers I immediately start seeing errors on the other server. What I mean by errors is that after getting my data into a ResultSet, I try to deserialize it row by row and get 'null' values or values from other columns instead of the ones I expect. After restarting the second server (the one that is getting the errors) everything gets back to normal.
I've tried investigating both the logs of datastax-agent and cassandra on both the servers but there is nothing to be found.
Is there a 'proper procedure' to altering the columnfamily? Does anyone have any idea as to what may be the problem?
Thanks!