I'm using Cassandra 3.11.6 on Centos 7 with a 3 node cluster, I ran some schema changes, drop tables/materialized views, alter tables, etc, after that one of the materialized views was failing with this error:
org.apache.cassandra.schema.SchemaKeyspace$MissingColumns: No partition key columns found in schema table for my_keyspace.my_materialized_view.
I wanted to replace that materialized view with a table, with the same name though, that might be why it's failing
I ran nodetool describecluster and found the schema version was different, I tried to run a repair, it didn't work, I restarted the nodes, but they didn't start.
This is the error that is showing up in cassandra.log
ERROR [main] 2020-12-09 10:13:15,827 SchemaKeyspace.java:1017 - No partition columns found for table my_keyspace.my_materialized_view in system_schema.columns. This may be due to corruption or concurrent dropping and altering of a table. If this table is supposed to be dropped, run the following query to cleanup: "DELETE FROM system_schema.tables WHERE keyspace_name = 'my_keyspace' AND table_name = 'my_materialized_view'; DELETE FROM system_schema.columns WHERE keyspace_name = 'my_keyspace' AND table_name = 'my_materialized_view';" If the table is not supposed to be dropped, restore system_schema.columns sstables from backups.
org.apache.cassandra.schema.SchemaKeyspace$MissingColumns: No partition key columns found in schema table for my_keyspace.my_materialized_view
at org.apache.cassandra.schema.SchemaKeyspace.fetchColumns(SchemaKeyspace.java:1106) [apache-cassandra-3.11.6.jar:3.11.6]
at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:1046) [apache-cassandra-3.11.6.jar:3.11.6]
at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:1000) [apache-cassandra-3.11.6.jar:3.11.6]
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:959) [apache-cassandra-3.11.6.jar:3.11.6]
at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:936) [apache-cassandra-3.11.6.jar:3.11.6]
at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:924) [apache-cassandra-3.11.6.jar:3.11.6]
at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:92) [apache-cassandra-3.11.6.jar:3.11.6]
at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:82) [apache-cassandra-3.11.6.jar:3.11.6]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:269) [apache-cassandra-3.11.6.jar:3.11.6]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:630) [apache-cassandra-3.11.6.jar:3.11.6]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:757) [apache-cassandra-3.11.6.jar:3.11.6]
I tried starting Cassandra with -Dcassandra.ignore_corrupted_schema_tables=true and it doesn't work
It looks like you've either made concurrent updates to your schema or at least made multiple DDL statements close to each other to cause a schema disagreement in your cluster.
You are supposed to wait for a single DDL change to propagate through the cluster and check that the schema is in agreement before making the next DDL change to prevent disagreement.
My suggestion is to attempt to start the node where you were performing the schema changes and leave the other nodes down temporarily. Hopefully it's also a seed node.
Remove the ignore_corrupted_schema_tables and see if you can bring it back online. If it does, then proceed to the next node and watch the startup sequence (do a tail -f on the system.log). Keep going until all nodes are back online.
The issue is that depending on the state of the schema on each node, it will be difficult to "unscramble the egg". Good luck!
Related
I have 4-node 2-dc cassandra cluster, and there I have faced an unexplainable behaviour.
dc1 - node1, node2, node3.
dc2 - node4.
In my database I have columnFamily1 with replication factor dc1:2 dc2:1. The column family contains 28 columns, and 11 columns are in primary key. I query node4 from dc2, that contains 100% data. So, the problem is:
'Select * from columnFamily' returns 5103 records.
'Select (any column from primaryKey) from columnFamily' returns 733 records.
If I export columnFamily to CSV and then import to truncated table, the problem disappears. Can anyone explain, how could this happen? Is there any solution without truncating columnFamily?
So it looks like from your above statements, putting aside your comment (the portion about the "node that has 100% of the data"), you must have some consistency issues. As you're using consistency one, you will query one node for the details. Which node could be any one of them. The fact that you do "*" or "any column from primary key" shouldn't matter, but each time you run the query, you could get a different node for the results. If you're using CQLSH, you could try to log onto each node, one at a time, set your consistency to LOCAL_ONE and run a "select " or "select count()" and see what results you get. I'm guessing it would be different. As you have truncated the table and re-loaded it, that could clean up the consistency problems. Another approach you could have done is run repair on the column family and then re-try your experiment. What did you do to export the data? Did you use the CQLSH COPY command? If so, that will query all nodes for the data. Did you count the number of rows in the CSV? If so, did it match any of the counts from the queries themselves?
After adding/removing tables and views to a keyspace a got problems with inconsistency and error referring to tables previous deleted. We tried to restart cluster nodes, only resulting in the nodes not starting due to java.lang.IllegalArgumentException: Unknown CF.
The current error is thrown from a View that refers to a non existing table (The table do exist but has a new id). Is it possible to some how fix this when Cassandra is not running?
It may be that you have a schema mismatch. Verify first that you're running the same schema versions nodetool describecluster and make sure all nodes are reachable.
The only other time i've seen something like this is when you have corrupt data on a node. In which case you'll want to nodetool removenode the appropriate node and provision a new one.
As an asside MATERIALIZED VIEWS are deprecated in 3.11 and will not be supported going forward. I would suggest that you roll your own.
I am facing duplicate records in the Cassandra table when the spark tasks fails and it is restarted again. Schema of the table that I am trying to insert.
CREATE TABLE duplicate_record (object_id bigint,entity_key timeuuid,
PRIMARY KEY (object_id, entity_key));
Sample Duplicate Records in the table
1181592431 uuid
1181592431 uuid1
8082869622 uuid2
8082869622 uuid3
I have a df which is produced by the left join between Oracle and Cassandra. So we have already existing records in Cassandra and new records which are generated from Oracle.
I apply a map of each record to see if the entity_id exists. If it exists then use it else for new records create fresh entity_id and then do the save. I am using saveToCassandra to insert this df to Cassandra.
When the task fails and is restarted, already inserted records are being inserted again with a different entity_key. I guess the inserted record during the successful execution is not available when the task is resubmitted resulting in duplicate records.
Spark Speculative Execution can cause duplicate records, so pls turn it to false.
still if this happens then it could be due to some spark nodes/tasks getting restarted… there was a Spark bug for duplicate record commit, its resolved in 2.1.3 version:
https://issues.apache.org/jira/browse/SPARK-24589
pls ensure you are running on 2.1.3+.
I have a six node cluster running cassandra 2.1.6. Yesterday I tried to drop a column family and received the message "Column family ID mismatch". I tried running nodetool repair but after repair was complete I got the same message. I then tried selecting from the column family but got the message "Column family not found". I ran the following query to get a list of all column families in my schema
select columnfamily_name from system.schema_columnfamilies where keyspace_name = 'xxx';
At this point I received the message
"Keyspace 'system' not found." I tried the command describe keyspaces and sure enough system was not in the list of keyspaces.
I then tried nodetool resetlocalshema on one of the nodes missing the system keyspace and when that failed to resolve the problem I tried nodetool rebuild but got the same messages after rebuild was complete. I tried stopping the nodes missing the system keyspace and restarted them, once the restart was completed the system keyspace was back and I was able to execute the above query successfully. However, the table I had tried to drop previously was not listed so I tried to recreate it and once again received the message Column family ID mismatch.
Finally, I shutdown the cluster and restarted it... and everything works as expected.
My questions are: How/why did the system keyspace disappear? What happened to the data being inserted into my column families while the system keyspace was missing from two of the six nodes? (my application didn't seem to have any problems) Is there a way I can detect problems like this automatically or do I have to manually check up on my keyspaces each day? Is there a way to fix the missing system keyspace and/or the Column family ID mismatch without restarting the entire cluster?
EDIT
As per Jim Meyers suggestion I queried the cf_id on each node of the cluster and confirmed that all nodes return the same value.
select cf_id from system.schema_columnfamilies where columnfamily_name = 'customer' allow filtering;
cf_id
--------------------------------------
cbb51b40-2b75-11e5-a578-798867d9971f
I then ran ls on my data directory and can see that there are multiple entries for a few of my tables
customer-72bc62d0ff7611e4a5b53386c3f1c9f9
customer-cbb51b402b7511e5a578798867d9971f
My application dynamically creates tables at run time (always using IF NOT EXISTS), seems likely that the application issued the same create table command on separate nodes at the same time resulting in the schema mismatch.
Since I've restarted the cluster everything seems to be working fine.
Is it safe to delete the extra file?
i.e. customer-72bc62d0ff7611e4a5b53386c3f1c9f9
1 The cause of this problem is a CREATE TABLE statement collision. Do not generate tables dynamically from multiple clients, even with IF NOT EXISTS. First thing you need to do is fix your code so that this does not happen. Just create your tables manually from cqlsh allowing time for the schema to settle. Always wait for schema agreement when modifying schema.
2 Here's the fix:
1) Change your code to not automatically re-create tables (even with IF NOT EXISTS).
2) Run a rolling restart to ensure schema matches across nodes. Run nodetool describecluster around your cluster. Check that there is only one schema version.
ON EACH NODE:
3) Check your filesystem and see if you have two directories for the table in question in the data directory.
If THERE ARE TWO OR MORE DIRECTORIES:
4)Identify from schema_column_families which cf ID is the "new" one (currently in use).
cqlsh -e "select * from system.schema_column_families"|grep
5) Move the data from the "old" one to the "new" one and remove the old directory.
6) If there are multiple "old" ones repeat 5 for every "old" directory.
7) run nodetool refresh
IF THERE IS ONLY ONE DIRECTORY:
No further action is needed.
Futures
Schema collisions will continue to be an issue until - CASSANDRA-9424
Here's an example of it occurring on Jira and closed as not a problem CASSANDRA-8387
When you create a table in Cassandra it is assigned a unique id that should be the same on all nodes. Somehow it sounds like your table did not have the same id on all nodes. I'm not sure how that might happen, but maybe there was a glitch when the table was created and it was created multiple times, etc.
You should always use the IF NOT EXISTS clause when creating tables.
To check if your id's are consistent, try this on each node:
In cqlsh, run "SELECT cf_id from system.schema_columnfamilies where columnfamily_name ='yourtablename' allow filtering;
Look in the data directory under the keyspace name the table was created in. You should see a single directory for the table that looks like table_name-cf_id.
If things are correct you should see the same cf_id in all these places. If you see different ones, then somehow things got out of sync.
The other symptoms like the system keyspace disappearing I don't have a suggestion other than you hit some kind of bug in the software. If you get a lot of strange symptoms like this then perhaps you have some kind of data corruption. You might want to think about backing up your data in case things go south and you need to rebuild the cluster.
I'm having the following issue when trying to alter cassandra:
I'm altering the table straight forward:
ALTER TABLE posts ADD is_black BOOLEAN;
on a single-node environment, both under EC2 server and on localhost everything work perfect - select, delete and so on.
When I'm altering on a cluster with 3 nodes - stuff are getting massy.
When I perform
select().all().from(tableName).where..
I'm getting the following exception:
java.lang.IllegalArgumentException: is_black is not a column defined in this metadata
at com.datastax.driver.core.ColumnDefinitions.getAllIdx(ColumnDefinitions.java:273)
at com.datastax.driver.core.ColumnDefinitions.getFirstIdx(ColumnDefinitions.java:279)
at com.datastax.driver.core.ArrayBackedRow.getIndexOf(ArrayBackedRow.java:69)
at com.datastax.driver.core.AbstractGettableData.getString(AbstractGettableData.java:137)
Apparently I'm not the only one who's having this behaviour:
reference
p.s - drop creating the keyspace is not a possibility for me since I cannot delete the data contained in the table.
The bug was resolved :-)
I issue was that DataStax maintains in memory cache that contains the configuration of each node, this cache wasn't update when I alter the table since I used cqlsh instead of their SDK.
After restarting all the node, the in memory cache was dropped and the bug was resolved.