One node of my cassandra cluster breaks very often like every day. It mostly breaks in the morning most probably because of memory consumption by other jobs. But I don't see any error msg in the logs. Following are the log messages when it breaks:
INFO [MemtableFlushWriter:313] 2016-04-07 07:19:19,629 Memtable.java:385 - Completed flushing /data/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-6666-Data.db (6945 bytes) for commitlog position ReplayPosition(segmentId=1459914506391, position=14934142)
INFO [MemtableFlushWriter:313] 2016-04-07 07:19:19,631 Memtable.java:346 - Writing Memtable-sstable_activity#823770964(21691 serialized bytes, 7605 ops, 0%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:313] 2016-04-07 07:19:19,650 Memtable.java:385 - Completed flushing /data/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-7558-Data.db (13689 bytes) for commitlog position ReplayPosition(segmentId=1459914506391, position=14934142)
INFO [CompactionExecutor:1176] 2016-04-07 07:19:19,651 CompactionTask.java:140 - Compacting [SSTableReader(path='/data/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-7557-Data.db'), SSTableReader(path='/data/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-7556-Data.db'), SSTableReader(path='/data/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-7555-Data.db'), SSTableReader(path='/data/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-7558-Data.db')]
INFO [CompactionExecutor:1176] 2016-04-07 07:19:19,696 CompactionTask.java:270 - Compacted 4 sstables to [/data/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-7559,]. 46,456 bytes to 9,197 (~19% of original) in 45ms = 0.194910MB/s. 1,498 total partitions merged to 216. Partition merge counts were {1:675, 2:33, 3:11, 4:181, }
INFO [MemtableFlushWriter:312] 2016-04-07 07:19:19,717 Memtable.java:385 - Completed flushing /data/cassandra/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/system-size_estimates-ka-8014-Data.db (849738 bytes) for commitlog position ReplayPosition(segmentId=1459914506391, position=14934142)
I know that the OS kills it but how to find the root cause of the problem and if any config changes can be done for cassandra?
Related
I have a very simple word-count-like program that generates (Long, Double) counts like that:
val lines = sc.textFile(directory)
lines.repartition(600).mapPartitions{lineIterator =>
// Generate iterator of (Long,Double) counts
}
.reduceByKey(new HashPartitioner(30), (v1, v2) => v1 + v2).saveAsTextFile(outDir, classOf[GzipCodec])
My problem: The last of the 30 partitions never gets written.
Here are a few details:
My input is 5 GB gz-compressed and I expect about 1B unique Long keys.
I run on a 32 core 1.5TB machine. Input and output come from a local disk with 2TB free. Spark is assigned to use all the ram and happily does so. This application occupies about 0.5 TB.
I can observe the following:
For 29 partitions the reduce and repartition (because of the HashPartitioner) takes about 2h. The last one does not finish, not even after a day. Two to four threads stay on 100%.
No error or warning appears in the log
Spark occupies about 100GB in /tmp which aligns with what the UI reports for shuffle write.
In the UI I can see the number of "shuffle read records" growing very, very slowly for the remaining task. After one day, still one magnitude away from what all the finished tasks show.
The last log looks like that:
15/08/03 23:26:43 INFO SparkHadoopWriter: attempt_201508031748_0002_m_000020_748: Committed
15/08/03 23:26:43 INFO Executor: Finished task 20.0 in stage 2.0 (TID 748). 865 bytes result sent to driver
15/08/03 23:27:50 INFO FileOutputCommitter: Saved output of task 'attempt_201508031748_0002_m_000009_737' to file:/output-dir/_temporary/0/task_201508031748_0002_m_000009
15/08/03 23:27:50 INFO SparkHadoopWriter: attempt_201508031748_0002_m_000009_737: Committed
15/08/03 23:27:50 INFO Executor: Finished task 9.0 in stage 2.0 (TID 737). 865 bytes result sent to driver
15/08/04 02:44:54 INFO BlockManager: Removing broadcast 3
15/08/04 02:44:54 INFO BlockManager: Removing block broadcast_3_piece0
15/08/04 02:44:54 INFO MemoryStore: Block broadcast_3_piece0 of size 2009 dropped from memory (free 611091153849)
15/08/04 02:44:54 INFO BlockManagerMaster: Updated info of block broadcast_3_piece0
15/08/04 02:44:54 INFO BlockManager: Removing block broadcast_3
15/08/04 02:44:54 INFO MemoryStore: Block broadcast_3 of size 3336 dropped from memory (free 611091157185)
15/08/04 02:44:54 INFO BlockManager: Removing broadcast 4
15/08/04 02:44:54 INFO BlockManager: Removing block broadcast_4_piece0
15/08/04 02:44:54 INFO MemoryStore: Block broadcast_4_piece0 of size 2295 dropped from memory (free 611091159480)
15/08/04 02:44:54 INFO BlockManagerMaster: Updated info of block broadcast_4_piece0
15/08/04 02:44:54 INFO BlockManager: Removing block broadcast_4
15/08/04 02:44:54 INFO MemoryStore: Block broadcast_4 of size 4016 dropped from memory (free 611091163496)
Imagine the first five lines repeated for 28 other partitions within a two minute time frame.
I have tried several things:
Spark 1.3.0 and 1.4.0
nio instead of netty
flatMap instead of mapPartitions
Just 30 instead of 600 input partitions
Still, I never get the last 1/30 of my data out of spark. Did anyone ever observe something similar? These two posts here and here seem to describe similar problems but no solution.
UPDATE
The task that never finishes is always the first task of the reduceKey+writeToTextFile. I have also removed the HashPartitioner and even tried on a bigger cluster with 400 cores and 6000 partitions. Only 5999 finish successfully, the last runs forever.
The UI shows for all tasks something like
Shuffle Read Size / Records: 20.0 MB / 1954832
but for the first it shows (at the moment)
Shuffle Read Size / Records: 150.1 MB / 711836
Numbers still growing....
It might be that your keys are very skewed. Depending on how they are distributed (or if you have a null or default key), a significant amount of the data might be going to a single executor and be no different than running in your local machine (plus overhead of a distributed platform). It might even be causing that machine to swap to disk, becoming intolerably slow.
Try using aggregateByKey instead of reduceByKey, since it will attempt to get partial sums distributed across executors instead of shuffling all the (potentially large) set of key-value pairs to a single executor. And maybe avoid fixing the number of output partitions to 30 just in case.
Edit: It is hard to detect the problem for "it just does not finish". One thing you can do is to introduce a timeout:
val result = Await.result(future {
// Your normal computation
}, timeout)
That way, whatever task is taking too long, you can detect it and gather some metrics on the spot.
Our production environment has two cassandra node(v2.0.5), and we want to add additional node to extend scalability. We followed step desc in Datastax doc
After bootstrap new node, we observed some exception log
ERROR [CompactionExecutor:42] 2015-03-25 19:01:01,821 CassandraDaemon.java (line 192) Exception in thread Thread[CompactionExecutor:42,1,main]
java.lang.RuntimeException: org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:154)
at org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:137)
And it repeat some compaction task and after two week it didn't complete bootstrap. Node remains not join state
INFO [CompactionExecutor:4468] 2015-03-30 09:18:20,288 ColumnFamilyStore.java (line 784) Enqueuing flush of Memtable-compactions_in_progress#1247174540(212/13568 serialized/live bytes, 7 ops)
INFO [FlushWriter:314] 2015-03-30 09:18:22,408 Memtable.java (line 373) Completed flushing /var/lib/cassandra/data/production_alarm_keyspace/alarm_history_data_new/production_alarm_keyspace-alarm_history_data_new-jb-118-Data.db (11216962 bytes) for commitlog position ReplayPosition(segmentId=1427280544702, position=24550137)
INFO [FlushWriter:314] 2015-03-30 09:18:22,409 Memtable.java (line 333) Writing Memtable-alarm_master_data#37361826(26718076/141982437 serialized/live bytes, 791595 ops)
INFO [FlushWriter:314] 2015-03-30 09:18:24,018 Memtable.java (line 373) Completed flushing /var/lib/cassandra/data/production_alarm_keyspace/alarm_master_data/production_alarm_keyspace-alarm_master_data-jb-346-Data.db (8407637 bytes) for commitlog position ReplayPosition(segmentId=1427280544702, position=24550137)
INFO [FlushWriter:314] 2015-03-30 09:18:24,018 Memtable.java (line 333) Writing Memtable-compactions_in_progress#1247174540(212/13568 serialized/live bytes, 7 ops)
INFO [FlushWriter:314] 2015-03-30 09:18:24,185 Memtable.java (line 373) Completed flushing /var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-jb-1019-Data.db (201 bytes) for commitlog position ReplayPosition(segmentId=1427280544702, position=24550511)
INFO [CompactionExecutor:4468] 2015-03-30 09:18:24,186 CompactionTask.java (line 115) Compacting [SSTableReader(path='/var/lib/cassandra/data/production_alarm_keyspace/alarm_common_dump_by_minutes/production_alarm_keyspace-alarm_common_dump_by_minutes-jb-356-Data.db'), SSTableReader(path='/var/lib/cassandra/data/production_alarm_keyspace/alarm_common_dump_by_minutes/production_alarm_keyspace-alarm_common_dump_by_minutes-jb-357-Data.db'), SSTableReader(path='/var/lib/cassandra/data/production_alarm_keyspace/alarm_common_dump_by_minutes/production_alarm_keyspace-alarm_common_dump_by_minutes-jb-355-Data.db'), SSTableReader(path='/var/lib/cassandra/data/production_alarm_keyspace/alarm_common_dump_by_minutes/production_alarm_keyspace-alarm_common_dump_by_minutes-jb-354-Data.db')]
INFO [CompactionExecutor:4468] 2015-03-30 09:18:39,189 ColumnFamilyStore.java (line 784) Enqueuing flush of Memtable-compactions_in_progress#810255650(0/0 serialized/live bytes, 1 ops)
INFO [FlushWriter:314] 2015-03-30 09:18:39,189 Memtable.java (line 333) Writing Memtable-compactions_in_progress#810255650(0/0 serialized/live bytes, 1 ops)
INFO [FlushWriter:314] 2015-03-30 09:18:39,357 Memtable.java (line 373) Completed flushing /var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-jb-1020-Data.db (42 bytes) for commitlog position ReplayPosition(segmentId=1427280544702, position=25306969)
INFO [CompactionExecutor:4468] 2015-03-30 09:18:39,367 CompactionTask.java (line 275) Compacted 4 sstables to [/var/lib/cassandra/data/production_alarm_keyspace/alarm_common_dump_by_minutes/production_alarm_keyspace-alarm_common_dump_by_minutes-jb-358,]. 70,333,241 bytes to 70,337,669 (~100% of original) in 15,180ms = 4.418922MB/s. 260 total partitions merged to 248. Partition merge counts were {1:236, 2:12, }
Nodetool status just show two node, and it's accept because 2.0.5 has bug in nodetool don't show join node.
[bpmesb#bpmesbap2 ~]$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 172.18.8.56 99 GB 256 51.1% 7321548a-3998-4122-965f-0366dd0cc47e rack1
UN 172.18.8.57 93.78 GB 256 48.9% bb306032-ff1c-4209-8300-d8c3de843f26 rack1
Can anybody help about this condition? Because datastax says bootstrap only take few minutes but our situation didn't complete after 2 weeks? We search stackoverflow and find This issue
may be related to our problem
After few days test and look at exception log. We found this may be key issue about this problem.
ERROR 19:01:01,821 Exception in thread Thread[NonPeriodicTasks:1,5,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
at org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndexes(SecondaryIndexManager.java:140)
at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:113)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
The log shows that one stream receiver task encounter tombstone overwhelming exception. And we think that's the key point cause cassandra never change state to normal.
And here is following step we done to fix this problem. We use nodetool to compact table and secondary index at two original node.
nohup nodetool compact production_alarm_keyspace object_daily_data &
nohup nodetool rebuild_index production_alarm_keyspace object_daily_data object_daily_data_alarm_no_idx_1 &
And we restart new node again, and after a hour, new node jump to normal and work fine until now.
Using top
8260 root 20 0 5163m 4.7g **133m** S 144.6 30.5 2496:46 java
Most of the time %CPU is >170.
I am trying to identify the issue. I think GC or flushing is too blame.
S0 S1 E O P YGC YGCT FGC FGCT GCT LGCC GCC
0.00 16.73 74.74 29.33 59.91 27819 407.186 206 10.729 417.914 Allocation Failure No GC
0.00 16.73 99.57 29.33 59.91 27820 407.186 206 10.729 417.914 Allocation Failure Allocation Failure
Also from Cassandra logs, it says Replaying position with the same segment ID and memtable is flushing too often.
INFO [SlabPoolCleaner] 2015-01-20 13:55:48,515 ColumnFamilyStore.java:840 - Enqueuing flush of bid_list: 112838010 (11%) on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:1587] 2015-01-20 13:55:48,516 Memtable.java:325 - Writing Memtable-bid_list#2003093066(23761503 serialized bytes, 211002 ops, 11%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:1587] 2015-01-20 13:55:49,251 Memtable.java:364 - Completed flushing /root/Cassandra/apache-cassandra-2.1.2/bin/./../data/data/bigdspace/bid_list-27b59f109fa211e498559b0947587867/bigdspace-bid_list-ka-3965-Data.db (4144688 bytes) for commitlog position ReplayPosition(segmentId=1421647511710, position=25289038)
INFO [SlabPoolCleaner] 2015-01-20 13:56:23,429 ColumnFamilyStore.java:840 - Enqueuing flush of bid_list: 104056985 (10%) on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:1589] 2015-01-20 13:56:23,429 Memtable.java:325 - Writing Memtable-bid_list#1124683519(21909522 serialized bytes, 194778 ops, 10%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:1589] 2015-01-20 13:56:24,130 Memtable.java:364 - Completed flushing /root/Cassandra/apache-cassandra-2.1.2/bin/./../data/data/bigdspace/bid_list-27b59f109fa211e498559b0947587867/bigdspace-bid_list-ka-3967-Data.db (3830733 bytes) for commitlog position ReplayPosition(segmentId=1421647511710, position=25350445)
INFO [SlabPoolCleaner] 2015-01-20 13:56:55,493 ColumnFamilyStore.java:840 - Enqueuing flush of bid_list: 95807739 (9%) on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:1590] 2015-01-20 13:56:55,494 Memtable.java:325 - Writing Memtable-bid_list#473510037(20170635 serialized bytes, 179514 ops, 9%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:1590] 2015-01-20 13:56:56,151 Memtable.java:364 - Completed flushing /root/Cassandra/apache-cassandra-2.1.2/bin/./../data/data/bigdspace/bid_list-27b59f109fa211e498559b0947587867/bigdspace-bid_list-ka-3968-Data.db (3531752 bytes) for commitlog position ReplayPosition(segmentId=1421647511710, position=25373052)
Any help or suggestion would be great. I have also disabled durable write false for the KeySpace. Thanks
Just found out after restarting all the nodes, YGC on one of the server is kicking in even if nothing is happening. Stopped the dumping of data etc.
What type of compaction do you use? Size tiered or Leveled?
If you are using leveled compaction, can you switch off over to Size tiered as you seem to have too many compactions. Increasing the sstable size for leveled compaction may also help.
sstable_size_in_mb (Default: 160MB)
The target size for SSTables that use the leveled compaction strategy. Although SSTable sizes
should be less or equal to sstable_size_in_mb, it is possible to have
a larger SSTable during compaction. This occurs when data for a given
partition key is exceptionally large. The data is not split into two
SSTables.
(http://www.datastax.com/documentation/cassandra/1.2/cassandra/reference/referenceTableAttributes.html#reference_ds_zyq_zmz_1k__sstable_size_in_mb)
If you are using size tiered compaction, increase the number of SS Tables before you see a minor compaction. This is set when the table is created, so you can change it using ALTER command. Example below:
ALTER TABLE users WITH
compaction_strategy_class='SizeTieredCompactionStrategy' AND
min_compaction_threshold = 6;
Compact after 6 SSTables are created
I use a cassandra cluster of 3 EC2 instance nodes on AWS.
Each node mount a EBS volume to the path(/usr/lib/cassandra/), so data of cassandra is stored in EBS volume.
All nodes have a dns to point the private ip of themselves. And all cassandra configure files use the DNS instead of the private ip.
My question is that: when an node is teminated, and i will lanuch another cassandra, and mount the EBS to the newly-launched instance, so the data will not be losted. But when starting cassandra service, cassandra service will fail. And the log is as bellow:
I don't know why it will use ip of other nodes ?
INFO 12:47:36,364 Writing Memtable-schema_keyspaces#1358300025(251/2510 serialized/live bytes, 8 ops)
INFO 12:47:36,367 Enqueuing flush of Memtable-schema_columns#1982787565(91991/919910 serialized/live bytes, 2375 ops)
INFO 12:47:36,367 Enqueuing flush of Memtable-schema_columnfamilies#59370809(38866/388660 serialized/live bytes, 1052 ops)
INFO 12:47:36,385 Completed flushing /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-jb-10-Data.db (214 bytes) for commitlog position ReplayPosition(segmentId=1382719654978, position=289)
INFO 12:47:36,391 Writing Memtable-schema_columns#1982787565(91991/919910 serialized/live bytes, 2375 ops)
INFO 12:47:36,627 Completed flushing /var/lib/cassandra/data/system/schema_columns/system-schema_columns-jb-24-Data.db (19358 bytes) for commitlog position ReplayPosition(segmentId=1382719654978, position=289)
INFO 12:47:36,628 Writing Memtable-schema_columnfamilies#59370809(38866/388660 serialized/live bytes, 1052 ops)
INFO 12:47:36,660 Completed flushing /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-jb-24-Data.db (8750 bytes) for commitlog position ReplayPosition(segmentId=1382719654978, position=289)
INFO 12:47:36,661 Log replay complete, 64 replayed mutations
INFO 12:47:36,746 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-jb-23-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-jb-22-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-jb-24-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-jb-21-Data.db')]
INFO 12:47:37,584 Compacted 4 sstables to [/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-jb-25,]. 34,990 bytes to 8,759 (~25% of original) in 809ms = 0.010325MB/s. 16 total rows, 4 unique. Row merge counts were {1:0, 2:0, 3:0, 4:4, }
INFO 12:47:38,411 Cassandra version: 2.0.1
INFO 12:47:38,412 Thrift API version: 19.37.0
INFO 12:47:38,425 CQL supported versions: 2.0.0,3.1.1 (default: 3.1.1)
INFO 12:47:38,485 Loading persisted ring state
ERROR 12:47:38,537 Exception encountered during startup
java.lang.RuntimeException: Unknown host /172.31.9.175 with no default configured
at org.apache.cassandra.locator.PropertyFileSnitch.getEndpointInfo(PropertyFileSnitch.java:90)
at org.apache.cassandra.locator.PropertyFileSnitch.getDatacenter(PropertyFileSnitch.java:113)
at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:127)
at org.apache.cassandra.locator.TokenMetadata$Topology.addEndpoint(TokenMetadata.java:1049)
at org.apache.cassandra.locator.TokenMetadata.updateNormalTokens(TokenMetadata.java:187)
at org.apache.cassandra.locator.TokenMetadata.updateNormalTokens(TokenMetadata.java:159)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:470)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:428)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:343)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
java.lang.RuntimeException: Unknown host /172.31.9.175 with no default configured
at org.apache.cassandra.locator.PropertyFileSnitch.getEndpointInfo(PropertyFileSnitch.java:90)
at org.apache.cassandra.locator.PropertyFileSnitch.getDatacenter(PropertyFileSnitch.java:113)
at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:127)
at org.apache.cassandra.locator.TokenMetadata$Topology.addEndpoint(TokenMetadata.java:1049)
at org.apache.cassandra.locator.TokenMetadata.updateNormalTokens(TokenMetadata.java:187)
at org.apache.cassandra.locator.TokenMetadata.updateNormalTokens(TokenMetadata.java:159)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:470)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:428)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:343)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
Exception encountered during startup: Unknown host /172.31.9.175 with no default configured
I may be wrong, but in your cassandra.yaml, you may want to look at the seeds variable, it probably is incorrectly referencing an old IP. Another way around this is to use some of the floating IPs in AWS. That way if you spin up a new machine you point the floating IPs to it, and the configuration wouldn't have to change the IPs.
Check if you have configured Default for unknown nodes in your snitch property file.
default =<Any of your DC name>:RAC1
After making the changes restart, hope this will help.
I'm pretty new to Cassandra. I've had a single-node cluster running for a few days with no problems but today it started ignoring some of my CQL commands. SELECTs work fine but if I run DROP TABLE foo; from cqlsh then nothing happens. After a half-second pause it returns me to the prompt but the table wasn't dropped. The same goes for creating an index using CREATE INDEX.
I'm running in a virtual machine, using the Cassandra distribution from OpenStax on Ubuntu 12.04.
I checked the Cassandra logs and I definitely get output when I run a CREATE INDEX, but no apparent errors:
CREATE INDEX number_uri_index ON numbers (number);
Produces:
INFO [MigrationStage:1] 2012-07-25 14:25:59,120 ColumnFamilyStore.java (line 643) Enqueuing flush of Memtable-schema_columnfamilies#15955724(1212/1515 serialized/live bytes, 20 ops)
INFO [FlushWriter:5] 2012-07-25 14:25:59,122 Memtable.java (line 266) Writing Memtable-schema_columnfamilies#15955724(1212/1515 serialized/live bytes, 20 ops)
INFO [FlushWriter:5] 2012-07-25 14:25:59,139 Memtable.java (line 307) Completed flushing /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hd-50-Data.db (1267 bytes) for commitlog position ReplayPosition(segmentId=140485087964, position=8551)
INFO [MigrationStage:1] 2012-07-25 14:25:59,141 ColumnFamilyStore.java (line 643) Enqueuing flush of Memtable-schema_columns#7576227(320/400 serialized/live bytes, 5 ops)
INFO [FlushWriter:5] 2012-07-25 14:25:59,141 Memtable.java (line 266) Writing Memtable-schema_columns#7576227(320/400 serialized/live bytes, 5 ops)
INFO [FlushWriter:5] 2012-07-25 14:25:59,172 Memtable.java (line 307) Completed flushing /var/lib/cassandra/data/system/schema_columns/system-schema_columns-hd-46-Data.db (367 bytes) for commitlog position ReplayPosition(segmentId=140485087964, position=8551)
Same problem here in a 3 nodes setup. Solved doing the same modification on a second node.
Investiganting CASSANDRA jira we discovered that it could be related with the way timestamps are managed by schema related commands and it should be fixed in 1.1.3:
CASSANDRA-4461
CASSANDRA-4432