why the rpc timeout occur in cassandra - cassandra

Question

I tried using cqlsh -3 version on my keyspace and used select query on a column family.
It's return data in some causes and throws RPC time out in some other causes,I don't know the exact root cause.
I used select query with single where condition
select * FROM date where date='2013-10-11 00:00:00+0000';
In this date column has secondary index with datatype text in UTF8 format
Request did not complete within rpc_timeout.
I checked with cassandra log.it throws
ERROR [ReadStage:117] 2013-12-03 19:21:46,813 CassandraDaemon.java (line 192) Exception in thread Thread[ReadStage:117,5,main]
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:119)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60)
at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:132)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1390)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1213)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1125)
at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:191)
at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:109)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1499)
at org.apache.cassandra.db.index.keys.KeysSearcher.search(KeysSearcher.java:82)
at org.apache.cassandra.db.index.SecondaryIndexManager.search(SecondaryIndexManager.java:548)
at org.apache.cassandra.db.ColumnFamilyStore.search(ColumnFamilyStore.java:1487)
at org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:44)
at org.apache.cassandra.service.StorageProxy$LocalRangeSliceRunnable.runMayThrow(StorageProxy.java:1055)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1547)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
why this happening?
I am checking in my local with single seed?
Update 1:
my date table Structure
CREATE TABLE date (
key text PRIMARY KEY,
date text,
date_id text,
day bigint,
day_name text
) WITH COMPACT STORAGE AND
bloom_filter_fp_chance=0.010000 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction={'min_sstable_size': '52428800', 'class': 'SizeTieredCompactionStrategy'} AND
compression={'chunk_length_kb': '64', 'sstable_compression': 'SnappyCompressor'};
I checked with cassandra log,Its shows
ERROR [ReadStage:94] 2013-12-03 22:07:17,116 CassandraDaemon.java (line 192) Exception in thread Thread[ReadStage:94,5,main]
java.lang.AssertionError: DecoratedKey(-8665312888645846270,.......................<!--some bytes of numbers------->
/var/lib/cassandra/data/keyspace/columnfamily/keyspace-columnfamily-ic-1-Data.db
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:119)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60)
at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:132)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1390)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1213)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1125)
at org.apache.cassandra.db.Table.getRow(Table.java:347)
at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1033)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1547)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
currently i am using cassandra-1.2.6.
I check with this link,is this cassandra issue?
https://issues.apache.org/jira/browse/CASSANDRA-4687

Answer 1

If your query is expensive this can lead to rpc-timeouts. There's a bunch of SO questions along this line for different types of query, e.g., Fetching all the records for a partitionID in cassandra gives RPC timeout, RPC timeout in cqlsh - Cassandra (select count(*) queries). However, your question relates specifically to secondary indices.
Querying on secondary indices should be avoided if the number of unique indexed entries is high, as they are much more expensive than querying by key (i suspect this is the case if you are using dates). Perhaps there is a better way of modelling your data? (If you add the data model to your question I can try to elaborate on this answer.)