I have a scylla table as shown below:
cqlsh:sampleks> describe table test;
CREATE TABLE test (
client_id int,
when timestamp,
process_ids list<int>,
md text,
PRIMARY KEY (client_id, when) ) WITH CLUSTERING ORDER BY (when DESC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
AND comment = ''
AND compaction = {'class': 'TimeWindowCompactionStrategy', 'compaction_window_size': '1', 'compaction_window_unit': 'DAYS'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 172800
AND max_index_interval = 1024
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
And I see this is how we are querying it. It's been a long time I worked on cassandra so this PER PARTITION LIMIT is new thing to me (looks like recently added). Can someone explain what does this do with some example in a layman language? I couldn't find any good doc on that which explains easily.
SELECT * FROM test WHERE client_id IN ? PER PARTITION LIMIT 1;
The PER PARTITION LIMIT clause can be helpful in a "wide partition scenario."
It returns only the first two rows in the partition.
Take this query:
aploetz#cqlsh:stackoverflow> SELECT client_id,when,md
FROM test PER PARTITION LIMIT 2 ;
Considering the PRIMARY KEY definition of (client_id,when), that query will iterate over each client_id. Cassandra will then return only the first two rows (clustered by when) from that partition, regardless of how many ocurences of when may be present.
In this case, I inserted 7 rows into your test table, using two different client_ids (2 partitions total). Using a PER PARTITION LIMIT of 2, I get 4 rows returned (2 client_id x PER PARTITION LIMIT 2) == 4 rows.
client_id | when | md
-----------+---------------------------------+-----
1 | 2020-05-06 12:00:00.000000+0000 | md1
1 | 2020-05-05 22:00:00.000000+0000 | md1
2 | 2020-05-06 19:00:00.000000+0000 | md2
2 | 2020-05-06 01:00:00.000000+0000 | md2
(4 rows)
Related
I am new to Cassandra so I had a few quick questions, suppose I do this:
CREATE TABLE my_keyspace.my_table (
id bigint,
year int,
datetime timestamp,
field1 int,
field2 int,
PRIMARY KEY ((id, year), datetime))
I imagine Cassandra as something like Map<PartitionKey, SortedMap<ColKey, ColVal>>,
My question is when querying for something from Cassandra using a WHERE, it will be like:
SELECT * FROM my_keyspace.my_table WHERE id = 1 AND year = 4,
This could return 2 or more records, how does this fit in with the data model of Cassandra?
If it really it a Big HashMap how come duplicate records for a partition key are allowed?
Thanks!
There is a batch of entries in the SortedMap<ColKey, ColVal> for each row, using its sorted nature.
To build on your mental model, while there is only 1 partition key for id = 1 AND year = 4 there are multiple cells:
(id, year) | ColKey | ColVal
------------------------------------------
1, 4 | datetime(1):field1 | 1 \ Row1
1, 4 | datetime(1):field2 | 2 /
1, 4 | datetime(5):field1 | 1 \
1, 4 | datetime(5):field2 | 2 / Row2
...
Cassandra version :- [cqlsh 5.0.1 | Cassandra 3.0.9 | CQL spec 3.4.0 | Native protocol v4]
My Table structure
CREATE TABLE test (
id1 text,
id2 text,
id3 text,
id4 text,
clinet_starttime timestamp,
avail_endtime timestamp,
starttime timestamp,
client_endtime timestamp,
code int,
status text,
total_time double,
PRIMARY KEY (id1, id2, id3, id4, client_starttime)
) WITH CLUSTERING ORDER BY (id2 ASC, id3 ASC, id4 ASC, client_starttime ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
Following query is working for me
SELECT * FROM test WHERE client_starttime<1522832400000 AND client_starttime>1522831800000 AND id1='data1' AND id2='data2' AND id3='data3' AND id4 IN ('data5','data6','data7') ALLOW FILTERING;
But when I query using starttime and avail_endTime instead of client_starttime like
SELECT * FROM test WHERE starttime<1522832400000 AND avail_endTime>1522831800000 AND id1='data1' AND id2='data2' AND id3='data3' AND id4 IN ('data5','data6','data7') ALLOW FILTERING;
getting the following error
InvalidRequest: Error from server: code=2200 [Invalid query] message="Cannot restrict clustering columns by IN relations when a collection is selected by the query"
If I am not using IN operator , If I use = operator then its working fine
SELECT * FROM test WHERE starttime<1522832400000 AND avail_endTime>1522831800000 AND id1='data1' AND id2='data2' AND id3='data3' AND id4 = 'data5'ALLOW FILTERING;
but I want to retrieve data for data5, data6 and data7 same time.
This is a known Cassandra bug: CASSANDRA-12654 - it's already fixed, but will be available only in version 4.0 for which there is no release date yet defined.
I am using pyspark 2.1 to create partitions dynamically from table A to table B. Below are the DDL's
create table A (
objid bigint,
occur_date timestamp)
STORED AS ORC;
create table B (
objid bigint,
occur_date timestamp)
PARTITIONED BY (
occur_date_pt date)
STORED AS ORC;
I am then using a pyspark code where I am trying to determine the partitions that need to be merged, below is the portion of code where I am actually doing that
for row in incremental_df.select(partitioned_column).distinct().collect():
path = '/apps/hive/warehouse/B/' + partitioned_column + '=' + format(row[0])
old_df = merge_df.where(col(partitioned_column).isin(format(row[0])))
new_df = incremental_df.where(col(partitioned_column).isin(format(row[0])))
output_df = old_df.subtract(new_df)
output_df = output_df.unionAll(new_df)
output_df.write.option("compression","none").mode("overwrite").format("orc").save(path)
refresh_metadata_sql = 'MSCK REPAIR TABLE ' + table_name
sqlContext.sql(refresh_metadata_sql)
On Execution of the code I am able to see the partitions in HDFS
Found 3 items
drwx------ - 307010265 hdfs 0 2017-06-01 10:31 /apps/hive/warehouse/B/occur_date_pt=2017-06-01
drwx------ - 307010265 hdfs 0 2017-06-01 10:31 /apps/hive/warehouse/B/occur_date_pt=2017-06-02
drwx------ - 307010265 hdfs 0 2017-06-01 10:31 /apps/hive/warehouse/B/occur_date_pt=2017-06-03
But when I am trying to access the table inside Spark I am getting array out of bound error
>> merge_df = sqlContext.sql('select * from B')
DataFrame[]
>>> merge_df.show()
17/06/01 10:33:13 ERROR Executor: Exception in task 0.0 in stage 200.0 (TID 4827)
java.lang.IndexOutOfBoundsException: toIndex = 3
at java.util.ArrayList.subListRangeCheck(ArrayList.java:1004)
at java.util.ArrayList.subList(ArrayList.java:996)
at org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161)
at org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66)
at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:202)
at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539)
at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.<init>(OrcRawRecordMerger.java:183)
at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.<init>(OrcRawRecordMerger.java:226)
at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:437)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113)
at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:252)
at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:251)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:211)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:102)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
Any help or pointers to resolve the issue will be appreciated
Posting the comment as an answer for easier reference:
Pls ensure that partitioned column is not included in dataframe.
I have a very simple Cassandra table called campaignvariantaction with a counter column called val. I encountered a very strange problem today. For some keys, Cassandra increments the counter just fine. For others, it just doesn't do it. I'm very confused as to how this could be.
Sample output from Cassandra Shell (cqlsh) is below. Note how incrementing a counter works fine for one key (first 3 examples) and doesn't work for another (last example).
Cassandra version 2.2.7 on Ubuntu.
cqlsh:z> UPDATE campaignvariantaction SET val = val + 1 WHERE campaignvariantaction = 'campaign_variant#54408#sent' AND date = 20161118;
cqlsh:z> select * from campaignvariantaction where campaignvariantaction = 'campaign_variant#54408#sent';
campaignvariantaction | date | val
-----------------------------------+----------+-----
campaign_variant#54408#sent | 20161118 | 1
(1 rows)
cqlsh:z> UPDATE campaignvariantaction SET val = val + 1 WHERE campaignvariantaction = 'campaign_variant#54408#sent' AND date = 20161118;
cqlsh:z> select * from campaignvariantaction where campaignvariantaction = 'campaign_variant#54408#sent';
campaignvariantaction | date | val
-----------------------------------+----------+-----
campaign_variant#54408#sent | 20161118 | 2
(1 rows)
cqlsh:z> UPDATE campaignvariantaction SET val = val + 1 WHERE campaignvariantaction = 'campaign_variant#979165#sent' AND date = 20161118;
cqlsh:z> select * from campaignvariantaction where campaignvariantaction = 'campaign_variant#979165#sent';
campaignvariantaction | date | val
-----------------------+------+-----
(0 rows)
Describe output:
cqlsh:z> describe table campaignvariantaction ;
CREATE TABLE z.campaignvariantaction (
campaignvariantaction text,
date int,
val counter,
PRIMARY KEY (campaignvariantaction, date)
) WITH CLUSTERING ORDER BY (date ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
I`m using Cassandra dsc 2.1.5 with 3 nodes, and the following table description:
cqlsh> DESCRIBE TABLE mykeyspace.mytable;
CREATE TABLE mykeyspace.mytable (
a text,
b text,
c timestamp,
d timestamp,
e text,
PRIMARY KEY ((a, b), c)
) WITH CLUSTERING ORDER BY (c ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
The first and the third nodes are working fine (all have the same cassandra.yaml) but the second node started to have more and more pending compaction tasks that I can see with nodetool compactionstats -H command. The situation is so bad that my spark jobs get stuck and work only when I completely shut down the second node.
I have around 130G free on the second node...
Also, here a cfstatss state:
> nodetool cfstatss mykeyspace.mytable
Keyspace: mykeyspace
Read Count: 0
Read Latency: NaN ms.
Write Count: 54316 [1/1863] Write Latency: 0.1877597945356801 ms.
Pending Flushes: 0
Table: mytable
SSTable count: 1249
Space used (live): 1125634027755
Space used (total): 1125634027755
Space used by snapshots (total): 0
Off heap memory used (total): 1202327957
SSTable Compression Ratio: 0.11699340657338655
Number of keys (estimate): 34300801
Memtable cell count: 758856
Memtable data size: 351011415
Memtable off heap memory used: 0
Memtable switch count: 10
Local read count: 0
Local read latency: NaN ms
Local write count: 54319
Local write latency: 0.188 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 48230904
Bloom filter off heap memory used: 48220912
Index summary off heap memory used: 11161093
Compression metadata off heap memory used: 1142945952
Compacted partition minimum bytes: 925
Compacted partition maximum bytes: 52066354
Compacted partition mean bytes: 299014
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
What can be the problem?