Financial data analysis modelling in Apache Cassandra? - cassandra

Question

I need to model and store financial data in Apache Cassandra.
Data is accessed by date and business unit, so currently my model uses the date and business unit id as a compound row key.
I want to use wide-rows so I can pull the figures for a whole day (and unit) in one query.
For any given day, for a particular business unit, I need to store a series of increasingly granular breakdowns, like so (ignore the figures, they're purely illustrative):
| rowkey | USD | GBP | JPY | etc ....
|-------------|-------|------|------|----------
| 31122014-1 | 112 | 3006 | 234 |
| 31122014-2 | 3378 | -12.4| 998 |
| 31122014-3 | -456 | 2034 | 127 |
And then a more detailed breakdown, using compound columns:
| rowkey | USD-D1 | USD-D2 | GBP-D1 | GBP-D2 | etc ....
|-------------|--------|--------|--------|------------------
| 31122014-1 | 65 | 54 | 175 | 29 |
| 31122014-2 | 2003 | -6.4 | 603 | 349 |
| 31122014-3 | -230 | -198 | -53 | 217 |
And then an even more detailed breakdown:
| rowkey | USD-D1-X1 | USD-D1-X2 | USD-D1-X3 | USD-D2-X1 | etc ....
|-------------|-----------|-----------|-----------|-----------|-------
| 31122014-1 | 23 | 16 | 98 | 29 |
| 31122014-2 | 389 | -3.2 | 237 | 119 |
| 31122014-3 | -105 | -67 | -28 | 178 |
Is this the best way to model these breakdowns using three separate column families (as shown here)?
Or does it make more sense to store only the most granular breakdown and then use some form of column aggregation (if it exists) to extract the less granular data-sets?
I know Cassandra's aggregation capability is limited / non existent, I haven't found anything in the API to suggest how I might aggregate across columns like this.
I know I could do the aggregation in the application tier, but then the question is about the trade offs between retrieving unnecessary data, moving computational overhead and maintaining additional column families. I'm hoping Cassandra provides some way of solving this at the data tier.

Answer 1

Depending on how you want you want the data to be modeled you can
Use your solution. In this you create a column family for more details
If you feel that there are far too column families or that you will always use the next column family, i would suggest making it part of the primary key as a clustering key or directly as part of the partition key
For example:
If according to your data model, if row key access is always going to include a currency you could model it like this
| rowkey |currency|
|---------------|--------|
| 31122014-1,GBP| 112 |
Obviously this will spread your data for a single rowkey much better, but will increase the number of row keys
You could use aggregation as well as custom types that cassandra allows.
Consider the following before you choose one of the stategies
a. Distribution of the rows across nodes
b. Sparse columns vs wide columns
c. Effects on row cache (if you are going to turn it on) and key cache
d. And the most important, your selection queries

Answer 2

I think your solution is likely to be effective. For Cassandra it's generally better to store data multiple in multiple places based on what queries you're expecting to run against it.
If you see each of these use cases as three separate use cases that will be queried at different times, then you've got a solid datamodel.
For what it's worth, this use case plays very well to the strengths of CQL which would model it as follows:
CREATE TABLE finance0 (
day DATE,
unit INT,
currency TEXT,
amount BIGINT,
PRIMARY KEY ((day, unit) currency)
);
CREATE TABLE finance1 (
day DATE,
unit INT,
currency TEXT,
sorter1 TEXT,
amount BIGINT,
PRIMARY KEY ((day, unit) currency, sorter1)
);
CREATE TABLE finance2 (
day DATE,
unit INT,
currency TEXT,
sorter1 TEXT,
sorter2 TEXT,
amount BIGINT,
PRIMARY KEY ((day, unit) currency, sorter1, sorter2)
);