How to get X% percentile in Cassandra - cassandra

Question

Consider a table with structure:
CREATE TABLE statistics (name text, when timestamp, value int,
PRIMARY KEY ((name, when)));
What is the best way to calculate, for example, 50% value percentile by name?
I thought about:
a) writing custom aggregate function + query like:
SELECT PERCENTILE(value, 0.5) FROM statistics WHERE name = '...'
b) count elements by name first
SELECT COUNT(value) FROM statistics WHERE name = '...'
then find (0.5/count)th row value with paging when it is sorted by value ascending. Say, if count is 100 it will be 50th row.
c) your ideas
I'm not sure if case A can handle the task. Case B might be tricky when there is odd number of rows.

Answer 1

As long as you always provide name - this request can be very expensive without specifying partition and having everything within one. I am assuming you mean ((name), when) not ((name, when)) in your table, otherwise what your asking is impossible without full table scans (using hadoop or spark).
The UDA would work - but it can be expensive unless your willing to accept an approximation. To have it perfectly accurate you need to do 2 pass (ie doing a count, than a 2nd pass to go X into set, but since no isolation this isnt gonna be perfect either). So if you need it perfectly accurate your best bet is probably to just pull entire statistics[name] partition locally or to have UDA build up entire set (or majority) in a map (not recommended if partitions get large at all) before calculating. ie:
CREATE OR REPLACE FUNCTION all(state tuple<double, map<int, int>>, val int, percentile double)
CALLED ON NULL INPUT RETURNS tuple<double, map<int, int>> LANGUAGE java AS '
java.util.Map<Integer, Integer> m = state.getMap(1, Integer.class, Integer.class);
m.put(m.size(), val);
state.setMap(1, m);
state.setDouble(0, percentile);
return state;';
CREATE OR REPLACE FUNCTION calcAllPercentile (state tuple<double, map<int, int>>)
CALLED ON NULL INPUT RETURNS int LANGUAGE java AS
'java.util.Map<Integer, Integer> m = state.getMap(1, Integer.class, Integer.class);
int offset = (int) (m.size() * state.getDouble(0));
return m.get(offset);';
CREATE AGGREGATE IF NOT EXISTS percentile (int , double)
SFUNC all STYPE tuple<double, map<int, int>>
FINALFUNC calcAllPercentile
INITCOND (0.0, {});
If willing to accept an approximation you can use a sampling reservoir, say 1024 elements you store and as your UDA gets elements you replace elements in it at a decreasingly statistical chance. (vitter's algorithm R) This is pretty easy to implement, and IF your data set is expected to have a normal distribution will give you a decent approximation. If your data set is not a normal distribution this can be pretty far off. With a normal distribution theres actually a lot of other options as well but R is I think easiest to implement in a UDA. like:
CREATE OR REPLACE FUNCTION reservoir (state tuple<int, double, map<int, int>>, val int, percentile double)
CALLED ON NULL INPUT RETURNS tuple<int, double, map<int, int>> LANGUAGE java AS '
java.util.Map<Integer, Integer> m = state.getMap(2, Integer.class, Integer.class);
int current = state.getInt(0) + 1;
if (current < 1024) {
// fill the reservoir
m.put(current, val);
} else {
// replace elements with gradually decreasing probability
int replace = (int) (java.lang.Math.random() * (current + 1));
if (replace <= 1024) {
m.put(replace, val);
}
}
state.setMap(2, m);
state.setDouble(1, percentile);
state.setInt(0, current);
return state;';
CREATE OR REPLACE FUNCTION calcApproxPercentile (state tuple<int, double, map<int, int>>)
CALLED ON NULL INPUT RETURNS int LANGUAGE java AS
'java.util.Map<Integer, Integer> m = state.getMap(2, Integer.class, Integer.class);
int offset = (int) (java.lang.Math.min(state.getInt(0), 1024) * state.getDouble(1));
if(m.get(offset) != null)
return m.get(offset);
else
return 0;';
CREATE AGGREGATE IF NOT EXISTS percentile_approx (int , double)
SFUNC reservoir STYPE tuple<int, double, map<int, int>>
FINALFUNC calcApproxPercentile
INITCOND (0, 0.0, {});
In above, the percentile function will get slower sooner, playing with size of sampler can give you more or less accuracy but too large and you start to impact performance. Generally a UDA over more than 10k values (even simple functions like count) starts to fail. Important to recognize in these scenarios too that while the single query returns a single value, theres a ton of work to get it. So a lot of these queries or much concurrency will put a lot of pressure on your coordinators. This does require >3.8 (I would recommend 3.11.latest+) for CASSANDRA-10783
note: I make no promises that I havent missed an off by 1 error in example UDAs - I did not test fully, but should be close enough you can make it work from there