Understanding the Token Function in Cassandra - cassandra

Question

Hello I was reading the Cassandra documentation on Token Function,
I am trying to achieve pagination for a Cassandra table, I am unable to understand the lines highlighted. The document speaks about the difference between k > 42 and TOKEN(k) > TOKEN(42), but I am not able to understand the "token based comparison"
Looking forward for a detailed explanation of what token function does when part of a WHERE clause.

Answer 1

In order to understand in which partition it should put your data, C* makes some calculations on the PARTITION KEYs of every row. Specifically, on each node, rows are sorted by the token generated by the partitioner, (and each partition have data sorted by the cluster key). Different partitioners perform different types of calculations.
While the Murmur3Partitioner calculates the MurmurHash of the partion key, the ByteOrderedPartitioner uses the raw data bytes of the partition key itself: when you use the Murmur3Partitioner, your rows are sorted by their hashes, while when you use the ByteOrderedPartitioner, your rows are sorted directly by their raw values.
As an example, assume you have a table like this:
CREATE TABLE test (
username text,
...
PRIMARY KEY (username)
);
And assume you're trying to locate where the rows corresponding to the usernames abcd and abce and abcf are stored. The hex representation of these strings are 0x61626364 and 0x61626365 and 0x61626366 respectively. Assuming we apply this MH3 implementation (x86, 32-bit for simplicity, no optional seed) on both strings we get ‭0x‭43ED676A‬‬ and 0x‭‭E297E8AA‬‬ and 0x‭‭87E62668‬‬ respectively. So, in the case of MH3, the tokens of the strings will be these 3 values, while in the case of the BOP the tokens will be the raw data values themselves: 0x61626364, 0x61626365 and 0x61626366.
Now you can see that storing data sorted by token produces different results when different partitioners are used. A SELECT * FROM test; query would return rows in different order. This can (but should not) be a problem if you have data already sorted by their raw values and you need to retrieve that in the same order because when you use MH3 the order is complelety unrelated to your data.
Back to the question, the TOKEN function allows you to filter directly by the tokens of your data instead of your data. The documentation says:
ordering with the TOKEN function does not always provide the expected
results. Use the TOKEN function to express a conditional relation on a
partition key column. In this case, the query returns rows based on
the token of the partition key rather than on the value.
As an example, you could issue:
SELECT * FROM test WHERE TOKEN(username) <= TOKEN('abcf');
and you'd get figure what? abcd and acbf rows!!! This is because order sometimes matters... Like in the case of the pagination you're trying to do, which will be handled flawlessy for you by any available C* driver (eg the Java driver).
That said, the recommended partitioner for new clusters is Murmur3Partitioner, you can check the documentation for both pros and cons of each partitioner. Please note that the partitioner is a cluster-wide settings, and once set you cannot change it without pushing all of your data into another cluster.
Make your choice carefully.

Answer 2

Cassandra data is partitioned based on the Token of row's PartitionKey. The token is gerenated using a Hash Function. The function Token generates the value which would have been created by applying the hash function to it's arguments.
That said, almost all drivers now page automatically by default.