what are cqlsh tracing entries meaning? - cassandra

Question

activity | timestamp | source | source_elapsed
------------------------------------------------------------------------------------------------------+--------------+---------------+----------------
execute_cql3_query | 06:30:52,479 | 192.168.11.23 | 0
Parsing select adid from userlastadevents where userid = '90000012' and type in (1,2,3) LIMIT 10000; | 06:30:52,479 | 192.168.11.23 | 44
Peparing statement | 06:30:52,479 | 192.168.11.23 | 146
Executing single-partition query on userlastadevents | 06:30:52,480 | 192.168.11.23 | 665
Acquiring sstable references | 06:30:52,480 | 192.168.11.23 | 680
Executing single-partition query on userlastadevents | 06:30:52,480 | 192.168.11.23 | 696
Acquiring sstable references | 06:30:52,480 | 192.168.11.23 | 704
Merging memtable tombstones | 06:30:52,480 | 192.168.11.23 | 706
Merging memtable tombstones | 06:30:52,480 | 192.168.11.23 | 721
Bloom filter allows skipping sstable 37398 | 06:30:52,480 | 192.168.11.23 | 758
Bloom filter allows skipping sstable 37426 | 06:30:52,480 | 192.168.11.23 | 762
Bloom filter allows skipping sstable 35504 | 06:30:52,480 | 192.168.11.23 | 768
Bloom filter allows skipping sstable 36671 | 06:30:52,480 | 192.168.11.23 | 771
Merging data from memtables and 0 sstables | 06:30:52,480 | 192.168.11.23 | 777
Merging data from memtables and 0 sstables | 06:30:52,480 | 192.168.11.23 | 780
Executing single-partition query on userlastadevents | 06:30:52,480 | 192.168.11.23 | 782
Acquiring sstable references | 06:30:52,480 | 192.168.11.23 | 791
Read 0 live and 0 tombstoned cells | 06:30:52,480 | 192.168.11.23 | 797
Read 0 live and 0 tombstoned cells | 06:30:52,480 | 192.168.11.23 | 800
Merging memtable tombstones | 06:30:52,480 | 192.168.11.23 | 815
Bloom filter allows skipping sstable 37432 | 06:30:52,480 | 192.168.11.23 | 857
Bloom filter allows skipping sstable 36918 | 06:30:52,480 | 192.168.11.23 | 866
Merging data from memtables and 0 sstables | 06:30:52,480 | 192.168.11.23 | 874
Read 0 live and 0 tombstoned cells | 06:30:52,480 | 192.168.11.23 | 898
Request complete | 06:30:52,479 | 192.168.11.23 | 990
Above is the tracing output from cassandra cqlsh for a single query, but I couln't understand some the entries, at first the column "source_elapsed" what does it mean, does it mean time elapsed to execute particular task or cumulative time elapsed up to this task. second "timestamp" doesn't maintain chronology like "Request Complete" timestamp is 06:30:52,479 but "Merging data from memtables and 0 sstables" is 06:30:52,480 which is suppose to happen earlier but timestamp shows it happens later.
And couldn't understand some of the activities as well,
Executing single-partition query -- doesn't it mean all the task as a whole or is it a starting point? what are the job it includes? And why is it repeating three times? is it link to replication factor.
Acquiring sstable references -- What does it mean, does it checks all the sstable's bloom filters whether that contains a particular key we search for? An then find the reference in data file with the help of "Partition Index".
Bloom filter allows skipping sstable -- when does it happen? How does it happen? it is taking same amount of time of finding sstable references.
Request complete -- what does it mean? is it the finishing line or it is some job which takes most amount of time?

Answer 1

Did you see the request tracing in Cassandra link that explains different tracing scenarios?
source_elapsed: is the cumulative execution time on a specific node (if you check the above link it will be clearer)
Executing single-partition query: (seems to represent) the start time
Request complete: all work has been done for this request
For the rest you'd be better off reading the Reads in Cassandra docs as that would be much more detailed than I could summarize it here.