I am trying to evaluate number of tombstones getting created in one of tables in our application. For that I am trying to use nodetool cfstats. Here is how I am doing it:
create table demo.test(a int, b int, c int, primary key (a));
insert into demo.test(a, b, c) values(1,2,3);
Now I am making the same insert as above. So I expect 3 tombstones to be created. But on running cfstats for this columnfamily, I still see that there are no tombstones created.
nodetool cfstats demo.test
Average live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Now I tried deleting the record, but still I don't see any tombstones getting created. Is there any thing that I am missing here? Please suggest.
BTW a few other details,
* We are using version 2.1.1 of the Java driver
* We are running against Cassandra 2.1.0
For tombstone counts on a query your best bet is to enable tracing. This will give you the in depth history of a query including how many tombstones had to be read to complete it. This won't give you the total tombstone count, but is most likely more relevant for performance tuning.
In cqlsh you can enable this with
cqlsh> tracing on;
Now tracing requests.
cqlsh> SELECT * FROM ascii_ks.ascii_cs where pkey = 'One';
pkey | ckey1 | data1
------+-------+-------
One | One | One
(1 rows)
Tracing session: 2569d580-719b-11e4-9dd6-557d7f833b69
activity | timestamp | source | source_elapsed
--------------------------------------------------------------------------+--------------+-----------+----------------
execute_cql3_query | 08:26:28,953 | 127.0.0.1 | 0
Parsing SELECT * FROM ascii_ks.ascii_cs where pkey = 'One' LIMIT 10000; | 08:26:28,956 | 127.0.0.1 | 2635
Preparing statement | 08:26:28,960 | 127.0.0.1 | 6951
Executing single-partition query on ascii_cs | 08:26:28,962 | 127.0.0.1 | 9097
Acquiring sstable references | 08:26:28,963 | 127.0.0.1 | 10576
Merging memtable contents | 08:26:28,963 | 127.0.0.1 | 10618
Merging data from sstable 1 | 08:26:28,965 | 127.0.0.1 | 12146
Key cache hit for sstable 1 | 08:26:28,965 | 127.0.0.1 | 12257
Collating all results | 08:26:28,965 | 127.0.0.1 | 12402
Request complete | 08:26:28,965 | 127.0.0.1 | 12638
http://www.datastax.com/dev/blog/tracing-in-cassandra-1-2
Related
I am trying to measure the query execution time for comparison purpose with other sql and NoSQL databases. For this, i have used tracing on command, but can't understand which is the query execution time. I can't find exact information on internet. For table creation, i am using query:
Tracing on;
CREATE TABLE statistics(country_name text, dt date, confirmed_cases bigint, deaths bigint,
PRIMARY KEY(country_name, deaths))with clustering order by (deaths DESC);
cqlsh is showing the result like this:
left
center
right
One
Two
Three
|activity| timestamp | source | source_elapsed | client |
|:---- |:------:| -----:| -----:|
|Execute CQL3 query | 2022-05-10 10:38:06.084000 | 172.24.2.2 | 0 |
41e1:cbdc:b845:42f6:aa06:27ea:d549:3af0
|Parsing CREATE TABLE statistics(country_name text, dt date, confirmed_cases bigint, deaths
bigint, PRIMARY KEY(country_name, deaths))with clustering order by (deaths DESC);
[CoreThread-6] | 2022-05-10 10:38:06.085000 | 172.24.2.2 | 254 |
41e1:cbdc:b845:42f6:aa06:27ea:d549:3af0
|Preparing statement [CoreThread-6] | 2022-05-10 10:38:06.085000 | 172.24.2.2 |
457 | 41e1:cbdc:b845:42f6:aa06:27ea:d549:3af0
Adding to tables memtable [SchemaUpdatesStage:1] | 2022-05-10 10:38:06.092000 | 172.24.2.2
| 8175 | 41e1:cbdc:b845:42f6:aa06:27ea:d549:3af0
Adding to keyspaces memtable [SchemaUpdatesStage:1] | 2022-05-10 10:38:06.092000 |
172.24.2.2 | 8244 | 41e1:cbdc:b845:42f6:aa06:27ea:d549:3af0
Adding to columns memtable [SchemaUpdatesStage:1] | 2022-05-10 10:38:06.092000 | 172.24.2.2
|8320 | 41e1:cbdc:b845:42f6:aa06:27ea:d549:3af0
Request complete | 2022-05-10 10:38:06.141445 | 172.24.2.2 | 57445 |
41e1:cbdc:b845:42f6:aa06:27ea:d549:3af0
So which one is the actual time for execution of table creation query? i also need to trace execution time for insert query and retrieval of highest value of a column by partition. Please help!
Note: The source_elapsed column value is the elapsed time of the event on the source node in microseconds.
source_elapsed is the cumulative execution time on a specific node
"Request complete | 2022-05-10 10:38:06.141445 | 172.24.2.2 | 57445 |
41e1:cbdc:b845:42f6:aa06:27ea:d549:3af0"
Hive version: 3.1.0.3.1.4.0-315
spark version: 2.3.2.3.1.4.0-315
Basically, i am trying to read transactional table data from spark. As per this page [https://stackoverflow.com/questions/50254590/how-to-read-orc-transaction-hive-table-in-spark][1], found that transactional table has to be compacted. Hence, i want to try this approach.
I am new to this and was trying compaction on delta files but it always shows "initiated" and never complete.
This is happening for both Major and Minor compaction. Any help will be highly appreciated.
I want to know whether is this good approach.
Also, how to monitor the compaction job process other than show compactions? i can only see the line "Compaction enqueued with id 1" from the hiveserver_stdout.log.
Generally, how long does this compaction takes to complete?
is there any way to stop the compactions?
TIA.
[Edited]
SHOW COMPACTIONS;
+---------------+-----------+----------------+----------------+--------+------------+-----------+-------------+---------------+--------------+
| compactionid | dbname | tabname | partname | type | state | workerid | starttime | duration | hadoopjobid |
+---------------+-----------+----------------+----------------+--------+------------+-----------+-------------+---------------+--------------+
| CompactionId | Database | Table | Partition | Type | State | Worker | Start Time | Duration(ms) | HadoopJobId |
| 1 | tmp | shop_na2 | dt=2014-00-00 | MAJOR | initiated | --- | --- | --- | --- |
| 2 | tmp | na2_check | dt=2014-00-00 | MINOR | initiated | --- | --- | --- | --- |
+---------------+-----------+----------------+----------------+--------+------------+-----------+-------------+---------------+--------------+
3 rows selected (0.408 seconds)
The same compactions result has been showing for past 36 hours, though retention period has been set as 86400 sec.
It is advised to perform this operation when the load on the cluster is less, maybe initiate over a weekend when there are less jobs running, it is a resource intensive operation and amount of time depends on the data but a moderate quantity of deltas would span multiple hours. You can use the query SHOW COMPACTIONS; to get an update on the status of compaction including the following details
Database name
Table name
Partition name
Major or minor compaction
Compaction state:
Initiated - waiting in queue
Working - currently compacting
Ready for cleaning - compaction completed and old files scheduled for removal
Thread ID
Start time of compaction
I'm implementing unique entry counter in Cassandra. The counter may be represented just as a set of tuples:
counter_id = broadcast:12345, token = user:123
counter_id = broadcast:12345, token = user:321
where value for counter broadcast:12345 may be counted as size of corresponding entries set. Such counter can be effectively stored as a table with counter_id being partition key. My first thought was that since single counter value is basically size of partition, i can do count(1) WHERE counter_id = ? query, which won't need to read data and would be super-duper fast. However, i see following trace output:
cqlsh > select count(1) from token_counter_storage where id = '1';
activity | timestamp | source | source_elapsed
-------------------------------------------------------------------------------------------------+----------------------------+------------+----------------
Execute CQL3 query | 2016-06-10 11:22:42.809000 | 172.17.0.2 | 0
Parsing select count(1) from token_counter_storage where id = '1'; [SharedPool-Worker-1] | 2016-06-10 11:22:42.809000 | 172.17.0.2 | 260
Preparing statement [SharedPool-Worker-1] | 2016-06-10 11:22:42.810000 | 172.17.0.2 | 565
Executing single-partition query on token_counter_storage [SharedPool-Worker-2] | 2016-06-10 11:22:42.810000 | 172.17.0.2 | 1256
Acquiring sstable references [SharedPool-Worker-2] | 2016-06-10 11:22:42.810000 | 172.17.0.2 | 1350
Skipped 0/0 non-slice-intersecting sstables, included 0 due to tombstones [SharedPool-Worker-2] | 2016-06-10 11:22:42.810000 | 172.17.0.2 | 1465
Merging data from memtables and 0 sstables [SharedPool-Worker-2] | 2016-06-10 11:22:42.810000 | 172.17.0.2 | 1546
Read 10 live and 0 tombstone cells [SharedPool-Worker-2] | 2016-06-10 11:22:42.811000 | 172.17.0.2 | 1826
Request complete | 2016-06-10 11:22:42.811410 | 172.17.0.2 | 2410
I guess that this trace confirms data being read from disk. Am i right in this conclusion, and if yes, is there any way to simply fetch partition size using index without any excessive disk hits?
I've got two nodes that are fully replicated. When I run a query on a table that contains 30 rows, cqlsh trace seems to indicate it is fetching some rows from one server and some rows from the other server.
So even though all the rows are available on both nodes, the query takes 250ms+ rather than 1ms for other queries.
I've already got consistency level set to "one" at the protocol level, what else do you have to do to make it only use one node for the query?
select * from organisation:
activity | timestamp | source | source_elapsed
-------------------------------------------------------------------------------------------------+--------------+--------------+----------------
execute_cql3_query | 04:21:03,641 | 10.1.0.84 | 0
Parsing select * from organisation LIMIT 10000; | 04:21:03,641 | 10.1.0.84 | 68
Preparing statement | 04:21:03,641 | 10.1.0.84 | 174
Determining replicas to query | 04:21:03,642 | 10.1.0.84 | 307
Enqueuing request to /10.1.0.85 | 04:21:03,642 | 10.1.0.84 | 1034
Sending message to /10.1.0.85 | 04:21:03,643 | 10.1.0.84 | 1402
Message received from /10.1.0.84 | 04:21:03,644 | 10.1.0.85 | 47
Executing seq scan across 0 sstables for [min(-9223372036854775808), min(-9223372036854775808)] | 04:21:03,644 | 10.1.0.85 | 461
Read 1 live and 0 tombstoned cells | 04:21:03,644 | 10.1.0.85 | 560
Read 1 live and 0 tombstoned cells | 04:21:03,644 | 10.1.0.85 | 611
………..etc….....
It turns out that there was a bug in Cassandra versions 2.0.5-2.0.9 that would make Cassandra more likely to request data on two nodes when it only needed to talk to one.
Upgrading to 2.0.10 or greater resolves this problem.
Refer: CASSANDRA-7535
My question is related to Logs of Queries ran in Cassandra.
I have a cassandra Cluster. Now , if i run any query on it which if takes good amount of time ( say 1 hour ) to completely execute, then is there any way with which I can trace the status of the query and that too without using any cassandra API.
What I found regarding this is that we can turn 'tracing ON;' in Cassandra-cli, and then if I run any query, then I'll get the proper step-by-step status of the query.
For example :
**cqlsh> use demo;
cqlsh:demo> CREATE TABLE test ( a int PRIMARY KEY, b text );
cqlsh:demo> tracing on;
Now tracing requests.
cqlsh:demo> INSERT INTO test (a, b) VALUES (1, 'example');
Unable to complete request: one or more nodes were unavailable.
Tracing session: 4dc5f950-6625-11e3-841a-b7e2b08eed3e
activity | timestamp | source | source_elapsed
--------------------------------------------------------+--------------+----------------+----------------
execute_cql3_query | 13:10:15,627 | 192.168.171.87 | 0
Parsing INSERT INTO test (a, b) VALUES (1, 'example'); | 13:10:15,640 | 192.168.171.87 | 13770
Preparing statement | 13:10:15,657 | 192.168.171.87 | 30090
Determining replicas for mutation | 13:10:15,669 | 192.168.171.87 | 42689
Unavailable | 13:10:15,682 | 192.168.171.87 | 55131
Request complete | 13:10:15,682 | 192.168.171.87 | 55303**
But it does not satisfy my requirement as I need to see the status of any previously ran query.
Please provide any solution.
Thanks
Saurabh
Take a look at the system_traces keyspace events and sessions tables.