Cassandra how to see active user connections - cassandra

In cassandra (am using DSE),
how do I check how many users are connected to the database? Any way to check node wise?
Is there any auditing info stored which will tell me which all users connected along with info such as IP address and driver used etc?
In Opscenter there is a metric called "Native clients", where is this info stored in the db to query for? Does this include internal communication between the nodes and backups etc?

How do I check how many users are connected to the database? Any way to check node wise?
Is there any auditing info stored which will tell me which all users connected along with info such as IP address and driver used etc?
DSE has a performance service feature which you can enable to make this information available via cql. To enable this particular capability, configure the following in dse.yaml as described in the docs:
user_level_latency_tracking_options:
enabled: true
With this enabled, you can now query a variety of tables, for example:
cqlsh> select * from dse_perf.user_io;
node_ip | conn_id | last_activity | read_latency | total_reads | total_writes | user_ip | username | write_latency
-----------+-----------------+---------------------------------+--------------+-------------+--------------+-----------+-----------+---------------
127.0.0.1 | 127.0.0.1:55116 | 2019-01-14 14:08:19.399000+0000 | 1000 | 1 | 0 | 127.0.0.1 | anonymous | 0
127.0.0.1 | 127.0.0.1:55252 | 2019-01-14 14:07:39.399000+0000 | 0 | 0 | 1 | 127.0.0.1 | anonymous | 1000
(2 rows)
cqlsh> select * from dse_perf.user_object_io;
node_ip | conn_id | keyspace_name | table_name | last_activity | read_latency | read_quantiles | total_reads | total_writes | user_ip | username | write_latency | write_quantiles
-----------+-----------------+---------------+------------+---------------------------------+--------------+----------------+-------------+--------------+-----------+-----------+---------------+-----------------
127.0.0.1 | 127.0.0.1:55252 | s | t | 2019-01-14 14:07:39.393000+0000 | 0 | null | 0 | 1 | 127.0.0.1 | anonymous | 1000 | null
127.0.0.1 | 127.0.0.1:55116 | s | t | 2019-01-14 14:08:19.393000+0000 | 1000 | null | 1 | 0 | 127.0.0.1 | anonymous | 0 | null
Note that there is a cost to enabling the performance service, and it can be enabled and disabled selectively using dsetool perf userlatencytracking [enable|disable].
In a future release of Apache Cassandra (4.0+) and DSE (likely 7.0+), there will be a nodetool clientstats command (CASSANDRA-14275), and a corresponding system_views.clients table (CASSANDRA-14458) that includes connection info. This will include the driver name, if the driver client provides one (newer ones do).
In Opscenter there is a metric called "Native clients", where is this info stored in the db to query for? Does this include internal communication between the nodes and backups etc?
I'm not too up to speed on OpsCenter. From what I know OpsCenter typically stores it's data in the OpsCenter keyspace, you can configure data collection parameters by following this doc.

Related

Adding a User to Already Created Database - Give Full Read Access

I have been stuck on this problem for a while now and cannot figure it out. Hoping someone can help me.
I think my situation is pretty simple, so I feel extra stupid for having to post this Nonetheless -- I have a database, lets call it tempdb, that was created by user ikaros on Postgres 13.3 (Ubuntu 13.3-1.pgdg16.04+1)
Here is the output from \l+ with irrelevant information omitted.
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges | Size | Tablespace | Description
-----------------------+----------+----------+-------------+-------------+-----------------------+---------+------------+--------------------------------------------
...
ikaros | ikaros | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | 8029 kB | pg_default |
tempdb | ikaros | UTF8 | C | C | =T/ikaros +| 13 GB | pg_default |
| | | | | ikaros=CTc/ikaros +| | |
| | | | | johndoe=CTc/ikaros | | |
...
Currently, johndoe can connect to the database tempdb, but when executing a query, gets a message about not having sufficient table level privilege's. Error: Unable to execute query: Fatal Error; Reason: Error: (ERROR: permission denied for table settings )
I want johndoe to have full read privilege's on the tempdb along with all tables inside. How can I go about that? Thanks in advance!
According to Postgres documents, You can use below queries to add permission to users:
-- This query used for access to the database
grant connect on database [YOUR_DATABASE] to [USERNAME];
-- This query used for access to the schema
grant usage on schema [SCHEMA1, SCHEMA2, ...] to [USERNAME];
-- This query is used for access to all tables of a schema (or can use just **select** instead of **all**
grant all on all tables in schema [SCHEMA1, SCHEMA2, ...] to [USERNAME];
grant select on all tables in schema [SCHEMA1, SCHEMA2, ...] to [USERNAME];
-- If need add select permission to a specific table
grant select on table [YOUR_SCHEMA].[YOUR_TABLE] to [USERNAME];

Timestamp data persistence in cassandra

In our crawling system we have 5 node cassandra cluster. I have a scenario in which I want to delete cassandra data as soon as it becomes older than x days.
Ex:
id | name | created_date
1 | Dan | "2017-08-01"
2 | Monk | "2017-08-02"
3 | Shibuya | "2017-08-03"
4 | Rewa | "2017-08-04"
5 | Himan | "2017-08-05"
if x = 3 Then following should be the scenario:
id | name | created_date
1 | Dan | "2017-08-01" --------------> DELETE
2 | Monk | "2017-08-02" --------------> DELETE
3 | Shibuya | "2017-08-03" -------------->(REMAIN)Latest 3 days data
4 | Rewa | "2017-08-04" -------------->(REMAIN)Latest 3 days data
5 | Himan | "2017-08-05" -------------->(REMAIN)Latest 3 days data
If new data is added then id=3 should be deleted.
Is there any Cassandra configuration or any approach to do this?
Cassandra has a TTL feature, that allows you to specify how long each CQL cell is valid. Details are available on the INSERT docs, but it also applies to UPDATE.
You can use TTL.
But be careful with tombstouns and compaction procedure

How to get tombstone count for a cql query?

I am trying to evaluate number of tombstones getting created in one of tables in our application. For that I am trying to use nodetool cfstats. Here is how I am doing it:
create table demo.test(a int, b int, c int, primary key (a));
insert into demo.test(a, b, c) values(1,2,3);
Now I am making the same insert as above. So I expect 3 tombstones to be created. But on running cfstats for this columnfamily, I still see that there are no tombstones created.
nodetool cfstats demo.test
Average live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Now I tried deleting the record, but still I don't see any tombstones getting created. Is there any thing that I am missing here? Please suggest.
BTW a few other details,
* We are using version 2.1.1 of the Java driver
* We are running against Cassandra 2.1.0
For tombstone counts on a query your best bet is to enable tracing. This will give you the in depth history of a query including how many tombstones had to be read to complete it. This won't give you the total tombstone count, but is most likely more relevant for performance tuning.
In cqlsh you can enable this with
cqlsh> tracing on;
Now tracing requests.
cqlsh> SELECT * FROM ascii_ks.ascii_cs where pkey = 'One';
pkey | ckey1 | data1
------+-------+-------
One | One | One
(1 rows)
Tracing session: 2569d580-719b-11e4-9dd6-557d7f833b69
activity | timestamp | source | source_elapsed
--------------------------------------------------------------------------+--------------+-----------+----------------
execute_cql3_query | 08:26:28,953 | 127.0.0.1 | 0
Parsing SELECT * FROM ascii_ks.ascii_cs where pkey = 'One' LIMIT 10000; | 08:26:28,956 | 127.0.0.1 | 2635
Preparing statement | 08:26:28,960 | 127.0.0.1 | 6951
Executing single-partition query on ascii_cs | 08:26:28,962 | 127.0.0.1 | 9097
Acquiring sstable references | 08:26:28,963 | 127.0.0.1 | 10576
Merging memtable contents | 08:26:28,963 | 127.0.0.1 | 10618
Merging data from sstable 1 | 08:26:28,965 | 127.0.0.1 | 12146
Key cache hit for sstable 1 | 08:26:28,965 | 127.0.0.1 | 12257
Collating all results | 08:26:28,965 | 127.0.0.1 | 12402
Request complete | 08:26:28,965 | 127.0.0.1 | 12638
http://www.datastax.com/dev/blog/tracing-in-cassandra-1-2

Is it possible to tell cassandra to run a query only on the local node

I've got two nodes that are fully replicated. When I run a query on a table that contains 30 rows, cqlsh trace seems to indicate it is fetching some rows from one server and some rows from the other server.
So even though all the rows are available on both nodes, the query takes 250ms+ rather than 1ms for other queries.
I've already got consistency level set to "one" at the protocol level, what else do you have to do to make it only use one node for the query?
select * from organisation:
activity | timestamp | source | source_elapsed
-------------------------------------------------------------------------------------------------+--------------+--------------+----------------
execute_cql3_query | 04:21:03,641 | 10.1.0.84 | 0
Parsing select * from organisation LIMIT 10000; | 04:21:03,641 | 10.1.0.84 | 68
Preparing statement | 04:21:03,641 | 10.1.0.84 | 174
Determining replicas to query | 04:21:03,642 | 10.1.0.84 | 307
Enqueuing request to /10.1.0.85 | 04:21:03,642 | 10.1.0.84 | 1034
Sending message to /10.1.0.85 | 04:21:03,643 | 10.1.0.84 | 1402
Message received from /10.1.0.84 | 04:21:03,644 | 10.1.0.85 | 47
Executing seq scan across 0 sstables for [min(-9223372036854775808), min(-9223372036854775808)] | 04:21:03,644 | 10.1.0.85 | 461
Read 1 live and 0 tombstoned cells | 04:21:03,644 | 10.1.0.85 | 560
Read 1 live and 0 tombstoned cells | 04:21:03,644 | 10.1.0.85 | 611
………..etc….....
It turns out that there was a bug in Cassandra versions 2.0.5-2.0.9 that would make Cassandra more likely to request data on two nodes when it only needed to talk to one.
Upgrading to 2.0.10 or greater resolves this problem.
Refer: CASSANDRA-7535

cassandra, bad performance of time series table

I got a 3x nodes cluster (on the same 16 core box, in virtual box via lxc but each node on a 3TB disk on it's own).
My table is this:
CREATE TABLE history (
id text,
idx bigint,
data bigint,
PRIMARY KEY (id, idx)
) WITH CLUSTERING ORDER BY (idx DESC)
id will store an id which is a string , idx is a time in ms and data are my data. According to all examples I found, this seems to be a correct schema for time series data.
My query is :
select idx,data from history where id=? limit 2
This returns the 2 most recent (based on idx) rows.
Since id is the partition key and idx the clustering key, docs I found claim that this is very performant with cassandra. But my benchmarks say otherwise.
I've populated a 400GB in total (split in those 3 nodes) and now I am running queries from a 2ndary box. Using 16 or 32 threads, I am running the mentioned query but the performance is really low for 3 nodes running on 3 separate disks:
throughput: 61 avg time: 614,808 μs
throughput: 57 avg time: 519,651 μs
throughput: 52 avg time: 569,245 μs
So , ~55 queries per second, each query taking half second (sometimes they do take 200ms)
I find this really low.
Can someone please tell me if my schema is correct and if not suggest a schema? If my schema is correct, how can I find what is going wrong?
Disk IO on the 16core box:
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 0.00 0.00 0.00 0 0
sdb 135.00 6.76 0.00 6 0
sdc 149.00 6.99 0.00 6 0
sdd 124.00 7.21 0.00 7 0
The cassandras don't use more than 1 cpu core each.
EDIT: With tracing on I get a lot of lines like the following when I run a simple query for 1 id:
Key cache hit for sstable 33259 | 20:16:26,699 | 127.0.0.1 | 5830
Seeking to partition beginning in data file | 20:16:26,699 | 127.0.0.1 | 5833
Bloom filter allows skipping sstable 33256 | 20:16:26,699 | 127.0.0.1 | 5923
Bloom filter allows skipping sstable 33255 | 20:16:26,699 | 127.0.0.1 | 5932
Bloom filter allows skipping sstable 33252 | 20:16:26,699 | 127.0.0.1 | 5938
Key cache hit for sstable 33247 | 20:16:26,699 | 127.0.0.1 | 5948
Seeking to partition beginning in data file | 20:16:26,699 | 127.0.0.1 | 5951
Bloom filter allows skipping sstable 33246 | 20:16:26,699 | 127.0.0.1 | 6072
Bloom filter allows skipping sstable 33243 | 20:16:26,699 | 127.0.0.1 | 6081
Key cache hit for sstable 33242 | 20:16:26,699 | 127.0.0.1 | 6092
Seeking to partition beginning in data file | 20:16:26,699 | 127.0.0.1 | 6095
Bloom filter allows skipping sstable 33240 | 20:16:26,699 | 127.0.0.1 | 6187
Key cache hit for sstable 33237 | 20:16:26,699 | 127.0.0.1 | 6198
Seeking to partition beginning in data file | 20:16:26,699 | 127.0.0.1 | 6201
Key cache hit for sstable 33235 | 20:16:26,699 | 127.0.0.1 | 6297
Seeking to partition beginning in data file | 20:16:26,699 | 127.0.0.1 | 6301
Bloom filter allows skipping sstable 33234 | 20:16:26,699 | 127.0.0.1 | 6393
Key cache hit for sstable 33229 | 20:16:26,699 | 127.0.0.1 | 6404
Seeking to partition beginning in data file | 20:16:26,699 | 127.0.0.1 | 6408
Bloom filter allows skipping sstable 33228 | 20:16:26,699 | 127.0.0.1 | 6496
Key cache hit for sstable 33227 | 20:16:26,699 | 127.0.0.1 | 6508
Seeking to partition beginning in data file | 20:16:26,699 | 127.0.0.1 | 6511
Key cache hit for sstable 33226 | 20:16:26,699 | 127.0.0.1 | 6601
Seeking to partition beginning in data file | 20:16:26,699 | 127.0.0.1 | 6605
Key cache hit for sstable 33225 | 20:16:26,700 | 127.0.0.1 | 6692
Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 | 6696
Key cache hit for sstable 33223 | 20:16:26,700 | 127.0.0.1 | 6785
Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 | 6789
Key cache hit for sstable 33221 | 20:16:26,700 | 127.0.0.1 | 6876
Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 | 6880
Bloom filter allows skipping sstable 33219 | 20:16:26,700 | 127.0.0.1 | 6967
Key cache hit for sstable 33377 | 20:16:26,700 | 127.0.0.1 | 6978
Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 | 6981
Key cache hit for sstable 33208 | 20:16:26,700 | 127.0.0.1 | 7071
Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 | 7075
Key cache hit for sstable 33205 | 20:16:26,700 | 127.0.0.1 | 7161
Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 | 7166
Bloom filter allows skipping sstable 33201 | 20:16:26,700 | 127.0.0.1 | 7251
Bloom filter allows skipping sstable 33200 | 20:16:26,700 | 127.0.0.1 | 7260
Key cache hit for sstable 33195 | 20:16:26,700 | 127.0.0.1 | 7276
Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 | 7279
Bloom filter allows skipping sstable 33191 | 20:16:26,700 | 127.0.0.1 | 7363
Key cache hit for sstable 33190 | 20:16:26,700 | 127.0.0.1 | 7374
Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 | 7377
Bloom filter allows skipping sstable 33189 | 20:16:26,700 | 127.0.0.1 | 7463
Key cache hit for sstable 33186 | 20:16:26,700 | 127.0.0.1 | 7474
Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 | 7477
Key cache hit for sstable 33183 | 20:16:26,700 | 127.0.0.1 | 7563
Seeking to partition beginning in data file | 20:16:26,700 | 127.0.0.1 | 7567
Bloom filter allows skipping sstable 33182 | 20:16:26,701 | 127.0.0.1 | 7663
Bloom filter allows skipping sstable 33180 | 20:16:26,701 | 127.0.0.1 | 7672
Bloom filter allows skipping sstable 33178 | 20:16:26,701 | 127.0.0.1 | 7679
Bloom filter allows skipping sstable 33177 | 20:16:26,701 | 127.0.0.1 | 7686
Maybe most important is the end of the trace:
Merging data from memtables and 277 sstables | 20:21:29,186 | 127.0.0.1 | 607001
Read 3 live and 0 tombstoned cells | 20:21:29,186 | 127.0.0.1 | 607205
Request complete | 20:21:29,186 | 127.0.0.1 | 607714
Do look at tracing to confirm, but if sdb,sdc, and sdd are spinning disks, you are seeing the correct order of magnitude of tps, and are very likely random disk I/O bound on the read-side.
If that is the case, then you only have two options (with any system, not specific to Cassandra):
Switch to SSDs. My personal testing has demonstrated up to 3 orders of magnitude increased random read performance when the workload was entirely bound by the tps of the disks.
Ensure that a very large percentage of your reads are cached. If you are doing random reads across 400GB of data, that is probably not going to be feasible.\
Cassandra can do roughly 3k-5K operations (read or write) per CPU core, but only if the disk subsystem isn't the limiting factor.

Resources