I have encountered this problem recently. When I populated my tables (called event and index) to more than 1 million, and tried to truncate them for new tests, the tables were not empty after the truncation. CQL shows something like
cqlsh> select count(*) from event limit 100000000;
count
---------
2033492
cqlsh> truncate event;
cqlsh> select count(*) from event limit 100000000;
count
-------
25
(1 rows)
cqlsh> select count(*) from event limit 100000000;
count
-------
27
(1 rows)
cqlsh> select count(*) from event limit 100000000;
count
-------
34
(1 rows)
cqlsh> select event_id, dateOf(time_token), unixTimestampOf(time_token), writetime(time_token) from event limit 100000000;
event_id | dateOf(time_token) | unixTimestampOf(time_token) | writetime(time_token)
--------------------------------------+--------------------------+-----------------------------+-----------------------
567c4f2b-c86a-4663-a8ec-50f70d183b62 | 2014-07-22 22:29:04-0400 | 1406082544416 | 1406082544416000
20a2f9e7-cdcb-4c2d-93e7-a646d0910e6b | 2014-07-22 15:12:29-0400 | 1406056349772 | 1406056349774000
... ...
0d983cec-4ba5-4df8-ada8-eb347add57bf | 2014-07-22 22:20:53-0400 | 1406082053926 | 1406082053930000
(34 rows)
cqlsh>
After the "truncate" command, the "select count(*)" returned numbers quickly changing, and stabilized at 34. To be sure there is no other program inserting records at the time, I ran a CQL statement showing all records were created on July 22 or 23, which is 4 - 5 days ago.
I tried "truncate" command several times, and the results were the same.
This happened in 2 environments. The first environment is on my laptop where I created 3 Cassandra instances cluster using localhost IPs (127.0.0.2, 127.0.0.3, and 127.0.0.4), while the second environment is 3 node Cassandra cluster, with each node on a separate Linux CentOS 6.5 machine. I am using Cassandra 2.0.6.
Could someone help me to figure out what is going on? Thanks in advance.
It is a bug in Cassandra 2.0.6, and got fixed in at least 2.0.10.
Apparently, it is not a well-known (well-published) bug as many DataStax experts did not know it either when I reproduced it to them at the Cassandra Summit 2014. They were also puzzled until the CQL architect dropped by and said he fixed a mysterious bug in recent release. He asked me to upgrade to 2.0.10, and the problem is gone. There are no more lingering records after "truncate" in 2.0.10.
Truncate doesn't truncate hints so hints awaiting delivery will still get delivered. This could be causing your issue, especially if you inserted lots of rows quickly that could have caused a few dropped mutations. However, hints are normally delivered in minutes, not days, so there must be something else wrong if hints are causing your issue. You can see when hints are delivered from the logs.
The safest way to delete all data is to drop the table and recreate under a different name (or in a different keyspace).
There is one thing you absolutely have to make sure before truncating is all the nodes are up.
If you are using Astyanax
/* keyspace variable is Keyspace Type */
keyspace.truncateColumnFamily(ColumnFamilyName);
Note: Even after truncating you will have to manually delete all the table metadata
Related
Need help to resolve below issue.
Error: ResponseError: Expected 8 or 0 byte long for date (23)\n
at FrameReader.readError (D:\\EACApp\\eac-app-management\\node_modules\\cassandra-driver\\lib\\readers.js:326:15)\n
DB Query-
SELECT * FROM user_request_by_country_by_processworkflow \
WHERE created_on <= ? (timestamp- 2022-07-26T15:39:52+05:30)
//passing as bind value
cqlsh> show version
[cqlsh 5.0.1 | Cassandra 3.11.2 | CQL spec 3.4.4 | Native protocol v4]
If the created_on column in the sample query you provided is a CQL timestamp type then there is no reason why the query wouldn't work.
I suspect that the underlying problem is that created_on is a different CQL type.
As a friendly note, I suggest you learn how to ask good questions. The general guidance is that you (a) provide a good summary of the problem that includes software/component versions, the full error message + full stack trace; (b) describe what you've tried to fix the problem, details of investigation you've done; and (c) minimal sample code that replicates the problem.
In your case, you need to provide:
the CQL table schema
the driver you're using + version
minimal sample code
If you don't provide sufficient information in your questions, you are less likely to get help from forums. Cheers!
I have a Data model like below,
CREATE TABLE appstat.nodedata (
nodeip text,
timestamp timestamp,
flashmode text,
physicalusage int,
readbw int,
readiops int,
totalcapacity int,
writebw int,
writeiops int,
writelatency int,
PRIMARY KEY (nodeip, timestamp)
) WITH CLUSTERING ORDER BY (timestamp DESC)
where, nodeip - primary key and timestamp - clustering key (Sorted by descinding oder to get the latest),
Sample data in this table,
SELECT * from nodedata WHERE nodeip = '172.30.56.60' LIMIT 2;
nodeip | timestamp | flashmode | physicalusage | readbw | readiops | totalcapacity | writebw | writeiops | writelatency
--------------+---------------------------------+-----------+---------------+--------+----------+---------------+---------+-----------+--------------
172.30.56.60 | 2017-12-08 06:13:07.161000+0000 | yes | 34 | 57 | 19 | 27 | 8 | 89 | 57
172.30.56.60 | 2017-12-08 06:12:07.161000+0000 | yes | 70 | 6 | 43 | 88 | 79 | 83 | 89
This is properly available and whenever I need to get the statistics I am able to get the data using the partition key like below,
(The above logic seems similar to my previous question : Aggregation in Cassandra across partitions) but expectation is different,
I have value for each column (like readbw, latency etc.,) populated for every one minute in all the 4 nodes.
Now, If I need to get the max value for a column (Example : readbw), It is possible using the following query,
SELECT max(readbw) FROM nodedata WHERE nodeip IN ('172.30.56.60','172.30.56.61','172.30.56.60','172.30.56.63') AND timestamp < 1512652272989 AND timestamp > 1512537899000;
1) First question : Is there a way to perform max aggregation on all nodes of a column (readbw) without using IN query?
2) Second question : Is there a way in Cassandra, whenever I insert the data in Node 1, Node 2, Node 3 and Node 4.
It needs to be aggregated and stored in another table. So that I will collect the aggregated value of each column from the aggregated table.
If any of my point is not clear, please let me know.
Thanks,
Harry
If you are dse Cassandra you can enable spark and write the aggregation queries
Disclaimer. In your question you should define restrictions to query speed. Readers do not know whether you're trying to show this in real time, or is it more for analytical purposes. It's also not clear on how much data you're operating and the answers might depend on that.
Firstly decide whether you want to do aggregation on read or write. This largely depends on your read/write patterns.
1) First question: (aggregation on read)
The short answer is no - it's not possible. If you want to use Cassandra for this, the best approach would be doing aggregation in your application by reading each nodeip with timestamp restriction. That would be slow. But Cassandra aggregations are also potentially slow. This warning exists for a reason:
Warnings :
Aggregation query used without partition key
I found C++ Cassandra driver to be the fastest option, if you're into that.
If your data size allows, I'd look into using other databases. Regular old MySQL or Postgres will do the job just fine, unless you have terabytes of data. There's also influx DB if you want a more exotic one. But I'm getting off-topic here.
2) Second question: (aggregation on write)
That's the approach I've been using for a while. Whenever I need some aggregations, I would do them in memory (redis) and then flush to Cassandra. Remember, Cassandra is super efficient at writing data, don't be afraid to create some extra tables for your aggregations. I can't say exactly how to do this for your data, as it all depends on your requirements. It doesn't seem feasible to provide results for arbitrary timestamp intervals when aggregating on write.
Just don't try to put large sets of data into a single partition. You're better of with traditional SQL databases then.
Maybe it is a stupid question, but I'm not able to determine the size of a table in Cassandra.
This is what I tried:
select count(*) from articles;
It works fine if the table is small but once it fills up, I always run into timeout issues:
cqlsh:
OperationTimedOut: errors={}, last_host=127.0.0.1
DBeaver:
Run 1: 225,000 (7477 ms)
Run 2: 233,637 (8265 ms)
Run 3: 216,595 (7269 ms)
I assume that it hits some timeout and just aborts. The actual number of entries in the table is probably much higher.
I'm testing against a local Cassandra instance which is completely idle. I would not mind if it has to do a full table scan and is unresponsive during that time.
Is there a way to reliably count the number of entries in a Cassandra table?
I'm using Cassandra 2.1.13.
Here is my current workaround:
COPY articles TO '/dev/null';
...
3568068 rows exported to 1 files in 2 minutes and 16.606 seconds.
Background: Cassandra supports to
export a table to a text file, for instance:
COPY articles TO '/tmp/data.csv';
Output: 3568068 rows exported to 1 files in 2 minutes and 25.559 seconds
That also matches the number of lines in the generated file:
$ wc -l /tmp/data.csv
3568068
As far as I see you problem connected to timeout of cqlsh: OperationTimedOut: errors={}, last_host=127.0.0.1
you can simple increase it with options:
--connect-timeout=CONNECT_TIMEOUT
Specify the connection timeout in seconds (default: 5
seconds).
--request-timeout=REQUEST_TIMEOUT
Specify the default request timeout in seconds
(default: 10 seconds).
Is there a way to reliably count the number of entries in a Cassandra table?
Plain answer is no. It is not a Cassandra limitation but a hard challenge for distributed systems to count unique items reliably.
That's the challenge that approximation algorithms like HyperLogLog address.
One possible solution is to use counter in Cassandra to count the number of distinct rows but even counters can miscount in some corner cases so you'll get a few % error.
This is a good utility for counting rows that avoids the timeout issues that happen when running a large COUNT(*) in Cassandra:
https://github.com/brianmhess/cassandra-count
The reason is simple:
When you're using:
SELECT count(*) FROM articles;
it has the same effect on the database as:
SELECT * FROM articles;
You have to query over all your nodes. Cassandra simply runs into a timeout.
You can change the timeout, but it isn't a good solution. (For one time it's fine but don't use it in your regular queries.)
There's a better solution: make your client count your rows. You can create a java app where you count your rows, when you inserting them, and insert the result using a counter column in a Cassandra table.
You can use copy to avoid cassandra timeout usually happens on count(*)
use this bash
cqlsh -e "copy keyspace.table_name (first_partition_key_name) to '/dev/null'" | sed -n 5p | sed 's/ .*//'
I have a 6 node cluster and I kept the replication factor as 3 and 3 for the two data centers. I inserted a single row as test and later more rows. As the replication factor is 6 I want to check if the data is written into all nodes. How can I individually check if the data is present in that node. Worst option i got is shutting down the remaining 5 nodes and checking select statement from the 1 node and repeating same on all nodes. Is there any better way to check this? THanks for your time.
You can use nodetool getendpoints for this. Here is a sample table to keep track of Blade Runners.
aploetz#cqlsh:stackoverflow> SELECT * FROm bladerunners;
id | type | datetime | data
--------+--------------+--------------------------+---------------------------------------------
B25881 | Blade Runner | 2015-02-16 18:00:03+0000 | Holden- Fine as long as nobody unplugs him.
B26354 | Blade Runner | 2015-02-16 18:00:03+0000 | Deckard- Filed and monitored.
(2 rows)
Now if I exit back out to my command prompt, I can use nodetool getendpoints, followed by my keyspace, table, and a partition key value. The list of nodes containing data for that key should be displayed underneath:
aploetz#dockingBay94:~$ nodetool getendpoints stackoverflow bladerunners 'B26354'
127.0.0.1
I've been testing out Cassandra to store observations.
All "things" belong to one or more reporting groups:
CREATE TABLE observations (
group_id int,
actual_time timestamp, /* 1 second granularity */
is_something int, /* 0/1 bool */
thing_id int,
data1 text, /* JSON encoded dict/hash */
data2 text, /* JSON encoded dict/hash */
PRIMARY KEY (group_id, actual_time, thing_id)
)
WITH compaction={'class': 'DateTieredCompactionStrategy',
'tombstone_threshold': '.01'}
AND gc_grace_seconds = 3600;
CREATE INDEX something_index ON observations (is_something);
All inserts are done with a TTL, and should expire 36 hours after
"actual_time". Something that is beyond our control is that duplicate
observations are sent to us. Some observations are sent in near real
time, others delayed by hours.
The "something_index" is an experiment to see if we can slice queries
on a boolean property without having to create separate tables, and
seems to work.
"data2" is not currently being written-- it is meant to be written by
a different process than writes "data1", but will be given the same
TTL (based on "actual_time").
Particulars:
Three nodes (EC2 m3.xlarge)
Datastax ami-ada2b6c4 (us-east-1) installed 8/26/2015
Cassandra 2.2.0
Inserts from Python program using "cql" module
(had to enable "thrift" RPC)
Running "nodetool repair -pr" on each node every three hours (staggered).
Inserting between 1 and 4 million rows per hour.
I'm seeing large numbers of data files:
$ ls *Data* | wc -l
42150
$ ls | wc -l
337201
Queries don't return expired entries,
but files older than 36 hours are not going away!
The large number SSTables is probably caused by the frequent repairs you are running. Repair would normally only be run once a day or once a week, so I'm not sure why you are running repair every three hours. If you are worried about short term downtime missing writes, then you could set the hint window to three hours instead of running repair so frequently.
You might have a look at CASSANDRA-9644. This sounds like it is describing your situation. Also CASSANDRA-10253 might be of interest.
I'm not sure why your TTL isn't working to drop old SSTables. Are you setting the TTL on a whole row insert, or individual column updates? If you run sstable2json on a data file, I think you can see the TTL values.
Full disclosure: I have a love/hate relationship with DTCS. I manage a cluster with hundreds of terabytes of data in DTCS, and one of the things it does absolutely horribly is streaming of any kind. For that reason, I've recommended replacing it ( https://issues.apache.org/jira/browse/CASSANDRA-9666 ).
That said, it should mostly just work. However, there are parameters that come into play, such as timestamp_resolution, that can throw things off if set improperly.
Have you checked the sstable timestamps to ensure they match timestamp_resolution (default: microseconds)?