I need to split my timestamp to date and time separately and insert then to db columns with 'date' and 'time' cqltypes.
I was trying to insert a time value as string to Cassandra table. The time was converted to UTC (05:27:00). But while I checked table using Datastax devcenter, column was populated with value '09:37:54.935541808'. I tried to retrieve the value in spring using repository, then it was returning value as '3473746674935541808'.
How to get the correct value from table for time?
It looks like the limitation of Spring-data. In Cassandra time value is encoded as a 64-bit signed integer representing the number of nanoseconds since midnight. But I don't see the time type listed as supported in spring-data-cassandra documentation, so you may need to write your custom converter for it, as described in documentation.
Related
I'm making a small employee management system in which I have to play with dates in many places like attendance, leaves etc.
I use luxon for dates in my node.js project. Ex: DateTime.fromISO("2020-10-15"); gives me this in the console: 2020-10-15T00:00:00.000+05:00
but when I save it in mongodb it becomes 2020-10-14T19:00:00.000+00:00 and this is problematic because i need to query objects to check if there is an attendance marked in the same day or to get today's attendance. How can I solve this issue?
Storage
When MongoDB stores time-type data, it is first converted to UTC time and then stored in the database. The date submitted by the client is East 5, and the conversion to UTC time is minus 5 hours;
Read
MongoDB will convert the UTC date stored in the database into the East 5 zone of the client according to the time zone of the client, and then return it to the client
MongoDB only has a timestamp type, it does not have a date type. Dates are converted to times when they are stored. Which timezone the date is interpreted to be in can vary, looks like your driver uses the beginning of specified date in UTC. Ruby works the same way and you can read more about the principle here.
To "solve this issue", either only store and query by timestamps in MongoDB, converting dates to timestamps in your application, or understand how your language and driver converts dates to timestamps (note that JS Date type is actually a timestamp).
I have the values stored for timestamp type column in cassandra table in format of
2018-10-27 11:36:37.950000+0000 (GMT date).
I get Unable to coerce '2018-10-27 11:36:37.950000+0000' to a formatted date (long) when I run below query to get data.
select create_date from test_table where create_date='2018-10-27 11:36:37.950000+0000' allow filtering;
How to get the query working if the data is already stored in the table (of format, 2018-10-27 11:36:37.950000+0000) and also perform range (>= or <=) operations on create_date column?
I tried with create_date='2018-10-27 11:36:37.95Z',
create_date='2018-10-27 11:36:37.95' create_date='2018-10-27 11:36:37.95'too.
Is it possible to perform filtering on this kind of timestamp type data?
P.S. Using cqlsh to run query on cassandra table.
In first case, the problem is that you specify timestamp with microseconds, while Cassandra operates with milliseconds - try to remove the three last digits - .950 instead of .950000 (see this document for details). The timestamps are stored inside Cassandra as 64-bit number, and then formatted when printing results using the format specified by datetimeformat options of cqlshrc (see doc). Dates without explicit timezone will require that default timezone is specified in cqlshrc.
Regarding your question about filtering the data - this query will work only for small amounts of data, and on bigger data sizes will most probably timeout, as it will need to scan all data in the cluster. Also, the data won't be sorted correctly, because sorting happens only inside single partition.
If you want to perform such queries, then maybe the Spark Cassandra Connector will be the better choice, as it can effectively select required data, and then you can perform sorting, etc. Although this will require much more resources.
I recommend to take DS220 course from DataStax Academy to understand how to model data for Cassandra.
This is works for me
var datetime = DateTime.UtcNow.ToString("yyyy-MM-dd HH:MM:ss");
var query = $"SET updatedat = '{datetime}' WHERE ...
I have json logs with timestamp(UTC TIME) in it. I map keys and values to Cassandra Table keys and Insert the record. However, Cassandra converts the already UTC timestamps to UTC again by subtracting 5 hours from the timestamp. The timezone here is (GMT + 5).
cqlsh> INSERT INTO myTable (id,time) VAlUES (abc123, 2018-01-12T12:32:31);
Now the time is already UTC time and its still inserts a timestamp of 5 hours ago.
How can I resolve this?
If you're using cqlsh to insert data, then you can specify default timezone in the cqlshrc file using the timezone parameter (see default cqlshrc as example).
If you insert dates programmatically, then you need to convert your time into corresponding type matching to the Cassandra's timestamp type (java.util.Date for Java, for example). In your case change could be simple - just append Z to timestamp string as pointed by Ralf
I am new in Cassandra. I have a Cassandra( V: 3.11 ) table (data). It is having a column timeStampCol of timestamp type and I am inserting a value in it.
insert into data (timeStampCol) values('2017-05-02 17:33:03');
While accessing the data from table
select * from data;
I got result like -
# Row 1
----------+------------------------------------
timeStampCol | 2017-05-02 08:33:03.000000+0000
Inserted value and retrieved values are different for time.
Reason might be timezone, how can I get it correct ?
Your selected timestamp value is correct, it's just showing in different timezone.
If you insert data into timestamp column without providing timezone like this one :
insert into data (timeStampCol) values('2017-05-02 17:33:03');
Cassandra will choose coordinator timezone
If no time zone is specified, the time zone of the Cassandra coordinator node handing the write request is used. For accuracy, DataStax recommends specifying the time zone rather than relying on the time zone configured on the Cassandra nodes.
You Need To Convert the String date into java.util.Date and set the timezone of coordinator node, In my case it was GMT+6
DateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd");
Date date = dateFormat.parse("2012-01-21");
dateFormat.setTimeZone(TimeZone.getTimeZone("GMT+6")); //Change this time zone
Source : https://docs.datastax.com/en/cql/3.0/cql/cql_reference/timestamp_type_r.html
Cassandra will assume incoming data in the timezone it is set up. For example if you have Cassandra set up in IST, and even though incoming data is UTC, Cassandra will convert it back to UTC, considering data to be in IST.
You might have to set Cassandra coordinator timezone in code or calculate the time difference between the incoming data timezone and Cassandra timezone and add/subtract that from incoming data before it is written to Cassandra. This way you will have the exact timestamps written to Cassandra.
I've written a program that reads a file containing the date (in yyyy/MM/dd format) and uses the Datastax Java Driver to read the date and add it to a cassandra table.
So for instance, if my record contains a date value of '2010/06/01', then this date value gets converted into a date object (using the SimpleDateFormat class).
However, when I view the data (containing the date) in the database, I see that the date (which in the cassandra table is a timestamp type) shows the following:
2010-06-01 00:00:00+0100
The issue here is that I don't want the timestamp to have "+0100" (to indicate that this is british summer time), rather I'd want to store the date just as "2010-06-01 00:00:00+0000".
I've done the following to my program to try and 'ignore' the timezone by doing the following:
SimpleTimeZone tz = new SimpleTimeZone(0, "Out Timezone");
TimeZone.setDefault(tz);
String dateStringFromFile = "2010/06/01";
SimpleDateFormat sdf = new SimpleDateFormat("yyyy/MM/dd");
Date theDate = sdf.parse(dateStringFromFile);
...Now when I add debug statements to my program, I can see that the date shows "2010-06-01 00:00:00+0000" on my log file (this is right for me). However when i see the date stored in Cassandra, i still see that the date shows as
"2010-06-01 00:00:00+0100" and not "2010-06-01 00:00:00+0000".
Is there anything on the cassandra side that I would have to change or update to ignore the timezone (i.e. not put +0100 on the date and to put +0000), so that the timestamp shows as "2010-06-01 00:00:00+0000"?
Please note that I am running Cassandra 3.0.5 on a Docker VM (Centos linux), Java 8.
Any advice is appreciated.
Thanks.
This solution has been sorted out. Nothing to do with cqlshrc. Its to do with forcing the timezone as 0 (like the code in the original post) and getting the time in milliseconds and writing that to the database - (which is a timestamp type column)