Doing a streaming insert into Google BigQuery, from a light Node.js app, using this package: https://www.npmjs.org/package/bigquery
I generated a timestamp on my server via this simple line of code:
jsonData['createdAt'] = new Date().getTime();
I then insert that into BigQuery, into a field with type 'timestamp'. There is no intermediate step (besides the Node package).
But many, although not all, of dates look waaaaaay off. For example:
46343-08-28 05:58:59 UTC
When that should say something like 11:45pm on 05-16-2014. However, some of my createdAt dates are correct, and I can't find a reason for the difference.
Any suggestions?
Without actually debugging the JS code, this seems to be an "off by a thousand" problem.
Check this out:
SELECT USEC_TO_TIMESTAMP(1400341611711851)
2014-05-17 15:46:51 UTC
SELECT USEC_TO_TIMESTAMP(1400341611711851*1000)
46345-01-22 13:01:51 UTC
SELECT MSEC_TO_TIMESTAMP(1400341611711851)
46345-01-22 13:01:51 UTC
SELECT MSEC_TO_TIMESTAMP(1400341611711851/1000)
2014-05-17 15:46:51 UTC
So to get a UNIX timestamp in seconds, divide the new Date().getTime() number by 1000.
I had a requirement where I am supposed to send Timestamp type to BigQuery table from NodeJS.
I did it as follows:
bigqueryClient.timestamp(new Date(Date.now()))
I hope this may help someone.
Use:
bigquery.datetime(new Date().toISOString())
Related
Inside my Docker-Container, which has the timezone Etc/UTC, I need to convert a Date-String which represents a Date in Europe/Berlin-timezone into a UTC timestamp.
So lets say the Europe/Berlin-Date is 2022-04-20T00:00:00.
Now the UTC-Timestamp should be equivalent to 2022-04-19T22:00:00.
But when I do
console.log(new Date("2022-04-20").getTime())
I get 1650412800000 which is equivalent to 2022-04-20T02:00:00 in Europe/Berlin-timezone.
How would I do this?
Edit:
I tried various libs, but still cant get that managed
const { DateTime } = require("luxon")
var f = DateTime.fromISO("2022-04-20").setZone('Europe/Berlin').toUTC()
console.log(f)
also the equivalent stamp in f is 2022-04-20T02:00:00 :/
I need to convert a Date-String which represents a Date in Europe/Berlin-timezone into a UTC timestamp.
Fundamentally, a date-only value cannot be converted into a timestamp, unless you arbitrarily associate a time with that date. It sounds like you meant to use midnight local time (00:00+02:00) but instead you are seeing midnight UTC (00:00Z).
This is due to how you are constructing the Date object. You specify new Date("2022-04-20"), which according to the ECMASCript spec will be treated as midnight UTC. The spec says:
... When the UTC offset representation is absent, date-only forms are interpreted as a UTC time and date-time forms are interpreted as a local time.
Yes, this is inconsistent with ISO 8601, and that has been discussed ad nauseum.
To solve this problem, append T00:00 to your input string, so that you are specifically asking for local time. In other words, new Date("2022-04-20T00:00").
That said, if you need it to not be "local time" but exactly Europe/Berlin, then yes - you'll need to use a library. In luxon, it's like this:
DateTime.fromISO('2022-04-20T00:00', {zone: 'Europe/Berlin'}).toUTC()
I'm using version 'TZDB: 2018c (mapping: 13290)' of the TZD db however when I ask for Pacific/Auckland timezone information using either
VTimeZone.FromTzId(tzdbId);
or
VTimeZone.FromTzId(tzdbId, earliestDateTimeToSupport, true);
If I use 2008-01-01 as earliestDateTimeToSupport I'll get the current daylight saving details and recurrence rules. If however I use any date after that or NO date I'll get UTC -11 (WHEN was that rule last in play??? not while I've been alive.) and the recurrence rule looks all wrong. Can anyone advise if this is the DB or something I'm likely to be doing?
Australia/Sydney seems to behave properly... they've had a timezone change during 2008 and that rule comes through... and I'm not going to test them all but I think this is just a fault with NZDST for dates post (ish) 2009-1-1.
iam filtering some records from my Rethinkdb . the code works just fine when i use the following code . but it works with date only
(new Date(filter.fromdate+' 00:00 GMT+00:00'));
the line above works perfect even after the server is on different region. By using above . It wont skipp any records and time and date just works fine .
but when i use date with time it throws error of rangeError invalid time
I am saving date in rethink in the following format Mon Feb 20 2017 07:25:27 GMT+00:00
SomeHow i just need time with my date to work with the above mentioned code or may be other method can also be used .
Ok, so you need to check if filter.fromdate is with time:
If with time don't add your time and modify it with the proper method of Date , otherwise if is a date without time add your time. And remember you can use momentjs instead of Date to handle dates. Hope help you man.
I have declared a date column in Postgres as date.
When I write the value with node's pg module, the Postgres Tool pgAdmin displays it correctly.
When I read the value back using pg, Instead of plain date, a date-time string comes with wrong day.
e.g.:
Date inserted: 1975-05-11
Date displayed by pgAdmin: 1975-05-11
Date returned by node's pg: 1975-05-10T23:00:00.000Z
Can I prevent node's pg to appy time-zone to date-only data? It is intended for day of birth and ihmo time-zone has no relevance here.
EDIT Issue response from Developer on github
The node-postgres team decided long ago to convert dates and datetimes
without timezones to local time when pulling them out. This is consistent
with some documentation we've dug up in the past. If you root around
through old issues here you'll find the discussions.
The good news is its trivially easy to over-ride this behavior and return
dates however you see fit.
There's documentation on how to do this here:
https://github.com/brianc/node-pg-types
There's probably even a module somewhere that will convert dates from
postgres into whatever timezone you want (utc I'm guessing). And if
there's not...that's a good opportunity to write one & share with everyone!
Original message
Looks like this is an issue in pg-module.
I'm a beginner in JS and node, so this is only my interpretation.
When dates (without time-part) are parsed, local time is assumed.
pg\node_modules\pg-types\lib\textParsers.js
if(!match) {
dateMatcher = /^(\d{1,})-(\d{2})-(\d{2})$/;
match = dateMatcher.test(isoDate);
if(!match) {
return null;
} else {
//it is a date in YYYY-MM-DD format
//add time portion to force js to parse as local time
return new Date(isoDate + ' 00:00:00');
But when the JS date object is converted back to a string getTimezoneOffset is applied.
pg\lib\utils.js s. function dateToString(date)
Another option is change the data type of the column:
You can do this by running the command:
ALTER TABLE table_name
ALTER COLUMN column_name_1 [SET DATA] TYPE new_data_type,
ALTER COLUMN column_name_2 [SET DATA] TYPE new_data_type,
...;
as descripted here.
I'd the same issue, I changed to text.
Just override node-postgres parser for the type date (1082) and return the value without parsing it:
import pg from pg
pg.types.setTypeParser(1082, value => value)
I've got a strange sybase behavior which I do not understand.
Situation
I have a table (MY_TABLE) with several columns of type smalldatetime. For illustration purposes let's assume the following table and data:
MY_TABLE||ID |TS_INIT |TS_LASTCHANGE |MY_TEXT |
||4711|3/31/2013 12:00:00 AM|3/31/2013 3:00:00 AM|someText|
TS_INIT and TS_LASTCHANGE are of type smalldatetime.
When executing the following statement I get the above result:
SELECT ID, TS_INIT, TS_LASTCHANGE MY_TEXT
FROM MY_TABLE
WHERE ID = 4711
go
My client is running in UTC+1 (Berlin) and has daylight savings time (DST) enabled.
I am not sure in what time zone the server is running and whether or not DST is enabled.
Problem
When I execute this (note that it is 03:00h):
SELECT ID, TS_INIT, TS_LASTCHANGE MY_TEXT
FROM MY_TABLE
WHERE ID = 4711 AND TS_LASTCHANGE = "2013-03-31 03:00:00:000"
go
I get NO results but when I execute this (note that it is 02:00h this time):
SELECT ID, TS_INIT, TS_LASTCHANGE MY_TEXT
FROM MY_TABLE
WHERE ID = 4711 AND TS_LASTCHANGE = "2013-03-31 02:00:00:000"
go
I do again get the above result which is saying TS_LASTCHANGE is
3/31/2013 3:00:00 AM
Note that the result prints 03:00h, even though I queried for 02:00h.
Why Is the first query returning no results even though there should be a match and why is the second query returning a result even though there should be no match?!
Note also that 3/31/2013 3:00:00 AM is the first moment in DST (at least in the year 2013) and 3/31/2013 2:00:00 AM should never ever exist at all because when transitioning from winter to summer time, the clock switches from 01:59:59 to 03:00:00 (as per this site).
Database: Adaptive Server Enterprise V15.0.3
Client: Aqua Data Studio V16.0.5
EDIT:
When querying whit the TS_INIT everything works as one would expect (only a result for 3/31/2013 12:00:00 AM)
Aqua Data Studio is written in Java.
The problem you are having has to do with the fact that Java is aware of timezones and databases don't have a concept of timezone when they store date and times. When the time comes back from the database, the database's JDBC driver puts it in a Java date and just assumes the timezone is irrelevant. The problem happens when you try to display a time which the JVM thinks is invalid, so a valid date is presented, which basically pushes the time by an hour. Daylight savings for 2015 started on March 08 2.00 AM and one of your rows contains a date which is invalid according to JVM.
This has been a known design issue with Java, and they are trying to fix this with JSR-310 for inclusion in Java SE 8. With this, they will have LocalDate, OffsetDate and ZonedDate. You can read more about it here ...
https://today.java.net/pub/a/today/2008/09/18/jsr-310-new-java-date-time-api.html#jsr-310-datetime-concepts
https://jcp.org/en/jsr/detail?id=310
http://docs.google.com/View?id=dfn5297z_8d27fnf
Workaround
The only workaround, is to probably trick the JVM by setting the timezone in the JVM to GMT. If you are running ADS 16 on Windows, and you launch ADS with the shortcut icon on the desktop (which runs datastudio.exe), then you need to modify the datastudio.ini file in your folder. Add a new entry for vmarg.5=-Duser.timezone=gmt
This link explains the location of where to find the data studio.ini
https://www.aquaclusters.com/app/home/project/public/aquadatastudio/wikibook/Documentation14/page/50/Launcher-Memory-Configuration#windows
Once you have made the change, Restart ADS. Then go to Help->About->System: and double check your user.timezone setting and make sure it is GMT. Then give it a try.
With the above change there might be side effects in the application where timezone are involved, For e.g. in the Table Data Editor->Insert Current Date&Time, which would display a GMT time ... so there would be an offset.