I have a script that requests a datapull from a sql server.
For the request I send string like:
start_of_pull="2018-09-13 11:00:00"
The sql script pulls the data using UTC timestamp.
It returns a UTC timestamp of
1536850800.
But when I use pandas' pandas.to_datetime(df['TIME'],unit='s')
it returns at timestamp of
2018-09-13 15:00:00
Any ideas why I'm losing 4 hours?
My solution to the problem is to subtract 14,400 seconds
Related
I have these to ways to convert an Epoch Time to date and time
f.to_date(f.from_unixtime(1625838240))
f.date_format(f.from_unixtime(1625838240), 'HH:mm:ss')
and from_unixtime is based on the local timezone.
How can I force the above commands to always return local time UTC only without UDF?
Change the spark session timezone. See documentation
spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles") <-- set timezone
time_df = spark.createDataFrame([(1428476400,)], ['unix_time'])
time_df.select(from_unixtime('unix_time').alias('ts')).collect()
[Row(ts='2015-04-08 00:00:00')] <-- timezone applied
spark.conf.unset("spark.sql.session.timeZone")
When converting a timestamp between timezones in databricks/spark sql, the timezone itself seems lost in the end result, and I can't seem to either keep it or add it back.
I have a bunch of UTC times and am using the from_utc_timetamp() to convert them to a different timezone based on another field. The result is calculated correctly, but if I output it with a timezone it shows as UTC. It seems the conversion is done correctly but the end result has no timezone stored with it (affirmed by this answer), so it uses the server zone for the timezone in all cases.
Example: Using the following SQL:
createTimestampUTC,
v.timezone,
date_format(from_utc_timestamp(createTimestampUTC, v.timezone),"yyyy-MM-dd'T'HH:mm:s Z") createTimestampLocal,
I get the following:
You can see that the third column has done the conversions correctly for the timezones, but the output itself still shows as being in UTC timezone.
Repeating this with a lowercase z in the date_format function shows the same; namely, the conversions occur but the end result is still treated as UTC.
createTimestampUTC,
v.timezone,
date_format(from_utc_timestamp(createTimestampUTC, v.timezone),"yyyy-MM-dd'T'HH:mm:s z") createTimestampLocal,
I can also use an O in the format output instead of a Z or z, but this just gives me GMT instead of UTC; same output basically.
All the databricks documentation or stackoverflow questions I can find seem to treat printing timezones as a matter of setting the spark server time and outputting that way, or doing the conversion without keeping the resulting timezone. I'm trying to convert to multiple different timezones though, and to keep the timezone in the output. I need to generate the end result in this format:
Is there a way to do this? How do I either keep the timezone after the conversion or add it back in the format I need based on the timezone column I have? Given that the conversion works, and that I can output the end result with a +0000 on it, all the functionality to do this seems there, how do I put it together?
Spark does not support TIMESTAMP WITH TIMEZONE datatype as defined by ANSI SQL. Even though there are some functions that convert the timestamp across timezones, this information is never stored. Databricks documentation on timestamps explains:
Spark SQL defines the timestamp type as TIMESTAMP WITH SESSION TIME
ZONE, which is a combination of the fields (YEAR, MONTH, DAY, HOUR,
MINUTE, SECOND, SESSION TZ) where the YEAR through SECOND field
identify a time instant in the UTC time zone, and where SESSION TZ is
taken from the SQL config spark.sql.session.timeZone.
In your case spark.sql.session.timeZone is UTC and Z symbol in datetime pattern will always return UTC. Therefore you will never get a correct behavior with date_format if you deal with multiple timezones in a single query.
The only thing you can do is to explicitly store timezone information in a column and manually append it for display.
concat(
date_format(from_utc_timestamp(createTimestampUTC, v.timezone), "yyyy-MM-dd'T'HH:mm:s "),
v.timezone
) createTimestampLocal
This will display 2022-03-01T16:47:22.000 America/New_York. If you need an offset (-05:00) you will need to write a UDF to do the conversion and use Python or Scala native libraries that handle datetime conversions.
This question already has answers here:
How to truncate the time on a datetime object?
(18 answers)
Closed 2 years ago.
I'm using Python 3.8. Is there a simpler way to get the current UTC time in which the hour, minute, and second are all zero? For example, if UTC now evalutes to "2020-09-24 12:34:45", I would want my result to be "2020-09-24 00:00:00". Right now I have
today = datetime.datetime.utcnow()
today_wo_time_str = datetime.datetime.strftime(today, '%Y-%m-%d 00:00:00')
today_wo_time_obj = datetime.datetime.stpftime(today_wo_time_str, '%Y-%m-%d 00:00:00')
which seems a little unecessarily wordy.
One option would be get the current UTC datetime, as you are already doing, and then combine its date only component with a midnight time component, using the combine() function.
import datetime
utc_now = datetime.datetime.utcnow()
midnight = datetime.time(0)
utc_midnight = datetime.datetime.combine(utc_now.date(), midnight)
print(utc_midnight)
This prints:
2020-09-24 00:00:00
I have a strange thing occurring and I hope someone can point out what i am missing.
In MongoDB I have a field DT that is of Type Date
An example of what the date looks like in MongoDB is 2014-10-01 10:28:04.329-04:00
When I query MongoDB from Node.js using MongoClient, Node.js is returning this:
2014-10-01T14:28:04.329Z
As i understand it the driver is suppose to convert UTC to local time. In my case it should be Eastern Time (EDT). Why would Node be adding 4 hours instead?
I am loading the date into MongoDB from Java using the Java driver. The variable is set using
new Date();
Node isn't adding 4 hours. Both show exactly the same instant.
2014-10-01 10:28:04.329-04:00
is exactly the same as
2014-10-01T14:28:04.329Z
only one is in a EDT timezone which has -04:00 offset to UTC (so it's four hours earlier there), and the other is in UTC.
Probably you have your server configured in EDT and your client is set to UTC or the other way around.
Unless you need the exact same strings, I wouldn't worry about it.
Or, even better, set both the client and server machine to the same timezone, preferably UTC.
i want to convert UTC date time to local date time by myself and do not want to use .net TimeZoneInfo or other classs about this.
i know Tehran is a GMT offset of +03:30 i use code below to convert UTC Date time to tehran (my local computer is in this location):
DateTime dt = DateTime.UtcNow.AddHours(3.30);
it shows time like 5/2/2014 8:32:05 PM but Tehran time is 5/2/2014 9:32:05 PM it has one Hour deference.
How can i fixed it?
i know Tehran is a GMT offset of +03:30
Well, that's its offset from UTC in standard time, but it's currently observing daylight saving time (details). So the current UTC offset is actually +04:30, hence the difference of an hour.
I suspect you're really off by more than an hour though, are you're adding an offset of 3.3 hours, which is 3 hours and 18 minutes. The literal 3.30 doesn't mean "3 hours and 30 minutes", it means 3.30 as a double literal. If you want 3 hours and 30 minutes, that's 3 and a half hours, so you'd need to use 3.5 instead. The time in Tehran when you posted was 9:46 PM... so I suspect you actually ran the code at 9:44 PM.
This sort of thing is why you should really, really, really use a proper time-zone-aware system rather than trying to code it yourself. Personally I wouldn't use TimeZoneInfo - I'd use my Noda Time library which allows you to either use the Windows time zones via TimeZoneInfo, or the IANA time zone database. The latter - also known as Olsen, or TZDB, or zoneinfo, is the most commonly-used time zone database on non-Windows platforms.