There seems to be a lot of confusion online on doing a very basic thing: create a datetime object with UTC timezone given seconds since unix epoch in the UTC timezone. Basically, I always want to work in absolute time/UTC.
I'm using python 3.5 (the latest right now) and want to simply get a datetime object in the context of UTC (+0/Zulu offset) from a floating point value of elapsed seconds since 1970 Jan 01.
This is wrong since the first time is created in my local timezone, and then I attempt to switch to UTC.
import datetime
import pytz
dt = datetime.datetime.fromtimestamp(my_seconds).replace(tzinfo=pytz.UTC)
Python provided the method utcfromtimestamp just for that case. utcfromtimestamp
import datetime
seconds = 0
utcdate_from_timestamp = datetime.datetime.utcfromtimestamp(seconds)
If my_seconds is a POSIX timestamp then to convert it to datetime in Python 3:
#!/usr/bin/env python3
from datetime import datetime, timedelta, timezone
utc_dt = datetime(1970, 1, 1, tzinfo=timezone.utc) + timedelta(seconds=my_seconds)
utc_dt = datetime.fromtimestamp(my_seconds, timezone.utc)
naive_utc_dt = datetime.utcfromtimestamp(my_seconds)
If your local timezone is "right" (non-POSIX) then only the first formula is correct (the others interpret my_seconds as TAI timestamp with datetime(1970, 1, 1, 0, 0, 10) TAI epoch in this case).
The first formula is more portable and may support a wider input range than the others.
The results of the 1st and 2nd expressions may differ due to rounding errors on some Python versions.
The 2nd and 3rd calls should differ only by tzinfo attibute (the latter returns a naive datetime object (.tzinfo is None)). You should prefer timezone-aware datetime objects, to avoid ambiguity.
Related
let's say I got a timestamp since epoch in microseconds 1611590898133828 how could I convert this easily into a datetime object considering the unit microseconds.
from datetime import datetime
timestamp_micro = 1611590898133828
dt = datetime.datetime.fromtimestamp(timestamp_micro / 1e6)
I would like to be able to do easy conversions since sometimes I have microseconds, sometimes seconds, sometimes nanoseconds to convert.
timestamp_micro = 1611590898133828
dt = datetime.datetime.fromtimestamp(timestamp_micro, unit="us")
Is this somehow possible? For me using python's datetime package is just one pain. Maybe you can also recommend another package in which timestamp handling is easier?
pandas.to_datetime provides the option to set the unit as a keyword:
import pandas as pd
t, UNIT = 1611590898133828, 'us'
dt = pd.to_datetime(t, unit=UNIT)
print(dt, repr(dt))
# 2021-01-25 16:08:18.133828 Timestamp('2021-01-25 16:08:18.133828')
You can now work with pandas' timestamps or convert to a regular Python datetime object like
dt.to_pydatetime()
# datetime.datetime(2021, 1, 25, 16, 8, 18, 133828)
Please also note that if you use fromtimestamp without setting a time zone, you'll get naive datetime, which will be treated as local time by Python (UTC offset might not be 0). See e.g. here.
You can create new javascript date objects by simply calling const dt = new Date(timestamp). The timestamp value here is an epoch up to milliseconds precision. JavaScript does have native support for higher precision.
If you constantly need to work with dates, I would recommend you to use a package such as momentJS, since native JS is quite a pain to handle dates/times.
I have a string in this format: 09/20/2020 10:30 AM
That timestamp is in Eastern Timezone.
I need to obtain the UTC equivalent, but in the following ISO format: 2020-09-20T14:30:00.0000Z
I've tried a few things but there doesn't seem to be a clean/short way of converting it.
So far I've tried:
dtSept = "09/20/2020 10:00 PM"
dtSeptTZ = pytz.timezone('US/Eastern').localize(datetime.datetime.strptime(dtSept, "%m/%d/%Y %I:%M %p")).isoformat(timespec='milliseconds')
dtSeptTZ at this point is a string object.
If I have to convert its TimeZone and format it, I have to execute the following which each take a datetime object but return a string.
dtSeptTZ.astimezone(pytz.timezone('Etc/UTC'))
dtSeptTZ.strftime("%Y-%m-%dT%I:%M.%fZ")
Is there a clean/short way of getting the proper output without coverting it back and forth between string and datetime?
Many thanks.
due to the immanent deprecation of pytz, I'd suggest to use dateutil. The usage of dateutil also transforms nicely to Python 3.9's zoneinfo.
from datetime import datetime, timezone
from dateutil.tz import gettz
dtSept = "09/20/2020 10:00 PM"
# string to datetime object
dt = datetime.strptime(dtSept, "%m/%d/%Y %I:%M %p")
# set tzinfo to appropriate time zone; (!) use "localize" instead with pytz timezone class
dt = dt.replace(tzinfo=gettz('US/Eastern'))
# to UTC (could use any other tz here)
dt_utc = dt.astimezone(timezone.utc)
# to ISO format string:
print(dt_utc.isoformat())
>>> 2020-09-21T02:00:00+00:00
I'm receiving an ISO 8601 format from an API GET request ("2020-02-25T00:02:43.000Z"). I'm trying to convert it to milliseconds, because that format is required in the payload of the API POST call. I've been successful running the code from a Linux system, but I get ValueError: Invalid format string from Windows.
From Linux:
import dateutil.parser
time = "2020-02-25T00:02:43.000Z"
parsed_time = dateutil.parser.parse(time)
t_in_millisec = parsed_time.strftime('%s%f')
t_in_millisec[:-3]
returns
'1582588963000'
From Windows:
import dateutil.parser
1 time = "2020-02-25T00:02:43.000Z"
2 parsed_time = dateutil.parser.parse(time)
----> 3 t_in_millisec = parsed_time.strftime('%s%f')
ValueError: Invalid format string
Is there a way around this?
Here is the list of what works on windows and indeed the %s is not present.
https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/strftime-wcsftime-strftime-l-wcsftime-l?redirectedfrom=MSDN&view=vs-2019
I always use datetime, if you have the opportunity to use it here is an example :
datetime.datetime(2020,2,25,0,2,43).timestamp()
or
import datetime
time = "2020-02-25T00:02:43.000Z"
date = datetime.datetime.strptime(time, '%Y-%m-%dT%H:%M:%S.%fZ')
timestamp = str((date - datetime.datetime(1970, 1, 1)).total_seconds()*1000)
print(timestamp[:-2])
The reason this doesn't work in Windows is that the strftime function calls the native OS's C library, and Unix ticks (i.e. seconds since midnight on Jan 1, 1970) aren't a part of the Windows operating system.
If you want to get the number of seconds since Jan 1, 1970, then you can simply subtract the original date and get the total seconds from the timedelta. Python makes this easier and provides a timestamp function that does the computation for you (and includes subseconds as a decimal component).
import dateutil.parser
time = "2020-02-25T00:02:43.000Z"
parsed_time = dateutil.parser.parse(time)
timestamp = parsed_time.timestamp() * 1000
return str(int(timestamp))
I have been struggling with this issue for a while now. I feel that I understand Timezones well, and I think I understand how to use pytz properly so I usually use that and I typically don't have any problems. Maybe I am trying to use the wrong tools for what I want.
With my current application, I need to work in abstract time objects. That is, I care about something that happens at 16:00. I don't care that it happens at 16:00 ON December 23rd. It seems that pytz prefers working with datetime objects: and that makes sense. It can't figure out the offset since it's different based on the day for daylight savings and historical reasons and so on.
I'm trying to allow users to coordinate daily events all around the world. If a user in Japan says the event is from 21:00 to 23:00 every day, I want users in US/Central to see 6:00 to 8:00 every day. And I thought I had this working... until two weeks ago that is. See, DST just ended in most of the United States, so what now is 6:00-8:00 was actually previously 7:00-9:00.
This breaks my idea that I need to typically store times in UTC, then convert them only for viewing. The timezone it was created in is actually very important. If we reverse this, and a user in a US time zone that observes Daylight Savings sets an event time, that time needs to change in Japan even though they don't observe it! If I store that time as UTC, nothing changes in Japan, but that is not the functionality I want.
So I would like to store as a time object with tzinfo. But you can't create a time object with an accurate pytz tzinfo without a date, but the date isn't important. If I use the current date to figure out the tzinfo then that will actually cease to be accurate once that time zone changes.
I guess my question is: what the best way to store "4 PM Eastern" in a way that can be retrieved anywhere in the world, at any time in the future? That includes Eastern! I want it to be 4PM during DST and also outside of DST. I can't store it as UTC, because 12:00 UTC is the same time as 12:00 UTC during the entire year, but I don't want that. What I think I want is, an "abstract" or "temporary" pytz.timezone that doesn't have an actual offset until the date (the date of viewing) is given. Is that a thing? I've read countless questions on this site, both the python and pytz docs, and can't find anything like this, or anyone with a similar problem. Everything seems to talk about specific datetimes or working only within datetimes but that doesn't seem relevant to my issue.
My application is pretty huge so it's hard to pull out the specific parts, but I can try to show what I've tried and why things don't work.
event_time = datetime.time(hour=12, tzinfo=pytz.timezone("US/Eastern")) would be my ideal solution. But using pytz for tzinfo creation is not a good idea (this will give me an offset like -5:04 because of historical reasons) - is there a way to specify the version of US/Eastern to use?
datetime.now(pytz.timezone("US/Eastern")).replace(hour=12, minute=0, second=0, microsecond=0).timetz() gives me something that looks like what I want, but it only functions correctly as long as US/Eastern doesn't change. If I apply this to a date that I have moved back before the DST change, it gives me 13:00, which is not what I want.
I built a tzinfo class that automatically handles the conversion of DST. I got the idea from the USTimeZone class example in the datetime documentation.
The trick here is the pytz timezone database has all the historical dates of when daylight savings goes into effect. That's also why when you create a datetime object without a date, it converts the DST incorrectly; it's going based off the first entry in the database.
from datetime import datetime, tzinfo, timedelta
import pytz
ZERO = timedelta(0)
def format_timedelta(td):
if td < timedelta(0):
return '-' + format_timedelta(-td)
else:
# Change this to format positive timedeltas the way you want
return str(td)
class WorldTimeZone(tzinfo):
"""
A self adjusting according to DST rules in the PYTZ database tzinfo class
See pytz.all_timezones for a list of all zone names and offsets.
"""
def __init__(self, zone):
"""
:param zone: (str) Proper tzdatabase timze zone name.
"""
# initialize the pytz timezone with current time
# this is done to avoid confusing tznames found in the start of the tz database
# _utcoffset should always be STD rather than DST.
self.pytzinfo = self.__getSTD(zone)
self._utcoffset = self.pytzinfo._utcoffset
#staticmethod
def __getSTD(tname):
"""
This returns a pytz timezone object normalized to standard time for the zone requested.
If the zone does not follow DST or a future transition time cannot be found, it normalizes to NOW instead.
:param tname: Proper timezone name found in the tzdatabase. example: "US/Central"
"""
# This defaults to the STD time for the zone rather than current time which could be DST
tzone = pytz.timezone(tname)
NOW = datetime.now(tz=pytz.UTC)
std_date = NOW
hasdst = False
try:
#transitions are in UTC. They need to be converted to localtime once we find the correct STD transition.
for utcdate, info in zip(tzone._utc_transition_times, tzone._transition_info):
utcdate = utcdate.replace(tzinfo=pytz.UTC)
utcoffset, dstoffset, tzname = info
if dstoffset == ZERO:
std_date = utcdate
if utcdate > NOW:
hasdst = True
break
except AttributeError:
std_date = NOW
if not hasdst:
std_date = NOW
std_date = tzone.normalize(std_date)
return std_date.tzinfo
# This needs to be dynamic because pytzinfo updates everytime .dst() is called; which is a lot.
#property
def _dst(self):
return self.pytzinfo._dst
def __repr__(self):
# return self.pytzinfo.__repr__()
if self._dst:
dst = 'DST'
else:
dst = 'STD'
if self._utcoffset > timedelta(seconds=0):
msg = '<WorldTimeZone %r %s+%s %s>'
else:
msg = '<WorldTimeZone %r %s%s %s>'
return msg % (self.pytzinfo.zone, self.pytzinfo._tzname,
format_timedelta(self._utcoffset + self._dst), dst)
def __str__(self):
return "%s %s" % (self.pytzinfo._tzname, self.pytzinfo)
def tzname(self, dt):
# print " TZNAME called"
return "%s %s" % (self.pytzinfo._tzname, self.pytzinfo)
def utcoffset(self, dt):
# print " UTCOFFSET CALLED"
return self._utcoffset + self.dst(dt)
def dst(self, dt):
# print " DST CALLED"
if dt is None or dt.tzinfo is None:
# An exception may be sensible here, in one or both cases.
# It depends on how you want to treat them. The default
# fromutc() implementation (called by the default astimezone()
# implementation) passes a datetime with dt.tzinfo is self.
return ZERO
assert dt.tzinfo is self # WE ASSUME THE TZINFO ON THE DATE PASSED IN IS OUR TZINFO OBJECT.
tmpdt = self.pytzinfo.normalize(dt)
self.pytzinfo = tmpdt.tzinfo
return tmpdt.tzinfo._dst
Sample code
EST = WorldTimeZone('US/Eastern')
PST = WorldTimeZone('US/Pacific')
dt_dst = datetime(2018, 11, 1, 1, 30, 00)
dt_std = datetime(2018, 11, 6, 1, 30, 00)
est_dst = dt_dst.replace(tzinfo=EST)
est_std = dt_std.replace(tzinfo=EST)
pst_dst = est_dst.astimezone(PST)
pst_std = est_std.astimezone(PST)
print(f"{dt_dst} >> {est_dst} >> {pst_dst}")
print(f"{dt_std} >> {est_std} >> {pst_std}")
outputs
2018-11-01 01:30:00 >> 2018-11-01 01:30:00-04:00 >> 2018-10-31 22:30:00-07:00
2018-11-06 01:30:00 >> 2018-11-06 01:30:00-05:00 >> 2018-11-05 22:30:00-08:00
It looks like pytz provides this through the 'localize' method, as suggested in `this answer.
From the pytz documentation:
If you insist on working with local times, this library provides a facility for constructing them unambiguously:
eastern = timezone('US/Eastern')
loc_dt = datetime(2002, 10, 27, 1, 30, 00)
est_dt = eastern.localize(loc_dt, is_dst=True)
edt_dt = eastern.localize(loc_dt, is_dst=False)
print(est_dt.strftime(fmt) + ' / ' + edt_dt.strftime(fmt))
2002-10-27 01:30:00 EDT-0400 / 2002-10-27 01:30:00 EST-0500`
By storing the local time, and the local time zone, you can then convert it when you use it, and convert it to another region's time zone.
Suppose I have a datetime variable:
dt = datetime.datetime(2001,1,1,0,0)
and I convert it to numpy as follows numpy.datetime64(dt) I get
numpy.datetime64('2000-12-31T19:00:00.000000-0500')
with dtype('<M8[us]')
But this automatically takes into account my time-zone (i.e. EST in this case) and gives me back a date of 2001-12-31 and a time of 19:00 hours.
How can I convert it to datetime64[D] in numpy that ignores the timezone information and simply gives me
numpy.datetime64('2001-01-01')
with dtype('<M8[D]')
The numpy datetime64 doc page gives no information on how to ignore the time-zone or give the default time-zone as UTC
I was just playing around with this the other day. I think there are 2 issues - how the datetime.datetime object is converted to np.datetime64, and how the later is displayed.
The numpy doc talks about creating a datatime64 object from a date string. It appears that when given a datetime.datetime object, it first produces a string.
np.datetime64(dt) == np.datetime64(dt.isoformat())
I found that I could add timezone info to that string
np.datetime64(dt.isoformat()+'Z') # default assumption
np.datetime64(dt.isoformat()+'-0500')
Numpy 1.7.0 reads ISO 8601 strings w/o TZ as local (ISO specifies this)
Datetimes are always stored based on POSIX time with an epoch of 1970-01-01T00:00Z
As for display, the test_datetime.py file offers some clues as to the undocumented behavior.
https://github.com/numpy/numpy/blob/280f6050d2291e50aeb0716a66d1258ab3276553/numpy/core/tests/test_datetime.py
e.g.:
def test_datetime_array_str(self):
a = np.array(['2011-03-16', '1920-01-01', '2013-05-19'], dtype='M')
assert_equal(str(a), "['2011-03-16' '1920-01-01' '2013-05-19']")
a = np.array(['2011-03-16T13:55Z', '1920-01-01T03:12Z'], dtype='M')
assert_equal(np.array2string(a, separator=', ',
formatter={'datetime': lambda x :
"'%s'" % np.datetime_as_string(x, timezone='UTC')}),
"['2011-03-16T13:55Z', '1920-01-01T03:12Z']")
So you can customize the print behavior of an array with np.array2string, and np.datetime_as_string. np.set_printoptions also takes a formatter parameter.
The pytz module is used to add further timezone handling:
#dec.skipif(not _has_pytz, "The pytz module is not available.")
def test_datetime_as_string_timezone(self):
# timezone='local' vs 'UTC'
a = np.datetime64('2010-03-15T06:30Z', 'm')
assert_equal(np.datetime_as_string(a, timezone='UTC'),
'2010-03-15T06:30Z')
assert_(np.datetime_as_string(a, timezone='local') !=
'2010-03-15T06:30Z')
....
Examples:
In [48]: np.datetime_as_string(np.datetime64(dt),timezone='local')
Out[48]: '2000-12-31T16:00:00.000000-0800'
In [49]: np.datetime64(dt)
Out[49]: numpy.datetime64('2000-12-31T16:00:00.000000-0800')
In [50]: np.datetime_as_string(np.datetime64(dt))
Out[50]: '2001-01-01T00:00:00.000000Z'
In [51]: np.datetime_as_string(np.datetime64(dt),timezone='UTC')
Out[51]: '2001-01-01T00:00:00.000000Z'
In [52]: np.datetime_as_string(np.datetime64(dt),timezone='local')
Out[52]: '2000-12-31T16:00:00.000000-0800'
In [81]: np.datetime_as_string(np.datetime64(dt),timezone=pytz.timezone('US/Eastern'))
Out[81]: '2000-12-31T19:00:00.000000-0500'