Python 3 Convert ISO 8601 to milliseconds - python-3.x

I'm receiving an ISO 8601 format from an API GET request ("2020-02-25T00:02:43.000Z"). I'm trying to convert it to milliseconds, because that format is required in the payload of the API POST call. I've been successful running the code from a Linux system, but I get ValueError: Invalid format string from Windows.
From Linux:
import dateutil.parser
time = "2020-02-25T00:02:43.000Z"
parsed_time = dateutil.parser.parse(time)
t_in_millisec = parsed_time.strftime('%s%f')
t_in_millisec[:-3]
returns
'1582588963000'
From Windows:
import dateutil.parser
1 time = "2020-02-25T00:02:43.000Z"
2 parsed_time = dateutil.parser.parse(time)
----> 3 t_in_millisec = parsed_time.strftime('%s%f')
ValueError: Invalid format string
Is there a way around this?

Here is the list of what works on windows and indeed the %s is not present.
https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/strftime-wcsftime-strftime-l-wcsftime-l?redirectedfrom=MSDN&view=vs-2019
I always use datetime, if you have the opportunity to use it here is an example :
datetime.datetime(2020,2,25,0,2,43).timestamp()
or
import datetime
time = "2020-02-25T00:02:43.000Z"
date = datetime.datetime.strptime(time, '%Y-%m-%dT%H:%M:%S.%fZ')
timestamp = str((date - datetime.datetime(1970, 1, 1)).total_seconds()*1000)
print(timestamp[:-2])

The reason this doesn't work in Windows is that the strftime function calls the native OS's C library, and Unix ticks (i.e. seconds since midnight on Jan 1, 1970) aren't a part of the Windows operating system.
If you want to get the number of seconds since Jan 1, 1970, then you can simply subtract the original date and get the total seconds from the timedelta. Python makes this easier and provides a timestamp function that does the computation for you (and includes subseconds as a decimal component).
import dateutil.parser
time = "2020-02-25T00:02:43.000Z"
parsed_time = dateutil.parser.parse(time)
timestamp = parsed_time.timestamp() * 1000
return str(int(timestamp))

Related

How to easily create a datetime object from time with unit since epoch?

let's say I got a timestamp since epoch in microseconds 1611590898133828 how could I convert this easily into a datetime object considering the unit microseconds.
from datetime import datetime
timestamp_micro = 1611590898133828
dt = datetime.datetime.fromtimestamp(timestamp_micro / 1e6)
I would like to be able to do easy conversions since sometimes I have microseconds, sometimes seconds, sometimes nanoseconds to convert.
timestamp_micro = 1611590898133828
dt = datetime.datetime.fromtimestamp(timestamp_micro, unit="us")
Is this somehow possible? For me using python's datetime package is just one pain. Maybe you can also recommend another package in which timestamp handling is easier?
pandas.to_datetime provides the option to set the unit as a keyword:
import pandas as pd
t, UNIT = 1611590898133828, 'us'
dt = pd.to_datetime(t, unit=UNIT)
print(dt, repr(dt))
# 2021-01-25 16:08:18.133828 Timestamp('2021-01-25 16:08:18.133828')
You can now work with pandas' timestamps or convert to a regular Python datetime object like
dt.to_pydatetime()
# datetime.datetime(2021, 1, 25, 16, 8, 18, 133828)
Please also note that if you use fromtimestamp without setting a time zone, you'll get naive datetime, which will be treated as local time by Python (UTC offset might not be 0). See e.g. here.
You can create new javascript date objects by simply calling const dt = new Date(timestamp). The timestamp value here is an epoch up to milliseconds precision. JavaScript does have native support for higher precision.
If you constantly need to work with dates, I would recommend you to use a package such as momentJS, since native JS is quite a pain to handle dates/times.

How fix the error "ValueError: Julian Day must be positive" in netCDF4.num2date?

Here is the partial code:
import netCDF4
import pandas as pd
import matplotlib.pyplot as plt
file='/Users/dedeco/Downloads/_grib2netcdf-atls12-95e2cf679cd58ee9b4db4dd119a05a8d-OzkfHp.nc'
nc = netCDF4.Dataset(file)
nc.variables.keys()
lat = nc.variables['latitude'][:]
lon = nc.variables['longitude'][:]
time_var = nc.variables['time']
dtime = netCDF4.num2date(time_var[:],time_var.units)
The file can be download in the link: https://stream.ecmwf.int/data/atls12/data/data01/scratch/84/bc/_grib2netcdf-atls12-95e2cf679cd58ee9b4db4dd119a05a8d-OzkfHp.nc
So, I got this error:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-47-3647c36af24c> in <module>()
2 lon = nc.variables['longitude'][:]
3 time_var = nc.variables['time']
----> 4 dtime = netCDF4.num2date(time_var[:],time_var.units)
cftime/_cftime.pyx in cftime._cftime.num2date()
cftime/_cftime.pyx in cftime._cftime.utime.num2date()
cftime/_cftime.pyx in cftime._cftime.DateFromJulianDay()
ValueError: Julian Day must be positive
How I can fix? Any ideas?
I fixed the problem set the parameter (the default is standard): calendar: describes the calendar used in the time calculations.
So replace this:
dtime = netCDF4.num2date(time_var[:],time_var.units)
by (in this case the year has 365 days):
dtime = netCDF4.num2date(time_var[:],time_var.units,'365_day')
Here is the documentation as follow:
def date2num(
...)
date2num(dates,units,calendar='standard')
Return numeric time values given datetime objects. The units of the
numeric time values are described by the units argument and the
calendar keyword. The datetime objects must be in UTC with no
time-zone offset. If there is a time-zone offset in units, it will be
applied to the returned numeric values.
dates: A datetime object or a sequence of datetime objects. The
datetime objects should not include a time-zone offset.
units: a string of the form since
describing the time units. can be days, hours, minutes,
seconds, milliseconds or microseconds. is the time
origin.
calendar: describes the calendar used in the time calculations. All
the values currently defined in the CF metadata convention Valid
calendars 'standard', 'gregorian', 'proleptic_gregorian' 'noleap',
'365_day', '360_day', 'julian', 'all_leap', '366_day'. Default is
'standard', which is a mixed Julian/Gregorian calendar.
returns a numeric time value, or an array of numeric time values with
approximately millisecond accuracy.
A complementar understand about the conversion can be found here.

python datetime - read a time interval/range from csv formatted as HH:MM-HH:MM

is there a readily-available command in Python's datetime to understand a discrete time range given as HH:MM-HH:MM or HH:MM:ss-HH:MM:ss (e.g. 07:30-12:45)? Such a range would be entered like that in a single cell from a CSV file that the script would access.
Or, might specifying just the start time and then a timedelta value be a better idea?
You can just use split() to separate the two time values, then parse each as a datetime.datetime type and then calculate the timedelta.
Example:
from datetime import datetime
time_string = "07:30-12:45"
separate_times = time_string.split("-")
parsed_times = [datetime.strptime(t, "%H:%M") for t in separate_times]
difference = parsed_times[1] - parsed_times[0]
Calling difference.total_seconds() will return the total seconds between the two times and if you aren't interested in the direction of the difference between the times, you can use abs(difference.total_seconds()).

How to create a datetime object given seconds since unix epoch?

There seems to be a lot of confusion online on doing a very basic thing: create a datetime object with UTC timezone given seconds since unix epoch in the UTC timezone. Basically, I always want to work in absolute time/UTC.
I'm using python 3.5 (the latest right now) and want to simply get a datetime object in the context of UTC (+0/Zulu offset) from a floating point value of elapsed seconds since 1970 Jan 01.
This is wrong since the first time is created in my local timezone, and then I attempt to switch to UTC.
import datetime
import pytz
dt = datetime.datetime.fromtimestamp(my_seconds).replace(tzinfo=pytz.UTC)
Python provided the method utcfromtimestamp just for that case. utcfromtimestamp
import datetime
seconds = 0
utcdate_from_timestamp = datetime.datetime.utcfromtimestamp(seconds)
If my_seconds is a POSIX timestamp then to convert it to datetime in Python 3:
#!/usr/bin/env python3
from datetime import datetime, timedelta, timezone
utc_dt = datetime(1970, 1, 1, tzinfo=timezone.utc) + timedelta(seconds=my_seconds)
utc_dt = datetime.fromtimestamp(my_seconds, timezone.utc)
naive_utc_dt = datetime.utcfromtimestamp(my_seconds)
If your local timezone is "right" (non-POSIX) then only the first formula is correct (the others interpret my_seconds as TAI timestamp with datetime(1970, 1, 1, 0, 0, 10) TAI epoch in this case).
The first formula is more portable and may support a wider input range than the others.
The results of the 1st and 2nd expressions may differ due to rounding errors on some Python versions.
The 2nd and 3rd calls should differ only by tzinfo attibute (the latter returns a naive datetime object (.tzinfo is None)). You should prefer timezone-aware datetime objects, to avoid ambiguity.

convertion of datetime to numpy datetime without timezone info

Suppose I have a datetime variable:
dt = datetime.datetime(2001,1,1,0,0)
and I convert it to numpy as follows numpy.datetime64(dt) I get
numpy.datetime64('2000-12-31T19:00:00.000000-0500')
with dtype('<M8[us]')
But this automatically takes into account my time-zone (i.e. EST in this case) and gives me back a date of 2001-12-31 and a time of 19:00 hours.
How can I convert it to datetime64[D] in numpy that ignores the timezone information and simply gives me
numpy.datetime64('2001-01-01')
with dtype('<M8[D]')
The numpy datetime64 doc page gives no information on how to ignore the time-zone or give the default time-zone as UTC
I was just playing around with this the other day. I think there are 2 issues - how the datetime.datetime object is converted to np.datetime64, and how the later is displayed.
The numpy doc talks about creating a datatime64 object from a date string. It appears that when given a datetime.datetime object, it first produces a string.
np.datetime64(dt) == np.datetime64(dt.isoformat())
I found that I could add timezone info to that string
np.datetime64(dt.isoformat()+'Z') # default assumption
np.datetime64(dt.isoformat()+'-0500')
Numpy 1.7.0 reads ISO 8601 strings w/o TZ as local (ISO specifies this)
Datetimes are always stored based on POSIX time with an epoch of 1970-01-01T00:00Z
As for display, the test_datetime.py file offers some clues as to the undocumented behavior.
https://github.com/numpy/numpy/blob/280f6050d2291e50aeb0716a66d1258ab3276553/numpy/core/tests/test_datetime.py
e.g.:
def test_datetime_array_str(self):
a = np.array(['2011-03-16', '1920-01-01', '2013-05-19'], dtype='M')
assert_equal(str(a), "['2011-03-16' '1920-01-01' '2013-05-19']")
a = np.array(['2011-03-16T13:55Z', '1920-01-01T03:12Z'], dtype='M')
assert_equal(np.array2string(a, separator=', ',
formatter={'datetime': lambda x :
"'%s'" % np.datetime_as_string(x, timezone='UTC')}),
"['2011-03-16T13:55Z', '1920-01-01T03:12Z']")
So you can customize the print behavior of an array with np.array2string, and np.datetime_as_string. np.set_printoptions also takes a formatter parameter.
The pytz module is used to add further timezone handling:
#dec.skipif(not _has_pytz, "The pytz module is not available.")
def test_datetime_as_string_timezone(self):
# timezone='local' vs 'UTC'
a = np.datetime64('2010-03-15T06:30Z', 'm')
assert_equal(np.datetime_as_string(a, timezone='UTC'),
'2010-03-15T06:30Z')
assert_(np.datetime_as_string(a, timezone='local') !=
'2010-03-15T06:30Z')
....
Examples:
In [48]: np.datetime_as_string(np.datetime64(dt),timezone='local')
Out[48]: '2000-12-31T16:00:00.000000-0800'
In [49]: np.datetime64(dt)
Out[49]: numpy.datetime64('2000-12-31T16:00:00.000000-0800')
In [50]: np.datetime_as_string(np.datetime64(dt))
Out[50]: '2001-01-01T00:00:00.000000Z'
In [51]: np.datetime_as_string(np.datetime64(dt),timezone='UTC')
Out[51]: '2001-01-01T00:00:00.000000Z'
In [52]: np.datetime_as_string(np.datetime64(dt),timezone='local')
Out[52]: '2000-12-31T16:00:00.000000-0800'
In [81]: np.datetime_as_string(np.datetime64(dt),timezone=pytz.timezone('US/Eastern'))
Out[81]: '2000-12-31T19:00:00.000000-0500'

Resources