dt_corr = dt[["date","mean_press", "maxwind", "sun_dur","mean_rel_hum", "mean_temp", "mean_cloud", "max_temp_2m", ]].dropna()
dt_corr["year"] = dt_corr.date.dt.year
dt_corr.set_index("date")
dt_corr.index = pd.to_datetime(dt_corr.index, format='%Y/%m-%d') #####why appears format green here ????? i don't get it
dt_corr
The index in any output is always like this:
1970-01-01 00:00:00.000033417 ##last for digits differ
I cannot coerce my index to datetime, to finally reasample my data and plot it.
The format parameter doesn't change anything and appears in green in my codeline.
What can I do?
I'm trying to generate a date range of monthly data where the day is always at the beginning of the month:
pd.date_range(start='1/1/1980', end='11/1/1991', freq='M')
This generates 1/31/1980, 2/29/1980, and so on. Instead, I just want 1/1/1980, 2/1/1980,...
I've seen other question ask about generating data that is always on a specific day of the month, with answers saying it wasn't possible, but beginning of month surely must be possible!
You can do this by changing the freq argument from 'M' to 'MS':
d = pandas.date_range(start='1/1/1980', end='11/1/1990', freq='MS')
print(d)
This should now print:
DatetimeIndex(['1980-01-01', '1980-02-01', '1980-03-01', '1980-04-01',
'1980-05-01', '1980-06-01', '1980-07-01', '1980-08-01',
'1980-09-01', '1980-10-01',
...
'1990-02-01', '1990-03-01', '1990-04-01', '1990-05-01',
'1990-06-01', '1990-07-01', '1990-08-01', '1990-09-01',
'1990-10-01', '1990-11-01'],
dtype='datetime64[ns]', length=131, freq='MS', tz=None)
Look into the offset aliases part of the documentation. There it states that 'M' is for the end of the month (month end frequency) while 'MS' for the beginning (month start frequency).
It is worth noting that pandas.date_range() only includes dates within the defined interval, which may not be expected :
start = "2020-03-08"
end = "2021-03-08"
pd.date_range(start, end, freq='MS')
results in
DatetimeIndex(['2020-04-01', '2020-05-01', '2020-06-01', '2020-07-01',
'2020-08-01', '2020-09-01', '2020-10-01', '2020-11-01',
'2020-12-01', '2021-01-01', '2021-02-01', '2021-03-01'],
dtype='datetime64[ns]', freq='MS')
For MS, a workaround to include the first day of the opening month is to work only with the year and month of the start date :
pd.date_range(start[:7], end, freq='MS')
will then give
DatetimeIndex(['2020-03-01', '2020-04-01', '2020-05-01', '2020-06-01',
'2020-07-01', '2020-08-01', '2020-09-01', '2020-10-01',
'2020-11-01', '2020-12-01', '2021-01-01', '2021-02-01',
'2021-03-01'],
dtype='datetime64[ns]', freq='MS')
If you wish to keep the same starting day for each month, you can then add the offset with pd.DateOffset() :
pd.date_range(start[:7], end, freq='MS') + pd.DateOffset(days=7)
will give
DatetimeIndex(['2020-03-08', '2020-04-08', '2020-05-08', '2020-06-08',
'2020-07-08', '2020-08-08', '2020-09-08', '2020-10-08',
'2020-11-08', '2020-12-08', '2021-01-08', '2021-02-08',
'2021-03-08'],
dtype='datetime64[ns]', freq=None)
As mentioned in comments, note that trouble may come with this workaround for offsets higher or equals to 28.
I have a variable called FOUNDATION_DATE which includes the following date observations in string format:
'01/Jan/12'
''
''
''
'01/Jan/08'
''
'01/Jan/44'
''
''
'14/Oct/08'
''
''
'12/Jul/04'
'03/Aug/05'
'20/Apr/10'
'30/Dec/98'
'09/Apr/16'
'01/Jan/10'
'01/Dec/01'
'01/Jan/93'
I am using the Matlab function datetime to transform the above observations in datetime data type. The code is
datetime(FOUNDATION_DATE,'InputFormat','dd/MMM/yy')
which provides the following results:
01-Jan-2012
NaT
NaT
NaT
01-Jan-2008
NaT
01-Jan-2044
NaT
NaT
14-Oct-2008
NaT
NaT
12-Jul-2004
03-Aug-2005
20-Apr-2010
30-Dec-1998
09-Apr-2016
01-Jan-2010
01-Dec-2001
01-Jan-1993
While for the majority of the cases the transformation is conducted properly, for the observation '01/Jan/44' this is not the case as the year becomes 2044. This issue appears in many other date observations of my variable (only a small sample is presented here) and it is quite strange that this issue appears for date observations for years before 1969.
Does anyone have a solution for accurately transforming these strings to datetime variables? Any explanation also why this happens?
You want the 'PivotYear' option, which defines which 100-year date range the 2 digit date refers to:
datetime( '01/Jan/44', 'inputformat', 'dd/MMM/yy', 'pivotyear', 1930 )
So here the 100-year range is 1930 - 2029
The default as documented (therefore not very "strange"), is
year(datetime('now'))-50 % = 1969 at time of writing (2019)
When only 2 years are represented matlab makes an assumption on what the first two digits are, you can override this by:
startYear = year(datetime('now')) - 99;
datetime('01/Jan/69', 'InputFormat', 'dd/MMM/yy', 'PivotYear', startYear)
That will make any dates in 2 digits up until today be historic.
I have a dataframe which contain a column 'trade_dt' like this
2009/12/1
2009/12/2
2009/12/3
2009/12/4
I got this problem
benchmark['trade_dt'] = pd.to_datetime(benchmark['trade_dt'], format='%Y-&m-%d')
ValueError: time data '2009/12/1' does not match format '%Y-&m-%d' (match)
how to solve it? Thanks~
Need change format for match - replace & and - to % and /:
benchmark['trade_dt'] = pd.to_datetime(benchmark['trade_dt'], format='%Y/%m/%d')
Also working with sample data removing format (but not sure with real data):
benchmark['trade_dt'] = pd.to_datetime(benchmark['trade_dt'])
print (benchmark)
trade_dt
0 2009-12-01
1 2009-12-02
2 2009-12-03
3 2009-12-04
There must be a quick solution for this, but after 30min I gave up and need help.
This is the format of source data
0h56m40s 0h57m10s 1h00m40s 1h02m15s 1h02m25s
52m47s 54m25s 54m52s 57m23s 57m43s
49m30s 54m31s 54m34s 56m35s 56m36s
47m45s 48m03s 51m02s 52m23s 53m05s
46m54s 49m29s 50m51s 51m02s 51m03s
46m09s 47m56s 50m16s 51m20s 51m53s
46m55s 47m08s 47m13s 48m16s 50m11s
and I need this in time format like 0h56m40s to 0:56:40
I tried search/replace, from h to :, m to : and removing s, works for when there's hour, but messes up for when only minutes are there.
Any tips?
You can concatenate 0: if the input string is too short:
=(IF(LEN(A1)<7,"0:","") & SUBSTITUTE(SUBSTITUTE(SUBSTITUTE(A1,"h",":"),"m",":"),"s",""))+0
The +0 part converts string to time value (change cell format to h:mm:ss). If you prefer to keep it in text format, remove +0.