Rearrange Dataframe with Datetime-Index to multiple columns - python-3.x

Question

I have a pandas Dataframe with a Datetime-Index and just one column with a measured value at that time:
Index
Value
2017-01-01 05:00:00
2.8
2017-01-01 05:15:00
3.2
I have data for several years now, one value every 15 minutes. I want to reorganize the df to this (I'm preparing the data to train a Neural Network, each line will be one input):
Index
0 days 05:00:00
0 days 05:00:00
...
1 days 04:45:00
2017-01-01
2.8
3.2
...
1.9
2017-01-02
...
The fastest, most "python" way I could find, was this (with df being the original data, df_result the empty target df):
# prepare df
df_result = pd.DataFrame(index=days_array, columns=times_array)
# fill df
df_result = df_result.apply(order_data_by_days, df=df, log=log, axis=1)
def order_data_by_days(row, df):
for col in row.index:
row[col] = df.loc[row.name + col].values[0]
return row
But this takes >20 seconds for 3.5 years of data! (~120k datapoints). Does anyone have any idea how I could do this a lot faster (I'm aiming at a couple of seconds).
If not, I would try to the the transformation with some other language before the import.

Answer 1

I found a solution, if anyone else has this issue:
Step 1: create target df_result with index (dates, e.g. 2018-01-01, 2018-01-02, ...) as datetime.date and columns (times, e.g. 0 days 05:00:00, 0 days 05:15:00, ..., 1 days 04:45:00) as timedelta.
Step 2: use a for-loop to go through all times. Filter the original df each time using the between_time-function, write the filtered df into the target df_result:
for j in range(0,len(times_array)):
this_time = get_str_from_timedelta(times_array[j], log)
df_this_time = df.between_time(this_time, this_time)
if df_result.empty:
df_result = pd.DataFrame(index=df_this_time.index.date, columns=times_array)
df_this_time.index = df_this_time.index.date
if times_array[j] >= timedelta(days=1):
df_this_time.index = df_this_time.index - timedelta(days=1)
df_result[times_array[j]] = df_this_time[pv]
Note that in my case I checked if the times are actually from next day (timedelta(days=1)), since my "day" starts at 05:00 a.m. and lasts until 04:45 a.m. the next day. To make sure they end up in the same row of df_result (even though, technically, the date-index is wrong here), I use the last if.