How to send many pickled file into a dataframe? - python-3.x

Question

I have many files which have been created using "pickle".
I want to send them to a dataframe, calculate the average (from the 2nd row until the end) of each one, multiply it by 1000 and round it to 2 decimals.
So far I have achieved this using 1 pickle file.
import pandas as pd
df = pd.read_pickle(r'C:\Users\file_inference_time')
df = pd.DataFrame(df)
df.rename(columns={0:'MobileNet'},inplace=True)
df_mean=(df.iloc[2::,:].mean()* 1000).round(decimals=2)
df_mean2=pd.DataFrame(df_mean)
df_mean2
Result I get from 1 file.
These are the files ("pickle") that I need to read
EDIT
This is the error that I get when running the 2nd option
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-2-b72e45d8bcfc> in <module>
16
17
---> 18 df_mean_all = pd.concat(df_mean_list).reset_index(drop=True)
19
20 print(df_mean_all)
~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\reshape\concat.py in concat(objs, axis, join, join_axes, ignore_index, keys, levels, names, verify_integrity, sort, copy)
253 verify_integrity=verify_integrity,
254 copy=copy,
--> 255 sort=sort,
256 )
257
~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\reshape\concat.py in __init__(self, objs, axis, join, join_axes, keys, levels, names, ignore_index, verify_integrity, copy, sort)
302
303 if len(objs) == 0:
--> 304 raise ValueError("No objects to concatenate")
305
306 if keys is None:
ValueError: No objects to concatenate
THIS IS A PLOT WITH THE RESULTS

Answer 1

Get a dict of dataframes
Save the calculated mean result for each file, into a dict
from pathlib import Path
dir_path = Path(r'C:\Users\path_to_files')
files = dir_path.glob('**/file_inference_time*') # get all pkl files in main dir and subdirectories
df_mean_dict = dict()
for i, file in enumerate(files):
df = pd.DataFrame(pd.read_pickle(file))
df.rename(columns={0:'MobileNet'}, inplace=True)
df_mean_dict[i] = pd.DataFrame((df.iloc[2::,:].mean()* 1000).round(decimals=2))
# if all the file names are unique, the dict key can be the file name (w/o the file extension)
# df_mean_dict[file.stem] = pd.DataFrame((df.iloc[2::,:].mean()* 1000).round(decimals=2))
Get a single dataframe - This is what I would do
The result df_mean_all will be a single, 2-column dataframe.
column 0 will be MobileNet
column 1 will be file
dir_path = Path(r'C:\Users\path_to_files')
files = dir_path.glob('**/file_inference_time*') # get all pkl files in main dir and subdirectories
# to check if the files are found
# if an empty list prints, no files are found
files = list(files)
print(files[:5]
df_mean_list = list()
for file in files:
df = pd.DataFrame(pd.read_pickle(file))
df_mean = pd.DataFrame((df.iloc[2::,:].mean()* 1000).round(decimals=2)).reset_index(drop=True).rename(columns={0: 'MobileNet'})
df_mean['file'] = file # or file.stem for just the file name
df_mean_list.append(df_mean)
# df_mean_list is a list of dataframes, pd.concat combines them all into one dataframe
df_mean_all = pd.concat(df_mean_list).reset_index(drop=True)
print(df_mean_all)
MobileNet file
0 3.24 C:\Users\file_inference_time\file1.pkl
1 2.34 C:\Users\file_inference_time\file2.pkl
2 4.23 C:\Users\file_inference_time\file3.pkl