Expanding dict entries into rows with Pandas - python-3.x

Question

I'm trying to expand dict keys and values into their own columns using Python3 and Pandas. Below is an example. Not all dicts have the same number of items and there's no guarantee that key names match for each metric type.
I want to convert this dataframe:
id metric dicts
1 some_metric_1 {'a': 161, 'b': 121}
2 some_metric_1 {'a': 152, 'c': 4}
2 some_metric_2 {'b': 162, 'a': 83}
3 some_metric_2 {'b': 103, 'z': 69}
Created by this:
data = {'id': [1, 2, 2, 3], 'metric': ['some_metric_1', 'some_metric_1', 'some_metric_2', 'some_metric_2'], 'dicts': [{'a': 161, 'b': 121}, {'a': 152, 'c': 4}, {'b': 162, 'a': 83}, {'b': 103, 'z': 69}]}
df = pd.DataFrame.from_dict(data)
into this:
id metric key value
1 some_metric_1 a 161
1 some_metric_1 b 121
2 some_metric_1 a 152
2 some_metric_1 c 4
2 some_metric_2 b 162
2 some_metric_2 a 83
3 some_metric_2 b 103
3 some_metric_2 z 69

Answer 1

You can simply iterate through the rows of your DataFrame and extract the values needed as shown below.
Now keep in mind that the code below assumes that each key will only have 1 value (i.e. no list of value will be passed to a dict key). Though, it will work regardless of the numbers of keys.
final_df = pd.DataFrame()
for row in df.iterrows():
metric = row[1][1] # get the value in the metric column
i = row[1][0] # get the id value
for key, value in row[1][2].items():
tmp_df = pd.DataFrame({
'id':i,
'metric':metric,
'key': key,
'value': value
}, index=[0])
final_df = final_df.append(tmp_df) # append the tmp_df to our final df
final_df.reset_index(drop=True) # Reset the final DF index sinze we assign index 0 to each tmp df
Output
id metric key value
0 1 some_metric_1 a 161
1 1 some_metric_1 b 121
2 1 some_metric_1 c 152
3 2 some_metric_1 a 152
4 2 some_metric_1 c 4
5 2 some_metric_2 b 162
6 2 some_metric_2 a 83
7 3 some_metric_3 b 103
8 3 some_metric_3 z 69
Here is more information regarding df.append().

Answer 2

I find this type of problem easier to solve in plain Python rather than Pandas - once you are storing dictionaries in your DataFrame, it's going to be difficult to perform the kind of fast vectorized operations which make Pandas so useful for simple numeric/string data.
Here's my solution which involves a couple of comprehensions, and zip.
metrics = df['metric']
dicts = df['dicts']
ids = df['id']
metrics, ids = zip(*((m, i) for m, d, i in zip(metrics, dicts, ids) for j in range(len(d))))
keys, values = zip(*((k, v) for d in dicts for k, v in d.items()))
new_data = {'id': ids, 'metric': metrics, 'keys': keys, 'values': values}
new_df = pd.DataFrame.from_dict(new_data)
Results:
id keys metric values
0 1 a some_metric_1 161
1 1 b some_metric_1 121
2 2 a some_metric_1 152
3 2 c some_metric_1 4
4 2 b some_metric_2 162
5 2 a some_metric_2 83
6 3 b some_metric_2 103
7 3 z some_metric_2 69