when combining features and then aggregating them featuretools returns some variables that don't make sense, how can this be avoided? - featuretools

Question

I've got a dataset that contains invoices, with a unique identifier, and customers with a unique identifier. Each customer can have 1 or more invoices.
I set up the entity sets as follows:
es = ft.EntitySet(id="data")
es = es.add_dataframe(
dataframe=df,
dataframe_name="data",
index="rows",
make_index=True,
time_index="invoice_date",
logical_types={
"customer_id": Categorical,
"description": NaturalLanguage,
}
)
es.normalize_dataframe(
base_dataframe_name="data",
new_dataframe_name="invoices",
index="invoice",
copy_columns=["customer_id"],
)
es.normalize_dataframe(
base_dataframe_name="invoices",
new_dataframe_name="customers",
index="customer_id",
)
So that customers is child of invoices which is child of the entire dataset.
Now, I want to combine the variables price and quantity at the entire dataframe level, to obtain price*quantity, which happens all good. But when aggregating, I see combinations of variables that don't make human sense (or maybe it is me who does not understand them).
I set up the dfs as follows:
date_primitives = ["month", "weekday"]
text_primitives = ["num_words"]
trans_primitives = date_primitives + text_primitives + ["multiply_numeric"]
agg_primitives = ["mean"]
feature_matrix, feature_defs = ft.dfs(
entityset=es,
target_dataframe_name="customers",
agg_primitives=agg_primitives,
trans_primitives=trans_primitives,
primitive_options={
("multiply_numeric"): {
'include_columns': {
'data': ['price', 'quantity']
}
}
},
max_depth=3,
)
and the result of dfs contains the following features:
[<Feature: MEAN(data.price)>,
<Feature: MEAN(data.quantity)>,
<Feature: MONTH(first_invoices_time)>,
<Feature: WEEKDAY(first_invoices_time)>,
<Feature: MEAN(invoices.MEAN(data.price))>,
<Feature: MEAN(invoices.MEAN(data.quantity))>,
<Feature: MEAN(data.NUM_WORDS(description))>,
<Feature: MEAN(data.price * quantity)>,
<Feature: MEAN(data.price) * MEAN(data.quantity)>,
<Feature: MEAN(invoices.MEAN(data.NUM_WORDS(description)))>,
<Feature: MEAN(invoices.MEAN(data.price * quantity))>,
<Feature: MEAN(invoices.MEAN(data.price) * MEAN(data.quantity))>,
<Feature: MEAN(data.price * quantity) * MEAN(data.price)>,
<Feature: MEAN(data.price * quantity) * MEAN(data.quantity)>,
<Feature: MEAN(data.price * quantity) * MEAN(invoices.MEAN(data.price))>,
<Feature: MEAN(data.price * quantity) * MEAN(invoices.MEAN(data.quantity))>,
<Feature: MEAN(data.price) * MEAN(invoices.MEAN(data.price))>,
<Feature: MEAN(data.price) * MEAN(invoices.MEAN(data.quantity))>,
<Feature: MEAN(data.quantity) * MEAN(invoices.MEAN(data.price))>,
<Feature: MEAN(data.quantity) * MEAN(invoices.MEAN(data.quantity))>,
<Feature: MEAN(invoices.MEAN(data.price)) * MEAN(invoices.MEAN(data.quantity))>]
From those, these feature MEAN(data.price * quantity)> makes sense to me, the variations of this feature MEAN(invoices.MEAN(data.price * quantity))> also make sense to me. But features like these ones MEAN(data.quantity) * MEAN(invoices.MEAN(data.price)), MEAN(invoices.MEAN(data.price)) * MEAN(invoices.MEAN(data.quantity)), don't make sense to me.
I was wondering if they could be prevented from the output? I tried reducing the depth, but that would prevent the text primitive from executing. So not sure what else I can try?
Thank you!

Answer 1

Thank you for your question.
You can use the drop_contains argument for dfs. It drops features that contain the specified string.
A sample call to dfs would be:
feature_matrix, feature_defs = ft.dfs(
entityset=es,
target_dataframe_name="customers",
agg_primitives=agg_primitives,
trans_primitives=trans_primitives,
primitive_options={
("multiply_numeric"): {
'include_columns': {
'data': ['Price', 'Quantity']
}
}
},
drop_contains=[") * MEAN("],
max_depth=3,
)