How to extract the actual values from the shap summary plot - python-3.x

Question

How to go about extracting the numerical values for the shap summary plot so that the data can be viewed in a dataframe?:
Here is a MWE:
from sklearn.datasets import make_classification
from shap import Explainer, waterfall_plot, Explanation
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
# Generate noisy Data
X, y = make_classification(n_samples=1000,
n_features=50,
n_informative=9,
n_redundant=0,
n_repeated=0,
n_classes=10,
n_clusters_per_class=1,
class_sep=9,
flip_y=0.2,
random_state=17)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
model = RandomForestClassifier()
model.fit(X_train, y_train)
explainer = Explainer(model)
sv = explainer.shap_values(X_test)
shap.summary_plot(shap_values, X_train, plot_type="bar")
I tried
np.abs(shap_values.values).mean(axis=0)
but I get a shape of (50,10). How do I get just the aggerated value for each feature to then sort for the feature importance?

Answer 1

You've done this:
from sklearn.datasets import make_classification
from shap import Explainer, waterfall_plot, Explanation
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from shap import summary_plot
# Generate noisy Data
X, y = make_classification(n_samples=1000,
n_features=50,
n_informative=9,
n_redundant=0,
n_repeated=0,
n_classes=10,
n_clusters_per_class=1,
class_sep=9,
flip_y=0.2,
random_state=17)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
model = RandomForestClassifier()
model.fit(X_train, y_train)
explainer = Explainer(model)
sv = explainer.shap_values(X_test)
summary_plot(sv, X_train, plot_type="bar")
Note, you have features 3, 29, 34 and so on at the top.
If you do:
np.abs(sv).shape
(10, 250, 50)
You'll find out you've got 10 classes for 250 datapoints for 50 features.
If you aggregate, you'll get everything you need:
aggs = np.abs(sv).mean(1)
aggs.shape
(10, 50)
You can draw it:
sv_df = pd.DataFrame(aggs.T)
sv_df.plot(kind="barh",stacked=True)
And if it still doesn't look familiar, you can rearrange and filter:
sv_df.loc[sv_df.sum(1).sort_values(ascending=True).index[-10:]].plot(kind="barh",stacked=True)
Conclusion:
sv_df are aggregated SHAP values, as in summary plot, arranged as features per row and classes per column.
Does it help?