I am trying to replicate Keras-LSTM DeepExplainer example. I am getting the following error when trying to compute the shap values:
This warning: keras is no longer supported, please use tf.keras instead.
Your TensorFlow version is newer than 2.4.0 and so graph support has been removed in eager mode and some static graphs may not be supported. See PR #1483 for discussion.
And this error:
TypeError Traceback (most recent call last)
in
1 import shap
2 explainer = shap.DeepExplainer(model, x_train[:100])
----> 3 shap_values = explainer.shap_values(x_test[:10])
~/miniconda3/envs/mtq/lib/python3.8/site-packages/shap/explainers/_deep/init.py
in shap_values(self, X, ranked_outputs, output_rank_order,
check_additivity)
122 were chosen as "top".
123 """
--> 124 return self.explainer.shap_values(X, ranked_outputs, output_rank_order, check_additivity=check_additivity)
~/miniconda3/envs/mtq/lib/python3.8/site-packages/shap/explainers/_deep/deep_tf.py
in shap_values(self, X, ranked_outputs, output_rank_order,
check_additivity)
306 # run attribution computation graph
307 feature_ind = model_output_ranks[j,i]
--> 308 sample_phis = self.run(self.phi_symbolic(feature_ind), self.model_inputs,
joint_input) 309
310 # assign the attributions to the right part of the output arrays
~/miniconda3/envs/mtq/lib/python3.8/site-packages/shap/explainers/_deep/deep_tf.py
in run(self, out, model_inputs, X)
363
364 return final_out
--> 365 return self.execute_with_overridden_gradients(anon)
366
367 def custom_grad(self, op, *grads):
~/miniconda3/envs/mtq/lib/python3.8/site-packages/shap/explainers/_deep/deep_tf.py
in execute_with_overridden_gradients(self, f)
399 # define the computation graph for the attribution values using a custom gradient-like computation
400 try:
--> 401 out = f()
402 finally:
403 # reinstate the backpropagatable check
~/miniconda3/envs/mtq/lib/python3.8/site-packages/shap/explainers/_deep/deep_tf.py
in anon()
356 shape = list(self.model_inputs[i].shape)
357 shape[0] = -1
--> 358 data = X[i].reshape(shape)
359 v = tf.constant(data, dtype=self.model_inputs[i].dtype)
360 inputs.append(v)
TypeError: 'NoneType' object cannot be interpreted as an integer
I have checked out the PR#1483, but couldn't find a relevant fix there. Please suggest on what tensorflow, keras, and shap versions are needed to successfully replicate the example.
Related
Currently, I'm exploring machine learning interpretability tools for one of my project. I found Shapash quite a new tool and many people suggesting to use it to create a few easily interpretable charts for ML model. When I tried it with RandomForestClassifier it worked fine and generate a webpage full of different charts but the same I cannot achieve while using SVM(just exploring this library, not focusing on the perfect ML model for a problem).
Note - using Shapash link here
#Fit blackbox model
svc = svm.SVC()
svc.fit(X_train_smote, y_train_smote)
y_pred = svc.predict(X_test)
print(f"F1 Score {f1_score(y_test, y_pred, average='macro')}")
print(f"Accuracy {accuracy_score(y_test, y_pred)}")
from shapash import SmartExplainer
xpl = SmartExplainer(model=svc)
error which I'm getting -
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
/tmp/ipykernel_13648/1233939729.py in <module>
----> 1 xpl = SmartExplainer(model=svc)
~/Python_AI/ai_env/lib/python3.8/site-packages/shapash/explainer/smart_explainer.py in __init__(self, model, backend, preprocessing, postprocessing, features_groups, features_dict, label_dict, title_story, palette_name, colors_dict, **kwargs)
194 if isinstance(backend, str):
195 backend_cls = get_backend_cls_from_name(backend)
--> 196 self.backend = backend_cls(
197 model=self.model, preprocessing=preprocessing, **kwargs)
198 elif isinstance(backend, BaseBackend):
~/Python_AI/ai_env/lib/python3.8/site-packages/shapash/backend/shap_backend.py in __init__(self, model, preprocessing, explainer_args, explainer_compute_args)
16 self.explainer_args = explainer_args if explainer_args else {}
17 self.explainer_compute_args = explainer_compute_args if explainer_compute_args else {}
---> 18 self.explainer = shap.Explainer(model=model, **self.explainer_args)
19
20 def run_explainer(self, x: pd.DataFrame) -> dict:
~/Python_AI/ai_env/lib/python3.8/site-packages/shap/explainers/_explainer.py in __init__(self, model, masker, link, algorithm, output_names, feature_names, **kwargs)
166 # if we get here then we don't know how to handle what was given to us
167 else:
--> 168 raise Exception("The passed model is not callable and cannot be analyzed directly with the given masker! Model: " + str(model))
169
170 # build the right subclass
Exception: The passed model is not callable and cannot be analyzed directly with the given masker! Model: SVC()
I have a strange problem relating to a topic model I am running with BERTopic. The model runs without any errors in Colab and vscode venv. However, when I run the same model in Jupyter Notebook using the same venv as I have in the vscode venv, the model returns an error, half-way through the run.
The error is below:
TypeError Traceback (most recent call last)
<timed exec> in <module>
c:\python\python39\lib\site-packages\bertopic\_bertopic.py in fit_transform(self, documents, embeddings, y)
285 # Reduce dimensionality with UMAP
286 if self.seed_topic_list is not None and self.embedding_model is not None:
--> 287 y, embeddings = self._guided_topic_modeling(embeddings)
288 umap_embeddings = self._reduce_dimensionality(embeddings, y)
289
c:\python\python39\lib\site-packages\bertopic\_bertopic.py in _guided_topic_modeling(self, embeddings)
1424 for seed_topic in range(len(seed_topic_list)):
1425 indices = [index for index, topic in enumerate(y) if topic == seed_topic]
-> 1426 embeddings[indices] = np.average([embeddings[indices], seed_topic_embeddings[seed_topic]], weights=[3, 1])
1427 return y, embeddings
1428
<__array_function__ internals> in average(*args, **kwargs)
c:\python\python39\lib\site-packages\numpy\lib\function_base.py in average(a, axis, weights, returned)
405 wgt = wgt.swapaxes(-1, axis)
406
--> 407 scl = wgt.sum(axis=axis, dtype=result_dtype)
408 if np.any(scl == 0.0):
409 raise ZeroDivisionError(
c:\python\python39\lib\site-packages\numpy\core\_methods.py in _sum(a, axis, dtype, out, keepdims, initial, where)
45 def _sum(a, axis=None, dtype=None, out=None, keepdims=False,
46 initial=_NoValue, where=True):
---> 47 return umr_sum(a, axis, dtype, out, keepdims, initial, where)
48
49 def _prod(a, axis=None, dtype=None, out=None, keepdims=False,
TypeError: No loop matching the specified signature and casting was found for ufunc add
Not sure what the source of the error could be, since the same code works in Colab and vscode venv. Any pointers in the right direction would be greatly appreciated.
I am trying to load an existing MODFLOW-USG model with FloPy (Windows environment). The model has a Voronoi mesh, and this seems to trip the "load" function:
m1=flopy.modflow.Modflow.load(model_name+".nam",model_ws=model_dir,verbose=True,check=False,exe_name="mfusg.exe",version='mfusg')
I get the following error, which appears to relate to the fact that FloPy is expecting a structured grid with rows and columns:
TypeError Traceback (most recent call last)
<ipython-input-33-62420c415719> in <module>
6 head_file = os.path.join(model_dir,model_name+'.hds')
7 print(head_file)
----> 8 m1=flopy.modflow.Modflow.load(model_name+".nam",model_ws=model_dir,verbose=True,check=False,exe_name="mfusg.exe",version='mfusg')
9 headobj = bf.HeadUFile(head_file,verbose=True,text='HEADU')
10 headobj.list_records()
~\Anaconda3\lib\site-packages\flopy\modflow\mf.py in load(f, version, exe_name, verbose, model_ws, load_only, forgive, check)
797 item.package.load(item.filehandle, ml,
798 ext_unit_dict=ext_unit_dict,
--> 799 check=False)
800 else:
801 item.package.load(item.filehandle, ml,
~\Anaconda3\lib\site-packages\flopy\modflow\mfrch.py in load(f, model, nper, ext_unit_dict, check)
408 print(txt)
409 t = Util2d.load(f, model, (nrow, ncol), np.float32, 'rech',
--> 410 ext_unit_dict)
411 else:
412 parm_dict = {}
~\Anaconda3\lib\site-packages\flopy\utils\util_array.py in load(f_handle, model, shape, dtype, name, ext_unit_dict, array_free_format, array_format)
2699
2700 elif cr_dict['type'] == 'internal':
-> 2701 data = Util2d.load_txt(shape, f_handle, dtype, cr_dict['fmtin'])
2702 u2d = Util2d(model, shape, dtype, data, name=name,
2703 iprn=cr_dict['iprn'], fmtin="(FREE)",
~\Anaconda3\lib\site-packages\flopy\utils\util_array.py in load_txt(shape, file_in, dtype, fmtin)
2376 elif len(shape) == 2:
2377 nrow, ncol = shape
-> 2378 num_items = nrow * ncol
2379 else:
2380 raise ValueError(
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'
I could not find any documentation or Jupyter notebooks with examples of loading an existing model with Voronoi mesh, only creating new triangular meshes or structured / local-grid-refined grids.
Try the code with forgive = True.
m1=flopy.modflow.Modflow.load(model_name+".nam",model_ws=model_dir,verbose=True,check=False,exe_name="mfusg.exe",version='mfusg', forgive = True)
I want to compile my python3 code on ubuntu, and also want to know about the problem, such that i can handle that in future.
It seems there is some problem with the imported library function.
## sample code
1 import numpy as np
2 x = np.array([[-1,-1],[-2,-1],[-3,-2],[1,1],[2,1],[3,2]])
3 y = np.array([1,1,1,2,2,2])
4 from sklearn.naive_bayes import GaussianNB
5 clf = GaussianNB(x, y)
6 clf = clf.fit(x,y) ###showing error on compiling
7 print(clf.predict([[-2,1]]))
## output shown
Traceback (most recent call last):
File "naive.py", line 7, in <module>
clf = clf.fit(x,y)
File "/home/abhihsek/.local/lib/python3.6/site-
packages/sklearn/naive_bayes.py", line 192, in fit
sample_weight=sample_weight)
File "/home/abhihsek/.local/lib/python3.6/site-
packages/sklearn/naive_bayes.py", line 371, in _partial_fit
raise ValueError('Number of priors must match number of'
ValueError: Number of priors must match number of classes.
## code of library function line 192
190 X, y = check_X_y(X, y)
191 return self._partial_fit(X, y, np.unique(y),
_refit=True,
192
sample_weight=sample_weight)
## code of library function line 371
369 # Check that the provide prior match the number of classes
370 if len(priors) != n_classes:
371 raise ValueError('Number of priors must
match
number of'
372 ' classes.')
373 # Check that the sum is 1
As #Suvan Pandey mentioned, then the code won't give any error when writing clf = GaussianNB() instead of clf = GaussianNB(x, y).
If we look at the GaussianNB class then the __init__() can take these parameters:
def __init__(self, priors=None, var_smoothing=1e-9): # <-- these have a default value
self.priors = priors
self.var_smoothing = var_smoothing
The documentation about the two parameters:
priors – Prior probabilities of the classes. If specified the priors are not adjusted according to the data.
var_smoothing – Portion of the largest variance of all features that is added to variances for calculation stability.
As your x and y variables both return an array object then they don't fit the parameters of the __init__(...).
I have a graph with 3000 nodes. I am trying to use the pydot layout engine to find a more pleasing layout than the default networkx layout layout = nx.fruchterman_reingold_layout(G)
The example from networkx doc
G_tst = nx.complete_graph(4)
pos = nx.nx_pydot.pydot_layout(G_tst )
pos = nx.nx_pydot.pydot_layout(G_tst , prog='dot')
works just fine. However when I use my own graph
pos = nx.nx_pydot.pydot_layout(G) I get a Type Error where it claims that G has the attibute name more than once.
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-72-1326868cc786> in <module>()
1
----> 2 pos = nx.nx_pydot.pydot_layout(G)
3 nx.draw(G, pos=pos)
C:\Anaconda3\lib\site-packages\networkx\drawing\nx_pydot.py in pydot_layout(G, prog, root, **kwds)
261 """
262 import pydotplus
--> 263 P=to_pydot(G)
264 if root is not None :
265 P.set("root",make_str(root))
C:\Anaconda3\lib\site-packages\networkx\drawing\nx_pydot.py in to_pydot(N, strict)
200 for n,nodedata in N.nodes_iter(data=True):
201 str_nodedata=dict((k,make_str(v)) for k,v in nodedata.items())
--> 202 p=pydotplus.Node(make_str(n),**str_nodedata)
203 P.add_node(p)
204
TypeError: __init__() got multiple values for argument 'name'
Here are the node attributes I do have:
`G.add_node(G.number_of_nodes(),
name=endNode.endWord, # string
teaching_text=endNode.tt_corpus, # string
definition=endNode.domainDef, # string
search_string=endNode.searchKey_obj.search_key_str,
name_len = len(endNode.endWord))` #int
I got the same error yesterday. I'm not 100 percent sure but it seems some internal variables conflict with your attribute "name". In my case, I change it to "name_" then it works.