I was using tensorflow + keras while trying to implement a "Text classification" model to classify different types of movie reviews. I am running into a error which tells that the shapes aren't equal.
Because I am not sure where the error might be hidden, I can't produce a reprex example as i am not certain how to isolate the problem. It may be worth your time if the line with variable(x_val) is there as there may be a problem with the partitioning.
note this is not the final code. as I had already encountered an error at this point, i stopped writing it.
from __future__ import absolute_import, division, print_function
import numpy as np
import tensorflow as tf
from tensorflow import keras
imdb = keras.datasets.imdb
(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)
#print ("The length if training data: "len(train_data[0]), "And labels: "len(test_data[0]))
word_index = imdb.get_word_index()
word_index = {k: (v+3) for k,v in word_index.items()}
word_index["<PAD>"] = 0
word_index["<START>"] = 1
word_index["<UNKNOWN>"] = 2
word_index["<END>"] = 3
reverse_word_index = dict([(value, key) for (key, value) in word_index.items() ])
def decode_review(text):
return (' '.join([reverse_word_index.get(i , "?") for i in text ]))
print (decode_review(train_data[0]))
train_data = keras.preprocessing.sequence.pad_sequences(train_data,
value=word_index["<PAD>"],
padding="post",
maxlen=256)
test_data = keras.preprocessing.sequence.pad_sequences(test_data,
value=word_index["<PAD>"],
padding="post",
maxlen=256)
#print ('train length :' ,len(train_data[0]), 'test length: ', len(train_data[1]))
vocab_size = 10000
model = keras.Sequential()
model.add(keras.layers.Embedding(vocab_size , 16))
model.add(keras.layers.GlobalAveragePooling1D())
model.add(keras.layers.Dense(16, activation=tf.nn.relu))
model.add(keras.layers.Dense(16, activation=tf.nn.sigmoid))
print ("the model summary is :======>>" , model.summary())
model.compile(optimizer="adam" , loss="binary_crossentropy", metrics=["acc"])
x_val = train_data[:10000]
partial_x_train = train_data[10000:]
y_val = train_labels[:10000]
partial_y_train = train_labels[10000:]
history = model.fit(partial_x_train , partial_y_train , epochs=40 , batch_size=512,
validation_data=(x_val, y_val), verbose=1)
This is the error message I was getting:-----
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
embedding_1 (Embedding) (None, None, 16) 160000
_________________________________________________________________
global_average_pooling1d_1 ( (None, 16) 0
_________________________________________________________________
dense_2 (Dense) (None, 16) 272
_________________________________________________________________
dense_3 (Dense) (None, 16) 272
=================================================================
Total params: 160,544
Trainable params: 160,544
Non-trainable params: 0
_________________________________________________________________
the model summary is :======>> None
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-18-02082e1f39d4> in <module>()
57
58 history = model.fit(partial_x_train , partial_y_train , epochs=40 , batch_size=512,
---> 59 validation_data=(x_val, y_val), verbose=1)
ValueError: A target array with shape (15000, 1) was passed for an output of shape (None, 16) while using as loss `binary_crossentropy`. This loss expects targets to have the same shape as the output.
You need to update the final/output layer of your model. Since it's a binary classification problem, the output Dense layer should have one node like follows:
model.add(keras.layers.Dense(1, activation=tf.nn.sigmoid))
You may want to check out this tutorial on text classification using IMDB dataset.
Related
New to the Keras model implementation and despite looking for answers:
https://stackoverflow.com/questions/60991253/invalidargumenterror-input-must-be-a-vector-got-shape
https://stackoverflow.com/questions/41736677/how-could-keras-model-predict-only-one-sample
But still couldnt make it work :(
I successfully trained my Keras model (words embeddings using "https://tfhub.dev/google/universal-sentence-encoder/4")
But when I try to predict with:
test_text = ["We are looking for Data Scientists"]
test_text = np.array(test_text , dtype=object)[:, np.newaxis]
predicts = model.predict(test_text , batch_size=32)
predicts
But I get the following error:
InvalidArgumentError Traceback (most recent call last)
<ipython-input-150-078cff510ad4> in <module>
----> 1 predicts = model.predict(test_text, batch_size=32)
2
3 predicts
1 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
53 ctx.ensure_initialized()
54 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 55 inputs, attrs, num_outputs)
56 except core._NotOkStatusException as e:
57 if name is not None:
InvalidArgumentError: Graph execution error:
input must be a vector, got shape: []
[[{{node text_preprocessor/tokenize/StringSplit/StringSplit}}]] [Op:__inference_predict_function_838262]
Below the model summary and the batch input shape required - but really dont get it to work :(
Any HELP more than welcome!!!
Thanks
>> model.summary()
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=
input_18 (InputLayer) \[(None, 1)\] 0
lambda_14 (Lambda) (None, 512) 0
dense_1 (Dense) (None, 256) 131328
dense_2 (Dense) (None, 788) 202516
=================================================================
Total params: 333,844
Trainable params: 333,844
Non-trainable params: 0
config = model.get_config() # Returns pretty much every information about your model
print(config\["layers"\]\[0\]\["config"\]\["batch_input_shape"\]) # returns a tuple of width, height and channels
(None, 1)
Looking at some answers like
Keras model input shape wrong
I also tried to reshape like that
predicts = model.predict(test_text.reshape((1, -1)))
predicts
But I get exactly the same error than previously
I am trying to learn multi modal distribution using Neural Network and Gaussian Mixture model.
but, in my modal summary the output layer has None output, and it is from using MixtureSameFamily.
Can you help me resolve this?
def zero_inf(out):
loc, scale, probs = tf.split(out, num_or_size_splits=3, axis=-1)
scale = tf.nn.softplus(scale)
probs = tf.nn.softmax(probs)
return tfd.MixtureSameFamily(
mixture_distribution = tfd.Categorical(probs=probs),#D
components_distribution = tfd.Normal(loc=loc, scale=scale))
## Definition of the custom parametrized distribution
inputs = tf.keras.layers.Input(shape=(5,))
out = Dense(6)(inputs)#A
p_y_zi = tfp.layers.DistributionLambda(lambda t: zero_inf(t))(out)
model_zi = Model(inputs=inputs, outputs=p_y_zi)
# def NLL(y_true, y_hat):
# return -y_hat.log_prob(tf.reshape(y_true,(-1,)))
# model_zi.compile(optimizer="adam", loss=NLL)
model_zi.summary()
Below is the model summary.
Model: "model_70"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_30 (InputLayer) [(None, 5)] 0
_________________________________________________________________
dense_130 (Dense) (None, 6) 36
_________________________________________________________________
distribution_lambda_36 (Dist ((None,), (None,)) 0
=================================================================
Total params: 36
Trainable params: 36
Non-trainable params: 0
_________________________________________________________________
Because of this empty/none output, I unable to train the model as I get some shape related errors.
InvalidArgumentError: Cannot update variable with shape [] using a Tensor with shape [32], shapes must be equal.
[[{{node metrics_60/mae/AssignAddVariableOp}}]]
Tensorflow version: 2.6.2
Tensorflow-probability: 0.14.0
Python: 3.6
I create a custom generator tensorflow.keras.utils.Sequence and tried to fit a simple model.
import tensorflow as tf
import keras
import numpy as np
from tensorflow.keras.utils import Sequence
from keras import Sequential
from keras.layers import InputLayer, Dense
class MyDataGenerator(Sequence):
def __init__(self, df, x_col='filename', y_col='class',
batch_size=32, path='./', num_classes=None, shuffle=True,
dim=(634,513,1), nfft=1024, hstep=256, sr=16000):
self.batch_size = batch_size
self.df = df
self.indices = self.df.index.tolist()
self.num_classes = num_classes
self.path = path
self.shuffle = shuffle
self.x_col = x_col
self.y_col = y_col
self.on_epoch_end()
self.dim = dim
self.nfft = nfft
self.hstep = hstep
self.sr = sr
def __getitem__(self, index):
index = self.index[index * self.batch_size:(index + 1) * self.batch_size]
batch = [self.indices[k] for k in index]
X = np.zeros((self.batch_size, *self.dim), dtype=np.float32)
y = np.zeros((self.batch_size,), dtype=np.uint32)
return X, y
def __len__(self):
return math.ceil(len(self.indices) / self.batch_size)
def on_epoch_end(self):
self.index = np.arange(len(self.indices))
if self.shuffle == True:
np.random.shuffle(self.index)
train_datagen = MyDataGenerator(df_training)
valid_datagen = MyDataGenerator(df_validation, shuffle=False)
x,y = train_datagen[27]
print(x.shape, y.shape, x.dtype, y.dtype, type(x), type(y))
print(y)
1. (32, 634, 513, 1) (32,) float32 uint32 <class 'numpy.ndarray'> <class 'numpy.ndarray'>
2. [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
model = Sequential(name='Test_model')
model.add(InputLayer((634, 513, 1), name='Input'))
model.add(Flatten(name='Flatten'))
model.add(Dense(1, activation='sigmoid', name='Output'))
model.summary()
model.compile(optimizer='adam', loss='binary_crossentropy',
metrics=['accuracy'])
model.fit(
train_datagen,
epochs=1,
verbose=0,
validation_data=valid_datagen
)
And i got ValueError
Model: "Test_model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
Flatten (Flatten) (None, 325242) 0
_________________________________________________________________
Output (Dense) (None, 1) 325243
=================================================================
Total params: 325,243
Trainable params: 325,243
Non-trainable params: 0
_________________________________________________________________
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-203-533c619e3ae0> in <module>()
11 epochs=1,
12 verbose=0,
---> 13 validation_data=valid_datagen
14 )
3 frames
/usr/local/lib/python3.7/dist-packages/keras/engine/data_adapter.py in select_data_adapter(x, y)
976 "Failed to find data adapter that can handle "
977 "input: {}, {}".format(
--> 978 _type_name(x), _type_name(y)))
979 elif len(adapter_cls) > 1:
980 raise RuntimeError(
ValueError: Failed to find data adapter that can handle input: <class '__main__.MyDataGenerator'>, <class 'NoneType'>
Can't figure out why the model doesn't want to train with a generator. The data types are correct. Inputs and outputs too.
If try to do it without a generator, then the model works correctly, but the generator cannot be removed due to the huge dataset.
X = np.zeros((32, 634, 513, 1), dtype=np.float32)
y = np.zeros((32,), dtype=np.uint32)
model = Sequential(name='Test_model')
model.add(InputLayer((634, 513, 1), name='Input'))
model.add(Flatten(name='Flatten'))
model.add(Dense(1, activation='sigmoid', name='Output'))
model.summary()
model.compile(optimizer='adam', loss='binary_crossentropy',
metrics=['accuracy'])
model.fit(
X,y,
epochs=1,
verbose=1,
#validation_data=valid_datagen
)
Model: "Test_model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
Flatten (Flatten) (None, 325242) 0
_________________________________________________________________
Output (Dense) (None, 1) 325243
=================================================================
Total params: 325,243
Trainable params: 325,243
Non-trainable params: 0
_________________________________________________________________
1/1 [==============================] - 14s 14s/step - loss: 0.6931 - accuracy: 1.0000
<keras.callbacks.History at 0x7f74adf34510>
Tensorflow version: 2.5.0
Keras version: 2.4.3
I upgraded to Tensorflow 2.0 and there is no tf.summary.FileWriter("tf_graphs", sess.graph). I was looking through some other StackOverflow questions on this and they said to use tf.compat.v1.summary etc. Surely there must be a way to graph and visualize a tf.keras model in Tensorflow version 2. What is it? I'm looking for a tensorboard output like the one below. Thank you!
You can visualize the graph of any tf.function decorated function, but first, you have to trace its execution.
Visualizing the graph of a Keras model means to visualize it's call method.
By default, this method is not tf.function decorated and therefore you have to wrap the model call in a function correctly decorated and execute it.
import tensorflow as tf
model = tf.keras.Sequential(
[
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(32, activation="relu"),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation="softmax"),
]
)
#tf.function
def traceme(x):
return model(x)
logdir = "log"
writer = tf.summary.create_file_writer(logdir)
tf.summary.trace_on(graph=True, profiler=True)
# Forward pass
traceme(tf.zeros((1, 28, 28, 1)))
with writer.as_default():
tf.summary.trace_export(name="model_trace", step=0, profiler_outdir=logdir)
According to the docs, you can use Tensorboard to visualise graphs once your model has been trained.
First, define your model and run it. Then, open Tensorboard and switch to the Graph tab.
Minimal Compilable Example
This example is taken from the docs. First, define your model and data.
# Relevant imports.
%load_ext tensorboard
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from datetime import datetime
from packaging import version
import tensorflow as tf
from tensorflow import keras
# Define the model.
model = keras.models.Sequential([
keras.layers.Flatten(input_shape=(28, 28)),
keras.layers.Dense(32, activation='relu'),
keras.layers.Dropout(0.2),
keras.layers.Dense(10, activation='softmax')
])
model.compile(
optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
(train_images, train_labels), _ = keras.datasets.fashion_mnist.load_data()
train_images = train_images / 255.0
Next, train your model. Here, you will need to define a callback for Tensorboard to use for visualising stats and graphs.
# Define the Keras TensorBoard callback.
logdir="logs/fit/" + datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = keras.callbacks.TensorBoard(log_dir=logdir)
# Train the model.
model.fit(
train_images,
train_labels,
batch_size=64,
epochs=5,
callbacks=[tensorboard_callback])
After training, in your notebook, run
%tensorboard --logdir logs
And switch to the Graph tab in the navbar:
You will see a graph that looks a lot like this:
Here is the solution for tf2.x with Graph visualization of subclassed model/layer
import tensorflow as tf
print("TensorFlow version:", tf.__version__)
from tensorflow.keras.layers import Dense, Flatten, Conv2D
from tensorflow.keras import Model,Input
class MyModel(Model):
def __init__(self, dim):
super(MyModel, self).__init__()
self.conv1 = Conv2D(16, 3, activation='relu')
self.conv2 = Conv2D(32, 3, activation='relu')
self.conv3 = Conv2D(8, 3, activation='relu')
self.flatten = Flatten()
self.d1 = Dense(128, activation='relu')
self.d2 = Dense(1)
def call(self, x):
x = self.conv1(x)
x = self.conv2(x)
x = self.conv3(x)
x = self.flatten(x)
x = self.d1(x)
return self.d2(x)
def build_graph(self):
x = Input(shape=(dim))
return Model(inputs=[x], outputs=self.call(x))
dim = (28, 28, 1)
# Create an instance of the model
model = MyModel((dim))
model.build((None, *dim))
model.build_graph().summary()
tf.keras.utils.plot_model(model.build_graph(), to_file="model.png",
expand_nested=True, show_shapes=True)
the output is
TensorFlow version: 2.5.0
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 28, 28, 1)] 0
_________________________________________________________________
conv2d (Conv2D) (None, 26, 26, 16) 160
_________________________________________________________________
conv2d_1 (Conv2D) (None, 24, 24, 32) 4640
_________________________________________________________________
conv2d_2 (Conv2D) (None, 22, 22, 8) 2312
_________________________________________________________________
flatten (Flatten) (None, 3872) 0
_________________________________________________________________
dense (Dense) (None, 128) 495744
_________________________________________________________________
dense_1 (Dense) (None, 1) 129
=================================================================
Total params: 502,985
Trainable params: 502,985
Non-trainable params: 0
Here is also a graph visualization
Here's what is working for me at the moment (TF 2.0.0), based on the tf.keras.callbacks.TensorBoard code:
# After model has been compiled
from tensorflow.python.ops import summary_ops_v2
from tensorflow.python.keras.backend import get_graph
tb_path = '/tmp/tensorboard/'
tb_writer = tf.summary.create_file_writer(tb_path)
with tb_writer.as_default():
if not model.run_eagerly:
summary_ops_v2.graph(get_graph(), step=0)
Another option is to use this website: https://lutzroeder.github.io/netron/
which generate a graph with a .h5 or .tflite file.
The github repo it's based on may be found here:
https://github.com/lutzroeder/netron
I am not able to solve the following error, please accept my apologies if it sounds naive, I am very new to Keras.
The output of the encoder is actually a complex value, so each output is real and imaginary part, input_h1 is also a complex value with real and imaginary parts represented as a vector. I want to multiply both of them.
# Input bits
input_bits1 = Input(shape=(2,))
input_bits2 = Input(shape=(2,))
# Input Channels
input_h1 = Input(shape=(2,))
input_h2 = Input(shape=(2,))
# Concatenate both inputs
input_bits = keras.layers.concatenate([input_bits1, input_bits2], axis=1)
print(input_bits)
# Create Encoder
m1 = Dense(64, activation='relu')(input_bits)
m2 = Dense(128, activation='relu')(m1)
encoded1 = Dense(2, activation='linear')(m2)
# Normalize the encoded value
encoded = Lambda(lambda x: K.l2_normalize(x, axis=1))(encoded1)
# The output of the encoder is actually a complex value, so each output is real and imaginary part,input_h1 is also a complex value with real and imaginary parts represented as a vector. I want to multiply both of them.
# mt1 is the real part of complex number multiplication
mt1 = encoded[:,0:1]*input_h1[:,0:1] - encoded[:,1:2]*input_h1[:,1:2]
print(mt1)
# nt1 is the imaginary part of the complex number multiplication
nt1 = encoded[:,0:1]*input_h1[:,1:2] + encoded[:,1:2]*input_h1[:,0:1]
print(nt1)
# Concatenate real and imaginary parts to feed into the decoder
mnt2 = keras.layers.concatenate([mt1, nt1], axis=1)
print(mnt2)
# Decoder 1
x5 = Dense(1024, activation='relu')(mnt2)
x6 = Dense(512, activation='relu')(x5)
x7 = Dense(64, activation='relu')(x6)
decoded_UP1 = Dense(2, activation='tanh')(x7)
# Decoder 2
a3 = Dense(1024, activation='relu')(mnt2)
a4 = Dense(512, activation='relu')(a3)
a5 = Dense(64, activation='relu')(a4)
decoded_UP2 = Dense(2, activation='tanh')(a5)
decoded = keras.layers.concatenate([decoded_UP1, decoded_UP2], axis=1)
autoencoder = Model([input_bits1, input_bits2, input_h1, input_h2], decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')
autoencoder.summary()
I am getting the following output/error :
AttributeError Traceback (most recent call last)
<ipython-input-9-c3710aa7e060> in <module>()
35 decoded = keras.layers.concatenate([decoded_UP1, decoded_UP2], axis=1)
36
---> 37 autoencoder = Model([input_bits1, input_bits2, input_h1, input_h2], decoded)
38 autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')
39 autoencoder.summary()
AttributeError: 'NoneType' object has no attribute '_inbound_nodes'
To improve the clarity of the code, you should create multiple models for instance one for the encoder and one for the decoder. That way you could have a model.summary() for each:
Example for the Bits Encoder:
from keras.layers import Input, Dense, concatenate
from keras.models import Model
# Input
input_bits1 = Input(shape=(2,))
input_bits2 = Input(shape=(2,))
input_bits = keras.layers.concatenate([input_bits1, input_bits2], axis=1)
# Hidden Layers
encoder_bits_h1 = Dense(64, activation='relu')(input_bits)
encoder_bits_h2 = Dense(128, activation='relu')(encoder_bits_h1)
encoder_bits_h3 = Dense(2, activation='linear')(encoder_bits_h2)
# Create the model
bits_encoder = Model(inputs=[input_bits1, input_bits2], outputs=[encoder_bits_h3])
bits_encoder.summary()
Which returns the bits encoder configuration:
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_5 (InputLayer) (None, 2) 0
__________________________________________________________________________________________________
input_6 (InputLayer) (None, 2) 0
__________________________________________________________________________________________________
concatenate_2 (Concatenate) (None, 4) 0 input_5[0][0]
input_6[0][0]
__________________________________________________________________________________________________
dense_4 (Dense) (None, 64) 320 concatenate_2[0][0]
__________________________________________________________________________________________________
dense_5 (Dense) (None, 128) 8320 dense_4[0][0]
__________________________________________________________________________________________________
dense_6 (Dense) (None, 2) 258 dense_5[0][0]
==================================================================================================
Total params: 8,898
Trainable params: 8,898
Non-trainable params: 0
__________________________________________________________________________________________________