I am creating a regression model with keras. I have 10 145 * 5 matrices of ten digits. I am facing problem to fit the 10 145 * 5 matrices in keras model.
X is input matrix
In: X.shape
Out: (10, 145, 5)
y is target matrix
In: y.shape
Out: (10,)
For each 145 * 5 matrix there will be one value in the target matrix
Making the model
In: model = Sequential([
Dense(32, input_dim=145),
Activation('sigmoid'),
Dense(output_dim=10)
])
Although the previous line is not throwing any error or warning but I am quite sure that it is not the correct way to fit the model in this case.
In: model.compile(optimizer='sgd',loss='mse')
No problem so far. But when I am trying to fit the matrices
In: model.fit(X, y.reshape(-1, 1))
After this line I am getting a long Traceback which ultimately says
ValueError: Error when checking model input: expected dense_input_1 to have 2 dimensions, but got array with shape (10, 145, 5)
Please help me to correctly fit the matrices in the model. Thanks!
Use input_shape instead of input_dim. Also since the number of dimensions of output is changing you need to use Flatten or Reshape as one of the dimensions.
from keras.layers import Flatten
model = Sequential([
Dense(32, input_shape=(145,5)),
Flatten(),
Activation('sigmoid'),
Dense(output_dim=10)
])
model.summary()
Use model.summary() to check the structure of your model for better understanding.
Related
I am new to tensor flow and i am trying to build a multivariate (two features for each time step) multi step (forecast 12 time step in the future) forecast model.
I created tensorflow data set to feed it to my model:
When i print the shape of my data set, i find the following:
<PrefetchDataset shapes: ((None, None, 2), (None, None)), types: (tf.float32, tf.float32)>
This is what i understand:
(None, None, 2) = Input tensor "features" : (batchSize, Timesteps input, Features by time step)
(None, None) =Output Tensor "label" (batchSize, future forecsated time steps )
I follow up by creating my model as following :
keras.backend.clear_session()
tf.random.set_seed(42)
np.random.seed(42)
model = keras.models.Sequential([
keras.layers.SimpleRNN(50),
keras.layers.SimpleRNN(100),
keras.layers.Dense(12),
])
optimizer = keras.optimizers.SGD(lr=1.5e-6, momentum=0.9)
model.compile(loss="mae",
optimizer=optimizer,
metrics=["mae"])
When i fit the model
model.fit(train_set, epochs=5,
validation_data=valid_set)
I have the following error:
ValueError: Input 0 of layer simple_rnn_1 is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: [None, 50]
I do understand that SimpleRNN layer is expecting a 3 dimension tensor. But i think that my input has this dimension.
Thanks a lot for the help.
If you need me to share with you how i am creating my dataset, i would gladly do it.
The issue was coming form the second layer not the first one. Basically, the activation of the first layer issue a vector rather than a sequence, so it will have a rank of 2, eg: (a, b). But the second layer requires a three dimension input. TO solve this i added the return_sequences=True in the first layer of RNN.
If train_set is a numpy array, pass train_set.reshape((1,50)) to model.fit()
model.fit(train_set.reshape((1,50)), epochs=5,
validation_data=valid_set)
Then you wouldn't need to apply return_sequences=True to the first RNN cell either.
I'm trying out different Neural Network architectures for a word based NLP.
So far I've used bidirectional-, embedded- and models with GRU's guided by this tutorial: https://towardsdatascience.com/language-translation-with-rnns-d84d43b40571 and it all worked out well.
When I tried using LSTM's however, I get an error saying:
logits and labels must have the same first dimension, got logits shape [32,186] and labels shape [4704]
How can I solve this?
My source and target dataset consists of 7200 sample sentences. They are integer tokenized and embedded. The source dataset is post padded to match the length of the target dataset.
Here is my model and the relevant code:
lstm_model = Sequential()
lstm_model.add(Embedding(src_vocab_size, 128, input_length=X.shape[1], input_shape=X.shape[1:]))
lstm_model.add(LSTM(128, return_sequences=False, dropout=0.1, recurrent_dropout=0.1))
lstm_model.add(Dense(128, activation='relu'))
lstm_model.add(Dropout(0.5))
lstm_model.add((Dense(target_vocab_size, activation='softmax')))
lstm_model.compile(optimizer=Adam(0.002), loss='sparse_categorical_crossentropy', metrics=['accuracy'])
history = lstm_model.fit(X, Y, batch_size = 32, callbacks=CALLBACK, epochs = 100, validation_split = 0.25) #At this line the error is raised!
With the shapes:
X.shape = (7200, 147)
Y.shape = (7200, 147, 1)
src_vocab_size = 188
target_vocab_size = 186
I've looked at similar question on here already and tried adding a Reshape layer
simple_lstm_model.add(Reshape((-1,)))
but this only causes the following error:
"TypeError: __int__ returned non-int (type NoneType)"
It's really weird as I preprocess the dataset the same way for all models and it works just fine except for the above.
You should have return_sequences=True and return_state=False in calling the LSTM constructor.
In your snippet, the LSTM only return its last state, instead of the sequence of states for every input embedding. In theory, you could have spotted it from the error message:
logits and labels must have the same first dimension, got logits shape [32,186] and labels shape [4704]
The logits should be three-dimensional: batch size × sequence length × number of classes. The length of the sequences is 147 and indeed 32 × 147 = 4704 (number of your labels). This could have told you the length of the sequences disappeared.
I have trained my LSTM with 3 time steps. Following is the Keras LSTM layer.
model.add(LSTM(32, return_sequences=True, input_shape=(None, 3))).
ex:
X Y
[[1,2,3],[2,3,4],[4,5,6]] [[4],[5],[7]]
Now I need to predict the next value of a sequence with different time_steps (ex: 2)
X= [[1,2]]
When I use X= [[1,2]] I am getting following error
ValueError: Error when checking input: expected lstm_1_input to have shape (None, 3)
but got array with shape (1, 2)
Should I provide the same shape while I used for training.
Or can I still use a different timesteps (input shape) for predicting.
Appreciate your help on this issue.
I believe you need to use the same shape when using your model to predict on new data. Your data was trained with 3 timesteps (train_X), so you should feed it a 3-timesteps input when you model.predict your test data (test_X).
I'm working on a speech recognition problem running on Colab using LSTM. The audio files were converted into spectrograms and then normalized. There are 6840 spectrograms in total and the shape of each one is (288, 864, 4).
I already tried a few examples with RNN and CNN and they worked, but when I try an example using a LSTM I get shape errors, every time either there is one more or one less dimension than expected. Here are some of these cases :
rnn = keras.Sequential()
rnn.add(keras.layers.SimpleRNN(500, input_shape = (864, 4)))
rnn.add(keras.layers.LSTM(500, return_sequences = True))
rnn.add(keras.layers.Dropout(0.2))
rnn.add(keras.layers.LSTM(500, return_sequences = True))
rnn.add(keras.layers.Dropout(0.2))
rnn.add(keras.layers.LSTM(500, return_sequences = True))
rnn.add(keras.layers.Dropout(0.2))
rnn.add(keras.layers.Dense(212, activation = 'softmax'))
rnn.compile(optimizer = 'adam', loss = 'sparse_categorical_crossentropy',metrics = ['accuracy'])
rnn.fit(X_train, y_train, epochs = 5, validation_data=(X_test, y_test))
scores = rnn.evaluate(X_test, y_test, verbose=1)
print('Test loss:', scores[0])
print('Test accuracy:', '%.2f' % (scores[1] * 100), '%')
The following error is raised on the first LSTM layer : ValueError: Input 0 of layer lstm_54 is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: [None, 500]
If I remove the SimpleRNN line and feed the input directly to the first LSTM like this
rnn.add(keras.layers.LSTM(500, return_sequences = True, input_shape = (288, 864, 4)))
I get : ValueError: Input 0 of layer lstm_56 is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: [None, 288, 864, 4]
I tried reshaping the images to (4, 288 * 864) and got the same error when trying to use the RNN layer, but with just the LSTM I got InvalidArgumentError: Incompatible shapes: [32] vs. [32,4].
No idea where the 32 came from, though.
One last thing, not really an issue but more of a request, is there any library that can resize images the simple way? 288x864 is too big for Colab, so I'll have to do it eventually to be able to load all 6840 images and feed it to the neural network. Right now I'm just using 100 samples to test.
Feel free to leave suggestions about other methods, cabalistic number of nodes/layers or anything like that.
LSTM input is 3 dimensions [n_samples, n_timesteps, n_features], so your first line also need to enable return sequences:
rnn.add(keras.layers.SimpleRNN(500, return_sequences = True, input_shape = (864, 4)))
Next, your Dense layer will complain from wrong input size, so you want to remove return_sequence on the last LSTM network:
rnn.add(keras.layers.LSTM(500))
If you still want to keep the return_sequences = True on the last LSTM layer, you might want to wrap the Dense layer in a TimeDistributed.
I tried it on the following input and they seems to work
X_train = np.random.rand(100, 864, 4)
y_train = np.random.rand(100, 1)
The PIL from pillow package has plenty of image manipulation methods.
I want to use an LSTM neural Network with keras to forecast groups of time series and I am having troubles in making the model match what I want. The dimensions of my data are:
input tensor: (data length, number of series to train, time steps to look back)
output tensor: (data length, number of series to forecast, time steps to look ahead)
Note: I want to keep the dimensions exactly like that, no
transposition.
A dummy data code that reproduces the problem is:
import numpy as np
from keras.models import Sequential
from keras.layers import Dense, TimeDistributed, LSTM
epoch_number = 100
batch_size = 20
input_dim = 4
output_dim = 3
look_back = 24
look_ahead = 24
n = 100
trainX = np.random.rand(n, input_dim, look_back)
trainY = np.random.rand(n, output_dim, look_ahead)
print('test X:', trainX.shape)
print('test Y:', trainY.shape)
model = Sequential()
# Add the first LSTM layer (The intermediate layers need to pass the sequences to the next layer)
model.add(LSTM(10, batch_input_shape=(None, input_dim, look_back), return_sequences=True))
# add the first LSTM layer (the dimensions are only needed in the first layer)
model.add(LSTM(10, return_sequences=True))
# the TimeDistributed object allows a 3D output
model.add(TimeDistributed(Dense(look_ahead)))
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])
model.fit(trainX, trainY, nb_epoch=epoch_number, batch_size=batch_size, verbose=1)
This trows:
Exception: Error when checking model target: expected
timedistributed_1 to have shape (None, 4, 24) but got array with shape
(100, 3, 24)
The problem seems to be when defining the TimeDistributed layer.
How do I define the TimeDistributed layer so that it compiles and trains?
The error message is a bit misleading in your case. Your output node of the network is called timedistributed_1 because that's the last node in your sequential model. What the error message is trying to tell you is that the output of this node does not match the target your model is fitting to, i.e. your labels trainY.
Your trainY has a shape of (n, output_dim, look_ahead), so (100, 3, 24) but the network is producing an output shape of (batch_size, input_dim, look_ahead). The problem in this case is that output_dim != input_dim. If your time dimension changes you may need padding or a network node that removes said timestep.
I think the problem is that you expect output_dim (!= input_dim) at the output of TimeDistributed, while it's not possible. This dimension is what it considers as the time dimension: it is preserved.
The input should be at least 3D, and the dimension of index one will
be considered to be the temporal dimension.
The purpose of TimeDistributed is to apply the same layer to each time step. You can only end up with the same number of time steps as you started with.
If you really need to bring down this dimension from 4 to 3, I think you will need to either add another layer at the end, or use something different from TimeDistributed.
PS: one hint towards finding this issue was that output_dim is never used when creating the model, it only appears in the validation data. While it's only a code smell (there might not be anything wrong with this observation), it's something worth checking.