I am new to Keras and LSTMs -- I want to train a model on 2-dimensional sequences (ie, movement in a grid-space), as opposed to 1-dimensional sequences (like characters of text).
As a test, I first tried just one dimension, and I am doing it successfully with the following setup:
model = Sequential()
model.add(LSTM(512, return_sequences=True, input_shape=X[0].shape, dropout=0.2, recurrent_dropout=0.2))
model.add(LSTM(512, return_sequences=False, dropout=0.2))
model.add(Dense(len(y[0]), activation="softmax"))
model.compile(loss="categorical_crossentropy", optimizer="rmsprop", metrics=['accuracy'])
model.fit(X, y, epochs=50)
I'm formatting the data like this:
data = ## list of integers (1D)
inputs = []
outputs = []
for i in range(len(data) - SEQUENCE_LENGTH):
inputs.append(data[i:i + SEQUENCE_LENGTH])
outputs.append(data[i + SEQUENCE_LENGTH])
X = np.array([to_categorical(np.array(input), CATEGORY_LENGTH) for input in inputs])
y = to_categorical(np.array(outputs), CATEGORY_LENGTH)
This is straightforward and converges quickly.
But if instead of a list of integers, my data consists of 2D tuples, I can no longer create categorical (one-hot) arrays to pass to the LSTM layers.
I've tried not using categorical arrays and simply passing the tuples to the model. In this case, I've changed my output layer to:
model.add(Dense(1, activation="linear"))
But that does not converge, or at least moves incredibly slowly.
How can I adapt this code to handle input with additional dimensions?
This previous answer should apply to your question as well. The only difference is that you will have to convert your tuple to a data frame beforehand.
Related
H!
I am working on predicting two variables using time series forecasting autoencoder model.
The dataset present the coordinates bounding box in video like x and y.
So let's say we have 6 people in video, so that mean we will have 12 variable as input in the model for each image.
The LSTM model doing a good job predicting just one variable. However, I want to predict two variables ( x and y) using autoencoder model.
I am using karas to this work. This is the model that I use.
model = Sequential()
model.add(LSTM(64, activation='relu', input_shape=(trainX.shape[1], trainX.shape[2]), return_sequences=True))
model.add(LSTM(32, activation='relu', return_sequences=False))
model.add(Dropout(0.2))
model.add(Dense(trainY.shape[2]))
cp1 = ModelCheckpoint('model1/', save_best_only=True)
model.compile(loss='mse', optimizer=Adam(learning_rate=0.0001), metrics=['mae', 'mape'])
model.summary()
I want to predict two variables ( x and y) using autoencoder model.
One of the suggestion was use two dimensional MSE, which is involved x and y values. But not sure how to do that.
I will really appropriate any suggestion or command that can be helpful
I have a recurrent network (RNN) whose task is to learn to classify vectors (float32) in two classes. My model is really simple so far:
model = Sequential([
SimpleRNN(units=10, input_shape=(None, len_vector)),
Dense(1, activation="relu")
])
model.compile(loss='mse', optimizer='Adam', metrics=['accuracy'])
history = model.fit(X_train, y_train, epochs=30)
To train this network, I create a dataset with 1000 instances of sequences of vectors. When I create sequences with the same length each, the training works perfectly and the dataset has shape:
[<number of sequences>, <number of vectors in each sequence>, <number of floats in each vector>]
The problem is that my model must be able to work on sequence with various length. I don't know how (or even if it is possible) to create a numpy array where one dimension is not constant.
While searching a solution, I saw that setting the array dtype=object made it possible to assign list of different shapes to element of a numpy array, but the keras model will only accept dtype="float32".
Is there a way I can make this numpy array dataset? Or should I change the algorithm to train the model? Or is the only solution to pad sequences with nul vectors to unify their length?
(Thanks for the help. I'm fairly new to deep learning so I apologize if I'm asking for something obvious.)
Use Ragged Tensors, they provide you to make variable length inputs,
import numpy as np
_input = tf.keras.layers.Input(shape=(None, 100))
lstm = tf.keras.layers.LSTM(20,)(_input)
func = tf.keras.backend.function(inputs=_input, outputs=lstm)
rt = tf.ragged.constant([np.random.randn(1,34,100),
np.random.randn(1,55,100) ,
np.random.randn(1,60,100) ,
np.random.randn(1,70,100)])
func(rt[1])
I understand how to use the Keras Embedding layer in case there is a single text feature like in IMDB review classification. However, I am confused how to use the Embedding Layers when I have a Classification problem, where there are more than a single text feature. For example, I have a dataset with 2 text features Diagnosis Text, and Requested Procedure and the label is binary class (1 for approved, 0 for not approved). In the example below, x_train has 2 columns Diagnosis and Procedure, unlike the IMDB dataset. Do I need to create 2 Embedding layers, one for Diagnosis, and Procedure? If so, what code changes would be required?
x_train = preprocessing.sequences.pad_sequences(x_train, maxlen=20)
x_test = preprocessing.sequences.pad_sequences(x_test, maxlen=20)
model = Sequential()
model.add(Embedding(10000,8,input_length=20)
model.add(Flatten())
model.add(Dense(1, activation='sigmoid')
model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])
model.fit(x_train, y_train, epochs=10, batch_size=32, validation_split=0.2)
You have some choices, you could concatenate the the two features into one
and create a single embedding for both of them. Here is the logic
all_features = np.hstack(X['diag'] + X['proc'])
X = pad_sequence(all_features, max_len)
# build model as usual, as you can see on a single embedding layer is
# needed.
or you can use the Functional api and build multiple input model
diag_inp = Input()
diag_emb = Embedding(512)(diag_input)
proc_inp = Input()
proc_emb = Embedding(512)(proc_input)
# concatenate them to makes a single vector per sample
merged = Concatenate()[diag_emb, proc_emb]
out = Dense(2, activation='sigmoid')(merged)
model = Model(inputs=[diag_inp, proc_inp], outputs=[out])
That is you can learn an embedding for the concatenation or you can learn
multiple embeddings and concatenate them while training.
I have read plenty of posts for this point. They are inconsistent with each other and every answer seems to have a different explanation so I thought to ask based on my analyzing of all of them.
As Keras RNN documentation states, the input shape is always in this form (batch_size, timesteps, input_dim). I am a bit confused about that but I guess, not sure though, that input_dim is always 1 while timesteps depends on your problem (could be the data dimension as well). Is that roughly correct?
The reason for this question is that I always get an error when trying to change the value of input_dim to be my dataset dimension (as input_dim sounds like that!!), so I made an assumption that input_dim represent the shape of the input vector to LSTM at a time. Am I wrong again?
C = C.reshape((C.shape[0], C.shape[1], 1))
tr_C, ts_C, tr_r, ts_r = train_test_split(C, r, train_size=.8)
batch_size = 1000
print('Build model...')
model = Sequential()
model.add(LSTM(8, batch_input_shape=(batch_size, C.shape[1], 1), stateful=True, activation='relu'))
model.add(Dense(1, activation='relu'))
print('Training...')
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(tr_C, tr_r,
batch_size=batch_size, epochs=1,
shuffle=True, validation_data=(ts_C, ts_r))
Thanks!
Indeed, input_dim is the shape of the input vector at a time. In other words, input_dim is the number of the input features.
It's not necessarily 1, though. If you're working with more than one var, it can be any number.
Suppose you have 10 sequences, each sequence has 200 time steps, and you're measuring just a temperature. Then you have one feature:
input_shape = (200,1) -- notice that the batch size (number of sequences) is ignored here
batch_input_shape = (10,200,1) -- only in specific cases, like stateful = True, you will need a batch input shape.
Now suppose you're measuring not only temperature, but also pressure and volume. Now you've got three input features:
input_shape = (200,3)
batch_input_shape = (10,200,3)
In other words, the first dimension is the number of different sequences. The second is the length of the sequence (how many measures along time). And the last is how many vars at each time.
my question is quite closely related to this question but also goes beyond it.
I am trying to implement the following LSTM in Keras where
the number of timesteps be nb_tsteps=10
the number of input features is nb_feat=40
the number of LSTM cells at each time step is 120
the LSTM layer is followed by TimeDistributedDense layers
From the question referenced above I understand that I have to present the input data as
nb_samples, 10, 40
where I get nb_samples by rolling a window of length nb_tsteps=10 across the original timeseries of shape (5932720, 40). The code is hence
model = Sequential()
model.add(LSTM(120, input_shape=(X_train.shape[1], X_train.shape[2]),
return_sequences=True, consume_less='gpu'))
model.add(TimeDistributed(Dense(50, activation='relu')))
model.add(Dropout(0.2))
model.add(TimeDistributed(Dense(20, activation='relu')))
model.add(Dropout(0.2))
model.add(TimeDistributed(Dense(10, activation='relu')))
model.add(Dropout(0.2))
model.add(TimeDistributed(Dense(3, activation='relu')))
model.add(TimeDistributed(Dense(1, activation='sigmoid')))
Now to my question (assuming the above is correct so far):
The binary responses (0/1) are heavily imbalanced and I need to pass a class_weight dictionary like cw = {0: 1, 1: 25} to model.fit(). However I get an exception class_weight not supported for 3+ dimensional targets. This is because I present the response data as (nb_samples, 1, 1). If I reshape it into a 2D array (nb_samples, 1) I get the exception Error when checking model target: expected timedistributed_5 to have 3 dimensions, but got array with shape (5932720, 1).
Thanks a lot for any help!
I think you should use sample_weight with sample_weight_mode='temporal'.
From the Keras docs:
sample_weight: Numpy array of weights for the training samples, used
for scaling the loss function (during training only). You can either
pass a flat (1D) Numpy array with the same length as the input samples
(1:1 mapping between weights and samples), or in the case of temporal
data, you can pass a 2D array with shape (samples, sequence_length),
to apply a different weight to every timestep of every sample. In this
case you should make sure to specify sample_weight_mode="temporal" in
compile().
In your case you would need to supply a 2D array with the same shape as your labels.
If this is still an issue.. I think the TimeDistributed Layer expects and returns a 3D array (kind of similar to if you have return_sequences=True in the regular LSTM layer). Try adding a Flatten() layer or another LSTM layer at the end before the prediction layer.
d = TimeDistributed(Dense(10))(input_from_previous_layer)
lstm_out = Bidirectional(LSTM(10))(d)
output = Dense(1, activation='sigmoid')(lstm_out)
Using temporal is a workaround. Check out this stack. The issue is also documented on github.