TimeDistributed of a KerasLayer in Tensorflow 2.0 - conv-neural-network

Question

I'm trying to build a CNN + RNN using a pre-trained model from tensorflow-hub:
base_model = hub.KerasLayer('https://tfhub.dev/google/imagenet/resnet_v2_50/feature_vector/4', input_shape=(244, 244, 3)
base_model.trainable = False
model = Sequential()
model.add(TimeDistributed(base_model, input_shape=(15, 244, 244, 3)))
model.add(LSTM(512))
model.add(Dense(256, activation='relu'))
model.add(Dense(3, activation='softmax'))
adam = Adam(learning_rate=learning_rate)
model.compile(loss='categorical_crossentropy' , optimizer=adam , metrics=['accuracy'])
model.summary()
and this what I get:
2020-01-29 16:1
6:37.585888: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2494000000 Hz
2020-01-29 16:16:37.586205: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3b553f0 executing computations on platform Host. Devices:
2020-01-29 16:16:37.586231: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Host, Default Version
Traceback (most recent call last):
File "./RNN.py", line 45, in <module>
model.add(TimeDistributed(base_model, input_shape=(None, 244, 244, 3)))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/base.py", line 457, in _method_wrapper
result = method(self, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/sequential.py", line 178, in add
layer(x)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 842, in __call__
outputs = call_fn(cast_inputs, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/wrappers.py", line 256, in call
output_shape = self.compute_output_shape(input_shape).as_list()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/wrappers.py", line 210, in compute_output_shape
child_output_shape = self.layer.compute_output_shape(child_input_shape)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 639, in compute_output_shape
raise NotImplementedError
NotImplementedError
any suggestions?
Is it possible to convert a KerasLayer to Conv2D,... layers?

Answer 1

It doesn't seem like you can use the TimeDistributed layer for this problem. However, as you don't want Resnet to train and just need the output, you can do the following to avoid the TimeDistributed layer.
Instead of model.add(TimeDistributed(base_model, input_shape=(15, 244, 244, 3))), do
Option 1
# 2048 is the output size
model.add(
Lambda(
lambda x: tf.reshape(base_model(tf.reshape(x, [-1, 244, 244,3])),[-1, 15, 2048])
, input_shape=(15, 244, 244, 3))
)
Option 2
If you don't want to depend too much on the output shape (this sacrifices performance though).
model.add(
Lambda(
lambda x: tf.stack([base_model(xx) for xx in tf.unstack(x, axis=1) ], axis=1)
, input_shape=(15, 244, 244, 3))
)

Answer 2

See my answer here. The error is thrown because it can't compute the output shape, you might be able to solve your problem by implementing compute_output_shape manually.

Answer 3

If someone faces this issue, as #josef mentioned, the problem is that compute_output_shape is not implemented. You can overcome the problem by specifying the output shape of the layer:
extractor = tfhub.KerasLayer("https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/feature_vector/4",
input_shape=(IMG_SIZE, IMG_SIZE, CHANNELS),
output_shape=(EXTRACTOR_SIZE),
trainable=False)
model.add(keras.layers.Lambda(lambda x: extractor(x)))
As you can see, I also had to wrap the layer inside of a Lambda function since it looks like you can't wrap KerasLayer directly inside of a TimeDistributed layer.
In the code, EXTRACTOR_SIZE is 1280 but that is specific of MobileNet.
This workaround worked for me.