The Reshape layer is not working how I would expect. In the example below, I think the last line should return a tensor object of shape [5,1]. However an error is thrown, stating that a shape [5] tensor cannot be reshaped into a size [5,5,1] tensor.
>>> from keras.layers import Reshape
>>> from keras import backend as K
>>> import numpy as np
>>> x = K.constant(np.array([1,2,3,4,5]))
>>> K.eval(x)
array([1., 2., 3., 4., 5.], dtype=float32)
>>> Reshape(target_shape=(5,1))(x)
...
ValueError: Cannot reshape a tensor with 5 elements to
shape [5,5,1] (25 elements) for 'reshape_3/Reshape' (op:
'Reshape') with input shapes: [5], [3] and with input
tensors computed as partial shapes: input[1] = [5,5,1].
Can someone kindly explain how the Reshape layer works (i.e. why it's adding the extra dim) and how to do the process of reshaping a vector into a matrix?
Thanks
User Reshape(target_shape=(1,))(x)
The batch_size is implied in the entire model and ignored from the beginning to the end.
If you do want to access the batch size, use a K.reshape(x,(5,1)).
Keras is not supposed to be used without creating a model made entirely of layers.
Related
1d-convolution is pretty simple when it is done by hand. However, I want to implement what is done here using nn.Conv1d and it is not simple for me to do it. In this example h=[1,2,-1], x=[4,1,2,5] and the output is going to be y=[4,9,0,8,8,-5]. To do it using Pytorch we need to define h=nn.Conv1d(in, out, k) and x=torch.tensor(*) and y=h(x) should be the result.
Note: please do not use nn.Conv2d to implement it.
First, you should be aware that the term "convolution" used in basically all literature related to convolutional neural networks (CNNs) actually corresponds to the correlation operation not the convolution operation.
The only difference (for real-valued inputs) between correlation and convolution is that in convolution the kernel is flipped/mirrored before sliding it across the signal, whereas in correlation no such flipping occurs.
There are also some extra operations that convolution layers in CNNs perform that are not part of the definition of convolution. They apply an offset (a.k.a. bias), they operate on mini-batches, and they map multi-channel inputs to multi-channel outputs.
Therefore, in order to recreate a convolution operation using a convolution layer we should (i) disable bias, (ii) flip the kernel, and (iii) set batch-size, input channels, and output channels to one.
For example, a PyTorch implementation of the convolution operation using nn.Conv1d looks like this:
import torch
from torch import nn
x = torch.tensor([4, 1, 2, 5], dtype=torch.float)
k = torch.tensor([1, 2, -1], dtype=torch.float)
# Define these constants to differentiate the various usages of "1".
BATCH_SIZE, IN_CH, OUT_CH = 1, 1, 1
# Pad with len(k)-1 zeros to ensure all non-zero outputs are computed.
h = nn.Conv1d(IN_CH, OUT_CH, kernel_size=len(k), padding=len(k) - 1, bias=False)
# Copy flipped k into h.weight.
# h.weight is shape (OUT_CH, IN_CH, kernel_size), reshape k accordingly.
# Perform copy inside no_grad context to avoid autograd issues.
with torch.no_grad():
h.weight.copy_(torch.flip(k, dims=[0]).reshape(OUT_CH, IN_CH, -1))
# Input shape to h is assumed to be (BATCH_SIZE, IN_CH, SIGNAL_LENGTH), reshape x accordingly.
# Output shape of h is (BATCH_SIZE, OUT_CH, OUTPUT_LENGTH), reshape output to 1D signal.
y = h(x.reshape(BATCH_SIZE, IN_CH, -1)).reshape(-1)
which results in
>>> print(y)
tensor([ 4., 9., 0., 8., 8., -5.], grad_fn=<ViewBackward>)
I'm using the estimator library of tensorflow on python. I want to train a student network by using a pre-trained teacher.I'm facing the following issue.
train_input_fn = tf.estimator.inputs.numpy_input_fn(
x={"x": train_data},
y=train_labels,
batch_size=100,
num_epochs=None,
shuffle=True)
student_classifier.train(
input_fn=train_input_fn,
steps=20,
hooks=None)
This code returns a generator object that is passed to a student classifier. Inside the generator, we have the inputs and labels (in batches of 100) as tensors. The problem is, I want to pass the same values to the teacher model and extract its softmax outputs. But unfortunately, the model input requires a numpy array as follows
student_classifier = tf.estimator.Estimator(
model_fn=student_model_fn, model_dir="./models/mnist_student")
def student_model_fn(features, labels, mode):
sess=tf.InteractiveSession()
tf.train.start_queue_runners(sess)
data=features['x'].eval()
out=labels.eval()
sess.close()
input_layer = tf.reshape(features["x"], [-1, 28, 28, 1])
eval_teacher_fn = tf.estimator.inputs.numpy_input_fn(
x={"x":data},
y=out,
num_epochs=1,
shuffle=False)
This requires x and y to be numpy arrays so I converted it via using such as ugly hack of using a session to convert tensor to numpy. Is there a better way of doing this?
P.S. I tried tf.estimator.Estimator.get_variable_value() but it retrieves weights from the model, not the input and output
Convert Tensor to Numpy_array using tf.make_ndarray.
tf.make_ndarray(), Create a numpy ndarray with the same shape and data as the tensor.
Sample working code:
import tensorflow as tf
a = tf.constant([[1,2,3],[4,5,6]])
proto_tensor = tf.make_tensor_proto(a)
tf.make_ndarray(proto_tensor)
output:
array([[1, 2, 3],
[4, 5, 6]], dtype=int32)
# output has shape (2,3)
As a follow-up from this question:
Concatenate input with constant vector in keras
I am trying to use the suggested solution:
constant=K.variable(np.ones((1,10, 5)))
constant = K.repeat_elements(constant,rep=batch_size,axis=0)
And got the following Error:
NameError: name 'batch_size' is not defined
I do not see how one define within the keras model the batch_size [not explicitly] so that one can concatenate a symbolic layer and a constant layer in order to use them as an input layer.
To get the dynamic batch size:
batch_size = K.shape(your_tensor)[0]
But K.repeat_elements() doesn't accept Tensor values for rep. You can however produce the same result using K.tile():
from keras.models import *
from keras import backend as K
import numpy as np
a = Input(shape=(10, 5))
batch_size = K.shape(a)[0]
constant = K.variable(np.ones((1,10, 5)))
constant = K.tile(constant, (batch_size, 1, 1))
print(constant)
# Tensor("Tile:0", shape=(?, 10, 5), dtype=float32)
I have a series of processed audio files I am using as input into a CNN using Keras. Does the Keras 1D Convolutional layer support variable sequence lengths? The Keras documentation makes this unclear.
https://keras.io/layers/convolutional/
At the top of the documentation it mentions you can use (None, 128) for variable-length sequences of 128-dimensional vectors. Yet at the bottom it declares that the input shape must be a
3D tensor with shape: (batch_size, steps, input_dim)
Given the following example how should I input sequences of variable length into the network
Lets say I have two examples (a and b) containing X 1 dimensional vectors of length 100 that I want to feed into the 1DConv layer as input
a.shape = (100, 100)
b.shape = (200, 100)
Can I use an input shape of (2, None, 100)? Do I need to concatenate these tensors into c where
c.shape = (300, 100)
Then reshape it to be something
c_reshape.shape = (3, 100, 100)
Where 3 is the batch size, 100, is the number of steps, and the second 100 is the input size? The documentation on the input vector is not very clear.
Keras supports variable lengths by using None in the respective dimension when defining the model.
Notice that often input_shape refers to the shape without the batch size.
So, the 3D tensor with shape (batch_size, steps, input_dim) suits perfectly a model with input_shape=(steps, input_dim).
All you need to make this model accept variable lengths is use None in the steps dimension:
input_shape=(None, input_dim)
Numpy limitation
Now, there is a numpy limitation about variable lengths. You cannot create a numpy array with a shape that suits variable lengths.
A few solutions are available:
Pad your sequences with dummy values until they all reach the same size so you can put them into a numpy array of shape (batch_size, length, input_dim). Use Masking layers to disconsider the dummy values.
Train with separate numpy arrays of shape (1, length, input_dim), each array having its own length.
Group your images by sizes into smaller arrays.
Be careful with layers that don't support variable sizes
In convolutional models using variable sizes, you can't for instance, use Flatten, the result of the flatten would have a variable size if this were possible. And the following Dense layers would not be able to have a constant number of weights. This is impossible.
So, instead of Flatten, you should start using GlobalMaxPooling1D or GlobalAveragePooling1D layers.
I would like to use Keras to train a multi-input NN with a batch of training data, but I'm not able to pass a set of input and output samples to execute a fit or a train_on_batch on the model.
My NN is defined as following:
i1 = keras.layers.Input(shape=(2,))
i2 = keras.layers.Input(shape=(2,))
i3 = keras.layers.Input(shape=(2,))
i_layer = keras.layers.Dense(2, activation='sigmoid')
embedded_i1 = i_layer(i1)
embedded_i2 = i_layer(i2)
embedded_i3 = i_layer(i3)
middle_concatenation = keras.layers.concatenate([embedded_i1, embedded_i2, embedded_i3], axis=1)
out = keras.layers.Dense(1, activation='sigmoid')(middle_concatenation)
model = keras.models.Model(inputs=[i1, i2, i3], outputs=out)
model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])
For example, an instance of the input (successfully used for predict the output) is the following:
[array([[0.1, 0.2]]), array([[0.3, 0.5]]), array([[0.1, 0.3]])]
But when I try to train my model with:
inputs = [[np.array([[0.1, 0.2]]), np.array([[0.3, 0.5]]), np.array([[0.1, 0.3]])],
[np.array([[0.2, 0.1]]), np.array([[0.5, 0.3]]), np.array([[0.3, 0.1]])]
]
outputs = np.ones(len(inputs))
model.fit(inputs, outputs)
I get this error:
ValueError: Error when checking model input: you are passing a list as input to your model, but the model expects a list of 3 Numpy arrays instead. The list you passed was: [[array([[ 0.1, 0.2]]), array([[ 0.3, 0.5]]), array([[ 0.1, 0.3]])], [array([[ 0.2, 0.1]]), array([[ 0.5, 0.3]]), array([[ 0.3, 0.1]])]]
What am I doing wrong?
How can I train a multi-input NN with a batch of input/output samples?
Thank you!
the problem is just incorrect formatting. You can't pass a list to keras, only numpy arrays, so when you have your data structured like
inputs = [[np.array([[0.1, 0.2]]), np.array([[0.3, 0.5]]), np.array([[0.1, 0.3]])],
[np.array([[0.2, 0.1]]), np.array([[0.5, 0.3]]), np.array([[0.3, 0.1]])]
]
You need to pass one list element into your model at a time. You will also need to pass one output value to the model at a time. To do this, structure you outputs like this
outputs = [np.ones(1) for x in inputs]
[array([ 1.]), array([ 1.])]
Then you can loop over the the fit function like this
for z in range(0,len(inputs)):
model.fit(inputs[z],outputs[z],batch_size=1)
you can also replace model.fit with model.train_on_batch() instead, see docs
however to avoid the loop, you could just have 3 numpy arrays stored in your inputs list and have you single outputs as a numpy array. If you only want to train on a single batch at a time, you could set your batch size to do that.
inputs = [np.array([[0.1, 0.2],[0.2, 0.1]]), np.array([[0.3, 0.5],[0.5, 0.3]]), np.array([[0.1, 0.3],[0.3, 0.1]])]
outputs = np.ones(inputs[0].shape[0])
model.fit(inputs,outputs,batch_size=1)
The problem is that right now you are using a list of lists as input, although keras expects a list of arrays.
You need to convert your list so that it looks like [array_inputs_1, array_inputs_2, array_inputs_3], where each input array is the array of inputs you would pass the model if it had only that input layer, you just put the 3 of them inside a list.
Using your data the correct input should be:
[np.array([[0.1, 0.2], [0.2, 0.1]]),
np.array([[0.3, 0.5], [0.5, 0.3]]),
np.array([[0.1, 0.3], [0.1, 0.3]])]
This way, as long as all 3 input arrays have the same number of elements, keras will know how to divide tham into batches.