I'm using the estimator library of tensorflow on python. I want to train a student network by using a pre-trained teacher.I'm facing the following issue.
train_input_fn = tf.estimator.inputs.numpy_input_fn(
x={"x": train_data},
y=train_labels,
batch_size=100,
num_epochs=None,
shuffle=True)
student_classifier.train(
input_fn=train_input_fn,
steps=20,
hooks=None)
This code returns a generator object that is passed to a student classifier. Inside the generator, we have the inputs and labels (in batches of 100) as tensors. The problem is, I want to pass the same values to the teacher model and extract its softmax outputs. But unfortunately, the model input requires a numpy array as follows
student_classifier = tf.estimator.Estimator(
model_fn=student_model_fn, model_dir="./models/mnist_student")
def student_model_fn(features, labels, mode):
sess=tf.InteractiveSession()
tf.train.start_queue_runners(sess)
data=features['x'].eval()
out=labels.eval()
sess.close()
input_layer = tf.reshape(features["x"], [-1, 28, 28, 1])
eval_teacher_fn = tf.estimator.inputs.numpy_input_fn(
x={"x":data},
y=out,
num_epochs=1,
shuffle=False)
This requires x and y to be numpy arrays so I converted it via using such as ugly hack of using a session to convert tensor to numpy. Is there a better way of doing this?
P.S. I tried tf.estimator.Estimator.get_variable_value() but it retrieves weights from the model, not the input and output
Convert Tensor to Numpy_array using tf.make_ndarray.
tf.make_ndarray(), Create a numpy ndarray with the same shape and data as the tensor.
Sample working code:
import tensorflow as tf
a = tf.constant([[1,2,3],[4,5,6]])
proto_tensor = tf.make_tensor_proto(a)
tf.make_ndarray(proto_tensor)
output:
array([[1, 2, 3],
[4, 5, 6]], dtype=int32)
# output has shape (2,3)
Related
I'm wondering about how to do the following thing:
If I have a torch.tensor x with shape (4,5,1) how can apply a neural network using PyTorch on the last dimension?
Using the standard procedure, the model is flattening the entire tensor into some new tensor of shape (20,1) but this is not actually what I want.
Let's say we want some output features of dimension 64, then I would like to obtain a new object of shape (4,5,64)
import torch
import torch.nn as nn
x = torch.randn(4, 5, 1)
print(x.size())
# https://pytorch.org/docs/stable/generated/torch.nn.Linear.html
m = nn.Linear(1, 64)
y = m(x)
print(y.size())
result:
torch.Size([4, 5, 1])
torch.Size([4, 5, 64])
I need to feed a neural network model with data read from a generator. For instance, consider this:
import numpy as np
def gener():
for i in range(100):
yield np.random.sample((28,28))
I am trying to use this:
import tensorflow as tf
train_data = tf.data.Dataset.from_generator(gener,output_types=tf.float32)
train_labels = np.random.randint(0, 10, size=100)
When creating a first model layer and fitting, say like this:
from keras.layers import Conv2D
from keras.models import Sequential
model = Sequential()
model.add(Conv2D(filters=32, kernel_size=(3,3), activation='relu', input_shape=(28, 28,1,)))
model.compile(optimizer=Adam(), loss='categorical_crossentropy', metrics=['accuracy'])
history = model.fit(x=train_data,y=labels_train, epochs=7, verbose=1
I get this error:
AttributeError: 'FlatMapDataset' object has no attribute 'ndim'
What am I missing ? What can I do to fit data from a generator to my model ?
When using generators each item should be a pair of the input data and the label. Like this:
def gener():
for i in range(100):
yield (np.random.sample((28, 28, 1)),
tf.one_hot(np.random.randint(0, 10), 10))
This corresponds to your train_data and train_labels which I have also one-hot encoded for you.
To create a dataset from it you need to explicitly declare the types that are involved:
train_ds = Dataset.from_generator(gener,
(tf.float32, tf.int32),
(tf.TensorShape([28, 28, 1]),
tf.TensorShape([10])))
Then you need to manually create mini-batches:
train_ds = train_ds.batch(4)
Then you need to call it like this:
model.fit(x = train_ds, epochs = 7, verbose = 1)
I.e don't specify the y parameter.
This question has been answered for Tensorflow 1, eg: How to Properly Combine TensorFlow's Dataset API and Keras?, but this answer hasn't helped for my use case.
Below is an example of a model with three float32 inputs and one float32 output. I have a large amount of data that doesn't all fit into memory at once, so it's split into separate files. I'm trying to use the Dataset API to train a model by bringing in a portion of the training data at once.
import tensorflow as tf
import tensorflow.keras.layers as layers
import numpy as np
# Create TF model of a given architecture (number of hidden layers, layersize, #outputs, activation function)
def create_model(h=2, l=64, activation='relu'):
model = tf.keras.Sequential([
layers.Dense(l, activation=activation, input_shape=(3,), name='input_layer'),
*[layers.Dense(l, activation=activation) for _ in range(h)],
layers.Dense(1, activation='linear', name='output_layer')])
return model
# Load data (3 X variables, 1 Y variable) split into 5 files
# (for this example, just create a list 5 numpy arrays)
list_of_training_datasets = [np.random.rand(10,4).astype(np.float32) for _ in range(5)]
validation_dataset = np.random.rand(30,4).astype(np.float32)
def data_generator():
for data in list_of_training_datasets:
x_data = data[:, 0:3]
y_data = data[:, 3:4]
yield((x_data,y_data))
# prepare model
model = create_model(h=2,l=64,activation='relu')
model.compile(loss='mse', optimizer=tf.keras.optimizers.Adam())
# load dataset
dataset = tf.data.Dataset.from_generator(data_generator,(np.float32,np.float32))
# fit model
model.fit(dataset, epochs=100, validation_data=(validation_dataset[:,0:3],validation_dataset[:,3:4]))
Running this, I get the error:
ValueError: Cannot take the length of shape with unknown rank.
Does anyone know how to get this working? I would also like to be able to use the batch dimension, to load two data files at a time, for example.
You need to need to specify the shapes of the your dataset along with the return data types like this.
dataset = tf.data.Dataset.from_generator(data_generator,
(np.float32,np.float32),
((None, 3), (None, 1)))
The following works, but I don't know if this is the most efficient.
As far as I understand, if your training dataset is split into 10 pieces, then you should set steps_per_epoch=10. This ensures that each epoch will step through all data once. As far as I understand, dataset.repeat() is needed because the dataset iterator is "used up" after the first epoch. .repeat() ensures that the iterator gets created again after being used up.
import numpy as np
import tensorflow.keras.layers as layers
import tensorflow as tf
# Create TF model of a given architecture (number of hidden layers, layersize, #outputs, activation function)
def create_model(h=2, l=64, activation='relu'):
model = tf.keras.Sequential([
layers.Dense(l, activation=activation, input_shape=(3,), name='input_layer'),
*[layers.Dense(l, activation=activation) for _ in range(h)],
layers.Dense(1, activation='linear', name='output_layer')])
return model
# Load data (3 X variables, 1 Y variable) split into 5 files
# (for this example, just create a list 5 numpy arrays)
list_of_training_datasets = [np.random.rand(10,4).astype(np.float32) for _ in range(5)]
steps_per_epoch = len(list_of_training_datasets)
validation_dataset = np.random.rand(30,4).astype(np.float32)
def data_generator():
for data in list_of_training_datasets:
x_data = data[:, 0:3]
y_data = data[:, 3:4]
yield((x_data,y_data))
# prepare model
model = create_model(h=2,l=64,activation='relu')
model.compile(loss='mse', optimizer=tf.keras.optimizers.Adam())
# load dataset
dataset = tf.data.Dataset.from_generator(data_generator,output_types=(np.float32,np.float32),
output_shapes=(tf.TensorShape([None,3]), tf.TensorShape([None,1]))).repeat()
# fit model
model.fit(dataset.as_numpy_iterator(), epochs=10,steps_per_epoch=steps_per_epoch,
validation_data=(validation_dataset[:,0:3],validation_dataset[:,3:4]))
Input tensors to a Model must be Keras tensors. Found:
Tensor("my_layer/Identity:0", shape=(?, 10, 1152, 16), dtype=float32)
(missing Keras metadata).
Hi, I get this error when trying to take one layer's intermediate variable to use it as input to a parallel network. Such that one layer's intermediate variable will be input to the other network.
def call(self, inputs, training=None):
inputs_expand = K.expand_dims(inputs, 1)
tensor_b = K.tile(inputs_expand, [1, 16, 1, 1])
tensor_a = K.map_fn(lambda x: K.batch_dot(x, self.Weights, [2, 3]), elems=tensor_b)
# I need this tensor_a
# I tried many things but ended putting it to member variable.
self.tensor_a = K.identity(inputs_hat)
....
outside when trying to build the parallel model I do this
a_model = models.Model([my_layer.tensor_a],[my_layer.c])
I could not find any good solution to this problem? How can I turn the tensor into K.tensor??
I would like to use Keras to train a multi-input NN with a batch of training data, but I'm not able to pass a set of input and output samples to execute a fit or a train_on_batch on the model.
My NN is defined as following:
i1 = keras.layers.Input(shape=(2,))
i2 = keras.layers.Input(shape=(2,))
i3 = keras.layers.Input(shape=(2,))
i_layer = keras.layers.Dense(2, activation='sigmoid')
embedded_i1 = i_layer(i1)
embedded_i2 = i_layer(i2)
embedded_i3 = i_layer(i3)
middle_concatenation = keras.layers.concatenate([embedded_i1, embedded_i2, embedded_i3], axis=1)
out = keras.layers.Dense(1, activation='sigmoid')(middle_concatenation)
model = keras.models.Model(inputs=[i1, i2, i3], outputs=out)
model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])
For example, an instance of the input (successfully used for predict the output) is the following:
[array([[0.1, 0.2]]), array([[0.3, 0.5]]), array([[0.1, 0.3]])]
But when I try to train my model with:
inputs = [[np.array([[0.1, 0.2]]), np.array([[0.3, 0.5]]), np.array([[0.1, 0.3]])],
[np.array([[0.2, 0.1]]), np.array([[0.5, 0.3]]), np.array([[0.3, 0.1]])]
]
outputs = np.ones(len(inputs))
model.fit(inputs, outputs)
I get this error:
ValueError: Error when checking model input: you are passing a list as input to your model, but the model expects a list of 3 Numpy arrays instead. The list you passed was: [[array([[ 0.1, 0.2]]), array([[ 0.3, 0.5]]), array([[ 0.1, 0.3]])], [array([[ 0.2, 0.1]]), array([[ 0.5, 0.3]]), array([[ 0.3, 0.1]])]]
What am I doing wrong?
How can I train a multi-input NN with a batch of input/output samples?
Thank you!
the problem is just incorrect formatting. You can't pass a list to keras, only numpy arrays, so when you have your data structured like
inputs = [[np.array([[0.1, 0.2]]), np.array([[0.3, 0.5]]), np.array([[0.1, 0.3]])],
[np.array([[0.2, 0.1]]), np.array([[0.5, 0.3]]), np.array([[0.3, 0.1]])]
]
You need to pass one list element into your model at a time. You will also need to pass one output value to the model at a time. To do this, structure you outputs like this
outputs = [np.ones(1) for x in inputs]
[array([ 1.]), array([ 1.])]
Then you can loop over the the fit function like this
for z in range(0,len(inputs)):
model.fit(inputs[z],outputs[z],batch_size=1)
you can also replace model.fit with model.train_on_batch() instead, see docs
however to avoid the loop, you could just have 3 numpy arrays stored in your inputs list and have you single outputs as a numpy array. If you only want to train on a single batch at a time, you could set your batch size to do that.
inputs = [np.array([[0.1, 0.2],[0.2, 0.1]]), np.array([[0.3, 0.5],[0.5, 0.3]]), np.array([[0.1, 0.3],[0.3, 0.1]])]
outputs = np.ones(inputs[0].shape[0])
model.fit(inputs,outputs,batch_size=1)
The problem is that right now you are using a list of lists as input, although keras expects a list of arrays.
You need to convert your list so that it looks like [array_inputs_1, array_inputs_2, array_inputs_3], where each input array is the array of inputs you would pass the model if it had only that input layer, you just put the 3 of them inside a list.
Using your data the correct input should be:
[np.array([[0.1, 0.2], [0.2, 0.1]]),
np.array([[0.3, 0.5], [0.5, 0.3]]),
np.array([[0.1, 0.3], [0.1, 0.3]])]
This way, as long as all 3 input arrays have the same number of elements, keras will know how to divide tham into batches.