I would like to save and restor my tf.estimator model. Although I tried to follow other related issues on stackoverflow, I could not be successful. The following input_fn can provide the data to be predicted. But I do not know how to use it to save and restore the model to predict.
Btw, my return dataset has a shape of [batch_size, dim] where dtype is float32
def predict_input_fn(path, dim, batch_size):
dataset = ds.get_dataset(path,
dim)
dataset = dataset.batch(batch_size)
dataset = dataset.prefetch(1)
return dataset
What I have tried until so far is the following but it did not work as expected, could you please help me to save and restore such a model?
Trial
def serving_input_receiver_fn():
features = tf.placeholder(
dtype=tf.float32, shape=[None, batch_size])
fn = lambda x : precict_input_fn(path, dim, batch_size)
mapped_fn = tf.map_fn(fn, features)
return tf.estimator.export.ServingInputReceiver(mapped_fn, features)
estimator.export_savedmodel(model_save_path, serving_input_receiver_fn)
Error:
Failed to convert object of type <class 'tensorflow.python.data.ops.dataset_ops.PrefetchDataset'> to Tensor. Contents: <PrefetchDataset shapes: (?, 1024), types: tf.float32>. Consider casting elements to a supported type
Related
I'm trying to build a ML production pipeline using TFX, and currently working on Trainer module. I need to implement modelling in a separate file.
Here is part of pipeline, which handle training:
trainer = Trainer(
module_file=module_file,
custom_executor_spec=executor_spec.ExecutorClassSpec(GenericExecutor),
transformed_examples=transform.outputs['transformed_examples'],
schema=schema_gen.outputs['schema'],
transform_graph=transform.outputs['transform_graph'],
train_args=trainer_pb2.TrainArgs(num_steps=10000),
eval_args=trainer_pb2.EvalArgs(num_steps=5000))
And here is part of module_file, which define model:
_DENSE_FLOAT_FEATURE_KEYS = ['number_of_likes', 'number_of_comments', 'owner_influence']
def _build_keras_model() -> tf.keras.Model:
inputs = [
keras.layers.Input(shape=(1,), name=_transformed_name(f))
for f in _DENSE_FLOAT_FEATURE_KEYS
]
d = keras.layers.concatenate(inputs)
for _ in range(int(4)):
d = keras.layers.Dense(8, activation='relu')(d)
output = keras.layers.Dense(1)(d)
model = keras.Model(inputs=inputs, outputs=output)
model.compile(loss='mean_absolute_error',
optimizer='adam',
metrics=['mean_absolute_error'])
model.summary(print_fn=absl.logging.info)
return model
And another function, which is called by Trainer:
def run_fn(fn_args: TrainerFnArgs):
tf_transform_output = tft.TFTransformOutput(fn_args.transform_output)
train_dataset = _input_fn(fn_args.train_files, tf_transform_output,
batch_size=_TRAIN_BATCH_SIZE)
eval_dataset = _input_fn(fn_args.eval_files, tf_transform_output,
batch_size=_EVAL_BATCH_SIZE)
mirrored_strategy = tf.distribute.MirroredStrategy()
with mirrored_strategy.scope():
model = _build_keras_model()
model.fit(
train_dataset,
steps_per_epoch=fn_args.train_steps,
validation_data=eval_dataset,
validation_steps=fn_args.eval_steps)
When I run this, I got such error:
TypeError: Failed to convert object of type <class 'tensorflow.python.framework.sparse_tensor.SparseTensor'> to Tensor. Contents: SparseTensor(indices=Tensor("DeserializeSparse_3:0", shape=(None, 2), dtype=int64, device=/job:localhost/replica:0/task:0/device:CPU:0), values=Tensor("DeserializeSparse_3:1", shape=(None,), dtype=float32, device=/job:localhost/replica:0/task:0/device:CPU:0), dense_shape=Tensor("stack_3:0", shape=(2,), dtype=int64, device=/job:localhost/replica:0/task:0/device:CPU:0)). Consider casting elements to a supported type.[while running 'Run[Trainer]']
Whole logic and code I took from official TFX examples and guides. After concatenation of inputs, data comes to model` layers in Tensor format, not in SparseTensor. I can not figure out what is going there, and there are no such cases in web :(
Will be very grateful for the help!
I am playing with Variational Autoencoders and would like to adapt a Keras example found on GitHub.
Basically, the example is very simple based on mnist dataset and I would like to implement on a more difficult set as it is more realistic.
Code I'm trying to modify:
vae_dfc.fit(
x_train,
epochs=epochs,
steps_per_epoch=train_size//batch_size,
validation_data=(x_val),
validation_steps=val_size//batch_size,
verbose=1
)
With more complex datasets it is nearly impossible to load everything on memory so we need to use fit_generator() to train the model. But it doesn't seem able to handle this:
image_generator = image.ImageDataGenerator(
rescale=1./255,
validation_split=0.2
)
train_generator = image_generator.flow_from_directory(
dir,
class_mode=None,
color_mode='rgb',
target_size=(ORIGINAL_SHAPE[0], ORIGINAL_SHAPE[1]),
batch_size=BATCH_SIZE,
subset='training'
)
vae.fit_generator(
train_generator,
epochs=EPOCHS,
steps_per_epoch=train_generator.samples // BATCH_SIZE,
validation_data=validation_generator,
validation_steps=validation_generator.samples // BATCH_SIZE
)
My understanding is that class_mode=None is producing an output similar to the original simple example, but the fit_generator() is unable to handle this. Are there any workarounds to deal with the fit generator error?
Configurations:
tensorflow-gpu==1.12.0
Python 3.6
Windows 10
Cuda 9.0
Full error:
File "xxx\venv\lib\site-packages\tensorflow\python\keras\engine\training.py",
line 2177, in fit_generator
initial_epoch=initial_epoch)
File "xxx\venv\lib\site-packages\tensorflow\python\keras\engine\training_generator.py",
line 162, in fit_generator
'or (x, y). Found: ' + str(generator_output)) ValueError: Output of generator should be a tuple (x, y, sample_weight) or (x, y).
Found: [[[[0.48627454 0.34901962 0.2901961 ] ....]]]
An autoencoder needs outputs = inputs. It's different from not having outputs.
I believe you can try class_mode='input'.
If this doesn't work, you can create a wrapper generator for outputting both:
class AutoencGenerator(keras.utils.Sequence):
def __init__(self, originalGenerator):
self.generator = originalGenerator
def __len__(self):
return len(self.generator)
def __getitem__(self, i):
x = self.generator[i]
return x, x
def on_epoch_end(self):
self.generator.on_epoch_end() #this only if there is an on_epoch_end in the original
train_autoenc_generator = AutoencGenerator(train_generator)
Both options will need that your model has outputs, of course. If the model was created without outputs (unusual), make it output the results and use the loss function in model.compile(loss=the_loss).
Example of VAE
inputs = Input(shape)
means, sigmas = encoder(inputs)
def encode(x):
means, sigmas = x
randomSamples = tf.random_normal(K.shape(means)) #samples
encoded = (sigmas * randomSamples) + means
return encoded
encodings = Lambda(encode)([means, sigmas])
outputs = decoder(encodings)
kl_loss = some_tensor_function(means, sigmas)
VAE = Model(inputs, outputs)
VAE.add_loss(kl_loss)
VAE.compile(loss = 'mse', optimizer='adam')
Train with the generator:
VAE.fit_generator(train_autoenc_generator, ...)
Based on this post. I need some basic implementation help. Below you see my model using a Dropout layer. When using the noise_shape parameter, it happens that the last batch does not fit into the batch size creating an error (see other post).
Original model:
def LSTM_model(X_train,Y_train,dropout,hidden_units,MaskWert,batchsize):
model = Sequential()
model.add(Masking(mask_value=MaskWert, input_shape=(X_train.shape[1],X_train.shape[2]) ))
model.add(Dropout(dropout, noise_shape=(batchsize, 1, X_train.shape[2]) ))
model.add(Dense(hidden_units, activation='sigmoid', kernel_constraint=max_norm(max_value=4.) ))
model.add(LSTM(hidden_units, return_sequences=True, dropout=dropout, recurrent_dropout=dropout))
Now Alexandre Passos suggested to get the runtime batchsize with tf.shape. I tried to implement the runtime batchsize idea it into Keras in different ways but never working.
import Keras.backend as K
def backend_shape(x):
return K.shape(x)
def LSTM_model(X_train,Y_train,dropout,hidden_units,MaskWert,batchsize):
batchsize=backend_shape(X_train)
model = Sequential()
...
model.add(Dropout(dropout, noise_shape=(batchsize[0], 1, X_train.shape[2]) ))
...
But that did just give me the input tensor shape but not the runtime input tensor shape.
I also tried to use a Lambda Layer
def output_of_lambda(input_shape):
return (input_shape)
def LSTM_model_2(X_train,Y_train,dropout,hidden_units,MaskWert,batchsize):
model = Sequential()
model.add(Lambda(output_of_lambda, outputshape=output_of_lambda))
...
model.add(Dropout(dropout, noise_shape=(outputshape[0], 1, X_train.shape[2]) ))
And different variants. But as you already guessed, that did not work at all.
Is the model definition actually the correct place?
Could you give me a tip or better just tell me how to obtain the running batch size of a Keras model? Thanks so much.
The current implementation does adjust the according to the runtime batch size. From the Dropout layer implementation code:
symbolic_shape = K.shape(inputs)
noise_shape = [symbolic_shape[axis] if shape is None else shape
for axis, shape in enumerate(self.noise_shape)]
So if you give noise_shape=(None, 1, features) the shape will be (runtime_batchsize, 1, features) following the code above.
I am learning TensorFlow and am trying to train a BoostedTreesClassifier (premade estimator). However, I cannot get it to work with my bucketized columns. Below is my bucketized column:
age_bucket_column = tf.feature_column.bucketized_column(tf.numeric_column(key='age'), [20, 40, 60])
Here is my train input function (note features is a Pandas DataFrame):
def train_input_fn(features, labels, batch_size):
dataset = tf.data.Dataset.from_tensor_slices((dict(features), labels))
dataset = dataset.shuffle(buffer_size=1000).repeat(count=None).batch(batch_size)
return dataset.make_one_shot_iterator().get_next()
Here is my estimator:
boosted_trees_classifier = tf.estimator.BoostedTreesClassifier(
feature_columns=[age_bucket_column],
n_batches_per_layer=100
)
And here is my code to train it:
classifier.train(
input_fn=lambda: train_input_fn(train_X, train_y, 100),
steps=1000
)
However, when I run it, I get the following error:
ValueError: Tensor conversion requested dtype float32 for Tensor with dtype int64: 'Tensor("IteratorGetNext:13", shape=(?,), dtype=int64, device=/device:CPU:0)'
Note that when I run the same code but with another model (say a LinearClassifier or DNNClassifier) it works perfectly. What am I doing wrong? Thank you in advance!
This is probably because of your labels are of type int64. Cast them to float32
train_y = pd.Series(train_y , index=np.array(range(1, train_y.shape[0] + 1)), dtype=np.float32)
I'm passing in a [64,240] tf.array.
I want to reshape this to a [64,10,24] tf.array.
I've tried various different methods but I always get the same error.
... raise ValueError("Shape %s must have rank %d" % (self, rank))
ValueError: Shape (64, 240) must have rank 1
The code fails at:
self.x_expand[i] = tf.reshape(self.input_x[i],[num_classes,data_size])
I guess i'm making a basic mistake, hopefully someone can point out what it is...
A larger fragment of the code is:
class CNN(object):
def __init__(
self, sequence_length, num_classes, data_size,
filter_sizes, num_filters, l2_reg_lambda=0.0, batch_size=64):
# Placeholders for input, output and dropout
self.input_x = tf.placeholder(tf.int32, [batch_size, sequence_length], name="input_x")
self.input_y = tf.placeholder(tf.float32, [batch_size, num_classes], name="input_y")
self.dropout_keep_prob = tf.placeholder(tf.float32, name="dropout_keep_prob")
with tf.device('/cpu:0'), tf.name_scope("reshaping"):
self.x_expand = tf.placeholder(tf.int32, [batch_size, num_classes,data_size], name="expand_x")
for i in range(batch_size):
self.x_expand[i] = tf.reshape(self.input_x[i],[num_classes,data_size])
self.x_expanded = tf.expand_dims(self.x_expanded, -1)
It seems you want self.x_expand to be the input reshaped I think the following might be more appropriate:
with tf.device('/cpu:0'), tf.name_scope("reshaping"):
self.x_expand = tf.reshape(self.input_x, [batch_size, num_classes, data_size])
Two things of note:
Did not use a tf.placeholder for self.x_expand. Placeholders are meant for feeding input from the user, but here it seems what you really wanted was to reshape existing feeds
Did not change shape on a per-element basis, but rather with a single operation.
NB: I was unable to reproduce the error message you saw, possibly because we're using different versions of TensorFlow and there have been changes to the error message reporting in recent versions. I am using 1.0.0.
I think you should try:
self.x_expand = tf.reshape(self.input_x, [batch_size, num_classes,data_size]