Wand to use keras.backend.get_session() to log scalars and images to tensorboard. Session closes with I do this and keras errors out. I think there must be some way of getting the keras graph and doing things in it without breaking it.
Related
I am trying out some examples using keras models, that are already available. Most of the examples are using keras with tensorflow (or pytorch or theano).
Due to limited available resource and cost cutting, I am using plaidml to work with amd gpu. As keras support pluggable backend, I think this may not be an issue.
Please share your thoughts about using keras api and later plugging in with desired backend.
I have this concern because the samples and this are using keras from tensorflow (import tensorflow.keras) and I am using plain from keras(import keras) with pluggable backend.
what is equivalent statement for
img = tf.io.decode_png(img, channels=1)
# 3. Convert to float32 in [0, 1] range
img = tf.image.convert_image_dtype(img, tf.float32)
Is there any limitation going with plain keras api?
I just used PIL Image to read and convert an image. It works the same as without using tensorflow api. Most of the keras api can be used irrespective of the backend. There are some caveat with PlaidML as well, there are some function like CTC Loss ctc_batch_cost cannot be found. I got an error like
The Keras backend function 'ctc_batch_cost' is not yet implemented in
Plaid. You can help us prioritize by letting us know if this function
is important to you, and as always, contributions are welcome!
There are some posts, which provide some sample implementation but it is not straight forward. From PLaidML, the response was that it may not be available soon.
I am trying to port two pre-trained keras models into the IPU machine. I managed to load and run them using IPUstrategy.scope but I dont know if i am doing it the right way. I have my pre-trained models in .h5 file format.
I load them this way:
def first_model():
model = tf.keras.models.load_model("./model1.h5")
return model
After searching your ipu.keras.models.py file I couldn't find any load methods to load my pre-trained models, and this is why i used tf.keras.models.load_model().
Then i use this code to run:
cfg=ipu.utils.create_ipu_config()
cfg=ipu.utils.auto_select_ipus(cfg, 1)
ipu.utils.configure_ipu_system(cfg)
ipu.utils.move_variable_initialization_to_cpu()
strategy = ipu.ipu_strategy.IPUStrategy()
with strategy.scope():
model = first_model()
print('compile attempt\n')
model.compile("sgd", "categorical_crossentropy", metrics=["accuracy"])
print('compilation completed\n')
print('running attempt\n')
res = model.predict(input_img)[0]
print('run completed\n')
you can see the output here:link
So i have some difficulties to understand how and if the system is working properly.
Basically the model.compile wont compile my model but when i use model.predict then the system first compiles and then is running. Why is that happening? Is there another way to run pre-trained keras models on an IPU chip?
Another question I have is if its possible to load a pre-trained keras model inside an ipu.keras.model and then use model.fit/evaluate to further train and evaluate it and then save it for future use?
One last question I have is about the compilation part of the graph. Is there a way to avoid recompilation of the graph every time i use the model.predict() in a different strategy.scope()?
I use tensorflow2.1.2 wheel
Thank you for your time
To add some context, the Graphcore TensorFlow wheel includes a port of Keras for the IPU, available as tensorflow.python.ipu.keras. You can access the API documentation for IPU Keras at this link. This module contains IPU-specific optimised replacement for TensorFlow Keras classes Model and Sequential, plus more high-performance, multi-IPU classes e.g. PipelineModel and PipelineSequential.
As per your specific issue, you are right when you mention that there are no IPU-specific ways to load pre-trained Keras models at present. I would encourage you, as you appear to have access to IPUs, to reach out to Graphcore Support. When doing so, please attach your pre-trained Keras model model1.h5 and a self-contained reproducer of your code.
Switching topic to the recompilation question: using an executable cache prevents recompilation, you can set that up with environmental variable TF_POPLAR_FLAGS='--executable_cache_path=./cache'. I'd also recommend to take a look into the following resources:
this tutorial gathers several considerations around recompilation and how to avoid it when using TensorFlow2 on the IPU.
Graphcore TensorFlow documentation here explains how to use the pre-compile mode on the IPU.
So right now I'm using keras and training my model works perfectly fine, but I have to pass my data as numpy ndarray. So I have to convert my list of data to numpy ndarray first and then pass it to keras for training. When I try to pass my python list/array, even tho it's the same shape as numpy array I get back errors. Is there any way to not use numpy for this or am I stuck with it?
Can you further explain your problem. What is the error message you are getting and are you getting this error during training or predicting?
Also if you could post some code samples that would help to
I am using sagemaker to train a keras model. I need to implement early stoping approach when training the model.
Is there a way to pass callbacks such as EarlyStopping, Histories..etc.
In traditional way, we used to pass this as a parameter to keras's fit function:
results = model.fit(train_x_trim, train_y_trim,
validation_data=(test_x, test_y),
epochs=FLAGS.epoch,
verbose=0,
callbacks=[tboard, checkpointer, early_stopping, history])
However, if using SageMaker, we need to call SageMaker's fit function instead which doesn't support callbacks.
from sagemaker.tensorflow import TensorFlow
iris_estimator = TensorFlow(entry_point='training_code.py',
role=role, output_path=model_location,
code_location=custom_code_upload_location,
train_instance_count=1,
train_instance_type='ml.c4.xlarge',
training_steps=1000,
evaluation_steps=100)
Any idea how to implement callbacks in SageMaker ?
I apologize for the late response.
It looks like the Keras code you specified above is essentially your algorithm code. This would be defined in your user script, which would be "training_code.py" in the SageMaker Python SDK example you provided.
Starting with TensorFlow 1.11, the SageMaker predefined TensorFlow containers have support for "script mode". You should be able to specify your Keras callbacks within your user script.
For more information: https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/tensorflow/README.rst#tensorflow-sagemaker-estimators-and-models
When training deep semantic segmentation models, it is often convenient to visualize a sample of predictions on the validation set during the training. Right now I'm simply saving some predictions to disk on my training server. I'm looking to migrate this task to TensorBoard. Simply put, I wan't to visualize a set of predictions (say 5) over each epoch.
I know there is a simple way to do it in pure TensorFlow like tf.summary.image(..) but I don't see any easy way to incorporate this into the Keras TensorBoard callback.
Any guidance would be much appreciated.
Fabio Perez provided an answer which should do exactly what you're looking for here How to display custom images in TensorBoard using Keras?