I'm trying to run a simple Keras script and use Google Colab with TensorBoard. Here's my code:
import tensorflow as tf
import tensorflow.keras as keras
from tensorflow.keras.datasets import cifar10
from tensorflow.keras.applications.mobilenet import MobileNet
from tensorboardcolab import TensorBoardColab, TensorBoardColabCallback
# Settings
num_classes = 10
batch_size = 16
epochs = 1
# Data setup
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
# Select model
model = MobileNet(weights=None, input_shape=x_train.shape[1:], classes=num_classes)
# Select loss, optimizer, metric
model.compile(loss='categorical_crossentropy',
optimizer=tf.train.AdamOptimizer(0.001),
metrics=['accuracy'])
# Train
tbc=TensorBoardColab()
model.fit(x_train, y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(x_test, y_test),
callbacks=[TensorBoardColabCallback(tbc)])
This is a suggestion I saw to use TensorBoard with Colab as referenced here: Can I use Tensorboard with Google Colab?
However, when adding the callback I get the error:
FailedPreconditionError: Error while reading resource variable
conv_dw_8_2/depthwise_kernel from Container: localhost. This could
mean that the variable was uninitialized. Not found: Resource
localhost/conv_dw_8_2/depthwise_kernel/N10tensorflow3VarE does not
exist. [[Node: conv_dw_8_2/depthwise/ReadVariableOp =
ReadVariableOpdtype=DT_FLOAT,
_device="/job:localhost/replica:0/task:0/device:GPU:0"]]
[[Node: loss_2/mul/_147 = _Recvclient_terminated=false,
recv_device="/job:localhost/replica:0/task:0/device:CPU:0",
send_device="/job:localhost/replica:0/task:0/device:GPU:0",
send_device_incarnation=1, tensor_name="edge_6752_loss_2/mul",
tensor_type=DT_FLOAT,
_device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Does anybody know what I'm doing wrong? This seems like a very useful way to run TensorBoard on Colab if I could get it working.
This is caused by conflicting versions of Keras. Tensorboardcolab uses the full keras library while you import the tf.keras implementation of the Keras API. So when you fit the model you end up using two different versions of keras.
You have a few options:
Use Keras libary and change your imports
import tensorflow as tf
import keras
from keras.datasets import cifar10
from keras.applications.mobilenet import MobileNet
from tensorboardcolab import TensorBoardColab, TensorBoardColabCallback
Although the code runs fine with these changes, you might consider using Keras's version of the Adam optimizer, so you don't need to import tensorflow explicitly anymore.
model.compile(loss='categorical_crossentropy',
optimizer=keras.optimizers.Adam(lr=0.001),
metrics=['accuracy'])`
Use tf.keras and patch TensorBoardColab
Your code runs fine, if you'd patch callbacks.py and core.py and fix the imports there:
from keras.callbacks import TensorBoard
from tensorflow.keras.callbacks import TensorBoard
You could also use this fork where I made these changes.
Related
Background of the Problem
I was trying to use a KerasRegressor model with the ML models (e.g. Lasso, Gradient Boost Regressor) for the purpose of building an ensemble method. I used the VotingRegressor() function of sklearn to group the models. However, when I add the KerasRegressor model in VotingRegressor(), I get the following error.
ValueError: The estimator KerasRegressor should be a regressor.
How Did I Try to Solve the Problem?
I searched on google by the error and I found only this page where I do not find the solution. Moreover, I tried to understand the document of the KerasRegressor. However, I do not know why I get the error as the document says that it is the implementation of the scikit-learn regressor API for Keras.
Then, My Question
Why did I get the error and what can I do to solve it?
Any help will be greatly appreciated :). Thanks!
From this issue there is no solution using keras as sklearn wrapper is not maintained and will be removed
Fortunately scikeras package solve this issue.
I advice you to read docs or tutorials but here a simple example using subclassing:
!pip install scikeras
import scikeras
from tensorflow import keras
from sklearn.datasets import make_regression
from sklearn.ensemble import VotingRegressor
from sklearn.linear_model import LinearRegression
class MLPRegressor(KerasRegressor):
def __init__(
self,
hidden_layer_sizes=(100, ),
optimizer="adam",
optimizer__learning_rate=0.001,
epochs=10,
verbose=0,
**kwargs,
):
super().__init__(**kwargs)
self.hidden_layer_sizes = hidden_layer_sizes
self.optimizer = optimizer
self.epochs = epochs
self.verbose = verbose
def _keras_build_fn(self, compile_kwargs):
model = keras.Sequential()
inp = keras.layers.Input(shape=(self.n_features_in_))
model.add(inp)
for hidden_layer_size in self.hidden_layer_sizes:
layer = keras.layers.Dense(hidden_layer_size, activation="relu")
model.add(layer)
out = keras.layers.Dense(1)
model.add(out)
model.compile(loss="mse", optimizer=compile_kwargs["optimizer"])
return model
# simple linear regression
r1 = LinearRegression()
# keras model wrapper
r2= MLPRegressor(epochs=20)
X = (y/2).reshape(-1, 1)
y = np.arange(100)
#defining votting classifier
vr = VotingRegressor([('lr', r1), ('MLPReg', r2)])
vr.fit(X,y)
VotingRegressor(estimators=[('lr',
LinearRegression(copy_X=True, fit_intercept=True,
n_jobs=None, normalize=False)),
('MLPReg',
MLPRegressor(batch_size=None, build_fn=None, callbacks=None, epochs=20, hidden_layer_sizes=(100,),
loss=None, metrics=None, model=None, optimizer='adam',
random_state=None, run_eagerly=False, shuffle=True,
validation_batch_size=None, validation_split=0.0, verbose=0,
warm_start=False))],
n_jobs=None, weights=None)
I am working on a project which uses imageai with YOLOv3 which works fast and accurately for my purpose. However this model is able to detect only 80 classes out of which I want some of them but want to add some more classes as well.
I referred to https://imageai.readthedocs.io/en/latest/customdetection/index.html to train my own custom model with 3 more classes. However, I am unable to detect the 80 classes that were provided by YOLOv3. Is there a way to generate a model that extends the existing YOLOv3 and can detect all 80 classes + extra classes that I want?
P.S. I am new to tensorflow and imageai so I don't know too much. Please bear with me.
I have not yet found a way to extend an existing model, but i can assure you that training your own model is far more efficient than using all the classes noones wants.
If it still interests you, this person had a similar question: Loading a trained Keras model and continue training
This is his finished code example:
"""
Model by: http://machinelearningmastery.com/
"""
import numpy
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.models import load_model
numpy.random.seed(7)
def baseline_model():
model = Sequential()
model.add(Dense(num_pixels, input_dim=num_pixels, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
return model
if __name__ == '__main__':
# load data
(X_train, y_train), (X_test, y_test) = mnist.load_data()
# flatten 28*28 images to a 784 vector for each image
num_pixels = X_train.shape[1] * X_train.shape[2]
X_train = X_train.reshape(X_train.shape[0], num_pixels).astype('float32')
X_test = X_test.reshape(X_test.shape[0], num_pixels).astype('float32')
# normalize inputs from 0-255 to 0-1
X_train = X_train / 255
X_test = X_test / 255
# one hot encode outputs
y_train = np_utils.to_categorical(y_train)
y_test = np_utils.to_categorical(y_test)
num_classes = y_test.shape[1]
# build the model
model = baseline_model()
#Partly train model
dataset1_x = X_train[:3000]
dataset1_y = y_train[:3000]
model.fit(dataset1_x, dataset1_y, nb_epoch=10, batch_size=200, verbose=2)
# Final evaluation of the model
scores = model.evaluate(X_test, y_test, verbose=0)
print("Baseline Error: %.2f%%" % (100-scores[1]*100))
#Save partly trained model
model.save('partly_trained.h5')
del model
#Reload model
model = load_model('partly_trained.h5')
#Continue training
dataset2_x = X_train[3000:]
dataset2_y = y_train[3000:]
model.fit(dataset2_x, dataset2_y, nb_epoch=10, batch_size=200, verbose=2)
scores = model.evaluate(X_test, y_test, verbose=0)
print("Baseline Error: %.2f%%" % (100-scores[1]*100))
I'm running an LSTM network that works fine (TF 2.0). My problem starts when trying to modify the loss function.
I planed to adjust some data manipulation over 'y_true' and 'y_pred' but since TF force to maintain the data as tensors (and not convert it to Pandas or NumPy) it is challenging.
To get better control of the data inside the loss function I've simulated tf.keras.losses.mae function.
My goal was to be able to see the data ('y_true' and 'y_pred') so I can make my desire adjustments.
The original function:
def mean_absolute_error(y_true, y_pred):
y_pred = ops.convert_to_tensor(y_pred)
y_true = math_ops.cast(y_true, y_pred.dtype)
return K.mean(math_ops.abs(y_pred - y_true), axis=-1)
And after adjustments for debugging:
from tensorflow.python.framework import ops
from tensorflow.python.ops import math_ops
import tensorflow.keras.backend as K
def mean_absolute_error_test(y_true, y_pred):
global temp_true
temp_true=y_true
print(y_true)
y_pred = ops.convert_to_tensor(y_pred)
y_true = math_ops.cast(y_true, y_pred.dtype)
return K.mean(math_ops.abs(y_pred - y_true), axis=-1)
when I run model.compile and print y_true I get:
Tensor("dense_target:0", shape=(None, None), dtype=float32)
type=tensorflow.python.framework.ops.Tensor
Does anyone know how can I see 'y_pred' and 'y_true' or what am I missing?
Seems like I can't see samples of y_true or the data is empty.
The main code part:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import Dropout,Dense
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential, load_model
from tensorflow.python.keras.layers.recurrent import LSTM
from tensorflow.keras.callbacks import EarlyStopping
K.clear_session()
model = Sequential()
model.add(LSTM(20,activation='relu',input_shape=(look_back,len(training_columns)),recurrent_dropout=0.4))
model.add(Dropout(0.1))
model.add(Dense(1, activation='linear'))
model.compile(optimizer='adam', loss=test2,experimental_run_tf_function=False)# mse,mean_squared_logarithmic_error
num_epochs = 20
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=3)
history=model.fit(X_train_lstm, y_train_lstm, epochs = num_epochs, batch_size = 128,shuffle=False,verbose=1,validation_data=[X_test_lstm,y_test_lstm],callbacks=[es])
I have built a simple ANN to do some simple upfront comparisons of different optimizer performance on my data. Code snippet is below. I'm just marching through different optimizers with default params and comparing performance on some basic measures.
I'm running tf keras on a CPU-only cluster on Azure Databricks as follows:
keras ver: 2.2.4-tf;
tensorflow ver: 1.15.0
All the keras optimizers (https://keras.io/api/optimizers/) work just fine in my code with the exception of Ftrl where I get the error:
ValueError: Unknown optimizer: Ftrl
I've tried "Ftrl", "ftrl" and setting the optimizer as a variable and calling that. No dice.
Thanks for your help
import tensorflow as tf
from tensorflow import keras
from keras.layers import Dense
from keras.models import Sequential
model = Sequential()
model.add(Dense(300, activation = 'relu', input_shape=(161,)))
model.add(Dense(300, activation = 'relu'))
model.add(Dense(300, activation = 'relu'))
model.add(Dense(2, activation = 'softmax'))
model.compile(loss="categorical_crossentropy",
optimizer = "Ftrl",
metrics=['accuracy'])
You can simply use tf.keras.optimizers.Ftrl()
model.compile(loss="categorical_crossentropy",
optimizer = tf.keras.optimizers.Ftrl(),
metrics=['accuracy'])
I have a Keras Nueral Network and I want to deploy this model using an wrapper in the spark environment. So I tried the following tutorial here
import tensorflow as tf
import keras
from keras.models import Sequential
from keras.layers import Input, Dense, Conv1D, Conv2D, MaxPooling2D, Dropout,Flatten
from keras import backend as K
from keras.models import Model
import numpy as np
import matplotlib.pyplot as plt
from keras.datasets import mnist
(X_train, y_train), (X_test, y_test) = mnist.load_data()
# Expect to see a numpy n-dimentional array of (60000, 28, 28)
type(X_train), X_train.shape, type(X_train)
#This time however, we flatten each of our 28 X 28 images to a vector of 1, 784
X_train = X_train.reshape(-1, 784)
X_test = X_test.reshape(-1, 784)
# expect to see a numpy n-dimentional array of : (60000, 784) for Traning Data shape and (10000, 784) for Test Data shape
type(X_train), X_train.shape, X_test.shape
#We also use sklearn's MinMaxScaler for normalizing
from sklearn.preprocessing import MinMaxScaler
def scaleData(data):
# normalize features
scaler = MinMaxScaler(feature_range=(0, 1))
return scaler.fit_transform(data)
X_train = scaleData(X_train)
X_test = scaleData(X_test)
# We define the same Keras model as earlier
input_shape = (1,28,28) if K.image_data_format() == 'channels_first' else (28,28, 1)
keras_model = Sequential()
keras_model.add(Conv2D(32, kernel_size=(5, 5), activation='relu', input_shape=input_shape, padding='same'))
keras_model.add(MaxPooling2D(pool_size=(2, 2)))
keras_model.add(Conv2D(64, (5, 5), activation='relu', padding='same'))
keras_model.add(MaxPooling2D(pool_size=(2, 2)))
keras_model.add(Flatten())
keras_model.add(Dense(512, activation='relu'))
keras_model.add(Dropout(0.5))
keras_model.add(Dense(10, activation='softmax'))
keras_model.summary()
# Import the Keras to DML wrapper and define some basic variables
from systemml.mllearn import Keras2DML
epochs = 5
batch_size = 100
samples = 60000
max_iter = int(epochs*math.ceil(samples/batch_size))
# Now create a SystemML model by calling the Keras2DML method and feeding it your spark session, Keras model, its input shape, and the # predefined variables. We also ask to be displayed the traning results every 10 iterations.
sysml_model = Keras2DML(spark, keras_model, input_shape=(1,28,28), weights='weights_dir', batch_size=batch_size, max_iter=max_iter, test_interval=0, display=10)
# Initiate traning. More spark workers and better machine configuration means faster training!
sysml_model.fit(X_train, y_train)
# Test your model's performance on the secluded test set, and re-iterate if required
sysml_model.score(X_test, y_test)
At the line from systemml.mllearn import Keras2DML
The error I got is
Traceback (most recent call last): File
"d:/SparkJarDirectory/./NNSpark.py", line 58,
in
from systemml.mllearn import Keras2DML File "C:\Users\xyz\AppData\Local\Continuum\anaconda3\lib\site-packages\systemml\mllearn__init__.py",
line 45, in
from .estimators import * File "C:\Users\xyz\AppData\Local\Continuum\anaconda3\lib\site-packages\systemml\mllearn\estimators.py",
line 917
def init(self, sparkSession, keras_model, input_shape, transferUsingDF=False, load_keras_weights=True, weights=None,
labels=None, batch_size=64, max_iter=2000, test_iter=10,
test_interval=500, display=100, lr_policy="step", weight_decay=5e-4,
regularization_type="L2"):
^ SyntaxError: import * only allowed at module level 2019-03-12 20:25:48 INFO ShutdownHookManager:54 - Shutdown hook called
2019-03-12 20:25:48 INFO ShutdownHookManager:54 - Deleting directory
C:\Users\xyz\AppData\Local\Temp\spark-2e1736f8-1798-42da-a157-cdf0ade1bf36
From my understanding I get that that there is an issue at the library I am using where they use
from .estimators import *
__all__ = estimators.__all__
I am not sure why the wrapper is not working or what fix is required. Any help is appreciated.
I think the systemml release 1.2.0 misses some fixes for python 3.5 (https://github.com/apache/systemml/commit/9e7ee19a45102f7cbb37507da25b1ba0641868fd) so you will need to install systemml from source (for my setup, which is different than yours, it would git clone and then "cd src/main/python; sudo python3.4 setup.py install")