I'm newbie in Neural Network. I'm going to do a text classification research using MLP model with keras. Input layer consisting of 900 nodes, 2 hidden layers, and 2 outputs.
The code I use is as follows:
#Split data training & testing (90:10)
Train_X, Test_X, Train_Y, Test_Y = model_selection.train_test_split(dataset['review'],dataset['sentimen'],test_size=0.2, random_state=8)
Encoder = LabelEncoder()
Train_Y = Encoder.fit_transform(Train_Y)
Test_Y = Encoder.fit_transform(Test_Y)
Tfidf_vect = TfidfVectorizer(max_features=None)
Tfidf_vect.fit(dataset['review'])
Train_X_Tfidf = Tfidf_vect.transform(Train_X)
Test_X_Tfidf = Tfidf_vect.transform(Test_X)
#ANN Architecture
model = Sequential()
model.add(Dense(units = 100, activation = 'sigmoid', input_shape=(32, 900)))
model.add(Dense(units = 100, activation = 'sigmoid'))
model.add(Dense(units = 2, activation = 'sigmoid'))
opt = Adam (learning_rate=0.001)
model.compile(loss = 'binary_crossentropy', optimizer = opt,
metrics = ['accuracy'])
print(model.summary())
#Hyperparameter
epochs= 100
batch_size= 32
es = EarlyStopping(monitor="val_loss",mode='min',patience=10)
model_prediction = model.fit(Train_X_Tfidf, Train_Y, epochs=epochs,
batch_size=batch_size, verbose=1,
validation_split=0.1, callbacks =[es])
But getting Error:
/usr/local/lib/python3.8/dist-packages/keras/engine/data_adapter.py in train_validation_split(arrays, validation_split)
1478 unsplitable = [type(t) for t in flat_arrays if not _can_split(t)]
1479 if unsplitable:
-> 1480 raise ValueError(
1481 "`validation_split` is only supported for Tensors or NumPy "
1482 "arrays, found following types in the input: {}".format(unsplitable))
ValueError: `validation_split` is only supported for Tensors or NumPy arrays, found following types in the input: [<class 'scipy.sparse.csr.csr_matrix'>]
How to Fix it? Thank you so much.
Related
I'm training the VGG16 block5 with code below over 100 iterations/epochs, adding up two dense and dropout layer after the output of block5.
seq = Sequential()
vgg_base = VGG16(weights='imagenet', input_shape=(224, 224, 3),include_top=False, pooling='avg')
seq.add(vgg_base)
for layer in vgg_base.layers:
layer.trainable = False
for layer in vgg_base.layers[:14]:
layer.trainable = False
for layer in vgg_base.layers[15:]:
layer.trainable = True
regularizer = tf.keras.regularizers.l2(1e-2)
for layer in vgg_base.layers[15:]:
for attr in ['kernel_regularizer']:
if hasattr(layer, attr):
setattr(layer, attr, regularizer)
vgg_base.summary()
#seq.add(Flatten())
input_a = Input(shape=(224, 224, 3))
out_a = seq(input_a)
hidden1 = Dense(128, activation='relu', name='dense_1', kernel_regularizer=tf.keras.regularizers.l2(0.001),
activity_regularizer=tf.keras.regularizers.l2(0.001))(out_a)
hidden_drp1 = Dropout(0.5)(hidden1)
hidden2 = Dense(32, activation='relu', name='dense_2',kernel_regularizer=tf.keras.regularizers.l2(0.001),
activity_regularizer=tf.keras.regularizers.l2(0.001))(hidden_drp1)
hidden_drp2 = Dropout(0.2)(hidden2)
out = Dense(1, activation='sigmoid', name='dense_3')(hidden_drp2)
model = Model(input_a, out)
Training settings:
if os.path.exists(filepath):
print('weights found... loading...')
model.load_weights(filepath)
train_datagen = ImageDataGenerator(rescale=1.0/255)
#,validation_split=0.4)
# validation_split=0.3) # set validation split
train_path = 'C:/fromtrainv4/'
train_set = train_datagen.flow_from_directory(
train_path,
target_size=(224,224),
shuffle=True,
color_mode='rgb',
batch_size=100,
class_mode='binary')
optim = Adam(lr=0.001)
loss = 'binary_crossentropy'
metrics = ['binary_accuracy', 'acc']
model.compile(loss=loss,
optimizer=optim,
metrics=metrics)
checkpoint = ModelCheckpoint(filepath,
monitor='loss',
verbose=1,
save_weights_only=False,
save_best_only=True,
mode='min')
my_callback = [tf.keras.callbacks.EarlyStopping(monitor='loss', patience=10),
checkpoint,tf.keras.callbacks.TensorBoard(log_dir='C:\\traning_logs\\')]
step_size_train = train_set.n//train_set.batch_size
history = model.fit_generator(train_set,
steps_per_epoch=step_size_train,
shuffle=False,
epochs=100,
callbacks=my_callback,
class_weight='balanced')
I have made sure that the data has same scaling as image_generator and everything else.
I load the model using code below:
from tensorflow.keras.models import load_model
model = load_model('C:/models/model.h5')
System Specs:
Processor: Intell Xeon
Ram: 16GB
GPU: NVIDIA RTX 2080 Super
Memory: M2. 256GB
Interpreter: Spyder Python 3.7.1
tensorflow version: 1.14.0
keras version: 2.3.1
Can someone guide me what I might be doing wrong. Please let me know if more information regarding this is required.
I would guess that your model is doing what is called "overfitting".
In short, by doing 100 train iterations of the same dataset, it kind of memorized by heart your train dataset and thus have a good result when tested on it. However it doesn't yet find a general answer to your problem which lead to lower results when testing with a new dataset with different inputs.
I'm trying to make a Face Recognition app using insightface, I wrote this code on Tensorflow 2.1.0 and Keras 2.3.1 and it worked well but due to some issues I have to migrate to TensorFlow 2.2.0 and Keras 2.4.3, I understand that my problem is my embeddings. they are sparse but in a meaningful way. How can I avoid changing the meaningfulness for my embeddings and avoid the spares data? From the error, I understand (Consider casting elements to a supported type.) that TensorFlow can't convert my np.array to tensor because it is sparse.
what I tried
these arent the commands but I wrote them so you would have a notion of what I tried. np.array(data["embeddings"]).todense(),csr_matrix(data["embeddings"]), tf.convert_to_tensor(data["embeddings"]) and also tried to follow along this but couldn't get to model.fit_generator work.
>>> print(type(embeddings))
<class 'numpy.ndarray'>
>>> print(embeddings.shape)
(49, 512)
>>> print(embeddings)
[[ 0.02751185 0.0143353 0.0324492 ... -0.00347222 0.0154978
-0.01304669]
[ 0.09154768 -0.04196533 0.01197386 ... -0.08363352 0.03335601
0.01748604]
[ 0.00182035 -0.00307933 0.00386595 ... -0.04442558 0.04434329
0.06080627]
...
[-0.01564891 -0.01510727 0.0345119 ... -0.01690779 -0.00816008
0.08056415]
[-0.00543963 -0.03811216 -0.01148985 ... -0.05366111 0.07108331
-0.00186215]
[ 0.00627459 -0.04221528 0.00426272 ... 0.02838095 0.02116473
0.00491964]]
This is my code:
class SoftMax():
def __init__(self, input_shape, num_classes):
self.input_shape = input_shape
self.num_classes = num_classes
def build(self):
#create model
model = Sequential()
#add model layers
model.add(Dense(1024, activation='relu', input_shape=self.input_shape))
model.add(Dropout(0.5))
model.add(Dense(1024, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(self.num_classes, activation='softmax'))
# loss and optimizer
optimizer=Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, amsgrad=False)
model.compile(loss=categorical_crossentropy,
optimizer=optimizer,
metrics=['accuracy'])
return model
def make_model(args, classifier=SoftMax):
# Load the face embeddings
data = pickle.loads(open(args.embeddings, "rb").read())
num_classes = len(np.unique(data["names"]))
ct = ColumnTransformer([('myŁName', OneHotEncoder(), [0])])
labels = np.array(data["names"]).reshape(-1, 1)
labels = ct.fit_transform(labels)
embeddings = np.array(data["embeddings"])
# Initialize Softmax training model arguments
BATCH_SIZE = 32
EPOCHS = 32
input_shape = embeddings.shape[1]
# Build classifier
init_classifier = classifier(input_shape=(input_shape,), num_classes=num_classes)
model = init_classifier.build()
# Create KFold
cv = KFold(n_splits = 5, random_state = None, shuffle=True)
history = {'acc': [], 'val_acc': [], 'loss': [], 'val_loss': []}
# Train
for train_idx, valid_idx in cv.split(embeddings):
X_train, X_val, y_train, y_val = embeddings[train_idx], embeddings[valid_idx], labels[train_idx], labels[valid_idx]
his = model.fit(X_train, y_train, batch_size=BATCH_SIZE, epochs=EPOCHS, verbose=1, validation_data=(X_val, y_val))
# write the face recognition model to output
model.save(args.mymodel)
f = open(args.le, "wb")
f.write(pickle.dumps(LabelEncoder()))
f.close()
Error
TypeError: Failed to convert object of type <class 'tensorflow.python.framework.sparse_tensor.SparseTensor'> to Tensor. Contents: SparseTensor(indices=Tensor("DeserializeSparse:0", shape=(None, 2), dtype=int64), values=Tensor("DeserializeSparse:1", shape=(None,), dtype=float32), dense_shape=Tensor("stack:0", shape=(2,), dtype=int64)). Consider casting elements to a supported type.
I passed two days trying to use Neural Structured language to adapt into me CNN Model I use ImageDataGenerator and flow_from_directory when I use model.fit_generator I got an error message:
ValueError:
When passing input data as arrays, do not specify
steps_per_epoch/steps argument. Please use batch_size instead.
I use Keras 2.3.1 and TensorFlow 2.0 as backend
This is a snipped of my code:
num_classes = 4
img_rows, img_cols = 224, 224
batch_size = 16
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=30,
width_shift_range=0.3,
height_shift_range=0.3,
horizontal_flip=True,
fill_mode='nearest')
validation_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(
train_data_dir,
target_size=(img_rows, img_cols),
batch_size=batch_size, shuffle=True,
class_mode='categorical')
validation_generator = validation_datagen.flow_from_directory(
validation_data_dir,
target_size=(img_rows, img_cols),
batch_size=batch_size, shuffle=True,
class_mode='categorical')
def vgg():
model1 = Sequential([ ])
return model1
base_model = vgg()
I adapte Datagenerated from (x,y) format to a dictionary format
def convert_training_data_generator():
for x ,y in train_generator:
return {'feature': x, 'label':y}
def convert_testing_data_generator():
for x ,y in validation_generator:
return {'feature': x, 'label': y}
adv_config = nsl.configs.make_adv_reg_config(multiplier=0.2, adv_step_size=0.05)
model = nsl.keras.AdversarialRegularization(base_model, adv_config=adv_config)
train= convert_training_data_generator()
test= convert_testing_data_generator()
history = model.fit_generator(train,
steps_per_epoch= nb_train_samples // batch_size,
epochs = epochs,
callbacks = callbacks,
validation_data = test,
validation_steps = nb_validation_samples // batch_size)
I think here there is the same error. Maybe you should consider using instead model.fit() function. You should define in that case your train input your train labels and the batch_size. In order to figure out the difference between fit and fit_generator, you can follow that link.
I am running MNIST prediction using Keras, with tensorflow backend.
I have code that runs with batches , using Keras fit() as
(X_train, y_train), (X_test, y_test) = mnist.load_data()
N1 = X_train.shape[0]
N2 = X_test.shape[0]
h = X_train.shape[1]
w = X_train.shape[2]
num_pixels = h*w
# reshape N1 samples to num_pixels
x_train = X_train.reshape(N1, num_pixels).astype('float32') # shape is now (60000,784)
x_test = X_test.reshape(N2, num_pixels).astype('float32') # shape is now (10000,784)
x_train = x_train / 255
x_test = x_test / 255
y_train = np_utils.to_categorical(y_train) #(60000,10)
y_test = np_utils.to_categorical(y_test) # (10000,10):
num_classes = y_test.shape[1]
def baseline_model():
# create model
model = Sequential()
model.add(Dense(num_pixels, input_dim=num_pixels, kernel_initializer='normal', activation='relu'))
model.add(Dense(num_classes, kernel_initializer='normal', activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
return model
model = baseline_model()
batch_size = 200
epochs = 20
max_batches = 2 * len(x_train) / batch_size # 2*60000/200
# reshape to be [samples][width][height][ channel] for ImageDataGenerator
x_t = X_train.reshape(N1, w, h, 1).astype('float32')
datagen = ImageDataGenerator(rescale= 1./255)
train_gen = datagen.flow(x_t, y_train, batch_size=batch_size)
for e in range(epochs):
batches = 0
for x_batch, y_batch in train_gen:
# x_batch is of size [batch_sz,w,h,ch]: resize to [bth_sz,pixel_sz]: (200,28,28,1)-> (200,784)
# for model.fit
x_batch = np.reshape(x_batch, [-1, num_pixels])
model.fit(x_batch, y_batch,validation_split=0.15,verbose=0)
batches += 1
print("Epoch %d/%d, Batch %d/%d" % (e+1, epochs, batches, max_batches))
if batches >= max_batches:
break
scores = model.evaluate(x_test, y_test, verbose=0)
However, when I try to implement similar code using fit_generator(), I get an error.
the code is as below:
(X_train, y_train), (X_test, y_test) = mnist.load_data()
# separate data into train and validation
from sklearn.model_selection import train_test_split
# Split the data
X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.15, shuffle= True)
# number of training samples
N1 = X_train.shape[0] # training size
N2 = X_test.shape[0] # test size
N3 = X_valid.shape[0] # valid size
h = X_train.shape[1]
w = X_train.shape[2]
num_pixels = h*w
y_train = np_utils.to_categorical(y_train)
y_valid = np_utils.to_categorical(y_valid)
y_test = np_utils.to_categorical(y_test)
num_classes = y_test.shape[1]
def baseline_model():
# create model
model = Sequential()
model.add(Dense(num_pixels, input_dim=num_pixels, kernel_initializer='normal', activation='relu'))
model.add(Dense(num_classes, kernel_initializer='normal', activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
return model
model = baseline_model()
batch_size = 200
epochs = 20
steps_per_epoch_tr = int(N1/ batch_size) # 51000/200
steps_per_epoch_val = int(N3/batch_size)
# reshape to be [samples][width][height][ channel] for ImageData Gnerator->datagen.flow
x_t = X_train.reshape(N1, w, h, 1).astype('float32')
x_v = X_valid.reshape(N3, w, h, 1).astype('float32')
# define data preparation
datagen = ImageDataGenerator(rescale=1./255) # scales x_t/x_v
train_gen = datagen.flow(x_t, y_train, batch_size=batch_size)
valid_gen = datagen.flow(x_v,y_valid, batch_size=batch_size)
model.fit_generator(train_gen,steps_per_epoch = steps_per_epoch_tr,validation_data = valid_gen,
validation_steps = steps_per_epoch_val,epochs=epochs)
This gives an error:
This is due to expected image dimension error, but I am not sure where/how to fix this. any help is greatly appreciated.
Thanks
sedy
In the model.fit() case, this line flattened the input before feeding it for training.
x_batch = np.reshape(x_batch, [-1, num_pixels])
But in the generator case, there is nothing to flatten the input before feeding it to the Dense layer. The Dense layer cannot process 2D input (28 x 28). Adding, a Flatten() layer to the model should do the trick as shown below.
def baseline_model():
# create model
model = Sequential()
model.add(Flatten(input_shape=(28,28,1)))
model.add(Dense(num_pixels, input_dim=num_pixels, kernel_initializer='normal', activation='relu'))
model.add(Dense(num_classes, kernel_initializer='normal', activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
return model
#10-Fold split
seed = 7
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed)
np.random.seed(seed)
cvscores = []
act = 'relu'
for train, test in kfold.split(X, Y):
model = Sequential()
model.add(Dense(43, input_shape=(8,)))
model.add(Activation(act))
model.add(Dense(500))
model.add(Activation(act))
#model.add(Dropout(0.4))
model.add(Dense(1000))
model.add(Activation(act))
#model.add(Dropout(0.4))
model.add(Dense(1500))
model.add(Activation(act))
#model.add(Dropout(0.4))
model.add(Dense(2))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
hist = model.fit(X[train], Y[train],
epochs=500,
shuffle=True,
batch_size=100,
validation_data=(X[test], Y[test]), verbose=2)
#model.summary()
When I call model.fit it reports the following error :
ValueError: Error when checking target: expected activation_5 to have shape (None, 2) but got array with shape (3869, 1)
I am using keras with TensorFlow backend. Please ask for any further clarification if needed.
The problem was solved when we used this statement
y = to_categorical(Y[:])