I am testing the MNIST dataset in Pytorch, and after I apply a transformation to the X data, it seems the DataLoader puts all values out of the original order, potentially messing up the training step.
My transformation is to divide all values by 255. One should notice that the transformation itself does not change positions, as shown by the first scatterplots. But after the data is passed to the DataLoader and I retrieve it back, they are out of order. If I make no transformation, everything is fine (not shown). The distribution of the values is the same among before, after1 (divided by 255/before DataLoader) and after2 (divided by 255/after DataLoader) (also not shown), only the order seems to be affected.
import torch
from torchvision import datasets
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
transform = transforms.ToTensor()
train = datasets.MNIST(root = '.', train = True, download = True, transform = transform)
test = datasets.MNIST(root = '.', train = False, download = True, transform = transform)
before = train.data[0]
train.data = train.data.float()/255
after1 = train.data[0]
train_loader = torch.utils.data.DataLoader(train, batch_size = 128)
test_loader = torch.utils.data.DataLoader(test, batch_size = 128)
fig, ax = plt.subplots(1, 2)
ax[0].scatter(range(len(before.view(-1))), before.view(-1))
ax[0].set_title('Before')
ax[1].scatter(range(len(after1.view(-1))), after1.view(-1))
ax[1].set_title('After1')
after2 = next(iter(train_loader))[0][0]
fig, ax = plt.subplots(1, 2)
ax[0].scatter(range(len(before.view(-1))), before.view(-1))
ax[0].set_title('Before')
ax[1].scatter(range(len(after2.view(-1))), after2.view(-1))
ax[1].set_title('After2')
fig, ax = plt.subplots(1, 3)
ax[0].imshow(before, cmap = 'gray')
ax[1].imshow(after1, cmap = 'gray')
ax[2].imshow(after2.view(28, 28), cmap = 'gray')
I know that this might not be the best way to deal with this data (transforms.Normalize should solve it), but I would really like to understand what is happening.
Thank you!
So... I posted this same question at Pytorch's GitHub page, and they answered the following:
It's unrelated to data loader. You are messing with an attribute of
the particular dataset object, however, the actual __getitem__ of that
object does much more:
https://github.com/pytorch/vision/blob/6de158c473b83cf43344a0651d7c01128c7850e6/torchvision/datasets/mnist.py#L92
In particular this line (mode='L') assumes uint8 input. Since you
replaced it with float, it is wrong.
Then I guess the preferred approach would be to apply a transform when preparing the dataset at the very beginning of my code.
Originally I haven't tested the code you wrote.
Rewrote the original:
import torch
from torchvision import datasets
import torchvision.transforms as transforms
from torch.utils.data import DataLoader, Dataset, TensorDataset
import matplotlib.pyplot as plt
transform = transforms.ToTensor()
train = datasets.MNIST(root = '.', train = True, download = True, transform = transform)
test = datasets.MNIST(root = '.', train = False, download = True, transform = transform)
dl = DataLoader(train)
images = dl.dataset.data.float()/255
labels = dl.dataset.targets
train_ds = TensorDataset(images, labels)
train_loader = DataLoader(train_ds, batch_size=128)
# img, target = next(iter(train_loader))
before = train.data[0]
train.data = train.data.float()/255
after1 = train.data[0]
# train_loader = torch.utils.data.DataLoader(train, batch_size = 128)
test_loader = torch.utils.data.DataLoader(test, batch_size = 128)
fig, ax = plt.subplots(1, 2)
ax[0].scatter(range(len(before.view(-1))), before.view(-1))
ax[0].set_title('Before')
ax[1].scatter(range(len(after1.view(-1))), after1.view(-1))
ax[1].set_title('After1')
after2 = next(iter(train_loader))[0][0]
fig, ax = plt.subplots(1, 2)
ax[0].scatter(range(len(before.view(-1))), before.view(-1))
ax[0].set_title('Before')
ax[1].scatter(range(len(after2.view(-1))), after2.view(-1))
ax[1].set_title('After2')
fig, ax = plt.subplots(1, 3)
ax[0].imshow(before, cmap = 'gray')
ax[1].imshow(after1, cmap = 'gray')
ax[2].imshow(after2.view(28, 28), cmap = 'gray')
Related
I am attempting to implement sequence learning in Python 3.5.2 | Anaconda 4.2.0 (64-bit) on my windows 10 machine. I have scoured the internet in order to get as far as I have, but documentation lacks the details I need. I will share my code and then ask my questions.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import tensorflow
import keras as krs
from keras.models import Sequential
from keras.layers import Dense, LSTM, Dropout
from keras.preprocessing import sequence
from sklearn.preprocessing import QuantileTransformer, MinMaxScaler
from sklearn.metrics import mean_squared_error
DF = pd.read_csv("./integer_seqn/train.csv", delimiter = ",")
N = DF.shape[0]
data = []
# turn pandas df into list
for rows in DF['Sequence']:
fix = rows.split(",")
fix = [int(elem) for elem in fix]
data.append(fix)
# for now I am working with one sequence at a time
D = data[0]
D = np.reshape(D, (len(D),1))
D = D.astype('float64')
# scaling needed since LSTMs are sensitive to large data values
# for this first sequence I use quantile scaling since the data is skewed
scaler = MinMaxScaler((0,1))
X_train_scaled = scaler.fit_transform(D)
# split each sequence into a training and a test set
train_size = int(len(X_train_scaled)*(0.90))
test_size = len(X_train_scaled) - train_size
train, test = X_train_scaled[0:train_size], X_train_scaled[train_size:len(X_train_scaled)]
# creates a data set so that the first column
# is every n-th element in a sequence
# and the second column is every (n+1)-th element in a sequence
dataX, dataY = [], []
time_step = 2
for i in range(0,(len(train)-time_step)):
dataX.append(train[i:(i+time_step),0])
dataY.append(train[i+time_step, 0])
dataX = np.array(dataX)
dataY = np.array(dataY)
testX, testY = [], []
for i in range(0,(len(test)-time_step)):
testX.append(test[i:(i+time_step),0])
testY.append(test[i+time_step,0])
testX = np.array(testX)
testY = np.array(testY)
# need to reshape the data so that model.fit will accept it
dataX = np.reshape(dataX, (dataX.shape[0],dataX.shape[1],1))
testX = np.reshape(testX, (testX.shape[0],testX.shape[1],1))
# build RNN-LSTM for a sequence
# the number of LSTM nodes is adjustable
model = Sequential()
model.add(LSTM(units = 20, input_shape = (1,1)))
model.add(Dense(1))
model.compile(loss="mean_squared_error", optimizer = "adam")
model.fit(dataX, dataY, epochs = 100, batch_size = 1, verbose = 0)
# predict using the model (validation with the train set and test with the test set)
trainPredict = model.predict(dataX)
testPredict = model.predict(testX)
# "unscale" the data
trainPredict = scaler.inverse_transform(trainPredict)
dataY = scaler.inverse_transform(np.reshape(dataY,(len(dataY),1)))
testPredict = scaler.inverse_transform(testPredict)
testY = scaler.inverse_transform(np.reshape(testY, (len(testY),1)))
I am testing this code on an integer sequence
data[0] = [1, 3, 13, 87, 1053, 28576, 2141733, 508147108, 402135275365, 1073376057490373, 9700385489355970183, 298434346895322960005291, 31479360095907908092817694945, 11474377948948020660089085281068730]
I am getting some pretty poor predictions. When I do model.predict(dataX), the order of magnitude is significantly larger than the true predictions. Additionally, the predictions provided are all the same. Most recently, I get a 7x1 numpy array filled with the value 1.2242328e+32. I get a similar occurrence with model.predict(testX), where I get a 3x1 numpy array filled with the value 1.2242328e+32, but at least this prediction is closer (though not as close as I would like) to the final element in testY.
My specific question is: why is my prediction array filled with the same value?
I'm trying to run a simple example with a NN using the MNIST dataset provided by tensorflow itself, running on Google Colab. I want to get the raw data and mount by myself the structure that has the data. I'm able to train the NN, but when I try to predict one example from the test set, I get the error
ValueError: Error when checking input: expected dense_input to have shape (784,) but got array with shape (1,).
Could somebody help me with this issue? I'm pretty new to Python and Keras/TensorFlow.
When I run
print(inp.shape)
I get (784,) and not the (1,) as the error says.
I have also tried to evaluate the test set using
test_loss, test_accuracy = model.evaluate(test_input.T)
, but I also get the error
ValueError: Arguments and signature arguments do not match: 25 27.
The source code is the following:
# Importing stuff
import tensorflow as tf
import tensorflow_datasets as tfds
import matplotlib.pyplot as plt
import numpy as np
import math
import time
import keras
tf.enable_eager_execution()
# Functions
def normalize(images, labels):
images = tf.cast(images, tf.float32)
images /= 255
return images, labels
# Getting dataset
ds, meta = tfds.load('fashion_mnist', as_supervised=True, with_info=True)
test_ds, train_ds = ds['test'], ds['train']
# Preprocess the data
train_ds = train_ds.map(normalize)
test_ds = test_ds.map(normalize)
num_train_examples = meta.splits['train'].num_examples
num_test_examples = meta.splits['test'].num_examples
# Making the train set
train_input = np.empty(shape=(784, num_train_examples))
train_label = np.empty(shape=(1, num_train_examples))
i = 0
for image, label in train_ds:
image = image.numpy().reshape((784, 1))
train_input[:, i] = image.ravel()
label = label.numpy().reshape(1)
train_label[:, i] = label
i = i + 1;
# Making the test set
test_input = np.empty(shape=(784, num_test_examples))
test_label = np.empty(shape=(1, num_test_examples))
i = 0
for image, label in test_ds:
image = image.numpy().reshape((784, 1))
test_input[:, i] = image.ravel()
label = label.numpy().reshape(1)
test_label[:, i] = label
i = i + 1;
# Network
input_layer = tf.keras.layers.Dense(units=784, input_shape=[784])
h1 = tf.keras.layers.Dense(128, activation=tf.nn.relu)
output_layer = tf.keras.layers.Dense(10, activation=tf.nn.softmax)
model = tf.keras.Sequential([input_layer, h1, output_layer])
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.fit(train_input.T, train_label.T, epochs=3, steps_per_epoch=100, batch_size=1)
test_loss, test_accuracy = model.evaluate(test_input.T)
inp = test_input[:, 0].T
res = model.predict(inp)
All the API functions expect an input shape containing the batch size first. In your case, you're trying to feed only 1 example and thus no batch size is given. You just need to specify your batch size as 1 by reshaping the data.
Using numpy:
res = model.predict(np.reshape(inp, len(inp))
The argument to the predict method now receives the array with shape (1, 784) in your case, specifying the batch size as 1.
When you give the function more examples to evaluate by stacking them on top of one another, the batch size is implicitly given by the shape of the array, so no further transformation is necessary.
I'm currently trying to implement a multi-layer autoencoder using Keras, but I'm starting to believe that my implementation is wrong, since I don't get any difference in accuracy nor variance by using 2 or 3 hidden layers. I tried using two hidden layers with 400/180 and then I tried using three hidden layers with nodes of 400/180/30 and I practically get the exact same result. My code is currently looking like this:
from keras.layers import Input, Dense, initializers
import numpy as np
from Dataset import Dataset
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.optimizers import Adam
import time
#global variables
d = Dataset()
num_features = d.X_train.shape[1]
#input = [784, 400, 100, 10, 100, 400]
#output = [400, 100, 10, 100, 400, 784]
#names = ['hidden1', 'hidden2', 'hidden3', 'hidden4', 'hidden5', 'hidden6']
list_of_nodes = [784, 400, 180]
def generate_hidden_nodes(list_of_nodes):
input = []
for j in range(len(list_of_nodes)):
input.append(list_of_nodes[j])
for i in range(len(list_of_nodes)-2):
input.append(list_of_nodes[-2-i])
output = input[::-1]
return input, output
input,output = generate_hidden_nodes(list_of_nodes)
def autoencoder(epochs):
w = initializers.RandomNormal(mean=0.0, stddev=0.05, seed=None)
model = Sequential()
input, output = generate_hidden_nodes(list_of_nodes)
for j in range(len(input)):
if j == (len(input)-1):
model.add(Dense(output[j], activation='sigmoid', kernel_initializer=w, input_dim=input[j]))
#model.add(Dropout(0.45))
else:
model.add(Dense(output[j], activation='relu', kernel_initializer=w, input_dim=input[j],
))
#model.add(Dropout(0.45))
model.compile(optimizer=Adam(lr=0.001), loss='binary_crossentropy', metrics=['acc'])
history = model.fit(d.X_train, d.X_train,
epochs=epochs,
batch_size=50,
shuffle=True,
validation_split = 0.2)
#validation_data=(d.X_test, d.X_test))
#print(history.history.keys())
#plt.plot(history.history['val_acc'])
#print(history.history['val_acc'])
plt.show()
return model
def cv():
accuracy = 0
size = 5
epochs = 20
variance = 0
storage = np.zeros((size, epochs))
for j in range(size):
ae = autoencoder(epochs)
#print(ae.history.history['val_acc'])
storage[j] = ae.history.history['val_acc']
for i in range(size):
accuracy += storage[i][-1]
mean = accuracy/size
for k in range(size):
variance += ((storage[k][-1] - mean)**2)
variance = variance/size
return mean, variance
mean, variance = cv()
print(mean)
print(variance)
time.sleep(10)
def finding_index():
elements, index = np.unique(d.Y_test, return_index=True)
return elements, index
def plotting():
ae = autoencoder(20)
elements, index = finding_index()
y_proba = ae.predict(d.X_test)
plt.figure(figsize=(20, 4))
# size = 20
for i in range(len(index)):
ax = plt.subplot(2, len(index), i + 1)
plt.imshow(d.X_test[index[i]].reshape(28, 28))
plt.gray()
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
ax = plt.subplot(2, len(index), i + 1 + len(index))
plt.imshow(y_proba[index[i]].reshape(28, 28))
plt.gray()
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)
plt.show()
plotting()
I was expecting a significant difference by sending it down to only 30 nodes in the end (and the decoding it back). Is there an obvious mistake in my code, the dataset I'm currently using is the Mnist dataset (handwritten digits). Thanks in advance!
I am trying to use 20 news groups data set available in sklearn to train a LSTM to do incremental learning (classification). I used the sklearn's TfidfVectorizer to pre-process the data. Then I turned the resulting sparse matrix into a numpy array before feeding it. After that when coding the below line:
outputs, final_state = tf.nn.dynamic_rnn(cell, inputs_, initial_state=initial_state)
It gave an error saying that the 'inputs_' should have 3 dimensions. so I used:
inputs_ = tf.expand_dims(inputs_, 0)
To expand the dimension. But when I do that i get the error:
ValueError: Input size (depth of inputs) must be accessible via shape
inference, but saw value None.
The shape of 'input_' is:
(1, 134410)
I already went through this post, but it did not help.
I cannot seem to understand how to solve this issue. Any help is much appreciated. Thank you in advance!
show below is my complete code:
import os
from collections import Counter
import tensorflow as tf
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.datasets import fetch_20newsgroups
import matplotlib as mplt
from matplotlib import cm
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
from sklearn.metrics import f1_score, recall_score, precision_score
from string import punctuation
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelBinarizer
def pre_process():
newsgroups_data = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))
vectorizer = TfidfVectorizer()
features = vectorizer.fit_transform(newsgroups_data.data)
lb = LabelBinarizer()
labels = np.reshape(newsgroups_data.target, [-1])
labels = lb.fit_transform(labels)
return features, labels
def get_batches(x, y, batch_size=1):
for ii in range(0, len(y), batch_size):
yield x[ii:ii + batch_size], y[ii:ii + batch_size]
def plot_error(errorplot, datapoint, numberOfWrongPreds):
errorplot.set_xdata(np.append(errorplot.get_xdata(), datapoint))
errorplot.set_ydata(np.append(errorplot.get_ydata(), numberOfWrongPreds))
errorplot.autoscale(enable=True, axis='both', tight=None)
plt.draw()
def train_test():
features, labels = pre_process()
#Defining Hyperparameters
epochs = 1
lstm_layers = 1
batch_size = 1
lstm_size = 30
learning_rate = 0.003
print(lstm_size)
print(batch_size)
print(epochs)
#--------------placeholders-------------------------------------
# Create the graph object
graph = tf.Graph()
# Add nodes to the graph
with graph.as_default():
tf.set_random_seed(1)
inputs_ = tf.placeholder(tf.float32, [None,None], name = "inputs")
# labels_ = tf.placeholder(dtype= tf.int32)
labels_ = tf.placeholder(tf.int32, [None,None], name = "labels")
#getting dynamic batch size according to the input tensor size
# dynamic_batch_size = tf.shape(inputs_)[0]
#output_keep_prob is the dropout added to the RNN's outputs, the dropout will have no effect on the calculation of the subsequent states.
keep_prob = tf.placeholder(tf.float32, name = "keep_prob")
# Your basic LSTM cell
lstm = tf.contrib.rnn.BasicLSTMCell(lstm_size)
# Add dropout to the cell
drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)
#Stack up multiple LSTM layers, for deep learning
cell = tf.contrib.rnn.MultiRNNCell([drop] * lstm_layers)
# Getting an initial state of all zeros
initial_state = cell.zero_state(batch_size, tf.float32)
inputs_ = tf.expand_dims(inputs_, 0)
outputs, final_state = tf.nn.dynamic_rnn(cell, inputs_, initial_state=initial_state)
#hidden layer
hidden = tf.layers.dense(outputs[:, -1], units=25, activation=tf.nn.relu)
logit = tf.contrib.layers.fully_connected(hidden, 1, activation_fn=None)
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logit, labels=labels_))
optimizer = tf.train.AdamOptimizer(learning_rate).minimize(cost)
saver = tf.train.Saver()
# ----------------------------online training-----------------------------------------
with tf.Session(graph=graph) as sess:
tf.set_random_seed(1)
sess.run(tf.global_variables_initializer())
iteration = 1
state = sess.run(initial_state)
wrongPred = 0
errorplot, = plt.plot([], [])
for ii, (x, y) in enumerate(get_batches(features, labels, batch_size), 1):
feed = {inputs_: x.toarray(),
labels_: y,
keep_prob: 0.5,
initial_state: state}
predictions = tf.round(tf.nn.softmax(logit)).eval(feed_dict=feed)
print("----------------------------------------------------------")
print("Iteration: {}".format(iteration))
print("Prediction: ", predictions)
print("Actual: ",y)
pred = np.array(predictions)
print(pred)
print(y)
if not ((pred==y).all()):
wrongPred += 1
if ii % 27 == 0:
plot_error(errorplot,ii,wrongPred)
loss, states, _ = sess.run([cost, final_state, optimizer], feed_dict=feed)
print("Train loss: {:.3f}".format(loss))
iteration += 1
saver.save(sess, "checkpoints/sentiment.ckpt")
errorRate = wrongPred/len(labels)
print("ERROR RATE: ", errorRate )
if __name__ == '__main__':
train_test()
ValueError: Input size (depth of inputs) must be accessible via shape inference, but saw value None.
This error is given because you don't specify the size nor the amount of inputs.
I got the script working like this:
inputs_ = tf.placeholder(tf.float32, [1,None], name = "inputs")
inputs_withextradim = tf.expand_dims(inputs_, 2)
outputs, final_state = tf.nn.dynamic_rnn(cell, inputs_withextradim, initial_state=initial_state)
I want to make a neural net which takes image+image+value as an input and performs convolution+pooling on images and then a linear transform on results. Can I do that in keras?
This is architecturally similar to Craig Li's answer but is in the image, image, value format and does not use VGG16 and just a vanilla CNN. These are 3 separate networks whose outputs are concatenated after being processed individually and the resulting concatenated vector is passed through the final layers, including information from all inputs.
input_1 = Input(data_1.shape[1:], name = 'input_1')
conv_branch_1 = Conv2D(filters, (kernel_size, kernel_size),
activation = LeakyReLU())(conv_branch_1)
conv_branch_1 = MaxPooling2D(pool_size = (2,2))(conv_branch_1)
conv_branch_1 = Flatten()(conv_branch_1)
input_2 = Input(data_2.shape[1:], name = 'input_2')
conv_branch_2 = Conv2D(filters, (kernel_size, kernel_size),
activation = LeakyReLU())(conv_branch_2)
conv_branch_2 = MaxPooling2D(pool_size = (2,2))(conv_branch_2)
conv_branch_2 = Flatten()(conv_branch_2)
value_input = Input(value_data.shape[1:], name = 'value_input')
fc_branch = Dense(80, activation=LeakyReLU())(value_input)
merged_branches = concatenate([conv_branch_1, conv_branch_2, fc_branch])
merged_branches = Dense(60, activation=LeakyReLU())(merged_branches)
merged_branches = Dropout(0.25)(merged_branches)
merged_branches = Dense(30, activation=LeakyReLU())(merged_branches)
merged_branches = Dense(1, activation='sigmoid')(merged_branches)
model = Model(inputs=[input_1, input_2, value_input], outputs=[merged_branches])
#if binary classification do this otherwise whatever loss you need
model.compile(loss='binary_crossentropy')
Suppose your image is RGB type, the shape of the image is (width,height,3), you can combine two images with numpy like:
import numpy as np
from PIL import Image
img1 = Image.open('image1.jpg')
img2 = Image.open('imgae2.jpg')
img1 = img1.resize((width,height))
img2 = img2.resize((width,height))
img1_arr = np.asarray(img1,dtype='int32')
img2_arr = np.asarray(img2,dtype='int32')
#shape of img_arr is (width,height,6)
img_arr = np.concatenate((img1_arr,img2_arr),axis=2)
Combine two images in this way, we only increase the channels, so we can still do convolution on the first two axis.
UPDATE:
I guess you mean Multi-Task Model, you want to merge two images after convolution,Keras has concatenate() can do that.
input_tensor = Input(shape=(channels, img_width, img_height))
# Task1 on image1
conv_model1 = VGG16(input_tensor=input_tensor, weights=None, include_top=False, classes=classes,
input_shape=(channels, img_width, img_height))
conv_output1 = conv_model1.output
flatten1 = Flatten()(conv_output1)
# Task2 on image2
conv_model2 = VGG16(input_tensor=input_tensor, weights=None, include_top=False, classes=classes,
input_shape=(channels, img_width, img_height))
conv_output2 = conv_model2.output
flatten2 = Flatten()(conv_output2)
# Merge the output
merged = concatenate([conv_output1, conv_output2], axis=1)
merged = Dense(classes,activation='softmax')(merged)
# add some Dense layers and Dropout,
final_model = Model(inputs=[input_tensor,input_tensor],outputs=merged)