fit() gives different result than fit_generator() on the same dataset - python-3.x

Question

I have been playing around with CNNs to try and remove coherent noise from relatively large images. Since the images are large, I cannot load too many into memory at once. Size of images: 1500x250. Because of this issue, I have tried to implement a generator which i feeding the images to the network. I have been struggling for a while getting bad results, but assumed the issues where in the network. I tried using fit() with a subset of my data and got extremely good results without doing anything with the network. Testing the generator on the same subset resulted in bad results. What is the catch that I cannot see? Why does my generator fail?
My dataset is approx 114 000 images, which is approximately 475 GB thus explaining why I cannot load all into memory at once. I get results and they are actual results recreating the images, but they are extremely bad. My generator class i here:
class genOne(k.utils.Sequence):
def __init__(self, img_rows, img_cols, channels, batch_size, clean_dir,
noisy_dir, clean_files, noisy_files, shuffle=True):
"""Initialize variables:
img_rows, img_cols, channels: the shape of the image
batch_size : Self explanatory
clean_dir, noisy_dir : directories with files
clean_files : Randomized list with clean images
noisy_file : Randomized list with noise"""
self.img_rows = img_rows
self.img_cols = img_cols
self.channels = channels
self.batch_size = batch_size
self.clean_dir = clean_dir
self.noisy_dir = noisy_dir
self.clean_files = clean_files.tolist()
self.noisy_files = noisy_files.tolist()
self.shuffled_noisy = []
self.tmp_noisy = []
self.tmp_clean = []
self.shuffle = shuffle
self.on_epoch_end()
def __len__(self):
"""Sets the number of batches per epoch"""
return floor((len(self.noisy_files)*len(self.clean_files))/self.batch_size)
def __getitem__(self, index):
"""Generates data for each batch
combine every type of noise with each image."""
X = np.empty((self.batch_size, self.img_rows, self.img_cols,
self.channels))
Y = np.zeros((self.batch_size, self.img_rows, self.img_cols,
self.channels))
for i in range(self.batch_size):
if not self.tmp_noisy:
self.tmp_noisy = self.shuffled_noisy
self.tmp_clean.pop(0)
x_test = self.tmp_noisy.pop(0)
X[i,] = np.expand_dims(np.load(self.noisy_dir + x_test).T[
:self.img_rows,:self.img_cols],-1)
Y[i,] = np.expand_dims(np.load(self.clean_dir + self.tmp_clean[0
]).T[:self.img_rows, :self.img_cols],-1)
y_test = self.tmp_clean[0]
# Input equals ground truth + noise
X[i,] += Y[i,]
# Normalize data between 0 and 1
X[i,] = ((X[i,]/np.amax(np.absolute(X[i,])))+1)/2
Y[i,] = ((Y[i,]/np.amax(np.absolute(Y[i,])))+1)/2
return X, Y
def on_epoch_end(self):
"""Refresh all data on epoch end"""
self.tmp_noisy = self.noisy_files
self.tmp_clean = self.clean_files
if self.shuffle == True:
np.random.shuffle(self.tmp_noisy)
np.random.shuffle(self.tmp_clean)
self.shuffled_noisy = self.tmp_noisy
I have 475 clean images, and 300 images consisting of pure noise. I combine them such that each image is fed into the network with each type of noise. The small case which worked with fit() was simply 300 images where every image was a different clean image with a different noise.
I am aware that my driver version is rather old, which requires an old version of tensorflow. I can not update this, so I'm stuck with tensorflow 1.4.1.
Specs:
2x Nvidia Geforce GTX 1080 7.9 GB
Nvidia Driver version 367.44
cuDNN 6.0.21
CUDA 8.0
Debian Wheezy 7
Tensorflow-gpu 1.4.1
Keras 2.0.8
Python 3.6.7