I want to change the way we shuffle training data in Keras. Conventionally we shuffle samples in each epoch and then we batch, and fit the model. Now I want to first batch the samples, and then shuffle these "BATCHES" (all samples in each batch should not shuffle). The reason is that I ordered all samples based on criteria (Curriculum Learning) and I want to preserve such an order.
Do you know how I can do this?
I found the answer according to this link:
How does shuffle = 'batch' argument of the .fit() layer work in the background?
Short answer: we need to set shuffle='batch' in fit.
Related
Why does the testing data use the mean and variance of the all training data? To keep the distribution consistent? What is the difference between the BN layer using model.train compared to model.val
It fixes the mean and var computed in the training phase by keeping estimates of it in running_mean and running_var. See PyTorch Documentation.
As noted there the implementation is based on the description in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. As one tries to use the whole training data one can get (given similar data train/test data) a better estimate of the mean/var for the (unseen) test set.
Also similar questions have been asked here: What does model.eval() do?
...or is there a way to circumvent this?
In stateful LSTMs, I have to define a batch size but Keras is forcing me to use the same batch size in training as in prediction, but I find that my modeling problem depends a lot on having larger batch sizes to see good performance.
I found out data augmentation can be done in PyTorch by using torchvision.transforms. I also read that transformations are apllied at each epoch. So I'm wondering whether or not the effect of copying each sample multiple times and then applying random transformation to them is same as using torchvision.transforms on original data set(unique images) and just training it for a longer time(more epochs).
Thanks in advance.
This is a question to be answered in a broad scale. don't get misunderstood that the TorchVision Transforms doesn't increase your dataset. It applies random or non-random transforms to your current data set at runtime. (hence unique each time and each epoch).
the effect of copying each sample multiple times and then applying random transformation to them is same as using torchvision.transforms on original data set(unique images) and just training it for a longer time(more epochs).
Answer-
To increase your dataset, you can copy paste, also use pyTorch or WEKA software. However, more epochs are a totally different concept to this. Of course, the more epochs you use, the better the model will be (only till the validation loss and training loss intersect each other)
Hope this helps.
I have repeated measurements on subjects, which I have structured as input to an LSTM model in Keras as follows:
batch_size = 1
model = Sequential()
model.add(LSTM(50, batch_input_shape=(batch_size, time_steps, features), return_sequences=True))
Where time_steps are the number of measurements on each subject, and features the number of available features on each measurement. Each row of the data is one subject.
My question is regarding the batch size with this type of data.
Should I only use a batch size of 1, or can the batch size be more than 1 subjects?
Related to that, would I benefit from setting stateful to True? Meaning that learning from one batch would inform the other batches too. Correct me if my understanding about this is not right too.
Great question! Using a batch size greater than 1 is possible with this sort of data and setup, provided that your rows are individual experiments on subjects and that your observations for each subject are ordered sequentially through time (e.g. Monday comes before Tuesday). Make sure that your observations between train and test are not split randomly and that your observations are ordered sequentially by subject in each, and you can apply batch processing. Because of this, set shuffle to false if using Keras as Keras shuffles observations in batches by default.
In regards to setting stateful to true: with a stateful model, all the states are propagated to the next batch. This means that the state of the sample located at index i, Xi will be used in the computation of the sample Xi+bs in the next batch. In the case of time series, this generally makes sense. If you believe that a subject measurement Si infleunces the state of the next subject measurement Si+1, then try setting stateful to true. It may be worth exploring setting stateful to false as well to explore and better understand if a previous observation in time infleunces the following observation for a particular subject.
Hope this helps!
What is the need for setting steps_per_epoch value when calling the function fit_generator() when ideally it should be number of total samples/ batch size?
Keras' generators are infinite.
Because of this, Keras cannot know by itself how many batches the generators should yield to complete one epoch.
When you have a static number of samples, it makes perfect sense to use samples//batch_size for one epoch. But you may want to use a generator that performs random data augmentation for instance. And because of the random process, you will never have two identical training epochs. There isn't then a clear limit.
So, these parameters in fit_generator allow you to control the yields per epoch as you wish, although in standard cases you'll probably keep to the most obvious option: samples//batch_size.
Without data augmentation, the number of samples is static as Daniel mentioned.
Then, the number of samples for training is steps_per_epoch * batch size.
By using ImageDataGenerator in Keras, we make additional training data for data augmentation. Therefore, the number of samples for training can be set by yourself.
If you want two times training data, just set steps_per_epoch as (original sample size *2)/batch_size.