Can anyone recommend a public Big-GAN-deep trainable with my own dataset, to do data augmentation, which produces 512x512 pixel images? possibly usable by Colab, Pytorch. Thanks in advance.
Related
I am currently working on fingers-count deep learning problem. When you look at the dataset, images in the training and validation set are very basic and are almost the same. The network can achieve high training and validation accuracies. But when it comes to prediction in real-life images, it performs very badly(this is because the model has been trained on very basic images).
To overcome this, I converted the training and validation images to HSV(Hue-Saturation-Value) and trained the model on new HSV images. Example of 1 such image from new training set is:
I then convert my image from real life to HSV and pass it to model for prediction. But still, the model is not able to predict correctly. I assumed that since the training images and predicting image are almost same after applying HSV, the model should be predicting good. Is there something which I am thinking incorrectly here? Can HSV images be actually used for training CNN?
It seems you have the overfitting issue, and your model only memorize the simple samples of the training set and in contrast it can not generalize to more complex and diverse data.
In the context of Deep Learning there are various methods to avoid overfitting and I think you don't need to transform your input to HSV necessarily. First of all you can apply various data augmentation methods like random crop or rotation to create various versions of your data. If this method does not work, you can use a smaller model or applying techniques such as Drop Out or Regularization.
Here is a good tutorial from TensorFlow.
I am training a CNN model(made using Keras). Input image data has around 10200 images. There are 120 classes to be classified. Plotting the data frequency, I can see that sample data for every class is more or less uniform in terms of distribution.
Problem I am facing is loss plot for training data goes down with epochs but for validation data it first falls and then goes on increasing. Accuracy plot reflects this. Accuracy for training data finally settles down at .94 but for validation data its around 0.08.
Basically its case of over fitting.
I am using learning rate of 0.005 and dropout of .25.
What measures can I take to get better accuracy for validation? Is it possible that sample size for each class is too small and I may need data augmentation to have more data points?
Hard to say what could be the reason. First you can try classical regularization techniques like reducing the size of your model, adding dropout or l2/l1-regularizers to the layers. But this is more like randomly guessing the models hyperparameters and hoping for the best.
The scientific approach would be to look at the outputs for your model and try to understand why it produces these outputs and obviously checking your pipeline. Did you had a look at the outputs (are they all the same)? Did you preprocess the validation data the same way as the training data? Did you made a stratified train/test-split, i.e. keeping the class distribution the same in both sets? Is the data shuffles when you feed it to your model?
In the end you have about ~85 images per class which is really not a lot, compare CIFAR-10 resp. CIFAR-100 with 6000/600 images per class or ImageNet with 20k classes and 14M images (~500 images per class). So data augmentation could be beneficial as well.
I was wondering if having images with watermarks in the training set when training a CNN (MobileNetv2) will affect its accuracy. Would simply removing them from the set be a better thing to do? Thank you!
I am trying to train a classification network. For my classification purpose, I don't have any publicly available data set. For deep ConvNets, large number of training data is needed. It is really time consuming to create such a large data set. Is there any way to generate large number of images from few images?
For example, 1000 images from a few 20 images for training.
If someone knows the answer, kindly help me.
Thanking you in advance.
You can use image augmentation for generating more images (for example flipping the image)
https://pytorch.org/docs/stable/torchvision/transforms.html
I am new to AI and I am using Keras and Tensorflow to train CNNs. My dataset is heavily unbalanced and I want to use class weights to counter that.
After a small search in the internet I found out that I can use scikit learn's class weight() and sample weight() to get the class weights and sample weights respectively and it can be passed to model.fit() in Keras. But I am unsure how to implement it programmatically for hot encoded outputs.
Can someone provide sample code explaining how to implement classweights for hot encoded outputs with Keras?
Thanks in advance 😁