I have been trying to build a machine learning model using Keras which predicts the radiation dose based on pre-treatment parameters. My dataset has approximately 2200 samples of which 20% goes into validation and testing.
The problem with the target variable is that it is very skewed since large radiation doses are much more rare than the small ones. Hence, I suspect that my regression model fails to predict the large values at all, and predicts everything around the mean, which is apparent from the figure. I have tried to log-normalise the target variable to make it more normally distributed, but it has had no effect.
Any suggestion how to fix this?
Target variable
Regression predictions
Computing individual sample weights based on 10 histogram bins helped in my case. See the code below:
import pandas as pd
import numpy as np
from sklearn.utils.class_weight import compute_sample_weight
hist, bin_edges = np.histogram(training_targets, bins = 10)
classes = training_targets.apply(lambda x: pd.cut(x, bin_edges, labels = False,
include_lowest = True)).values
sample_weights = compute_sample_weight('balanced', classes)
Related
I don't understand why I have three different behaviors depending on the classifier I use, even though they should go hand in hand.
This is the code in order to go deeply in the question:
from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from lightgbm import LGBMClassifier
from sklearn.model_selection import cross_validate
import matplotlib.pyplot as plt
import numpy as np
#load data
wine = datasets.load_wine()
X = wine.data
y = wine.target
# some helper functions
def repeat_feature(X,which=1,times=1):
return np.hstack([X,np.hstack([X[:, :which]]*times)])
def do_the_job(X,y,clf):
return np.mean(cross_validate(clf, X, y,cv=5)['test_score'])
# define the classifiers
clf1=DecisionTreeClassifier(max_depth=25,random_state=42)
clf2=RandomForestClassifier(n_estimators=5,random_state=42)
clf3=LGBMClassifier(n_estimators=5,random_state=42)
# repeat up to 50 times the same feature and test the classifiers
clf1_result=[]
clf2_result=[]
clf3_result=[]
for i in range(1,50):
my_x=repeat_feature(X,times=i)
clf1_result.append(do_the_job(my_x,y,clf1))
clf2_result.append(do_the_job(my_x,y,clf2))
clf3_result.append(do_the_job(my_x,y,clf3))
# plot the mean of the cv-scores for each classifier
plt.figure(figsize=(12,7))
plt.plot(clf1_result,label='tree')
plt.plot(clf2_result,label='forest')
plt.plot(clf3_result,label='boost')
plt.legend()
The result of the previous script is the following graph:
What I want to verify is that by adding the same information (like a repeated feature) I would get a decrease in the score (which happens as expected for random forest).
The question is why does this not happen with the other two classifiers instead?
Why do their scores remain stable?
Am I missing something from the theoretical point of view?
Ty all
When fitting a single decision tree (sklearn.tree.DecisionTreeClassifier)
or a LightGBM model using its default behavior (lightgbm.LGBMClassifier), the training algorithm considers all features as candidates for every split, and always chooses the split with the best "gain" (reduction in the training loss).
Because of this, adding multiple identical copies of the same feature will not change the fit to the training data.
For random forest, on the other hand, the training algorithm randomly selects a subset of features to consider at each split. The random forest learns how to explain the training data by ensembling together multiple slightly-different models, and this can be effective because the different models explain different characteristics of the target. If you hold the number of trees + the number of leaves per tree constant, then adding copies of a feature reduces the diversity of the trees in the forest, which reduces the forest's fit to the training data.
I have trained the resnet50_v1b_voc for two different objects and created the params file from the training. While the params files are working absolutely fine when I am doing prediction by loading only one throughout but getting issue in results if I load both of them together.
from gluoncv import model_zoo, data, utils
from matplotlib import pyplot as plt
net = model_zoo.get_model('faster_rcnn_resnet50_v1b_voc', pretrained=True, ctx=mx.gpu())
net.load_parameters('/home/ubuntu/abc/faster_rcnn_resnet50_v1b_voc_best.params', ctx=mx.gpu())
net.load_parameters('/home/ubuntu/xyz/faster_rcnn_resnet50_v1b_voc_best.params', ctx=mx.gpu())
net.reset_class(['abc'], reuse_weights={'abc': 'abc'})
class_IDs, scores, bounding_boxs = net(x)
# This works fine if I predict the abc object
# But when I do this the confidence score is too low for the xyz object
# Individually if I perform this task with xyz params the results are perfect
net.reset_class(['xyz'], reuse_weights={'xyz': 'xyz'})
class_IDs, scores, bounding_boxs = net(x)
Not sure what I am doing wrong here.
The probable cause of the issue is the reset_class():
reset_class(classes, reuse_weights=None): Resets class categories and class predictors.
The pre-trained model supports many classes in the training dataset.
After call net.reset_class(['abc'],…), the net output will be changed to only support one class: ‘abc’.
So, it would perform worse when predicting on ‘xyz’.
I built various ML models using sklearn for a binary classification problem. The data-set is provided to me by my professor for this comparative study.
my jupyter notebook and dataset can be found here
As I am getting very low accuracy, I fear that I must be doing something wrong while building the model. So I tested my decision tree on the inbuilt data-set in sklearn (breast cancer data-set) which is very similar to my data-set as both are binary classifications. Here I get an mean accuracy of 95 %. So I think right now that the problem might be my data-set. Can I get some help on how do I pre-process my data or any other steps that I might look into to improve accuracy.
Encode labels
Categorical data are variables that contain label values rather than numeric values.The number of possible values is often limited to a fixed set.
For example, users are typically described by country, gender, age group etc. We will use Label Encoder to label the categorical data. Label Encoder is the part of SciKit Learn library in Python and used to convert categorical data, or text data, into numbers, which our predictive models can better understand.
#Encoding categorical data values
from sklearn.preprocessing import LabelEncoder
labelencoder_Y = LabelEncoder()
Y = labelencoder_Y.fit_transform(Y)
Feature scaling
Most of the times, your dataset will contain features highly varying in magnitudes, units and range. But since, most of the machine learning algorithms use Eucledian distance between two data points in their computations. We need to bring all features to the same level of magnitudes. This can be achieved by scaling. This means that you’re transforming your data so that it fits within a specific scale, like 0–100 or 0–1. We will use StandardScaler method from SciKit-Learn library.
#Feature Scalingfrom sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
Choosing Right model
You kight also want to vhoose the appropriate model. You can't just use neural nets or so for all problems it's the no free luch theorem. For this you could use K-fold cross validation, AIC and BIC
Imagine a simple regression problem, where you are using Gradient Descent. For correct implementation you will need to scale values using mean of entire training dataset. Imagine your model is already trained and you feed it another example you wish to predict. How do you scale it correctly with respect to previous dataset? Do you include new example in training set and then scale it with mean of this training dataset + new data points? How this should be done the right way?
By referring to new data points i mean something the model hasn't seen before, neither in training nor testing. How do you handle scaling for anything you pass to regr.predict() if scaling of training set is done with respect to the whole set and not a single observation.
Imagine you have ndarray of features:
to_predict = [10, 12, 1, 330, 1311, 225].
The dataset used for train and test is already oscillating around 0 for every feature. Taking into account the below answer (pseudo code, that's why I'm asking how to do it right):
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
new_Xs = X_train.append(to_predict)
X_train_std_with_new = scalar.fit_transform(new_Xs)
scaled_to_predit = X_train_std_with_new[-1]
regr.predict(scaled_to_predict) ??
I am trying to run Spark MLlib packages in pyspark with a test machine learning data set. I am splitting the data sets into half training data set and half test data set. Below is my code that builds the model. However, it shows weight of NaN, NaN.. across all dependent variables. Couldn't figure out why. But it works when I try to standardize the data with the StandardScaler function.
model = LinearRegressionWithSGD.train(train_data, step = 0.01)
# evaluate model on test data set
valuesAndPreds = test_data.map(lambda p: (p.label, model.predict(p.features)))
Thank you very much for the help.
Below is the code that I used to do the scaling.
scaler = StandardScaler(withMean = True, withStd = True).fit(data.map(lambda x:x.features))
feature = [scaler.transform(x) for x in data.map(lambda x:x.features).collect()]
label = data.map(lambda x:x.label).collect()
scaledData = [LabeledPoint(l, f) for l,f in zip(label, feature)]
Try scaling the features
StandardScaler standardizes features by scaling to unit variance and/or removing the mean using column summary statistics on the samples in the training set. This is a very common pre-processing step.
Standardization can improve the convergence rate during the optimization process, and also prevents against features with very large variances exerting an overly large influence during model training. Since you have some variables that are large numbers (eg: revenue) and some variables that are smaller (eg: number of clients), this should solve your problem.