Let's say that we have a dataset with 3 variables: age_banded,number_of_accounts_banded and country
These 3 variables can have the value: "missing" when there is no information about that customer. For logistic regression with categorical variables we want to take dummy variables. What do we do with the dummies for the missing category? age_banded:missing,number_of_accounts_banded:missing and country:missing will all have correlation equal to 1 so there is no need to have all three dummies in the dataset.
Related
I have several features in my Logistic regression model. One of the features has multiple NULL values. I want these rows to be included in my model.
How does Logistic Regression handle null values when computing? Does it take into account the null value as part of the feature? I did not come across errors when running.
If Logistic Regression ignores it, would I have to change the null values to an actual string value in order for logistic regression to recognize it as part of the model?
I am attempting to create a prediction model using multiple linear regression.
One of the predictor variables I want to use is a percentage, so it ranges from 0 - 100. I hypothesize that when it’s <50% there will be a negative effect on the target variable and when >50% a positive effect.
The mean of the predictor variable isn’t exactly 50 in my data set so I am unsure if I centre or Standardize this variable, or just subtract 50 from it to create the split I am looking for.
I am very new to statistics and self teaching myself at the moment, any help is greatly appreciated.
I calculated a correlation of two dependent variables (size of plot/house vs cost), the correlation stands at 0.87. I want to use this index to measure the increase or decrease in cost if size is increased or decreased. Is it possible using correlation? How?
Correlation only tells us how much two variables are linearly related based on the data we have, but in it does not provide a method to calculate the value of variable given the value of another.
If the variables are linearly related we can predict the actual values that a variable Y will assume when a variable X has some value using Linear Regression:
The idea is to try and fit the data to a linear function, and use it to predict the values:
Y = bX + a
Usually we first discover if two variables are related using a Correlation Coefficient(ex. Pearson Coefficient), then we use a Regression method(ex. Linear) to predict values of a variable of interest given another.
Here is an easy to follow tutorial on Linear Regression in Python with some theory:
https://realpython.com/linear-regression-in-python/#what-is-regression
Here a tutorial on the typical problem of house price prediction:
https://blog.akquinet.de/2017/09/19/predicting-house-prices-on-kaggle-part-i/
i validated that the label field has only 2 possible values by running select distinct on it.
but spark gives an error that it can't assign binomial family in logistic regression for 3 classes in label field
I have a data with some categorical variables and I want to run a logistic regression using Mllib , it seems like the model support only continous variables.
Does anyone know how to deal with this please ?
Logistic regression, like the other linear models, takes as input an RDD whereas a LabeledPoint is a Double (the label) and the associated Vector (a double Array).
Categorical values (Strings) are not supported, however you could convert those to binary columns.
For example if you have a column RAG taking values Red, Amber and Green, you would add three binary column isRed, isAmber and isGreen of which only one of them is 1 (true) and the others are 0 (zero) for each sample.
See as further explanation: http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.DictVectorizer.html