I have two data sets, let's say A & B. I have an OLS regression model prepared by using dataset A, and now I want to update this model by using new dataset B.
Bayesian updating can be one way to update such model with the given formula:
here the defined parameters are :
๐ฝ๐ก and ๐ฝ๐ are the parameter vectors of the estimation context models and the locally estimated models respectively; ๐๐ก and ๐๐ are corresponding vectors of variance in estimation and application contexts.
Here I am not able to understand what should be the terms, ๐๐ก and ๐๐. Are they variances of the predicted valules by the models?
Related
TL;DR How can the Pearson correlation coefficient between ground truth labels and cosine similarity scores evaluate the performance of a sentence embedding model? A positive/negative linear relationship between the two doesn't necessarily indicate that a model is accurate, just that they move together, which to me doesn't seem like a good way to evaluate the performance of a sentence embedding model.
I'm training a model to be able to tell if two questions are similar or not. I first continue pre-training using MLM (masked language modeling) and finally fine-tune on the STS dataset. For fine-tuning, I'm using this example python file https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/sts/training_stsbenchmark.py. At the end of the file, it says to "load the stored model and evaluate its performance on STS benchmark dataset", and it uses this file to evaluate the performance of the model https://github.com/UKPLab/sentence-transformers/blob/master/sentence_transformers/evaluation/EmbeddingSimilarityEvaluator.py.
The second file has a few metrics for evaluation (cosine similarity being one of them), and it uses the Pearson correlation coefficient and Spearman correlation coefficient for each metric to evaluate the performance of the model. What I'm not understanding is: how does calculating the relationship (correlation coefficient) between the ground truth labels and the cosine similarity contribute to measuring the performance of the model? Even if the two have similar movement patterns i.e. a high correlation coefficient, that doesn't mean the model is performing well, does it?
I have this dataset in which the positive class consists of component failures for a specific component of the APS system.
I am doing Predictive Maintenance using Microsoft Azure Machine Learning Studio.
As you can see from the pictures below, I am using 4 algorithm: Logistic Regression, Random Forest, Decision Tree and SVM. And you can see that the Output dataset in the score model node consists of 16k rows. However, when I see the output of the Evaluate Model, in the confusion matrix there are only 160 observations for the Logistic Regression, and the correct number, 16k for Random Forest. I have the same problem, only 160 observations in the models of Decision Tree and SVM. And the same problem is repeated in other experiments for example after feature selection, normalization etc.: some evaluate model does not use all the rows of the test dataset, and some other node does it.
How can I fix this problem? Because I am interested in the real number of false positive and false negatives.
The output metrics shown are based on the validation set (e.g. โvalidation metricโ, โval-accuracyโ).All the metrics computed and displayed are on validation set and not on the original training set. All those metrics are calculated only over the validation set without considering the training set, otherwise we would inflate the performances of the model by considering data already used to train the model.
Can anyone please explain the difference between generalized linear model and logistic regression model table with statsmodels. Why do I get different results with both the models while performing logistic regression??
GLM is a generalized linear model and Logit Model is specific to models with binary classification. While using GLM model you have to mention the parameter family which can be binomial (logit model), Poisson etc. This parameter is not required in Logit model as its only for binary output. The difference in the output can be because of regularization parameters.
GLM with Binomial family and Logit link and the discrete Logit model represent the same underlying model and both fit by maximum likelihood estimation
Implementation, default optimization method, options for extensions and the availability of some results statistics differs between GLM and their discrete counterparts.
For regular, well defined cases and well behaved data, both model produce the same results, up to convergence tolerance in the optimization and numerical noise.
However, the behavior of the estimation methods can differ in corner cases and for singular or near-singular data.
Related aside:
For models with non-canonical links using GLM there can be differences in the definitions used across optimization methods, e.g. whether standard errors are based on expected or observed information matrix. With canonical links like logit those two are the same.
I have trained two models on different datasets and saved weights of each model as ModelA.h5 and ModelB.h5
I want to average these weights and create a new folder called ModelC.h5 and load it on the same model architechture.
How do I do it?
Model trained on different datasets can't just be added like this. It looks something like this. Let's say like this, train one person to classify 1000 images into 5 classes, then, train another person to classify another 1000 images into same 5 classes. Now, you want to combine them into one.
Rather, what you can do is take ensemble of both the networks. There are multiple ways to ensemble the predictions of both models using Max Voting, Averaging or Weighted Average, Bagging and Boosting, etc. Ensemble helps to boost the weak classifiers into one strong classifier.
You can refer to this link to read more about different types of ensemble: Link
How should one decide between using a linear regression model or non-linear regression model?
My goal is to predict Y.
In case of simple x and y dataset I could easily decide which regression model should be used by plotting a scatter plot.
In case of multi-variant like x1,x2,...,xn and y. How can I decide which regression model has to be used? That is, How will I decide about going with simple linear model or non linear models such as quadric, cubic etc.
Is there any technique or statistical approach or graphical plots to infer and decide which regression model has to be used? Please advise.
That is a pretty complex question.
You start visually first: if the data is normally distributed, and satisfy conditions for classical linear model, you use linear model. I normally start by making a scatter plot matrix to observe the relationships. If it is obvious that the relationship is non linear then you use non-linear model. But, a lot of times, I visually inspect, assuming that the number of factors are just not too many.
For example, this would be a non linear model:
However, if you want to use data mining (and computationally demanding methods), I suggest starting with stepwise regression. What you do is set a model evaluation criteria first: could be R^2 for example. You start a model with nothing and sequentially add predictors or permutations of them until your model evaluation criteria is "maximized". However, adding new predictor almost always increases R^2, a type of over-fitting.
The solution is to split the data into training and testing. You should make model based on the training and evaluate the mean error on testing. The best model will be the one that that minimized mean error on the testing set.
If your data is sparse, try integrating ridge or lasso regression in model evaluation.
Again, this is a kind of a complex question. The answer also kind of depends on whether you are building descriptive or explanatory model.