I am using Weka IBk for text classificaiton. Each document basically is a short sentence. The training dataset contains 15,000 documents. While testing, I can see that k=1 gives the best accuracy? How can this be explained?
If you are querying your learner with the same dataset you have trained on with k=1, the output values should be perfect barring you have data with the same parameters that have different outcome values. Do some reading on overfitting as it applies to KNN learners.
In the case where you are querying with the same dataset as you trained with, the query will come in for each learner with some given parameter values. Because that point exists in the learner from the dataset you trained with, the learner will match that training point as closest to the parameter values and therefore output whatever Y value existed for that training point, which in this case is the same as the point you queried with.
The possibilities are:
The data training with data tests are the same data
Data tests have high similarity with the training data
The boundaries between classes are very clear
The optimal value for K is depends on the data. In general, the value of k may reduce the effect of noise on the classification, but makes the boundaries between each classification becomes more blurred.
If your result variable contains values of 0 or 1 - then make sure you are using as.factor, otherwise it might be interpreting the data as continuous.
Accuracy is generally calculated for the points that are not in training dataset that is unseen data points because if you calculate the accuracy for unseen values (values not in training dataset), you can claim that my model's accuracy is the accuracy that is been calculated for the unseen values.
If you calculate accuracy for training dataset, KNN with k=1, you get 100% as the values are already seen by the model and a rough decision boundary is formed for k=1. When you calculate the accuracy for the unseen data it performs really bad that is the training error would be very low but the actual error would be very high. So it would be better if you choose an optimal k. To choose an optimal k you should be plotting a graph between error and k value for the unseen data that is the test data, now you should choose the value of the where the error is lowest.
To answer your question now,
1) you might have taken the entire dataset as train data set and would have chosen a subpart of the dataset as the test dataset.
(or)
2) you might have taken accuracy for the training dataset.
If these two are not the cases than please check the accuracy values for higher k, you will get even better accuracy for k>1 for the unseen data or the test data.
Related
I am just curious-- the chi square test-statistic can be sensitive to large count data. Has anyone ever seen/heard of someone:
1. Counting the raw frequencies
2. Taking the logarithm of the frequencies
3. Rounding that to be an integer
4. Performing the chi-square test on the modified count data
?
I think a better approach would be through an ANOVA model or linear regression with interactions, but still am curious.
Thanks!
I've tried the problem using the above approach.
I am trying to implement a scoring model following the link https://rstudio-pubs-static.s3.amazonaws.com/376828_032c59adbc984b0ab892ce0026370352.html#1_introduction.
Post the entire implementation though, When I create pivot with my generated scores and the original labels, the average scores for "good' labels is significantly lower than the ones for " high" labels.
Hence, my problem can be oversimplified to why would logistic regression give reversed probabilities for 0-1 target variable( In my model I am using 0 for bad and 1 for good).
Any suggestions and solutions would be welcome.
I am trying to implement VGGNet-16 for depth map prediction from single image. In the training the RMSE loss comes out to be 0.1599.
That loss value, is it in percentage or not?
No, if you want a percentage of a correctly classified data you can look at a value of accuracy.
Definition of RMSE from Wikipedia:
The root-mean-square deviation (RMSD) or root-mean-square error (RMSE) is a frequently used measure of the differences between values (sample and population values) predicted by a model or an estimator and the values actually observed.
It's always non-negative, and values closer to zero are better.
I have constructed a GMM-UBM model for the speaker recognition purpose. The output of models adapted for each speaker some scores calculated by log likelihood ratio. Now I want to convert these likelihood scores to equivalent number between 0 and 100. Can anybody guide me please?
There is no straightforward formula. You can do simple things like
prob = exp(logratio_score)
but those might not reflect the true distribution of your data. The computed probability percentage of your samples will not be uniformly distributed.
Ideally you need to take a large dataset and collect statistics on what acceptance/rejection rate do you have for what score. Then once you build a histogram you can normalize the score difference by that spectrogram to make sure that 30% of your subjects are accepted if you see the certain score difference. That normalization will allow you to create uniformly distributed probability percentages. See for example How to calculate the confidence intervals for likelihood ratios from a 2x2 table in the presence of cells with zeroes
This problem is rarely solved in speaker identification systems because confidence intervals is not what you want actually want to display. You need a simple accept/reject decision and for that you need to know the amount of false rejects and accept rate. So it is enough to find just a threshold, not build the whole distribution.
I've trained a model, the test results on test-set are okay.
Now I have saved the model as 'Trained model' and made a new experiment into a new dataset, for making predictions where I don't have the actual value's.
Normally, the trained model gives me a scored label result per instance.
But now, the scored label results are empty. Also when I convert the score results to CSV the scored labels column is empty.
Even stranger, when I take a look at the Statistics of the score Visualize tab, I DO see the statistics of the scored values. But no actual scored values...
Is this a bug? Or am I forgetting something important? Whats going on ;) ?
If your test dataset is missing the dependent values, your predictive experiment may fail for some models. The solution is to pad your csv file with zero values instead of blank values.
I had this same issue and it was frustrating but I think I finally understand why this is happening.
When I was training my experiment, part of the cleaning process was populating missing values or trimming existing data with R.
The problem is if one of those features is optional. For instance if you have a column that is not filled in, the scoring model will fail in the web service.
To see if this problem affects you, go to your Predictive Experiment and visualize the Score Model results. If you see empty Predicted Label & Predicted Score values, you can easily see which data points are missing.