Pointwise Mutual Information or PMI for short is given as
https://latex.codecogs.com/svg.image?%5Cfrac%7BP(bigram)%7D%7BP(1st%20Word)%20*%20P(2nd%20Word)%7D
Which is the same as:
https://latex.codecogs.com/svg.image?log_%7B2%7D%5Cfrac%7B%5Cfrac%7BBigramOccurrences%7D%7BN%7D%7D%7B%5Cfrac%7B1stWordOccurrences%7D%7BN%7D%20*%20%5Cfrac%7B2ndWordOccurrences%7D%7BN%7D%7D
Where BigramOccurrences is number of times bigram appears as feature, 1stWordOccurrences is number of times 1st word in bigram appears as feature and 2ndWordOccurrences is number of times 2nd word from the bigram appears as feature. Finally N is given as number of total words.
We can tweak the following formula a bit and get the following:
https://latex.codecogs.com/svg.image?log_%7B2%7D%5Cfrac%7BBigramOccurrences*%20N%7D%7B1stWordOccurrences%20*%202ndWordOccurrences%7D
Now the part that confuses me a bit is the N in the formula. From what I understand it should be a total number of feature occurrences, even though it is described as total number of words. So essentially I wouldn't count total number of words in dataset (as that after some preprocessing doesn't seem like it makes sense to me), but rather I should count the total number of times all bigrams that are features have appeared as well as single words, is this correct?
Finally, one other thing that confuses me a bit is when I work with more than bigrams, so for example trigrams are also part of features. I would then, when calculating PMI for a specific bigram, not consider count of trigrams for N in the given formula? Vice-versa when calculating PMI for a single trigram, the N wouldn't account for number of bigrams, is this correct?
If I misunderstood something about formula, please let me know, as the resources I found online don't make it really clear to me. Also I am sorry for not including formulas directly but rather as urls, I don't have enough reputation to link images and show them properly...
Related
I have two sets of tokenised sentences A and B and I want to calculate the overlap between them in terms of common tokens. For example, the overlap between two individual sentences a1 today is a good day and b1 today I went to a park is 2 (today and a). I need a simple string matching method, without fuzzy or advanced methods. So the result is a matrix between all sentences in A and B with an overlap count for each pair.
The problem is that, while trivial, it is a quadratic operation (size of A x size of B pair-wise comparisons). With large data, the computation gets very slow very quickly. What would be a smart way of computing this avoiding pair-wise comparisons or doing them very fast? Are there packages/data structures particularly good for this?
I have n players to assign to n games. 10 <= n <= 20. Each player can sign up for up to 3 games but will only get one. Different players have different score for each game they sign up for.
Example with 10 players:
It's always possible to assign players x to game x but it will not always give the highest score in total.
My goal is to get as high score as possible and I therefore want to test the different permutations. I could teoretically test all permutations and throw away the unfeasible ones but it will give me a hughe number of possibilities (n!).
Is it possible to reduce the problem with the sign up limit of max 3 games? Maybe this can be done more easily than my approach? Any thoughts?
I'm working in Excel VBA.
I hope you find this as interesting as I do ...
Sorry if you find this unclear! My question is if it's possible to generate a subset of all the permutations. More precise only the feasible ones (which are the ones without any zero score).
Well, just set this up in the solver using Linear Programming as you can see in the image. Have shown the formulae so you can build it as well, along with the solver settings.
Won't give the permutations, but does solve for the highest combination.
Edit, updated image... it now shows correct ranges for the calculations, after trying to make it fit a reasonable size...
How can I generate those numbers in Excel.
I have to generate 8 random numbers whose sum is always 320. I need around 100 sets or so.
http://en.wikipedia.org/wiki/User:Skinnerd/Simplex_Point_Picking. Two methods are explained here.
Or any other way so I can do it in Excel.
You could use the RAND() function to generate N numbers (8 in your case) in column A.
Then, in column B you could use the following formula B1=A1/SUM(A:A)*320, B2=A2/SUM(A:A)*320 and so on (where 320 is the sum that you are interested into).
So you can just enter =RAND() in A1, then drag it down to A8. Then enter =A1/SUM(A:A)*320 in B1 and drag it to B8. B1:B8 now contains 8 random numbers that sum up to 320.
Sample output:
I'm a bit late to the game here - but fyi if only integers required then:
=LET(x_,RANDARRAY(8,1,1,1000000,1),y_,ROUND(x_*320/SUM(x_),0),y_)
is somewhat similar to the favourite soln above, albeit parsimonious (formula in single cell required to produce desired array , no helper column). Also addresses insignificant decimal points, albeit you may need to allocate back the deficit / surplus due to the occasional rounding error which may yield a sum total of 321 or 319. Could do this in a random fashion again using something like index(y_,randbetween(1,8))+320-sum(y_) in formula above - or resort to the infamous helper fn..
Someone commented the favourite soln above (and thus mine, since it stems from a similar concept/approach) is not uniform - I'm not sure this was required; a uniform spread would impede the random nature (and is arguably far simpler as you simply divide a sizeable range into distinct octiles, and follow the same approach already laid out here - not sure where/why the notion that a random spread should be arbitrarily/mechanically 'forced' to adopting some type of non-random spread.. anyways... I obviously haven't read the problem properly (ehem).
I'm a bit late to the game here - but fyi if only integers required then:
=LET(x_,RANDARRAY(8,1,1,1000000,1),y_,ROUND(x_*320/SUM(x_),0),y_)
is somewhat similar to the favourite soln above, albeit parsimonious (formula in single cell required to produce desired array , no helper column). Also addresses insignificant decimal points, albeit you may need to allocate back the deficit / surplus due to the occasional rounding error which may yield a sum total of 321 or 319. Could do this in a random fashion again using something like index(y_,randbetween(1,8))+320-sum(y_) in formula above - or resort to the infamous helper fn..
I am not sure whether this is the right place to ask this question.
As this is more like a logic question.. but hey no harm in asking.
Suppose I have a huge list of data (customers)
and they all have a data_id
Now I want to select lets say split the data in ratio lets say 10:90 split.
Now rather than stating a condition that (example)
the sum of digits is even...go to bin 1
the sum of digits is odd.. go to bin 2
or sum of last three digits are x then go to bin 1
sum of last three digits is not x then go to bin 2
Now this might result in uneven data collection..sometimes it might be able to find the data.. more (which is fine) but sometimes it might not be able to find enough data
Is there a way (probabilistically speaking)
which says.. sample size is always greater than x%
Thanks
You want to partition your data by a feature that is uniformly distributed. Hash functions are designed to have this property ... so if you compute a hash of your customer ID, and then partition by the first n bits to get 2^n bins, each bin should have approximately the same number of items. (You can then select, say, 90% of your bins to get 90% of the data.) Hope this helps.
We have a application where users enter prices all day. These prices are recorded in a table with a timestamp and then used for producing charts of how the price has moved... Every now and then the user enters a price wrongly (eg. puts in a zero to many or to few) which somewhat ruins the chart (you get big spikes). We've even put in an extra confirmation dialogue if the price moves by more than 20% but this doesn't stop them entering wrong values...
What statistical method can I use to analyse the values before I chart them to exclude any values that are way different from the rest?
EDIT: To add some meat to the bone. Say the prices are share prices (they are not but they behave in the same way). You could see prices moving significantly up or down during the day. On an average day we record about 150 prices and sometimes one or two are way wrong. Other times they are all good...
Calculate and track the standard deviation for a while. After you have a decent backlog, you can disregard the outliers by seeing how many standard deviations away they are from the mean. Even better, if you've got the time, you could use the info to do some naive Bayesian classification.
That's a great question but may lead to quite a bit of discussion as the answers could be very varied. It depends on
how much effort are you willing to put into this?
could some answers genuinely differ by +/-20% or whatever test you invent? so will there always be need for some human intervention?
and to invent a relevant test I'd need to know far more about the subject matter.
That being said the following are possible alternatives.
A simple test against the previous value (or mean/mode of previous 10 or 20 values) would be straight forward to implement
The next level of complexity would involve some statistical measurement of all values (or previous x values, or values of the last 3 months), a normal or Gaussian distribution would enable you to give each value a degree of certainty as to it being a mistake vs. accurate. This degree of certainty would typically be expressed as a percentage.
See http://en.wikipedia.org/wiki/Normal_distribution and http://en.wikipedia.org/wiki/Gaussian_function there are adequate links from these pages to help in programming these, also depending on the language you're using there are likely to be functions and/or plugins available to help with this
A more advanced method could be to have some sort of learning algorithm that could take other parameters into account (on top of the last x values) a learning algorithm could take the product type or manufacturer into account, for instance. Or even monitor the time of day or the user that has entered the figure. This options seems way over the top for what you need however, it would require a lot of work to code it and also to train the learning algorithm.
I think the second option is the correct one for you. Using standard deviation (a lot of languages contain a function for this) may be a simpler alternative, this is simply a measure of how far the value has deviated from the mean of x previous values, I'd put the standard deviation option somewhere between option 1 and 2
You could measure the standard deviation in your existing population and exclude those that are greater than 1 or 2 standard deviations from the mean?
It's going to depend on what your data looks like to give a more precise answer...
Or graph a moving average of prices instead of the actual prices.
Quoting from here:
Statisticians have devised several methods for detecting outliers. All the methods first quantify how far the outlier is from the other values. This can be the difference between the outlier and the mean of all points, the difference between the outlier and the mean of the remaining values, or the difference between the outlier and the next closest value. Next, standardize this value by dividing by some measure of scatter, such as the SD of all values, the SD of the remaining values, or the range of the data. Finally, compute a P value answering this question: If all the values were really sampled from a Gaussian population, what is the chance of randomly obtaining an outlier so far from the other values? If the P value is small, you conclude that the deviation of the outlier from the other values is statistically significant.
Google is your friend, you know. ;)
For your specific question of plotting, and your specific scenario of an average of 1-2 errors per day out of 150, the simplest thing might be to plot trimmed means, or the range of the middle 95% of values, or something like that. It really depends on what value you want out of the plot.
If you are really concerned with the true max and true of a day's prices, then you have to deal with the outliers as outliers, and properly exclude them, probably using one of the outlier tests previously proposed ( data point is x% more than next point, or the last n points, or more than 5 standard deviations away from the daily mean). Another approach is to view what happens after the outlier. If it is an outlier, then it will have a sharp upturn followed by a sharp downturn.
If however you care about overall trend, plotting daily trimmed mean, median, 5% and 95% percentiles will portray history well.
Choose your display methods and how much outlier detection you need to do based on the analysis question. If you care about medians or percentiles, they're probably irrelevant.