I have written a code which takes list of items and outputs a json with unique items as keys and frequency as value.
The code below works fine when I test it
const tokenFrequency = tokens =>{
const setTokens=[...new Set(tokens)]
return setTokens.reduce((obj, tok) => {
const frequency = tokens.reduce((count, word) =>word===tok?count+1:count, 0);
const containsDigit = /\d+/;
if (!containsDigit.test(tok)) {
obj[tok.toLocaleLowerCase()] = frequency;
}
return obj;
}, new Object());
}
like
const x=["hello","hi","hi","whatsup","hey"]
console.log(tokenFrequency(x))
produces the output
{ hello: 1, hi: 2, whatsup: 1, hey: 1 }
but when i try with huge data corpus's list of words it seem to produce wrong result.
say if i feed a list words with the length of list being 14000+ it produces wrong results.
example:
https://github.com/Nahdus/word2vecDataParsing/blob/master/corpous/listOfWords.txt when this list in this page(linked above) to function the frequency of word "is" comes out to be 4, but the actual frequency is 907.
why does it behave like this for large data?
how can this be fixed?
You would need to normalize your tokens first by applying toLowerCase() to them, or a way to diferentiate between words that are the same but only differ in capitalization.
Reason:
Your small dataset has no Is words (with uppercase 'i'). The large dataset does have occurences of Is (with uppercase 'i'), which apparently has a frequency 4, which in turn overwrites your lowercase is's frequency.
Related
Imagine an object with strings as keys and their frequency count of occurrence as the value.
{"bravo charlie": 10, "alpha bravo charlie": 10, "delta echo foxtrot": 15, "delta echo": 7}
I am trying to optimize an algorithm such that A) any key that is a substring of another key AND has the same frequency value should be eliminated. The longer containing key should remain. B) Allow keys that are only a single word to remain even if contained by another key
The following pairwise comparison approach works but becomes very very slow on large objects. For example, an object with 560k keys is taking ~30 mins to complete the pairwise comparison:
// for every multi word key
// if a part of a longer key in candidates AND have same length delete
let keys = Object.keys(candidates)
.sort((a, b) => b.length - a.length)
.filter(key => {
if (key.split(" ").length === 1) {
return false
}
return true
});
// ^ order keys by length to speed up comparisons and filter out single words
checkKeyI: for (const keyi of keys) {
checkKeyJ: for (const keyj of keys) {
// because we pre-sorted if length is less then we are done with possible matches
if (keyj.length <= keyi.length) {
continue checkKeyI
}
// keys must not match exactly
if (keyj === keyi) {
continue checkKeyJ
}
// keyi must be a substring of keyj
if (!keyj.includes(keyi)) {
continue checkKeyJ
}
// they must have the same freq occurr values
if (candidates[keyj] === candidates[keyi]) {
delete candidates[keyi]
continue checkKeyI
}
}
}
The desired result would be:
{"alpha bravo charlie": 10, "delta echo foxtrot": 15, "delta echo": 7}
because bravo charlie was eliminated. Are they any obvious or clever ways to speed this up?
Sort the keys based on their length (use alphabetical ordering as a tie-breaker)
Starting with the longest key (and going in a descending order of length), add the keys to a trie. The leaf node for the key will store the value.
For inserting a key, start traversing the trie. If the key has a single word, insert it. If it has multiple words, and a match is found with the same frequency, don't insert it.
This will reduce your complexity from O(n^2) to O(n*m), where n = number of strings and m = length of longest string
I have an index array of nested two-dimensional arrays whose element pairs consist each of a phrase and a number.
In the absence of bidirectional text the phrase appears before the number for each element of the index array. In the presence of bidirectional text the phrase appears after the number.
If the phrase-number pairs are mixed -- namely, some phrase-number pairs appear with LTR text and some phrase-number pairs appear with RTL text, then the resulting order is mixed and makes for a messy list that cannot be used for input into the wordcloud.js function.
I have tried reversing the order of the elements of the phrase-number pairs whose phrases are written RTL before they are pushed onto the index array, but to no avail. The resulting index array is rendered confused.
var listItem = [];
var list = [];
$.each(countedPhrases, function(phrase, count) {
console.log('phrase: ' + phrase);
console.log('count: ' + count);
listItem = [phrase, count];
console.log('listItem: ' + listItem);
list.push(listItem);
});
console.log('list: ' + list);'
Sample console.log output.
phrase: النادر
count: 321
listItem: النادر,321
phrase: وتتلقاه
count: 321
listItem: وتتلقاه,321
phrase: 終結
count: 321
listItem: 終結,321
phrase: Podcast
count: 45
listItem: Podcast,45
list: النادر,321, وتتلقاه,321,終結,321,Podcast,45
The problem was resolved by wrapping the Arabic text in the following two Unicode controls \u2067 Arabic Text \u2069 before reading it into the array elements.
Without going into a lot of detail the code that corrected the problem appears as follows:
if (name === 'searchKeyword') {
if (agex.test(value)) {
nakedArabic = value.match(agex);
searchItem.target = '\u2067' + nakedArabic.join('') + '\u2069';
} else {
searchItem.target = value;
}
}
where agex.text(value) is a Regex test for the presence of Arabic characters.
In brief, the problem was brought about by something called spillover. The two Unicode control characters isolate the Arabic text while preserving its natural right-to-left directional ordering.
You can learn more about the problem and solution here.
Roddy
I am looking for an algorithm that will find the number of repeating substrings in a single string.
For this, I was looking for some dynamic programming algorithms but didn't find any that would help me. I just want some tutorial on how to do this.
Let's say I have a string ABCDABCDABCD. The expected output for this would be 3, because there is ABCD 3 times.
For input AAAA, output would be 4, since A is repeated 4 times.
For input ASDF, output would be 1, since every individual character is repeated 1 time only.
I hope that someone can point me in the right direction. Thank you.
I am taking the following assumptions:
The repeating substrings must be consecutive. That is, in case of ABCDABC, ABC would not count as a repeating substring, but it would in case of ABCABC.
The repeating substrings must be non-overalpping. That is, in case of ABCABC, ABC would not count as a repeating substring.
In case of multiple possible answers, we want the one with the maximum value. That is, in the case of AAAA, the answer should be 4 (a is the substring) rather than 2 (aa is the substring).
Under these assumptions, the algorithm is as follows:
Let the input string be denoted as inputString.
Calculate the KMP failure function array for the input string. Let this array be denoted as failure[]. This operation if of linear time complexity with respect to the length of the string. So, by definition, failure[i] denotes the length of the longest proper-prefix of the substring inputString[0....i] that is also a proper-suffix of the same substring.
Let len = inputString.length - failure.lastIndexValue. At this point, we know that if there is any repeating string at all, then it has to be of this length len. But we'll need to check for that; First, just check if len perfectly divides inputString.length (that is, inputString.length % len == 0). If yes, then check if every consecutive (non-overlapping) substring of len characters is the same or not; this operation is again of linear time complexity with respect to the length of the input string.
If it turns out that every consecutive non-overlapping substring is the same, then the answer would be = inputString.length/ len. Otherwise, the answer is simply inputString.length, as there is no such repeating substring present.
The overall time complexity would be O(n), where n is the number of characters in the input string.
A sample code for calculating the KMP failure array is given here.
For example,
Let the input string be abcaabcaabca.
Its KMP failure array would be - [0, 0, 0, 1, 1, 2, 3, 4, 5, 6, 7, 8].
So, our len = (12 - 8) = 4.
And every consecutive non-overlapping substring of length 4 is the same (abca).
Therefore the answer is 12/4 = 3. That is, abca is repeated 3 times repeatedly.
The solution for this with C# is:
class Program
{
public static string CountOfRepeatedSubstring(string str)
{
if (str.Length < 2)
{
return "-1";
}
StringBuilder substr = new StringBuilder();
// Length of the substring cannot be greater than half of the actual string
for (int i = 0; i < str.Length / 2; i++)
{
// We will iterate through half of the actual string and
// create a new string by appending the current character to the previous character
substr.Append(str[i]);
String clearedOfNewSubstrings = str.Replace(substr.ToString(), "");
// We will remove the newly created substring from the actual string and
// check if the length of the actual string, cleared of the newly created substring, is 0.
// If 0 it tells us that it is only made of its substring
if (clearedOfNewSubstrings.Length == 0)
{
// Next we will return the count of the newly created substring in the actual string.
var countOccurences = Regex.Matches(str, substr.ToString()).Count;
return countOccurences.ToString();
}
}
return "-1";
}
static void Main(string[] args)
{
// Input: {"abcdaabcdaabcda"}
// Output: 3
// Input: { "abcdaabcdaabcda" }
// Output: -1
// Input: {"barrybarrybarry"}
// Output: 3
var s = "asdf"; // Output will be -1
Console.WriteLine(CountOfRepeatedSubstring(s));
}
}
How do you want to specify the "repeating string"? Is it simply the first group of characters up until either a) the first character is found again, b) the pattern begins to repeat, or c) some other criteria?
So, if your string is "ABBAABBA", is that a 2 because "ABBA" repeats twice or is it 1 because you have "ABB" followed by "AAB"? What about "ABCDABCE" -- does "ABC" count (despite the "D" in between repetitions?) In "ABCDABCABCDABC", is the repeating string "ABCD" (1) or "ABCDABC" (2)?
What about "AAABBAAABB" -- is that 3 ("AAA") or 2 ("AAABB")?
If the end of the repeating string is another instance of the first letter, it's pretty simple:
Work your way through the string character by character, putting each character into another variable as you go, until the next character matches the first one. Then, given the length of the substring in your second variable, check the next bit of your string to see if it matches. Continue until it doesn't match or you hit the end of the string.
If you just want to find any length pattern that repeats regardless of whether the first character is repeated within the pattern, it gets more complicated (but, fortunately, it's the sort of thing computers are good at).
You'll need to go character by character building a pattern in another variable as above, but you'll also have to watch for the first character to reappear and start building a second substring as you go, to see if it matches the first. This should probably go in an array as you might encounter a third (or more) instance of the first character which would trigger the need to track yet another possible match.
It's not difficult but there is a lot to keep track of and it's a rather annoying problem. Is there a particular reason you're doing this?
What is an efficient way to produce phrase anagrams given a string?
The problem I am trying to solve
Assume you have a word list with n words. Given an input string, say, "peanutbutter", produce all phrase anagrams. Some contenders are: pea nut butter, A But Ten Erupt, etc.
My solution
I have a trie that contains all words in the given word list. Given an input string, I calculate all permutations of it. For each permutation, I have a recursive solution (something like this) to determine if that specific permuted string can be broken in to words. For example, if one of the permutations of peanutbutter was "abuttenerupt", I used this method to break it into "a but ten erupt". I use the trie to determine if a string is a valid word.
What sucks
My problem is that because I calculate all permutations, my solution runs very slow for phrases that are longer than 10 characters, which is a big let down. I want to know if there is a way to do this in a different way.
Websites like https://wordsmith.org/anagram/ can do the job in less than a second and I am curious to know how they do it.
Your problem can be decomposed to 2 sub-problems:
Find combination of words that use up all characters of the input string
Find all permutations of the words found in the first sub-problem
Subproblem #2 is a basic algorithm and you can find existing standard implementation in most programming language. Let's focus on subproblem #1
First convert the input string to a "character pool". We can implement the character pool as an array oc, where oc[c] = number of occurrence of character c.
Then we use backtracking algorithm to find words that fit in the charpool as in this pseudo-code:
result = empty;
function findAnagram(pool)
if (pool empty) then print result;
for (word in dictionary) {
if (word fit in charpool) {
result = result + word;
update pool to exclude characters in word;
findAnagram(pool);
// as with any backtracking algorithm, we have to restore global states
restore pool;
restore result;
}
}
}
Note: If we pass the charpool by value then we don't have to restore it. But as it is quite big, I prefer passing it by reference.
Now we remove redundant results and apply some optimizations:
Assuming A comes before B in the dictionary. If we choose the first word is B, then we don't have to consider word A in following steps, because those results (if we take A) would already be in the case where A is chosen as the first word
If the character set is small enough (< 64 characters is best), we can use a bitmask to quickly filter words that cannot fit in the pool. A bitmask mask which character is in a word, no matter how many time it occurs.
Update the pseudo-code to reflect those optimizations:
function findAnagram(charpool, minDictionaryIndex)
pool_bitmask <- bitmask(charpool);
if (pool empty) then print result;
for (word in dictionary AND word's index >= minDictionaryIndex) {
// bitmask of every words in the dictionary should be pre-calculated
word_bitmask <- bitmask(word)
if (word_bitmask contains bit(s) that is not in pool_bitmask)
then skip this for iteration
if (word fit in charpool) {
result = result + word;
update charpool to exclude characters in word;
findAnagram(charpool, word's index);
// as with any backtracking algorithm, we have to restore global states
restore pool;
restore result;
}
}
}
My C++ implementation of subproblem #1 where the character set contains only lowercase 'a'..'z': http://ideone.com/vf7Rpl .
Instead of a two stage solution where you generate permutations and then try and break them into words, you could speed it up by checking for valid words as you recursively generate the permutations. If at any point your current partially-complete permutation does not correspond to any valid words, stop there and do not recurse any further. This means you don't waste time generating useless permutations. For example, if you generate "tt", there is no need to permute "peanubuter" and append all the permutations to "tt" because there are no English words beginning with tt.
Suppose you are doing basic recursive permutation generation, keep track of the current partial word you have generated. If at any point it is a valid word, you can output a space and start a new word, and recursively permute the remaining character. You can also try adding each of the remaining characters to the current partial word, and only recurse if doing so results in a valid partial word (i.e. a word exists starting with those characters).
Something like this (pseudo-code):
void generateAnagrams(String partialAnagram, String currentWord, String remainingChars)
{
// at each point, you can either output a space, or each of the remaining chars:
// if the current word is a complete valid word, you can output a space
if(isValidWord(currentWord))
{
// if there are no more remaining chars, output the anagram:
if(remainingChars.length == 0)
{
outputAnagram(partialAnagram);
}
else
{
// output a space and start a new word
generateAnagrams(partialAnagram + " ", "", remainingChars);
}
}
// for each of the chars in remainingChars, check if it can be
// added to currentWord, to produce a valid partial word (i.e.
// there is at least 1 word starting with these characters)
for(i = 0 to remainingChars.length - 1)
{
char c = remainingChars[i];
if(isValidPartialWord(currentWord + c)
{
generateAnagrams(partialAnagram + c, currentWord + c,
remainingChars.remove(i));
}
}
}
You could call it like this
generateAnagrams("", "", "peanutbutter");
You could optimize this algorithm further by passing the node in the trie corresponding to the current partially completed word, as well as passing currentWord as a string. This would make your isValidPartialWord check even faster.
You can enforce uniqueness by changing your isValidWord check to only return true if the word is in ascending (greater or equal) alphabetic order compared to the previous word output. You might also need another check for dupes at the end, to catch cases where two of the same word can be output.
First, I am very new to nodejs. I want a function in nodejs that divides 2^128 into 'n' equal spaces and returns a list of length n. I should be able to use this list to determine in which range a given integer belongs. I expect the code to like the following:
function divideEqually(n){
/**
code here
**/
return aListOfRanges;
}
function findIndex(hashDigest, aListOfRanges){
/*
inspect ranges and find index
*/
return someIndexInList;
}
var list = divideEqually(15); //Returns a list of 15 equally spaced ranges
var index = findIndex('d9729feb74992cc3482b350163a1a010', list) //Find index of a hex digest
How do I do this efficiently. The ranges should be computed lazily as 2^128 will be a huge number whereas 'n' is expected to be less than 20.