How to prepare bidirectional text for use with wordcloud2.js - text

I have an index array of nested two-dimensional arrays whose element pairs consist each of a phrase and a number.
In the absence of bidirectional text the phrase appears before the number for each element of the index array. In the presence of bidirectional text the phrase appears after the number.
If the phrase-number pairs are mixed -- namely, some phrase-number pairs appear with LTR text and some phrase-number pairs appear with RTL text, then the resulting order is mixed and makes for a messy list that cannot be used for input into the wordcloud.js function.
I have tried reversing the order of the elements of the phrase-number pairs whose phrases are written RTL before they are pushed onto the index array, but to no avail. The resulting index array is rendered confused.
var listItem = [];
var list = [];
$.each(countedPhrases, function(phrase, count) {
console.log('phrase: ' + phrase);
console.log('count: ' + count);
listItem = [phrase, count];
console.log('listItem: ' + listItem);
list.push(listItem);
});
console.log('list: ' + list);'
Sample console.log output.
phrase: النادر
count: 321
listItem: النادر,321
phrase: وتتلقاه
count: 321
listItem: وتتلقاه,321
phrase: 終結
count: 321
listItem: 終結,321
phrase: Podcast
count: 45
listItem: Podcast,45
list: النادر,321, وتتلقاه,321,終結,321,Podcast,45

The problem was resolved by wrapping the Arabic text in the following two Unicode controls \u2067 Arabic Text \u2069 before reading it into the array elements.
Without going into a lot of detail the code that corrected the problem appears as follows:
if (name === 'searchKeyword') {
if (agex.test(value)) {
nakedArabic = value.match(agex);
searchItem.target = '\u2067' + nakedArabic.join('') + '\u2069';
} else {
searchItem.target = value;
}
}
where agex.text(value) is a Regex test for the presence of Arabic characters.
In brief, the problem was brought about by something called spillover. The two Unicode control characters isolate the Arabic text while preserving its natural right-to-left directional ordering.
You can learn more about the problem and solution here.
Roddy

Related

converting list of words into frequency json

I have written a code which takes list of items and outputs a json with unique items as keys and frequency as value.
The code below works fine when I test it
const tokenFrequency = tokens =>{
const setTokens=[...new Set(tokens)]
return setTokens.reduce((obj, tok) => {
const frequency = tokens.reduce((count, word) =>word===tok?count+1:count, 0);
const containsDigit = /\d+/;
if (!containsDigit.test(tok)) {
obj[tok.toLocaleLowerCase()] = frequency;
}
return obj;
}, new Object());
}
like
const x=["hello","hi","hi","whatsup","hey"]
console.log(tokenFrequency(x))
produces the output
{ hello: 1, hi: 2, whatsup: 1, hey: 1 }
but when i try with huge data corpus's list of words it seem to produce wrong result.
say if i feed a list words with the length of list being 14000+ it produces wrong results.
example:
https://github.com/Nahdus/word2vecDataParsing/blob/master/corpous/listOfWords.txt when this list in this page(linked above) to function the frequency of word "is" comes out to be 4, but the actual frequency is 907.
why does it behave like this for large data?
how can this be fixed?
You would need to normalize your tokens first by applying toLowerCase() to them, or a way to diferentiate between words that are the same but only differ in capitalization.
Reason:
Your small dataset has no Is words (with uppercase 'i'). The large dataset does have occurences of Is (with uppercase 'i'), which apparently has a frequency 4, which in turn overwrites your lowercase is's frequency.

How to manipulate strings in GO to reverse them?

I'm trying to invert a string in go but I'm having trouble handling the characters. Unlike C, GO treats strings as vectors of bytes, rather than characters, which are called runes here. I tried to do some type conversions to do the assignments, but so far I could not.
The idea here is to generate 5 strings with random characters of sizes 100, 200, 300, 400 and 500 and then invert their characters. I was able to make C work with ease, but in GO, the language returns an error saying that it is not possible to perform the assignment.
func inverte() {
var c = "A"
var strs, aux string
rand.Seed(time.Now().UnixNano())
// Gera 5 vetores de 100, 200, 300, 400, e 500 caracteres
for i := 1; i < 6; i++ {
strs = randomString(i * 100)
fmt.Print(strs)
for i2, j := 0, len(strs); i2 < j; i2, j = i+1, j-1 {
aux = strs[i2]
strs[i2] = strs[j]
strs[j] = aux
}
}
}
If you want to take into account unicode combining characters (characters that are intended to modify other characters, like an acute accent ´ + e = é), Andrew Sellers has an interesting take in this gist.
It starts by listing the Unicode block range for all combining diacritical marks (CDM) (the Unicode block containing the most common combining characters)
regulars (inherited), so the usual ◌̀ ◌́ ◌̂ ◌̃ ◌̄ ◌̅ ◌̆ ◌̇ ◌̈, ...;
extended (containing diacritical marks used in German dialectology -- Teuthonista)
supplement (or the Uralic Phonetic Alphabet, Medievalist notations, and German dialectology -- again, Teuthonista)
for symbols (arrows, dots, enclosures, and overlays for modifying symbol characters)
Half Marks (diacritic mark parts for spanning multiple characters, as seen here)
var combining = &unicode.RangeTable{
R16: []unicode.Range16{
{0x0300, 0x036f, 1}, // combining diacritical marks
{0x1ab0, 0x1aff, 1}, // combining diacritical marks extended
{0x1dc0, 0x1dff, 1}, // combining diacritical marks supplement
{0x20d0, 0x20ff, 1}, // combining diacritical marks for symbols
{0xfe20, 0xfe2f, 1}, // combining half marks
},
}
You can then read, rune after rune, your initial string:
sv := []rune(s)
But if you do so in reverse order, you will encounter combining diacritical marks (CDMs) first, and those need to preserve their order, to not be reversed
for ix := len(sv) - 1; ix >= 0; ix-- {
r := sv[ix]
if unicode.In(r, combining) {
cv = append(cv, r)
fmt.Printf("Detect combining diacritical mark ' %c'\n", r)
}
(note the space around the %c combining rune: '%c' without space would means combining the mark with the first 'ͤ': instead of ' ͤ '. I tried to use the CGJ Combining Grapheme Joiner \u034F, but that does not work)
If you encounter finally a regular rune, you need to combine with those CDMs, before adding it to your reverse final rune array.
} else {
rrv := make([]rune, 0, len(cv)+1)
rrv = append(rrv, r)
rrv = append(rrv, cv...)
fmt.Printf("regular mark '%c' (with '%d' combining diacritical marks '%s') => '%s'\n", r, len(cv), string(cv), string(rrv))
rv = append(rv, rrv...)
cv = make([]rune, 0)
}
Where it gets even more complex is with emojis, and, for instance more recently, modifiers like the Medium-Dark Skin Tone, the type 5 on the Fitzpatrick Scale of skin tones.
If ignored, Reverse '👩🏾‍🦰👱🏾🧑🏾‍⚖️' will give '️⚖‍🏾🧑🏾👱🦰‍🏾👩', loosing the skin tone on the last two emojis.
And don't get me started on the ZERO WIDTH JOINER (200D), which, from Wisdom/Awesome-Unicode, forces adjacent characters to be joined together (e.g., Arabic characters or supported emoji). It Can be used this to compose sequentially combined emoji.
Here are two examples of composed emojis, whose inner elements order should remain in the same order when "reversed":
👩🏾‍🦰 alone is (from Unicode to code points converter):
👩: women (1f469)
dark skin (1f3fe)
ZERO WIDTH JOINER (200d)
🦰red hair (1f9b0)
Those should remain in the exact same order.
The "character" "judge" (meaning an abstract idea of the semantic value for "judge") can be represented with several glyphs or one glyph.
🧑🏾‍⚖️ is actually one composed glyph (composed here of two emojis), representing a judge. That sequence should not be inverted.
The program below correctly detect the "zero width joiner" and do not invert the emojis it combines.
It you inspect that emoji, you will find it composed of:
🧑: Adult (1F9D1)
🏾: dark skin (1f3fe)
ZERO WIDTH JOINER (200d) discussed above
⚖: scale (2696)
VARIATION SELECTOR (FE0F), part of the unicode combining characters (characters that are intended to modify other characters), here requesting that character 'scale' to be displayed emoji-style (with color) ⚖️, using VS16 (U+FE0F), instead of text style (monochrome) '⚖', using VS15 (U+FE0E).
Again, that sequence order needs to be preserved.
Note: the actual judge emoji 👨🏾‍⚖️ uses a MAN 🧑 (1F468), instead of an Adult 🧑 (1F9D1) (plus the other characters listed above: dark skin, ZWJ, scale), and is therefore represented as one glyph, instead of a cluster of graphemes.
Meaning: the single glyph, official emoji for "judge", needs to combine "man" with "scale" (resulting in one glyph 👨🏾‍⚖️) instead of "adult" + "scale".
The latter, "adult" + "scale", is still considered as "one character": you cannot select just the scale, because of the ZWJ (Zero Width Joiner).
But that "character" is represented as a composed glyph 🧑🏾‍⚖️, two glyphs, each one a concrete written representations a corresponding grapheme through codepoint+font)
Obviously, using the first combination ("man"+"scale") results in a more expressive character 👨🏾‍⚖️.
See "The relationship between graphemes and abstract characters for textual representation"
Graphemes and orthographic characters are fairly concrete objects, in the sense that they are familiar to common users—non-experts, who are typically taught to work in terms of them from the time they first learn their “ABCs” (or equivalent from their writing system, of course).
In the domain of information systems, however, we have a different sense of character: abstract characters which are minimal units of textual representation within a given system.
These are, indeed, abstract in two important senses:
first, some of these abstract characters may not correspond to anything concrete in an orthography, as we saw above in the case of HORIZONTAL TAB.
Secondly, the concrete objects of writing (graphemes and orthographic characters) can be represented by abstract characters in more than one way, and not necessarily in a one-to-one manner, as we saw above in the case of “ô” being represented by a sequence <O, CIRCUMFLEX>.
Then: "From grapheme to codepoint to glyph":
Graphemes are the units in terms of which users are usually accustomed to thinking.
Within the computer, however, processes are done in terms of characters.
We don’t make any direct connection between graphemes and glyphs.
As we have defined these two notions here, there is no direct connection between them. They can only be related indirectly through the abstract characters.
This is a key point to grasp: the abstract characters are the element in common through which the others relate.
Full example in Go playground.
Reverse 'Hello, World' => 'dlroW ,olleH'
Reverse '👽👶⃠🎃' => '🎃👶⃠👽'
Reverse '👩🏾‍🦰👱🏾🧑🏾‍⚖️' => '🧑🏾‍⚖️👱🏾👩🏾‍🦰'
Reverse 'aͤoͧiͤ š́ž́ʟ́' => 'ʟ́ž́š́ iͤoͧaͤ'
Reverse 'H̙̖ell͔o̙̟͚͎̗̹̬ ̯W̖͝ǫ̬̞̜rḷ̦̣̪d̰̲̗͈' => 'd̰̲̗͈ḷ̦̣̪rǫ̬̞̜W̖͝ ̯o̙̟͚͎̗̹̬l͔leH̙̖'
As you correctly identified, go strings are immutable, so you cannot assign to rune/character values at given indices.
Instead of reversing the string in-place one must create a copy of the runes in the string and reverse those instead, and then return the resulting string.
For example (Go Playground):
func reverse(s string) string {
rs := []rune(s)
for i, j := 0, len(rs)-1; i < j; i, j = i+1, j-1 {
rs[i], rs[j] = rs[j], rs[i]
}
return string(rs)
}
func main() {
fmt.Println(reverse("Hello, World!"))
// !dlroW ,olleH
fmt.Println(reverse("Hello, 世界!"))
// !界世 ,olleH
}
There are problems with this approach due to the intricacies of Unicode (e.g. combining diacritical marks) but this will get you started.

How to print a list containing strings with a specific number of spaces between each string?

I'm working on making code that prints strings from a list, however I need to have it so that their are a specific number of spaces between each word. I'm also working with java.
You can do this like so:
for (String item : list) {
System.out.print(item + " ");
}
System.out.println();
assuming the number of spaces is low you can just add it to the end.

Efficient algorithm for phrase anagrams

What is an efficient way to produce phrase anagrams given a string?
The problem I am trying to solve
Assume you have a word list with n words. Given an input string, say, "peanutbutter", produce all phrase anagrams. Some contenders are: pea nut butter, A But Ten Erupt, etc.
My solution
I have a trie that contains all words in the given word list. Given an input string, I calculate all permutations of it. For each permutation, I have a recursive solution (something like this) to determine if that specific permuted string can be broken in to words. For example, if one of the permutations of peanutbutter was "abuttenerupt", I used this method to break it into "a but ten erupt". I use the trie to determine if a string is a valid word.
What sucks
My problem is that because I calculate all permutations, my solution runs very slow for phrases that are longer than 10 characters, which is a big let down. I want to know if there is a way to do this in a different way.
Websites like https://wordsmith.org/anagram/ can do the job in less than a second and I am curious to know how they do it.
Your problem can be decomposed to 2 sub-problems:
Find combination of words that use up all characters of the input string
Find all permutations of the words found in the first sub-problem
Subproblem #2 is a basic algorithm and you can find existing standard implementation in most programming language. Let's focus on subproblem #1
First convert the input string to a "character pool". We can implement the character pool as an array oc, where oc[c] = number of occurrence of character c.
Then we use backtracking algorithm to find words that fit in the charpool as in this pseudo-code:
result = empty;
function findAnagram(pool)
if (pool empty) then print result;
for (word in dictionary) {
if (word fit in charpool) {
result = result + word;
update pool to exclude characters in word;
findAnagram(pool);
// as with any backtracking algorithm, we have to restore global states
restore pool;
restore result;
}
}
}
Note: If we pass the charpool by value then we don't have to restore it. But as it is quite big, I prefer passing it by reference.
Now we remove redundant results and apply some optimizations:
Assuming A comes before B in the dictionary. If we choose the first word is B, then we don't have to consider word A in following steps, because those results (if we take A) would already be in the case where A is chosen as the first word
If the character set is small enough (< 64 characters is best), we can use a bitmask to quickly filter words that cannot fit in the pool. A bitmask mask which character is in a word, no matter how many time it occurs.
Update the pseudo-code to reflect those optimizations:
function findAnagram(charpool, minDictionaryIndex)
pool_bitmask <- bitmask(charpool);
if (pool empty) then print result;
for (word in dictionary AND word's index >= minDictionaryIndex) {
// bitmask of every words in the dictionary should be pre-calculated
word_bitmask <- bitmask(word)
if (word_bitmask contains bit(s) that is not in pool_bitmask)
then skip this for iteration
if (word fit in charpool) {
result = result + word;
update charpool to exclude characters in word;
findAnagram(charpool, word's index);
// as with any backtracking algorithm, we have to restore global states
restore pool;
restore result;
}
}
}
My C++ implementation of subproblem #1 where the character set contains only lowercase 'a'..'z': http://ideone.com/vf7Rpl .
Instead of a two stage solution where you generate permutations and then try and break them into words, you could speed it up by checking for valid words as you recursively generate the permutations. If at any point your current partially-complete permutation does not correspond to any valid words, stop there and do not recurse any further. This means you don't waste time generating useless permutations. For example, if you generate "tt", there is no need to permute "peanubuter" and append all the permutations to "tt" because there are no English words beginning with tt.
Suppose you are doing basic recursive permutation generation, keep track of the current partial word you have generated. If at any point it is a valid word, you can output a space and start a new word, and recursively permute the remaining character. You can also try adding each of the remaining characters to the current partial word, and only recurse if doing so results in a valid partial word (i.e. a word exists starting with those characters).
Something like this (pseudo-code):
void generateAnagrams(String partialAnagram, String currentWord, String remainingChars)
{
// at each point, you can either output a space, or each of the remaining chars:
// if the current word is a complete valid word, you can output a space
if(isValidWord(currentWord))
{
// if there are no more remaining chars, output the anagram:
if(remainingChars.length == 0)
{
outputAnagram(partialAnagram);
}
else
{
// output a space and start a new word
generateAnagrams(partialAnagram + " ", "", remainingChars);
}
}
// for each of the chars in remainingChars, check if it can be
// added to currentWord, to produce a valid partial word (i.e.
// there is at least 1 word starting with these characters)
for(i = 0 to remainingChars.length - 1)
{
char c = remainingChars[i];
if(isValidPartialWord(currentWord + c)
{
generateAnagrams(partialAnagram + c, currentWord + c,
remainingChars.remove(i));
}
}
}
You could call it like this
generateAnagrams("", "", "peanutbutter");
You could optimize this algorithm further by passing the node in the trie corresponding to the current partially completed word, as well as passing currentWord as a string. This would make your isValidPartialWord check even faster.
You can enforce uniqueness by changing your isValidWord check to only return true if the word is in ascending (greater or equal) alphabetic order compared to the previous word output. You might also need another check for dupes at the end, to catch cases where two of the same word can be output.

How to detect a number in my Linked List of Strings, and get the value

I need to sort my Linked List, the problem is that each of my Linked List elements are Strings with sentences. So the question is... how to detect each number in my Linked List and get the value?.
I tried to split my linked list so I can pass trough each element.
private LinkedList<String> list = new LinkedList<String>();
list.add("Number One: 1")
list.add("Number Three: 3")
list.add("Number two:2")
for(Iterator<String> iterator =list.iterator(); iterator.hasNext(); )
{
String string = iterator.next();
for (String word : string.split(" ")){
}
I also tried with "if((word.contains("1") || (word.contains("2")...." inside the for loop, and then pass the value "word" to Double... but I think is not very smart
So my goal is this Output (Number One: 1 , Number Two: 2, Number Three: 3), therefore I need the value of each number first.
why not use tryParse on the string,
for (String word : string.split(" ")){
int outvalue
if(int.TryParse(word, outvalue)){
//DoSomething with result
}
}

Resources