Tokenize Myanmar names, typed using Pyidaungsu Unicode font, in MS Excel, using VBA UDF - excel

Question

I searched everywhere for a way to sort Myanmar names, typed up in Pyidaungsu unicode font, by the last consonant in MS Excel.
Doing the same in English is relatively easy using Excel's builtin formulae/functions.
But it is hard for Myanmar names in Burmese because Myanmar names do not require a white space between each word and the first, middle and last names are not that distinct as in, eg. John W. Smith.
In a Myanmar name, eg. အောင်မြင့်မြတ်=Aung Myint Myat, there is no distinct first/last name and no white space is required if it is written in Myanmar font!
Thus, it is pretty hard to find the word boundary between each word, i.e, where အောင် starts and ends and မြင့် starts and ends etc. and so on!
So I need a VBA UDF to be able to tokenize Myanmar names!

Answer 1

After much searching and reading through NLP literature, a lot of which I don't really understand, I realized that the Myanmar font, Pyidaungsu by name, has a character binding method where all Myanmar characters: consonants and diacritics were bound together like: the consonants come first for each word, followed by diacritics (or may be I am wrong about how it is called).
So if only I could place a delimiter/separator just before each consonant, I should be able to tokenize each word!
Fortunately, it helps me write VBA code like:
Const kagyi = 4096
Const ah = 4129 '+9 to include ou
Const athat = 4154
Const shiftF = 4153 'for typing something under something
Const witecha = 4140
Const moutcha = 4139
'Return a tokenized Myanmar String
Function MMRTokenizer(target As Range) As String
Dim ch As String
Dim returnString As String
Dim charCounter As Integer
Dim previousChIsAthat As Boolean
Dim shiftFfound As Boolean
Dim previousCharAt As Long
If target.Cells.CountLarge > 1 Then MMRTokenizer = ">1Cell!": Exit Function
returnString = "": previousChIsAthat = False: shiftFfound = False: previousCharAt = Len(target.Value) + 1
If target.CountLarge = 1 Then
If target.Value <> "" Then
For charCounter = Len(target.Value) To 1 Step -1
ch = Mid(target.Value, charCounter, 1)
If AscW(ch) <> shiftF Then
If Not shiftFfound Or AscW(ch) = athat Then
If AscW(ch) <> athat Then
If AscW(ch) >= kagyi And AscW(ch) < ah + 9 Then
If Not previousChIsAthat Then
returnString = Mid(target.Value, charCounter, previousCharAt - charCounter) & IIf(Len(returnString) > 0, "|", "") & returnString
previousCharAt = charCounter
Else
previousChIsAthat = False
End If
Else
If AscW(ch) = witecha Or AscW(ch) = moutcha Then
previousChIsAthat = False
End If
End If
Else
previousChIsAthat = True
If shiftFfound Then shiftFfound = False
End If
Else
shiftFfound = False
If previousChIsAthat Then previousChIsAthat = False
End If
Else
shiftFfound = True
End If
Next charCounter
End If
End If
MMRTokenizer = returnString
End Function
In theory, it should be pretty simple since I am not using any NLP or ML methods but employed some string manipulations only.
I took out each character of the name/word from the right (it may be ok to start from the left) then go left until I found a consonant and place a separator/delimiter to the left of it and then keep going left and repeating the same process until the left-most character is reached.
The caveat here is, that, sometimes, there could be a consonant, which in Myanmar language is part of a combination of a consonant and a diacritic (pretty common behavior), eg. in အောင်=‌ေ+အ+ ာ+င+် though it looks like that way, the Pyidaungsu font bound it like အ+‌ေ+ာ+င+် ,if it were entered using Windows Burmese keyboard (Visual Order), the rightmost two, င+် where င=consonant called nga and ် =diacritic called Athat.
In such cases, we just skip over that renegade consonant (if we encountered that specific diacritic just right of it) as it should not be counted as such, according the Burmese way of spelling words.
I used chrW and ascW functions because Myanmar font cannot be rendered in VBIDE (even after tweaking in the Regional settings) and thus, I am forced to check the unicode character codes instead of directly comparing Burmese characters.
Above is just a gist of how the whole thing works.
Further details are available on my GitHub.
After we tokenized like above, we got something like: အောင်|မြင့်|မြတ် which is now pretty easy to be splitted up or reversed using builtin Excel formulae to become မြတ်|မြင့်|အောင် so that it can now be sorted by the last word (or last name) or separated into a last name/first name basis!
NB: This whole tokenization process could/may be achieved by using a combination of various formulae in Excel as nothing is impossible, especially in Excel365 (where arrays just spill without CSE), IMHO, however, I hope that we can easily see the benefits vs. complexity and effort in this case.
I, hereby, admit that the above code may not be the most elegant, but, it is a proven-working proof-of-concept tool, so employ it at your own risk but bugs can be reported to my GitHub provided above.