Generating word boundaries from string with no spaces - nlp

Question

I'm starting the process of developing an algorithm to determine the gender of an individual based on their email address. I can have emails such as the following:
johnsonsam#example.com
samjohnson#example.com
sjohnson#example.com
john#example.com
My plan is to try to do an index search against the most common first and last names based on the US census. This is meant to apply to the US demographic. However, I think it would be much more efficient if I could first decompose the above e-mail addresses into the following:
<wb>johnson</wb><wb>sam</wb>#example.com
<wb>sam</wb><wb>johnson</wb>#example.com
<wb>s</wb><wb>johnson</wb>#example.com
<wb>john</wb>#example.com
Are there any algorithms (preferably in Python) that you know of that can do this annotation? Any other suggestions towards solving this are welcome.

Answer 1

The problem you've described is called "word segmentation." The wordsegment package will do this for you. It uses the Google Web Trillion Word Corpus, and works well even on names.
To install it:
pip install wordsegment
Here's an example program:
import sys
import wordsegment
def main():
for line in sys.stdin:
print '%s -> %s' % (line.strip(), wordsegment.segment(line))
if __name__ == '__main__':
main()
And here's the output on some examples (assuming you've already separated out the part before the "#" in the email address):
johnsonsam -> ['johnson', 'sam']
samjohnson -> ['sam', 'johnson']
sjohnson -> ['s', 'johnson']
john -> ['john']
johnson_sam -> ['johnson', 'sam']
You could try using lists of names from census data and see if that gives you even better performance. For more information about how you might implement the algorithm yourself with a custom list of words, see the "Word Segmentation" section of this chapter by Norvig: Natural Language Corpus Data.

Answer 2

Here's a basic start, you need to consider also separators (such as dots, underscores, etc), middle names, and initials.
import re
def is_name_list(cands, refs):
for c in cands:
if (len(c) > 1) and (not c in refs):
return False
return True
emails = [
'johnsonsam#example.com',
'samjohnson#example.com',
'sjohnson#example.com',
'john#example.com'
]
names = ['john', 'sam', 'johnson']
for e in emails:
print '\n' + e
at_ind = e.index('#')
user = e[0:at_ind]
for n in names:
finals = []
parts = filter(None, user.split(n))
if is_name_list(parts, names):
all_parts = re.split('(' + n + ')', user)
all_parts.append(e[at_ind:])
strs = ["<wb>" + s + "</wb>" for s in all_parts if s != '']
if len(strs) > 0:
final = ''.join(strs)
if not final in finals:
finals.append(final)
print finals