Extract keywords from a title,relevant, and final column math - string

Question

I have a DataFrame that is structured in the following way:
Title; Total Visits; Rank
The dog; 8 ; 4
The cat; 9 ; 4
The dog cat; 10 ; 3
The second DataFrame contains:
Keyword; Rank
snail ; 5
dog ; 1
cat ; 2
What I am trying to accomplish is:
Title; Total Visits; Rank ; Keywords ; Score
The dog; 8 ; 4 ; dog ; 1
The cat; 9 ; 4 ; cat ; 2
The dog cat; 10 ; 3 ; dog,cat ; 1.5
I have made use of the following reference, but for some
df['Tweet'].map(lambda x: tuple(re.findall(r'({})'.format('|'.join(w.values)), x)))
return null. Any help would be appreciated.

Answer 1

You can use:
#create list of all words
wants = df2.Keyword.tolist()
#dict for maping
d = df2.set_index('Keyword')['Rank'].to_dict()
#split all values by whitespaces, create series
s = df1.Title.str.split(expand=True).stack()
#filter by list wants
s = s[s.isin(wants)]
print (s)
0 1 dog
1 1 cat
2 1 dog
2 cat
dtype: object
#create new columns
df1['Keywords'] = s.groupby(level=0).apply(','.join)
df1['Score'] = s.map(d).groupby(level=0).mean()
print (df1)
Title Total Visits Rank Keywords Score
0 The dog 8 4 dog 1.0
1 The cat 9 4 cat 2.0
2 The dog cat 10 3 dog,cat 1.5
Another solution with lists manipulation:
wants = df2.Keyword.tolist()
d = df2.set_index('Keyword')['Rank'].to_dict()
#create list from each value
df1['Keywords'] = df1.Title.str.split()
#remove unnecessary words
df1['Keywords'] = df1.Keywords.apply(lambda x: [item for item in x if item in wants])
#maping each word
df1['Score'] = df1.Keywords.apply(lambda x: [d[item] for item in x])
#create ne columns
df1['Keywords'] = df1.Keywords.apply(','.join)
#mean
df1['Score'] = df1.Score.apply(lambda l: sum(l) / float(len(l)))
print (df1)
Title Total Visits Rank Keywords Score
0 The dog 8 4 dog 1.0
1 The cat 9 4 cat 2.0
2 The dog cat 10 3 dog,cat 1.5
Timings:
In [96]: %timeit (a(df11, df22))
100 loops, best of 3: 3.71 ms per loop
In [97]: %timeit (b(df1, df2))
100 loops, best of 3: 2.55 ms per loop
Code for testing:
df11 = df1.copy()
df22 = df2.copy()
def a(df1, df2):
wants = df2.Keyword.tolist()
d = df2.set_index('Keyword')['Rank'].to_dict()
s = df1.Title.str.split(expand=True).stack()
s = s[s.isin(wants)]
df1['Keywords'] = s.groupby(level=0).apply(','.join)
df1['Score'] = s.map(d).groupby(level=0).mean()
return (df1)
def b(df1,df2):
wants = df2.Keyword.tolist()
d = df2.set_index('Keyword')['Rank'].to_dict()
df1['Keywords'] = df1.Title.str.split()
df1['Keywords'] = df1.Keywords.apply(lambda x: [item for item in x if item in wants])
df1['Score'] = df1.Keywords.apply(lambda x: [d[item] for item in x])
df1['Keywords'] = df1.Keywords.apply(','.join)
df1['Score'] = df1.Score.apply(lambda l: sum(l) / float(len(l)))
return (df1)
print (a(df11, df22))
print (b(df1, df2))
EDIT by comment:
You can apply list comprhension if there are Keywords with more as one words:
print (df1)
Title Total Visits Rank
0 The dog 8 4
1 The cat 9 4
2 The dog cat 10 3
print (df2)
Keyword Rank
0 snail 5
1 dog 1
2 cat 2
3 The dog 8
4 the Dog 1
5 The Dog 3
wants = df2.Keyword.tolist()
print (wants)
['snail', 'dog', 'cat', 'The dog', 'the Dog', 'The Dog']
d = df2.set_index('Keyword')['Rank'].to_dict()
df1['Keywords'] = df1.Title.apply(lambda x: [item for item in wants if item in x])
df1['Score'] = df1.Keywords.apply(lambda x: [d[item] for item in x])
df1['Keywords'] = df1.Keywords.apply(','.join)
df1['Score'] = df1.Score.apply(lambda l: sum(l) / float(len(l)))
print (df1)
Title Total Visits Rank Keywords Score
0 The dog 8 4 dog,The dog 4.500000
1 The cat 9 4 cat 2.000000
2 The dog cat 10 3 dog,cat,The dog 3.666667