How to count specific terms in tokenized sentences wthin a pandas df - python-3.x

Question

I'm new to Python and nltk, so I would really appreciate your input on the following problem.
Goal:
I want to search and count the occurrence of specific terminology in tokenized sentences which are stored in a pandas DataFrame. The terms I'm searching for are stored in a list of strings. The output should be saved in a new column.
Since the words I'm searching for are grammatically inflected (e.g. cats instead of cat) I need a solution which not only displays exact matches. I guess stemming the data and searching for specific stems would be a proper approach but let's assume this is not an option here, as we would still have semantic overlaps.
What I tried so far:
In order to further handle the data I preprocessed the data while following these steps:
Put everything in lower case
Remove punctuation
Tokenization
Remove stop words
I tried searching for single terms with str.count('cat') but this doesn't do the trick and the data is marked as missing with NaN. Additionally, I don't know how to iterate over the search word list in an efficient way while using pandas.
My code so far:
import numpy as np
import pandas as pd
import re
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
# Function to remove punctuation
def remove_punctuation(text):
return re.sub(r'[^\w\s]','',text)
# Target data where strings should be searched and counted
data = {'txt_body': ['Ab likes dogs.', 'Bc likes cats.',
'De likes cats and dogs.', 'Fg likes cats, dogs and cows.',
'Hi has two grey cats, a brown cat and two dogs.']}
df = pd.DataFrame(data=data)
# Search words stored in a list of strings
search_words = ['dog', 'cat', 'cow']
# Store stopwords from nltk.corpus
stop_words = set(stopwords.words('english'))
# Data preprocessing
df['txt_body'] = df['txt_body'].apply(lambda x: x.lower())
df['txt_body'] = df['txt_body'].apply(remove_punctuation)
df['txt_body'] = df['txt_body'].fillna("").map(word_tokenize)
df['txt_body'] = df['txt_body'].apply(lambda x: [word for word in x if word not in stop_words])
# Here is the problem space
df['search_count'] = df['txt_body'].str.count('cat')
print(df.head())
Expected output:
txt_body search_count
0 [ab, likes, dogs] 1
1 [bc, likes, cats] 1
2 [de, likes, cats, dogs] 2
3 [fg, likes, cats, dogs, cows] 3
4 [hi, two, grey, cats, brown, cat, two, dogs] 3

Answer 1

A very simple solution would be this:
def count_occurence(l, s):
counter = 0
for item in l:
if s in item:
counter += 1
return counter
df['search_count'] = df.apply(lambda row: count_occurence(row.txt_body, 'cat'),1)
You could then further decide how to define the count_occurence function. And, to search for the whole search_words, something like this will do the job, although it is probably not the most efficient:
def count_search_words(l, search_words):
counter = 0
for s in search_words:
counter += count_occurence(l, s)
return counter
df['search_count'] = df.apply(lambda row: count_search_words(row.txt_body, search_words),1)