Pandas DataFrame remove strings having a certain characters - python-3.x

Question

I have a pandas DataFrame with a lot of text data. I want to remove all lines starting with "*" mark. Therefore, I tried a small example as the following.
string1 = '''* This needs to be gone
But this line should stay
*remove
* this too
End'''
string2 = '''* This needs to be gone
But this line should stay
*remove
* this too
End'''
df = pd.DataFrame({'a':[string1,string2]})
df['a'] = df['a'].map(lambda a: (re.sub(r'(?m)^\*.*\n?', '', a, flags=re.MULTILINE)))
It could perfectly do the job. However, when I applied the same function to my original DataFrame it is not working. Can you help me to identify the issue?
df2['NewsText'] = df2['NewsText'].map(lambda a: (re.sub(r'(?m)^\*.*\n?', '', a, flags=re.MULTILINE)))
df2.head()
Pease see the attached image of my original DataFrame

Answer 1

Given your example data
.str.split('\n') creates a list of each section
.apply(lambda x: '\n'.join([y for y in x if '*' not in y])) uses a list comprehension to remove each sentence with * and then joins it back into a string.
You can join with ' '.join or ''.join
.apply(lambda x: [y for y in x if '*' not in y]) if you want to have a list instead of a long string.
| | a |
|---:|:--------------------------|
| 0 | * This needs to be gone |
| | But this line should stay |
| | *remove |
| | * this too |
| | End |
| 1 | * This needs to be gone |
| | But this line should stay |
| | *remove |
| | * this too |
| | End |
# remove sections with '*'
df['a'] = df['a'].str.split('\n').apply(lambda x: '\n'.join([y for y in x if '*' not in y]))
# final
| | a |
|---:|:--------------------------|
| 0 | But this line should stay |
| | End |
| 1 | But this line should stay |
| | End |