Spacy ent.label_ cannot define organization - nlp

Question

I am using spacy to analyze the terrorist and it is weird that spacy cannot find the organization such as fatah. The code is below
import spacy
nlp = spacy.load('en')
def read_file_to_list(file_name):
with open(file_name, 'r') as file:
return file.readlines()
terrorism_articles = read_file_to_list('data/rand-terrorism-dataset.txt')
terrorism_articles_nlp = [nlp(art) for art in terrorism_articles]
common_terrorist_groups = [
'taliban',
'al - qaeda',
'hamas',
'fatah',
'plo',
'bilad al - rafidayn'
]
common_locations = [
'iraq',
'baghdad',
'kirkuk',
'mosul',
'afghanistan',
'kabul',
'basra',
'palestine',
'gaza',
'israel',
'istanbul',
'beirut',
'pakistan'
]
location_entity_dict = defaultdict(Counter)
for article in terrorism_articles_nlp:
article_terrorist_groups = [ent.lemma_ for ent in article.ents if ent.label_=='PERSON' or ent.label_ =='ORG']#人或者组织
article_locations = [ent.lemma_ for ent in article.ents if ent.label_=='GPE']
terrorist_common = [ent for ent in article_terrorist_groups if ent in common_terrorist_groups]
locations_common = [ent for ent in article_locations if ent in common_locations]
for found_entity in terrorist_common:
for found_location in locations_common:
location_entity_dict[found_entity][found_location] += 1
location_entity_dict
I simply get nothing from the file.
Here is The text data link
Thank you!

Answer 1

I reproduced your example and it looks like you will get empty lists for article_terrorist_groups and terrorist_common. Therefore, you won't get the output (that I assume) you require. I changed the model (for my machine) to en_core_web_sm and I observed that the ent.label is different from ones that you are specifying in the if statement in your list comprehensions. I am almost certain this is the case whether you use spacy.load('en') or spacy.load('en_core_web_sm').
You are using if ent.label_=='PERSON' or ent.label_ =='ORG' which is leading to empty lists. You would need to change this in order for it to work. Basically, in your list comprehension for article_terrorist_groups and terrorist_common, the for loop is trying to iterate through an empty list.
If you look at the output that I posted, you will see that ent.label is not 'PERSON' or 'ORG'
Note: I would recommend adding print statements (or using a debugger) in your code to check from time to time.
My Code
import spacy
from collections import defaultdict, Counter
nlp = spacy.load('en_core_web_sm') # I changed this
def read_file_to_list(file_name):
with open(file_name, 'r') as file:
return file.readlines()
terrorism_articles = read_file_to_list('rand-terrorism-dataset.txt')
terrorism_articles_nlp = [nlp(art) for art in terrorism_articles]
common_terrorist_groups = [
'taliban',
'al - qaeda',
'hamas',
'fatah',
'plo',
'bilad al - rafidayn'
]
common_locations = [
'iraq',
'baghdad',
'kirkuk',
'mosul',
'afghanistan',
'kabul',
'basra',
'palestine',
'gaza',
'israel',
'istanbul',
'beirut',
'pakistan'
]
location_entity_dict = defaultdict(Counter)
for article in terrorism_articles_nlp:
print([(ent.lemma_, ent.label) for ent in article.ents])
Output
[('CHILE', 383), ('the Santiago Binational Center', 383), ('21,000', 394)]
[('ISRAEL', 384), ('palestinian', 381), ('five', 397), ('Masada', 384)]
[('GUATEMALA', 383), ('U.S. Marines', 381), ('Guatemala City', 384)]
truncated output in the interest of length of this answer

Answer 2

Because groups & locations in common_terrorist_groups and common_locations are lowercase while finded data terrorist_common and locations_common are uppercase. So just change the code if ent in common_terrorist_groups to if ent.lower() in common_terrorist_groups
common_terrorist_groups = [
'taliban',
'al - qaeda',
'hamas',
'fatah',
'plo',
'bilad al - rafidayn'
]
common_locations = [
'iraq',
'baghdad',
'kirkuk',
'mosul',
'afghanistan',
'kabul',
'basra',
'palestine',
'gaza',
'israel',
'istanbul',
'beirut',
'pakistan'
]
location_entity_dict = defaultdict(Counter)
for article in terrorism_articles_nlp:
article_terrorist_cands = [ent.lemma_ for ent in article.ents if ent.label_ == 'PERSON' or ent.label_ == 'ORG']
article_location_cands = [ent.lemma_ for ent in article.ents if ent.label_ == 'GPE']
terrorist_candidates = [ent for ent in article_terrorist_cands if ent.lower() in common_terrorist_groups]
location_candidates = [loc for loc in article_location_cands if loc.lower() in common_locations]
for found_entity in terrorist_candidates:
for found_location in location_candidates:
location_entity_dict[found_entity][found_location] += 1