I am using spaCy to find location from string - python-3.x

Question

I wrote a code to find locations present in a string.
import spacy
nlp= spacy.load('en')
doc1='Pune, India'
doc2='India, Pune'
doc3='Pune India'
doc4='India Pune'
print([(X.text, X.label_) for X in nlp(doc1).ents])
print([(X.text, X.label_) for X in nlp(doc2).ents])
print([(X.text, X.label_) for X in nlp(doc3).ents])
print([(X.text, X.label_) for X in nlp(doc4).ents])
and my output is:
[('India', 'GPE')]
[('India', 'GPE'), ('Pune', 'GPE')]
[('India', 'GPE')]
[('India', 'GPE')]
How can I get [('India', 'GPE'), ('Pune', 'GPE')] same output for all?

Answer 1

SpaCy is able to understand that 'Pune' is probably a GPE only because of the comma (,) after India in doc2. This isn't the case for other three examples, and it only detected a known GPE, India.
You may have to write some custom code based on the case you have in your dataset. You may try considering the words 'before' and 'after' the detected GPEs and check them if they come in English dictionary. For example, in your cases when only 'India' is detected, you may check 'Pune' in the English word list (you can get it in SpaCy), and if it doesn't exist, it can be considered as a candidate GPE.
Do let me know if you need help in coding this?

Answer 2

This is because the model you are using does not know that 'Pune' is an entity with label 'GPE'.
You can add an extra component to your pipeline with add_pipe() to identify 'Pune' as an entity as well.
Note that the presence of a comma is not the reason why 'Pune' is classified an entity.
Since you got a list of text, let's make a list so match nlp.pipe() can be used.
texts = ['Pune, India', 'India, Pune', 'Pune India', 'India Pune']
LOCATIONS = ["Pune", "India"]
First, you got to find the places where 'Pune' occurs in the text. PhraseMatcher can be used for that.
from spacy.matcher import PhraseMatcher
Create a matcher object and add the patterns to be recognized to it.
nlp = spacy.load('en')
matcher = PhraseMatcher(nlp.vocab)
matcher.add("LOCATIONS", None, *list(nlp.pipe(LOCATIONS)))
Now define the component function.
def places_component(doc):
doc.ents = [Span(doc, start, end, label="GPE") for match_id, start, end in matcher(doc)]
return doc
Add the component to the pipeline.
nlp.add_pipe(places_component) #last=True
Now try
for doc in nlp.pipe(texts):
print([(ent.text, ent.label_) for ent in doc.ents])
You'll get
[('Pune', 'GPE'), ('India', 'GPE')]
[('India', 'GPE'), ('Pune', 'GPE')]
[('Pune', 'GPE'), ('India', 'GPE')]
[('India', 'GPE'), ('Pune', 'GPE')]
The whole program would be something like
import spacy
from spacy.tokens import Span
from spacy.matcher import PhraseMatcher
LOCATIONS = ["Pune", "India"]
texts = ['Pune, India', 'India, Pune', 'Pune India', 'India Pune']
nlp = spacy.load('en')
matcher = PhraseMatcher(nlp.vocab)
matcher.add("LOCATIONS", None, *list(nlp.pipe(LOCATIONS)))
def places_component(doc):
doc.ents = [Span(doc, start, end, label="GPE") for match_id, start, end in matcher(doc)]
return doc
nlp.add_pipe(places_component) #last=True
for doc in nlp.pipe(texts):
print([(ent.text, ent.label_) for ent in doc.ents])