LingPipe Named Entity Recognizer outputs a lot of mismatches - nlp

Question

I'm trying to extract named entities (people, persons and organizations) using LingPipe and following this tutorial. Here is the full text that I am trying to extract names from and here is the code (exception handling omitted for brevity):
Chunker chunker = readChunker("/path-to-chunker"); // custom method for
reading the model
String article = "Some long news article spanning multiple lines...";
Chunking chunking = chunker.chunk(article);
Set<Chunk> chunkingSet = chunking.chunkSet();
for (Chunk chunk : chunkingSet) {
String name = article.substring(chunk.start(), chunk.end()));
System.out.println(name);
}
And this is (part of) the output I get:
Tony Abbott
Indonesia
Joko Widodo
Sukumaran
Andrew Chan
Bali.
pair
the Bali
Nusa Kambangan
Indonesian
Indonesian
I’
Widodo. I
” Abbott
Julie Bishop
Widodo
al-Jazeera
Sukumaran
Chan
Bishop
”
As you can see, there are a lot of mismatches/partial matches like Bali., pair, the Bali, I', Widodo. I, " Abbott, ". I assume library's NER is working just fine and the problem is that the above code is somehow misusing the classes/methods from this library. But I just can't seem to find what is wrong about the code?
Any ideas?

Answer 1

It seems that the problem is in the model of chunker that you read in the first line. Probably, it uses wrong tokenizer, which is the source of Bali., I', Widodo. I, " Abbott, ".
pair and the Bali can be explained by ordinary errors (taggers usually have not more than 80-90% precision). However, the source of such errors can also be explained by bad model - for example, it can be trained for another domain.
Btw, why do you use longpipe, but not Stanford NER? It shows better results, as a rule; e.g. the first available (i.e. random) article.
Also, here is a good step-by-step tutorial for Standford NER.