So my requirement is verbally simple. I need StanfordCoreNLP default models along with my custom trained model, based on custom entities. In a final run, I need to be able to isolate specific phrases from a given sentence (RegexNER will be used)
Following are my efforts :-
EFFORT I :-
So I wanted to use the StanfordCoreNLP CRF files, tagger files and ner model files, along with my custom trained ner models.
I tried to find if there is any official way of doing this, but didnt get anything. There is a property "ner.model" for StanfordCoreNLP pipeline, but it will skip the default ones if used.
EFFORT II :-
Next (might not be the smartest thing ever. Sorry! Just a guy trying to make ends meet!) , I extracted the model stanford-corenlp-models-3.7.0.jar , and copied all :-
*.ser.gz (Parser Models)
*.tagger (POS Tagger)
*.crf.ser.gz (NER CRF Files)
and tried to put Comma Separated Values with properties "parser.model", "pos.model" and "ner.model" respectively, as follows :-
parser.model=models/ner/default/anaphoricity_model.ser.gz,models/ner/default/anaphoricity_model_conll.ser.gz,models/ner/default/classification_model.ser.gz,models/ner/default/classification_model_conll.ser.gz,models/ner/default/clauseSearcherModel.ser.gz,models/ner/default/clustering_model.ser.gz,models/ner/default/clustering_model_conll.ser.gz,models/ner/default/english-embeddings.ser.gz,models/ner/default/english-model-conll.ser.gz,models/ner/default/english-model-default.ser.gz,models/ner/default/englishFactored.ser.gz,models/ner/default/englishPCFG.caseless.ser.gz,models/ner/default/englishPCFG.ser.gz,models/ner/default/englishRNN.ser.gz,models/ner/default/englishSR.beam.ser.gz,models/ner/default/englishSR.ser.gz,models/ner/default/gender.map.ser.gz,models/ner/default/md-model-dep.ser.gz,models/ner/default/ranking_model.ser.gz,models/ner/default/ranking_model_conll.ser.gz,models/ner/default/sentiment.binary.ser.gz,models/ner/default/sentiment.ser.gz,models/ner/default/truecasing.fast.caseless.qn.ser.gz,models/ner/default/truecasing.fast.qn.ser.gz,models/ner/default/word_counts.ser.gz,models/ner/default/wsjFactored.ser.gz,models/ner/default/wsjPCFG.ser.gz,models/ner/default/wsjRNN.ser.gz
ner.model=models/ner/default/english.all.3class.caseless.distsim.crf.ser.gz,models/ner/default/english.all.3class.distsim.crf.ser.gz,models/ner/default/english.all.3class.nodistsim.crf.ser.gz,models/ner/default/english.conll.4class.caseless.distsim.crf.ser.gz,models/ner/default/english.conll.4class.distsim.crf.ser.gz,models/ner/default/english.conll.4class.nodistsim.crf.ser.gz,models/ner/default/english.muc.7class.caseless.distsim.crf.ser.gz,models/ner/default/english.muc.7class.distsim.crf.ser.gz,models/ner/default/english.muc.7class.nodistsim.crf.ser.gz,models/ner/default/english.nowiki.3class.caseless.distsim.crf.ser.gz,models/ner/default/english.nowiki.3class.nodistsim.crf.ser.gz
pos.model=models/tagger/default/english-left3words-distsim.tagger
But, I get the following exception :-
Caused by: edu.stanford.nlp.io.RuntimeIOException: Error while loading a tagger model (probably missing model file)
Caused by: java.io.StreamCorruptedException: invalid stream header: EFBFBDEF
EFFORT III :-
I thought I will be able to handle with RegexNER, and I was successful to some extent. Just that the entities that it learns through RegexNER, it doesn't apply to forthcoming expressions. Eg: It will find the entity "CUSTOM_ENTITY" inside a text, but if i put a RegexNER like ( [ {ner:CUSTOM_ENTITY} ] /with/ [ {ner:CUSTOM_ENTITY} ] ) it never succeeds in finding the right phrase.
Really need help here!!! I don't wanna train the complete model again, Stanford guys got over a GB of model information which is useful to me. Just that I want to add custom entities too.
First of all make sure your CLASSPATH has the proper jars in it.
Here is how you should include your custom trained NER model:
java -Xmx8g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner -ner.model <csv-of-model-paths> -file example.txt
-ner.model should be set to a comma separated list of all models you want to use.
Here is an example of what you could put:
edu/stanford/nlp/models/ner/english.all.3class.distsim.crf.ser.gz,edu/stanford/nlp/models/ner/english.muc.7class.distsim.crf.ser.gz,edu/stanford/nlp/models/ner/english.conll.4class.distsim.crf.ser.gz,/path/to/custom_model.ser.gz
Note in my example that all of the standard models will be run, and then finally your custom model will be run. Make sure your custom model is in the CLASSPATH.
You also probably need to add this to your command: -ner.combinationMode HIGH_RECALL. By default the NER combination will only use the tags for a particular class from the first model. So if you have model1,model2,model3 only model1's LOCATION will be used. If you set things to HIGH_RECALL then model2 and model3's LOCATION tags will be used as well.
Another thing to keep in mind, model2 can't overwrite decisions by model1. It can only overwrite "O". So if model1 says that a particular token is a LOCATION, model2 can't say it's an ORGANIZATION or a PERSON or anything. So the order of the models in your list matters.
If you want to write rules that use entities found by previous rules, you should look at my answer to this question:
TokensRegex rules to get correct output for Named Entities
from your given context
use this instead of comma separated values and try to have all the jars within the same directory:
parser.model=models/ner/default/anaphoricity_model.ser.gz
parser.model=models/ner/default/anaphoricity_model_conll.ser.gz
parser.model=models/ner/default/classification_model.ser.gz
parser.model=models/ner/default/classification_model_conll.ser.gz
parser.model=models/ner/default/clauseSearcherModel.ser.gz
parser.model=models/ner/default/clustering_model.ser.gz
parser.model=models/ner/default/clustering_model_conll.ser.gz
parser.model=models/ner/default/english-embeddings.ser.gz
parser.model=models/ner/default/english-model-conll.ser.gz
parser.model=models/ner/default/english-model-default.ser.gz
parser.model=models/ner/default/englishFactored.ser.gz
parser.model=models/ner/default/englishPCFG.caseless.ser.gz
parser.model=models/ner/default/englishPCFG.ser.gz
parser.model=models/ner/default/englishRNN.ser.gz
parser.model=models/ner/default/englishSR.beam.ser.gz
parser.model=models/ner/default/englishSR.ser.gz
parser.model=models/ner/default/gender.map.ser.gz
parser.model=models/ner/default/md-model-dep.ser.gz
parser.model=models/ner/default/ranking_model.ser.gz
parser.model=models/ner/default/ranking_model_conll.ser.gz
parser.model=models/ner/default/sentiment.binary.ser.gz
parser.model=models/ner/default/sentiment.ser.gz
parser.model=models/ner/default/truecasing.fast.caseless.qn.ser.gz
parser.model=models/ner/default/truecasing.fast.qn.ser.gz
parser.model=models/ner/default/word_counts.ser.gz
parser.model=models/ner/default/wsjFactored.ser.gz
parser.model=models/ner/default/wsjPCFG.ser.gz
parser.model=models/ner/default/wsjRNN.ser.gz
now copy the above lines,and similarly make the other models too and paste it in a server.properties file.
if u don't have server.properties file then create it.
and use the following command too start you server:
java -Xmx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout 15000 -serverProperties server.properties
Related
Please bear with me as I am new to semantic technologies.
I am trying to use the package rdflib to extract labels from classes in ontologies. However some ontologies don't contain the labels themselves but have the URIs of classes from other ontologies. How does one extract the labels from URIs of the external ontologies?
The intuition behind my attempts center on identifying classes that don't contain labels locally (if that is the right way of putting it) and then "following" their URIs to the external ontologies to extract the labels. However the way I have implemented it does not work.
import rdflib
g = rdflib.Graph()
# I have no trouble extracting labels from this ontology:
# g.load("http://purl.obolibrary.org/obo/po.owl#")
# However, this ontology contains no labels locally:
g.load("http://www.bioassayontology.org/bao/bao_complete.owl#")
owlClass = rdflib.namespace.OWL.Class
rdfType = rdflib.namespace.RDF.type
for s in g.subjects(predicate=rdfType, object=owlClass):
# Where label is present...
if g.label(s) != '':
# Do something with label...
print(g.label(s))
# This is what I have added to try to follow the URI to the external ontology.
elif g.label(s) == '':
g2 = rdflib.Graph()
g2.parse(location=s)
# Do something with label...
print(g.label(s))
Am I taking completely the wrong approach? All help is appreciated! Thank you.
I think you can be much more efficient than this. You are trying to do a web request, remote ontology download and search every time you encounter a URI that doesn't have a label given in http://www.bioassayontology.org/bao/bao_complete.owl which is most of them and it's a very large number. So your script will take forever and thrash the web servers delivering those remote ontologies.
Looking at http://www.bioassayontology.org/bao/bao_complete.owl, I see that most of the URIs without labels there are from OBO, and perhaps a couple of other ontologies, but mostly OBO.
What you should do is download OBO once and load that with RDFlib. Then if you run your script above on the joined (union) graph of http://www.bioassayontology.org/bao/bao_complete.owl & OBO, you'll have all OBO's content at your fingertips so that g.label(s) will find a much higher proportion of labels.
Perhaps there are a couple of other source ontologies providing labels for http://www.bioassayontology.org/bao/bao_complete.owl you may need as well but my quick browsing sees only OBO.
I'm trying to get Stanford Core NLP to recognise an identification code. The problem is the code has punctuation in it. e.g. 01.A01.01 which causes the input to be separated into three sentences.
The matching expression for this code would be [0-9][0-9][.][a-z,A-Z][0-9][0-9][.][0-9][0-9]. I've tried adding this into my regexner.txt file but it doesn't identify it (presumably because the tokens are across separate sentences?)
I've also tried to match it using a TokenRegex similar to the following (also without any success).
/tell/ /me/ /about/ (?$refCode /[0-9][0-9]/ /./ /[a-z,A-Z][0-9][0-9]/ /./ /[0-9][0-9]/ )
Some example uses...
The user has resource 02.G36.63 reserved.
Is 21.J83.02 available?
Does anyone have any ideas or suggestions?
I took your sample input and replaced "\n" with " ", to create:
The user has resource 02.G36.63 reserved. Is 21.J83.02 available?
I created this rules file (sample-rules.txt):
02.G36.63 ID_CODE MISC 2
And I ran this command:
java -Xmx8g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner,regexner -regexner.mapping sample-rules.txt -ssplit.eolonly -tokenize.whitespace -file sample-sentence.txt -outputFormat text
I got this output:
Sentence #1 (9 tokens):
The user has resource 02.G36.63 reserved. Is 21.J83.02 available?
[Text=The CharacterOffsetBegin=0 CharacterOffsetEnd=3 PartOfSpeech=DT Lemma=the NamedEntityTag=O]
[Text=user CharacterOffsetBegin=4 CharacterOffsetEnd=8 PartOfSpeech=NN Lemma=user NamedEntityTag=O]
[Text=has CharacterOffsetBegin=9 CharacterOffsetEnd=12 PartOfSpeech=VBZ Lemma=have NamedEntityTag=O]
[Text=resource CharacterOffsetBegin=13 CharacterOffsetEnd=21 PartOfSpeech=NN Lemma=resource NamedEntityTag=O]
[Text=02.G36.63 CharacterOffsetBegin=22 CharacterOffsetEnd=31 PartOfSpeech=NN Lemma=02.g36.63 NamedEntityTag=ID_CODE]
[Text=reserved. CharacterOffsetBegin=32 CharacterOffsetEnd=41 PartOfSpeech=NN Lemma=reserved. NamedEntityTag=O]
[Text=Is CharacterOffsetBegin=43 CharacterOffsetEnd=45 PartOfSpeech=VBZ Lemma=be NamedEntityTag=O]
[Text=21.J83.02 CharacterOffsetBegin=46 CharacterOffsetEnd=55 PartOfSpeech=NN Lemma=21.j83.02 NamedEntityTag=O]
[Text=available? CharacterOffsetBegin=56 CharacterOffsetEnd=66 PartOfSpeech=NN Lemma=available? NamedEntityTag=O]
This said to just tokenize on whitespace, so it stopped breaking on the periods. Also it said to only split sentences on newline, so it is important in the input file to put the entire user request on one line. You won't get sentences, but you can get a token stream and identify your product codes.
Now if you really want the full power of Stanford CoreNLP and you don't want to have these codes split, you could take the ambitious route and alter the tokenizer PTBLexer.flex file to include all of your id codes.
That file is here in the repo:
https://github.com/stanfordnlp/CoreNLP/blob/master/src/edu/stanford/nlp/process/PTBLexer.flex
You'll have to Google around to find instructions on compiling the PTBLexer.flex file into PTBLexer.java. This site should have the info you need:
http://www.jflex.de/
This would basically mean adding in your id codes and making a few slight edits, and then rebuilding PTBLexer. Then with your custom tokenizer Stanford CoreNLP would treat your product codes like complete tokens and you could have normal sentence splitting if you want to do something like analyze the dependency structure of your user requests.
I currently have a VM running Titan over a local Cassandra backend and would like the ability to use ElasticSearch to index strings using CONTAINS matches and regular expressions. Here's what I have so far:
After titan.sh is run, a Groovy script is used to load in the data from separate vertex and edge files. The first stage of this script loads the graph from Titan and sets up the ES properties:
config.setProperty("storage.backend","cassandra")
config.setProperty("storage.hostname","127.0.0.1")
config.setProperty("storage.index.elastic.backend","elasticsearch")
config.setProperty("storage.index.elastic.directory","db/es")
config.setProperty("storage.index.elastic.client-only","false")
config.setProperty("storage.index.elastic.local-mode","true")
The second part of the script sets up the indexed types:
g.makeKey("property").dataType(String.class).indexed("elastic",Edge.class).make();
The third part loads in the data from the CSV files, this has been tested and works fine.
My problem is, I don't seem to be able to use the ElasticSearch functions when I do a Gremlin query. For example:
g.E.has("property",CONTAINS,"test")
returns 0 results, even though I know this field contains the string "test" for that property at least once. Weirder still, when I change CONTAINS to something that isn't recognised by ElasticSearch I get a "no such property" error. I can also perform exact string matches and any numerical comparisons including greater or less than, however I expect the default indexing method is being used over ElasticSearch in these instances.
Due to the lack of errors when I try to run a more advanced ES query, I am at a loss on what is causing the problem here. Is there anything I may have missed?
Thanks,
Adam
I'm not quite sure what's going wrong in your code. From your description everything looks fine. Can you try the follwing script (just paste it into your Gremlin REPL):
config = new BaseConfiguration()
config.setProperty("storage.backend","inmemory")
config.setProperty("storage.index.elastic.backend","elasticsearch")
config.setProperty("storage.index.elastic.directory","/tmp/es-so")
config.setProperty("storage.index.elastic.client-only","false")
config.setProperty("storage.index.elastic.local-mode","true")
g = TitanFactory.open(config)
g.makeKey("name").dataType(String.class).make()
g.makeKey("property").dataType(String.class).indexed("elastic",Edge.class).make()
g.makeLabel("knows").make()
g.commit()
alice = g.addVertex(["name":"alice"])
bob = g.addVertex(["name":"bob"])
alice.addEdge("knows", bob, ["property":"foo test bar"])
g.commit()
// test queries
g.E.has("property",CONTAINS,"test")
g.query().has("property",CONTAINS,"test").edges()
The last 2 lines should return something like e[1t-4-1w][4-knows-8]. If that works and you still can't figure out what's wrong in your code, it would be good if you can share your full code (e.g. in Github or in a Gist).
Cheers,
Daniel
I am working with stanford parser to extract grammetical dependency structures from review sentences. My problem is that for some reason the output generated by my code is not similar to the one generated my stanford online tool. Below is an example.
Review Sentence: The picture quality of the camera is not good.
My Code output (It used EnglishPCFG model and typedDependenciesCollapsed structure)
root(ROOT-0, -LSB--1),
det(quality-4, The-2),
nn(quality-4, picture-3),
nsubj(-RSB--11, quality-4),
det(camera-7, the-6),
prep_of(quality-4, camera-7),
cop(-RSB--11, is-8),
neg(-RSB--11, not-9),
amod(-RSB--11, good-10),
ccomp(-LSB--1, -RSB--11)
Stanford Online tool Output:
det(quality-3, The-1)
nn(quality-3, picture-2)
nsubj(good-9, quality-3)
det(camera-6, the-5)
prep_of(quality-3, camera-6)
cop(good-9, is-7)
neg(good-9, not-8)
root(ROOT-0, good-9)
I am looking for the reason for this difference. What kind of model and dependency structure does online parser use ? I apologies if I am missing something obvious. Any help would be highly appreciated.
I can add code snippet if required
Update:
I changed my code to ignore the LSB and RSB generated by the SP tokenizer but still the grammatical structure generated is different from that of online tool. Here is an example:
Review Sentence: The size and picture quality of the camera is perfect.
My Code Output:
det(quality-5, The-1),
nn(quality-5, size-2),
conj_and(size-2, picture-4),
nsubj(perfect-10, quality-5),
det(camera-8, the-7),
prep_of(quality-5, camera-8),
cop(perfect-10, is-9),
root(ROOT-0, perfect-10)
Stanford Online Tool Output:
det(quality-5, The-1)
nn(quality-5, size-2)
conj_and(size-2, picture-4)
**nn(quality-5, picture-4)**
nsubj(perfect-10, quality-5)
det(camera-8, the-7)
prep_of(quality-5, camera-8)
cop(perfect-10, is-9)
root(ROOT-0, perfect-10)
Note the missing nn dependency in my code output. I am trying to get my head around why this is happening. Any help would be appreciated.
Update (Relevant code snippet below):
rawWords2 = [-LSB-, The, size, and, picture, quality, of, the, camera, is, perfect, -RSB-]
lp = LexicalizedParser using EnglishPCFG model
Tree parse = lp.apply(rawWords2.subList(1,rawWords2.size() - 1));
TreebankLanguagePack tlp = new PennTreebankLanguagePack();
GrammaticalStructureFactory gsf = tlp.grammaticalStructureFactory();
GrammaticalStructure gs = gsf.newGrammaticalStructure(parse);
tdl = (List<TypedDependency>) gs.typedDependenciesCollapsed();
System.out.println(tdl.toString());
Output to screen is as mentioned earlier in the post.
Another observation.
I worked around with Stanford library to show me the dependency relation between quality and picture which as shown in the Stanford online tool is nn but the dependency shown by the library is dep (i.e. can't find more suitable dependency). Now the question is why is Stanford online tool showing nn dependency between quality and picturewhere as Stanford library showing dep as dependency.
The major issue for whether you get that extra nn dependency or not is whether there is propagation of dependencies across coordination (size is a nn of quality and it is coordinated with picture, therefore we make it an nn of quality too). The online output is showing the collapsed output with propagation, whereas you are calling the API method that doesn't include propagation. You can see either from the command-line using options as shown at the bottom of this post. In the API, to get coordination propagation, you should instead call
gs.typedDependenciesCCprocessed()
(instead of gs.typedDependenciesCollapsed()).
Other comments:
Where are the square brackets (-LSB-) coming from? They shouldn't be introduced by the tokenizer. If they are, it's a bug. Can you say what you do for them to be generated? I suspect they may be coming from your preprocessing? Unexpected things like that in a sentence will tend to cause the parse quality to degrade very badly.
The online parser isn't always up-to-date with the latest released version. I'm not sure if it is up-to-date right now. But I don't think that is the main issue here.
We are doing some work evolving the dependencies representation. This is deliberate, but will create problems if you have code that depends substantively on how the dependencies were defined in an older version. We would be interested to know (perhaps by email to the parser-user list) if your accuracy was coming down for reasons other than your code was written to expect the dependency names as they were in an earlier version.
Example of difference using the command line:
[manning]$ cat > camera.txt
The size and picture quality of the camera is perfect.
[manning]$ java edu.stanford.nlp.parser.lexparser.LexicalizedParser -outputFormat typedDependencies -outputFormatOptions collapsedDependencies edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz camera.txt
Loading parser from serialized file edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz ... done [2.4 sec].
Parsing file: camera.txt
Parsing [sent. 1 len. 11]: The size and picture quality of the camera is perfect .
det(quality-5, The-1)
nn(quality-5, size-2)
conj_and(size-2, picture-4)
nsubj(perfect-10, quality-5)
det(camera-8, the-7)
prep_of(quality-5, camera-8)
cop(perfect-10, is-9)
root(ROOT-0, perfect-10)
Parsed file: camera.txt [1 sentences].
Parsed 11 words in 1 sentences (6.94 wds/sec; 0.63 sents/sec).
[manning]$ java edu.stanford.nlp.parser.lexparser.LexicalizedParser -outputFormat typedDependencies -outputFormatOptions CCPropagatedDependencies edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz camera.txt
Loading parser from serialized file edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz ... done [2.2 sec].
Parsing file: camera.txt
Parsing [sent. 1 len. 11]: The size and picture quality of the camera is perfect .
det(quality-5, The-1)
nn(quality-5, size-2)
conj_and(size-2, picture-4)
nn(quality-5, picture-4)
nsubj(perfect-10, quality-5)
det(camera-8, the-7)
prep_of(quality-5, camera-8)
cop(perfect-10, is-9)
root(ROOT-0, perfect-10)
Parsed file: camera.txt [1 sentences].
Parsed 11 words in 1 sentences (12.85 wds/sec; 1.17 sents/sec).
According to my observations it seems like, stanford online parser still uses older versions at its backend.
I have been using stanford parser for an year now. We have been using version 3.2.0 for a long time now. When version 3.3.0 was released with additional feature of sentimental analysis I have tried using the newer version. But, its dependencies where observed to be slightly varying from version 3.2.0 and the efficiency of our product has come down.
If your requirement is just extract dependencies and not use sentiment analysis. I would suggest you to use version 3.2.0.
Check the end of this page to download earlier versions of parser.
I'm having a problem inputting tab delimited files into the stanford classifier.
Although I was able to successfully walk through all the included stanford tutorials, including the newsgroup tutorial, when I try to input my own training and test data it doesn't load properly.
At first I thought the problem was that I was saving the data into a tab delimited file using an Excel spreadsheet and it was some kind of encoding issue.
But then I got exactly the same results when I did the following. First I literally typed the demo data below into gedit, making sure to use a tab between the politics/sports class and the ensuing text:
politics Obama today announced a new immigration policy.
sports The NBA all-star game was last weekend.
politics Both parties are eyeing the next midterm elections.
politics Congress votes tomorrow on electoral reforms.
sports The Lakers lost again last night, 102-100.
politics The Supreme Court will rule on gay marriage this spring.
sports The Red Sox report to spring training in two weeks.
sports Messi set a world record for goals in a calendar year in 2012.
politics The Senate will vote on a new budget proposal next week.
politics The President declared on Friday that he will veto any budget that doesn't include revenue increases.
I saved that as myproject/demo-train.txt and a similar file as myproject/demo-test.txt.
I then ran the following:
java -mx1800m -cp stanford-classifier.jar edu.stanford.nlp.classify.ColumnDataClassifier
-trainFile myproject/demo-train.txt -testFile myproject/demo-test.txt
The good news: this actually ran without throwing any errors.
The bad news: since it doesn't extract any features, it can't actually estimate a real model and the probability defaults to 1/n for each item, where n is the number of classes.
So then I ran the same command but with two basic options specified:
java -mx1800m -cp stanford-classifier.jar edu.stanford.nlp.classify.ColumnDataClassifier
-trainFile myproject/demo-train.txt -testFile myproject/demo-test.txt -2.useSplitWords =2.splitWordsRegexp "\s+"
That yielded:
Exception in thread "main" java.lang.RuntimeException: Training dataset could not be processed
at edu.stanford.nlp.classify.ColumnDataClassifier.readDataset(ColumnDataClassifier.java:402)
at edu.stanford.nlp.classify.ColumnDataClassifier.readTrainingExamples (ColumnDataClassifier.java:317)
at edu.stanford.nlp.classify.ColumnDataClassifier.trainClassifier(ColumnDataClassifier.java:1652)
at edu.stanford.nlp.classify.ColumnDataClassifier.main(ColumnDataClassifier.java:1628)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
at edu.stanford.nlp.classify.ColumnDataClassifier.makeDatum(ColumnDataClassifier.java:670)
at edu.stanford.nlp.classify.ColumnDataClassifier.makeDatumFromLine(ColumnDataClassifier.java:267)
at edu.stanford.nlp.classify.ColumnDataClassifier.makeDatum(ColumnDataClassifier.java:396)
... 3 more
These are exactly the same results I get when I used the real data I saved from Excel.
Even more though, I don't know how to make sense of the ArrayIndexOutOfBoundsException. When I used readline in python to print out the raw strings for both the demo files I created and the tutorial files that worked, nothing about the formatting seemed different. So I don't know why this exception would be raised with one set of files but not the other.
Finally, one other quirk. At one point I thought maybe line breaks were the problem. So I deleted all line breaks from the demo files while preserving tab breaks and ran the same command:
java -mx1800m -cp stanford-classifier.jar edu.stanford.nlp.classify.ColumnDataClassifier
-trainFile myproject/demo-train.txt -testFile myproject/demo-test.txt -2.useSplitWords =2.splitWordsRegexp "\s+"
Surprisingly, this time no java exceptions are thrown. But again, it's worthless: it treats the entire file as one observation, and can't properly fit a model as a result.
I've spent 8 hours on this now and have exhausted everything I can think of. I'm new to Java but I don't think that should be an issue here -- according to Stanford's API documentation for ColumnDataClassifier, all that's required is a tab delimited file.
Any help would be MUCH appreciated.
One last note: I've run these same commands with the same files on both Windows and Ubuntu, and the results are the same in each.
Use a properties file. In the example Stanford classifier example
trainFile=20news-bydate-devtrain-stanford-classifier.txt
testFile=20news-bydate-devtest-stanford-classifier.txt
2.useSplitWords=true
2.splitWordsTokenizerRegexp=[\\p{L}][\\p{L}0-9]*|(?:\\$ ?)?[0-9]+(?:\\.[0-9]{2})?%?|\\s+|[\\x80-\\uFFFD]|.
2.splitWordsIgnoreRegexp=\\s+
The number 2 at the start of lines 3, 4 and 5 signifies the column in your tsv file. So in your case you would use
trainFile=20news-bydate-devtrain-stanford-classifier.txt
testFile=20news-bydate-devtest-stanford-classifier.txt
1.useSplitWords=true
1.splitWordsTokenizerRegexp=[\\p{L}][\\p{L}0-9]*|(?:\\$ ?)?[0-9]+(?:\\.[0-9]{2})?%?|\\s+|[\\x80-\\uFFFD]|.
1.splitWordsIgnoreRegexp=\\s+
or if you want to run with command line arguments
java -mx1800m -cp stanford-classifier.jar edu.stanford.nlp.classify.ColumnDataClassifier -trainFile myproject/demo-train.txt -testFile myproject/demo-test.txt -1.useSplitWords =1.splitWordsRegexp "\s+"
I've faced the same error as you.
Pay attention on tabs in the text you are classifying.
Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
This means, that at some point classifier expects array of 3 elements, after it splits the string with tabs.
I've run a method, that counts amount of tabs in each line, and if at some line you have not two of them - here is an error.