solr 4.1 java.lang.IllegalArgumentException: first position increment must be > 0 (got 0) - search

Question

all!
After i update my solr version to 4.1, there is a such error when rebuild index:
Warning: Error creating document : SolrInputDocument[dop_pos_state=, dop_country=, dop_first_name_onlySort=Rick, dop_first_name=Rick, dop_sync_flag=true, dop_orgid=1522402, dop_last_name=King, dop_last_name_onlySort=King, dop_invite_flag=true, dop_name=Rick King, dop_metro_area=, dop_create_date=2012-12-15 08:53:55.0, dop_address=Greater Boston Area, dop_job_level=1, dop_id=343218, dop_title=at A & J Engineering Inc., dop_update_date=2013-02-19 09:38:38.0, dop_metromap_id=210, dop_facebook_linked=0, dop_linkedin_linked=0, dop_crunchBase_linked=0, dop_twitter_linked=0]
java.lang.IllegalArgumentException: first position increment must be > 0 (got 0)
at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:125)
at org.apache.lucene.index.DocFieldProcessor.processDocument(DocFieldProcessor.java:306)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:250)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:376)
When i delete the field dop_title (which is "at A & J Engineering Corp." ) in schema.xml, it works fine. The dop_title's analyzer is below:
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
</fieldType>
Then I put in solr's analysis page, the result is:
How did this happen and in which way can i avoid this? Thanks for your help!

Answer 1

There is an ongoing migration from stream-based to graph-based processing of tokens. This has uncovered some strange edge-cases in Solr 4.1. It looks like yours is one of those (a regression). You can open an issue if you want and somebody will look at it.
In a meanwhile, you may find it useful to know that if you tick the little "Verbose Output" button on the right side of the analysis page, it shows a lot more information about each step in the pipeline, including position values. That could help you to debug this issue faster and/or help to avoid it.