Solr: Wildcards and Case sensitivity search - search

Question

I've been looking around trying to figure out what's going on here but have thus far come up empty. I'm hoping someone can offer me guidance as to where I can look for a solution. I have a text field that is defined as such:
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<!-- in this example, we will only use synonyms at query time
<filter class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
-->
<filter class="solr.StopFilterFactory"
ignoreCase="true"
words="stopwords.txt"
/>
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="0"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory"
ignoreCase="true"
words="stopwords.txt"
/>
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="0"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
</fieldType>
I have a few records that have the following key/values
"text":[
"NOFX_SiteTest_4",
"NOFX_SiteTest_4\nNOFX_SiteTest_4\n Fourteen\n Ten\n Thirteen\n Fifteen\n Two\n 3\n Select Fields"
]
"text":[
"NOFX_SiteTest_44",
"NOFX_SiteTest_44\nNOFX_SiteTest_44\n Fourteen\n Ten\n Thirteen\n Fifteen\n Two\n 3\n Select Fields"
]
"text":[
"NOFX_SiteTest_445",
"NOFX_SiteTest_445\nNOFX_SiteTest_445\n Fourteen\n Ten\n Thirteen\n Fifteen\n Two\n 3\n Select Fields"
]
I'm trying various searches to get Solr to return those records. The problem is, depending on how I structure the query (based on where I add the wildcard, if I add a wildcard, and where leave the search text with regards to the underscores), the results I get are unexpected and incorrect. Here are the searches I ran from the Solr Admin query page:
SEARCH
text:(( NOFX_SiteTest_4* )) OR text_exact:(( NOFX_SiteTest_4* ))
RESULT
3 Records (correct)
SEARCH
text:(( NOFX_SiteTest_ )) OR text_exact:(( NOFX_SiteTest_ ))
RESULT
3 Records (correct)
SEARCH
text:(( NOFX_SiteTest )) OR text_exact:(( NOFX_SiteTest ))
RESULT
3 Records (correct)
SEARCH
text:(( NOFX_SiteTest* )) OR text_exact:(( NOFX_SiteTest* ))
RESULT
3 Records (correct)
SEARCH
text:(( nofx_sitetest_4 )) OR text_exact:(( nofx_sitetest_4 ))
RESULT
1 Record (correct)
SEARCH
text:(( nofx_sitetest_4* )) OR text_exact:(( nofx_sitetest_4* ))
RESULT
0 Records (incorrect)
SEARCH
text:(( nofx_sitetest_ )) OR text_exact:(( nofx_sitetest_ ))
RESULT
3 Records (correct)
SEARCH
text:(( nofx_sitetest* )) OR text_exact:(( nofx_sitetest* ))
RESULT
0 Records (incorrect)
From what it seems to me, based on the configuration for this field, Solr should be seeing these two queries as identical :
text:(( NOFX_SiteTest_4* )) OR text_exact:(( NOFX_SiteTest_4* ))
and
text:(( nofx_sitetest_4* )) OR text_exact:(( nofx_sitetest_4* ))
Why is it the case that the first search, where the letters are properly capitalized, the appropriate number of records are returned but the second search, where it's all lower case, they are not. Yet, when running these queries:
text:(( NOFX_SiteTest_ )) OR text_exact:(( NOFX_SiteTest_ ))
and
text:(( nofx_sitetest_ )) OR text_exact:(( nofx_sitetest_ ))
the proper number of records are returned. Why is the inclusion of the wildcard causing an issue? Particularly when the search consists entirely of lower case letters?
I'm hoping that someone can point me in the right direction. I've been looking through the docs and searching on similar problems but nothing I've run across seems to help me with my issue or helping me understand why this is occurring in the first place.
EDIT: Some additional information.
Here is the definition of the two fields I'm using in my search above:
<field name="text" type="text" indexed="true" stored="true" multiValued="true"/>
<field name="text_exact" type="text_exact" indexed="false" stored="false" multiValued="true"/>
<!-- copy all fields to the default search field -->
<copyField source="title" dest="text"/>
<copyField source="content" dest="text"/>
<copyField source="Comment" dest="text"/>
<!-- copy all fields to the exact match search field -->
<copyField source="title" dest="text_exact"/>
<copyField source="content" dest="text_exact"/>
<copyField source="Comment" dest="text_exact"/>
The only difference between the text and the text_exact is how the field types are defined. When my search is
text:(( NOFX_SiteTest_4* )) OR text_exact:(( NOFX_SiteTest_4* ))
it will find the 3 records (as I state above) but it does so because of the text_exact field, not the text field. I find that odd. Running the search
text_exact:(( NOFX_SiteTest_4* ))
returns 3 records but running the search
text:(( NOFX_SiteTest_4* ))
returns 0 records. I can see why text_exact is returning data. Because there is that exact text in the text_exact field. But I'm not sure why the search against text yields no records. Shouldn't that field be a bit more open and lenient? And be even more allowing of wildcard searches? Because if I remove the asterisk, it does return the one record where that exact text is in the text field. Why isn't it honoring the asterisk as a wildcard?
Finally, if I remove the wildcard and change the text to all lower case, it will find that record without difficulty when searching against the text field. So, again, whatever the issue may be, it appears that it has something to do with using the asterisk as a wildcard.

Answer 1

First of all LowerCaseFilterFactory filter, should go before WordDelimiterFilterFactory filter
<filter class="solr.LowerCaseFilterFactory"/>
It would convert all chars into lowercase, then it will be split by wordDelimiterFilter
When you're using wildcards, add additional text:(( NOFX_SiteTest_4 )) that would be the exact math.
Final
text:( NOFX_SiteTest_4* ) OR text_exact:( NOFX_SiteTest_4*) OR text:( NOFX_SiteTest_4 )
Please use the analysis solr tool to see what's happening.

Answer 2

When you're using a wildcard, the analysis chain doesn't run as it usually does.
The only filters invoked are those that implement MultiTermAwareComponent, so the analysis page won't do much good to tell you what's happening there.
This means that when you're doing a wildcard search, if the indexing pipeline has changed the tokens (split them, etc.), that processing will not happen when querying. That's probably why you don't get the hits as you'd like, but without wildcards it works. The cause here is that the WordDelimiterFilter isn't multitermaware, so when you're indexing, the input text is split into multiple tokens, while when you're querying, that doesn't happen. Since the tokens doesn't match (I'd wager a guess that just NOFX* might match, since that would be a single token on both sides), you don't get a hit.
If you do require wildcard matching for analyzed text, you're probably going to have to do a NgramFilter instead, and then tweak that filter to get the results you want for each token. But this will, yet again, behave differently, depending on where you add the NgramFilter in your chain (i.e. before or after word delimiter, etc.).