I have a field containing a lot of words, for example:
"hello my name is Nicole and I am working with Solr"
and I need Solr to return this document if I search for this words (note that the word order is not as in the indexed text):
"am name with"
I am using this configuration
<fieldType name="propertiesField" class="solr.TextField">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory" />
</analyzer>
<analyzer type="query">
<tokenizer class="solr.PatternTokenizerFactory" pattern="-" />
<filter class="solr.LowerCaseFilterFactory" />
</analyzer>
</fieldType>
and this query:
select/?q=properties_all:am-name-with&version=2.2&start=0&rows=10&indent=on
when I analize it with the analyzer, those words are highlighted but no document is found when I do the search.
Thanks for your help!!!!
If there is no good reason to use different index and query time analyzer, do not.
I would fieldType like:
<fieldType name="propertiesField" class="solr.TextField">
<analyzer>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory" />
</analyzer>
</fieldType>
Additionally set default operator as AND (schema file)
<solrQueryParser defaultOperator="AND"/>
Then query Solr with:
select/?q=properties_all:(am name with)&version=2.2&start=0&rows=10&indent=on
Related
I am using EdgeNgramFilterFactory for partial search
<filter class="solr.EdgeNGramFilterFactory" minGramSize="4" maxGramSize="15" side="front"/>
so if the word is "sfx" this will not be stored in index as its length is < 4 and i cannot search it , is there any way to store the words with lenghth < minGramSize in index and make it searchable ???
Using Solr 4.3
Thanks ...
you have to change the minGramSize to lower length like to 3 your case.
Or may be you can keep it as 2.
In earlier version (I used 3.3) had the side front and back so I Used it like this.
<fieldType name="text_reference" class="solr.TextField" sortMissingLast="true" omitNorms="true" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="2" maxGramSize="50" side="front"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="2" maxGramSize="50" side="back"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
or you can index the text using another field type where you will keep the text as it without ngraming and search in that field as well. Where their wont be any n-gram filter so it the text with length 3 will get indexed.
<fieldType name="text_delimeter" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" preserveOriginal="1" catenateAll="1" splitOnCaseChange="0"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
My stopwords don't works as expected.
Here is part of my schema:
<fieldType name="text_general" class="solr.TextField">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<fieldType class="solr.TextField" name="text_auto">
<analyzer type="index">
<charFilter class="solr.HTMLStripCharFilterFactory"/>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="false"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
<filter class="solr.ShingleFilterFactory" maxShingleSize="3" outputUnigrams="true" outputUnigramsIfNoShingles="false"/>
</analyzer>
<analyzer type="query">
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="false"/>
</analyzer>
</fieldType>
<field name="deal_title_terms" type="text_auto" indexed="true" stored="false" required="false" multiValued="true"/>
<field name="deal_description" type="text_general" indexed="true" stored="true" required="false" multiValued="false"/>
In stopwords.txt I have next words: the, is, a;
Also I have next data in my fields:
deal_description - This is the my description
deal_title_terms - This is the deal title a terms (will be splitted in terms)
When I try to search deal_description:
Example 1: "deal_description: his is the m" - I expect that document with deal_description "This is the my description" will be returned
Example 2: "deal_description: is th" - I expect that nothing will be found because "is" and "the" are stopwords.
When I try to search deal_title_terms:
Example 1: "deal_title_terms: is" - I expect that nothing will be found because "is" is stopword.
Example 2: "deal_title_terms: is the deal" - I expect that "is" and "the" will be ignored and term "deal" will be found.
Example 3: "deal_title_terms: title a terms" - I expect that "a" will be ignored and term "title terms" will be found.
Question 1: Why stopwords don't works for "deal_description" field ?
Question 2: Why for field "deal_title_terms" stopwords not removed for my query ?(When I am trying to find title a terms it will not find "title terms" term)
Question 3: Is there any way to show stopwords in search result but prevent them from searching ? Example:
data: This is cool search engine
search query : "is coo" -> return "This is cool search engine"
search query : "is" -> return nothing
search query : "This coll" -> return "This is cool search engine"
Question 4: Where I can find detailed description (maybe with examples) how stopwords works in solr ? Because it looks like magic.
Answer to Question 1 : Replace the "KeywordTokenizerFactory" as it does no actual tokenizing, so the entire input string is preserved as a single token.Use StandardTokenizerFactory instead.
Or use the below fieldType.
<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
Stopwords will work as expected for the "deal_description" field.
Answer to Question 3 : Yes. Add the StopFilterFactory in analyzer of type="query" only. It will prevent them from searching and not adding them while indexing.
Answer to Quesion 4 : https://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters
Answer to Quesion 2 : The custom field created by you seems incorrect. The text has to tokenised first using the tokenizers but you are using filters first.
Check the analysis of it with solr analysis page.
I want to create a field that will only match if the document's value for that field matches the query term with no additions. For instance, a query for "john" should only return results where the name is "john", not "johnson", "johns", etc.
I've seen other posts about exact matching in solr, and the prevailing answer seems to be to create a new field in schema.xml with type string. I've tried it, but that approach seems to also match when the exact query is contained within a field (results containing "johnson" still appear with the query "john").
The schema has fields lastName and lastName_ngram (which we're currently searching with):
<field name="lastName_ngram" type="text_token_ngram" indexed="true" stored="false" omitNorms="true" omitTermFreqAndPositions="true"/>
<fieldType name="text_token_ngram" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.TrimFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory" maxGramSize="20" minGramSize="1"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.TrimFilterFactory"/>
</analyzer>
</fieldType>
<field name="lastName" type="text_token" indexed="true" stored="true" omitNorms="true" omitTermFreqAndPositions="true"/>
<fieldType name="text_token" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.TrimFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.TrimFilterFactory"/>
</analyzer>
</fieldType>
And I'd like to include a field lastNameExact so that documents that exactly match the entire field can be boosted:
<field name="lastNameExact" type="string" indexed="true" stored="false" omitNorms="true" omitTermFreqAndPositions="true"/>
<copyField source="lastName" dest="lastNameExact"/>
Is there a modification I can make to this so that the lastNameExact field will only hit on documents containing a field with the entirety of the search query?
I could propose you a fix for that. Do not use type string for lastNameExact and use exact_match field type instead.
<fieldType name="exact_match" class="solr.TextField" positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.TrimFilterFactory"/>
</analyzer>
</fieldType>
Copy field should remain the same.
Link for working schema.xml - https://github.com/MysterionRise/information-retrieval-adventure/blob/dadb683820fe4f1eaf6081185a933a28a5e1e481/lucene5/src/main/resources/solr/cores/test/conf/schema.xml
My scheme.xlm looks like this:
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
<!-- The searched field -->
<field name="product_name" type="text" indexed="true" stored="true"/>
This should index the field in lowercase and also transform search query into the lowercase.
Data I want to find is: "Nokia Lumia 610"
When I search "nokia" I get the expected result but
when searching only "Nokia"(upper case N) there aren't any results.
Above "analyzer" performs lowercase only on index but not on search query.
Is this an error?
How to force SOLR indexes and search query to be in lowercase?
Transformation of the search query also depends on the type of query and the analyzer that you are using. For example, the above will not convert your search query to lowercase if you are sending request to the select analyzer. If you are sending request :-
http://url/solr/select?q=Nokia
then the above will not be converted to lowercase since the select analyzer is not present in your fieldtype definition. You will have to modify your code as follows :-
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="select">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
if the above does not work, then please post the request that you are sending and the output of adding debugQuery=true to the request.
Along with
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="select">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
in schema.xml.
In head.vm change return $("#q").val();
to
return $("#q").val().toLowerCase(); for InCaseSensitive autocomplete feature.
So that you can get result if you search with Capital Letters.
When I search like this: q=*6205* I got much more results then searching q=6205, and it's good
but my problem is:
when I search for q=6205-2RS or q=6205\-2RS I got some results but when I put * in a search string I receive no results (q=*6205-2RS* or q=*6205\-2RS*) Why?
I want to search for *6205-2RS* but I want solr to search this string also in a middle of items names.
Wildcards queries does not undergo any analysis.
So when you are searching for 6205-2RS with wildcards, it would be searched as is, without any analysis like lower case filter, worddelimiters.
Whats the schema defination for the field. Is it a text field or a String field type ?
The definition for text_general is as below -
<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
<!-- in this example, we will only use synonyms at query time
<filter class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
-->
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
The tokenizer and lower case filter in analysis at index time for 6205-2RS would generate tokens 6205,2rs
As no analysis takes place during search, its searching for 6205-2RS as is, and will not find any results.
Change the field type to string and that should match the results.