I am using apache solr for bulding search for a website.
I am using nested entities to import data from different tables. Dataimport is successfull and all the documents are being added to the index. My dataConfig goes like this :
<dataConfig>
<dataSource type="JdbcDataSource" driver ="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost/purplle_purplle2" user="purplle_purplle" password="purplle123" />
<document name="doc">
<entity name="offer" query="SELECT * FROM service_offering WHERE module LIKE 'location' ">
<field column="name" name="name"/>
<field column="id" name="id" />
<field column="type_id" name="type_id" />
<entity name="offer_type" query=" select name from service_offeringtype where id='${offer.type_id}'" >
<field column="name" name="offer_type" />
</entity>
<entity name="offer_location" query=" select name from service_location where id='${offer.module_id}'" >
<field column="name" name="location_name" />
</entity>
<entity name="offer_address" query=" select * from service_address where module_id='${offer.module_id}' AND module LIKE 'location'" >
<entity name="loc_city" query=" select name from loc_city where id='${offer_address.city}'" >
<field column="name" name="loc_city" />
</entity>
<entity name="loc_area" query=" select name from loc_area where id='${offer_address.area}'" >
<field column="name" name="loc_area" />
</entity>
<entity name="loc_zone" query=" select name from loc_zone where id='${offer_address.zone}'" >
<field column="name" name="loc_zone" />
</entity>
</entity>
</entity>
</document>
</dataConfig>
Now if i directly do a search on this index. The results are fetched only for "name" field. It returns null for other fields namely "loc_area","location_name","loc_city" etc.
My schema looks like this
<field name="id" type="int" indexed="true" stored="true" />
<field name="name" type="string" indexed="true" stored="true" />
<field name="offer_type" type="string" indexed="true" stored="true" />
<field name="location_name" type="string" indexed="true" stored="true" />
<field name="type_id" type="string" indexed="true" stored="true" />
<field name="loc_city" type="string" indexed="true" stored="true" />
<field name="loc_area" type="string" indexed="true" stored="true" />
<field name="loc_zone" type="string" indexed="true" stored="true" />
However if i copy these fields into a "text" field which is present by default in the schema.xml. Then by searching on "text" field i easily get the relevant results.
<copyField source="name" dest="text"/>
<copyField source="offer_type" dest="text"/>
<copyField source="location_name" dest="text"/>
<copyField source="loc_city" dest="text"/>
<copyField source="loc_area" dest="text"/>
<copyField source="loc_zone" dest="text"/>
But i cannot do it like this because i have to assign boost levels to different fields for calcuation of score. The moment i append this in the query syntax "&defType=edismax&qf=name^1.0+location_name^10.0+loc_area^50.0" it returns null results.
What is wrong?
My guess is that your problem is the type of your fields. I don't know exactly what your fields contain, but there is a difference between type="string" and type = "text".
The String type indexes an untokenized String value of the entire field input. Text type tokenizes and analyzes the the field. For example, if I search for "john" against a string field containing "John Smith" I would not expect a hit, where if the field were a text field, I would get a hit.
Since your query seems to work against a text field, and not a string field, changing the types and reindexing seems to be a likely solution.
Related
In Solr, I want to index string field as list by splitting it.
Below is my indexing query in data_config.xml file.
<document name="Example">
<entity dataSource="example_table" name="Example"
query="select id, text from example_table"
pk="id"
transformer="RegexTransformer"
>
<field column="id" name="id" />
<field column="text" name="text" />
</entity>
Field text is a comma separated string. Example: "A, B, C"
Below is the field definition in schema.xml file
<field name="text" type="string" indexed="true" stored="true" required="false" multiValued="true" />
When I'm querying Solr the output is:
"text":["A, B, C"]
Could someone explain me how can I get the result as below?
"text":["A","B","C"]
To do it in your DataImportHandler definition (since you've already added the RegexTransformer):
<field column="text" name="text" splitBy=", " />
Or do it in your field definition by using a TextField with a Regular Expression Pattern Tokenizer:
<analyzer>
<tokenizer class="solr.PatternTokenizerFactory" pattern=","/>
<filter class="solr.TrimFilterFactory"/>
</analyzer>
I have configured solr for my ecommerce application (That mostly contains books data). The search result does not seem to return what I expect.
Following is the configuration.
schema.xml
`
<field name="namespace" type="string" indexed="true" stored="false" />
<field name="id" type="string" indexed="true" stored="true" />
<field name="productId" type="long" indexed="true" stored="true" />
<field name="skuId" type="long" indexed="true" stored="true" />
<field name="category" type="long" indexed="true" stored="false" multiValued="true" />
<field name="explicitCategory" type="long" indexed="true" stored="false" multiValued="true" />
<field name="searchable" type="text_general" indexed="true" stored="false" />
<dynamicField name="*_searchable" type="text_general" indexed="true" stored="false" />
<dynamicField name="*_i" type="int" indexed="true" stored="false" />
<dynamicField name="*_is" type="int" indexed="true" stored="false" multiValued="true" />
<dynamicField name="*_s" type="string" indexed="true" stored="false" />
<dynamicField name="*_ss" type="string" indexed="true" stored="false" multiValued="true" />
<dynamicField name="*_l" type="long" indexed="true" stored="false" />
<dynamicField name="*_ls" type="long" indexed="true" stored="false" multiValued="true" />
<dynamicField name="*_t" type="text_general" indexed="true" stored="false" />
<dynamicField name="*_txt" type="text_general" indexed="true" stored="false" multiValued="true" />
<dynamicField name="*_b" type="boolean" indexed="true" stored="false" />
<dynamicField name="*_bs" type="boolean" indexed="true" stored="false" multiValued="true" />
<dynamicField name="*_d" type="double" indexed="true" stored="false" />
<dynamicField name="*_ds" type="double" indexed="true" stored="false" multiValued="true" />
<dynamicField name="*_p" type="double" indexed="true" stored="false" />
<dynamicField name="*_dt" type="date" indexed="true" stored="false" />
<dynamicField name="*_dts" type="date" indexed="true" stored="false" multiValued="true" />
<!-- some trie-coded dynamic fields for faster range queries -->
<dynamicField name="*_ti" type="tint" indexed="true" stored="false" />
<dynamicField name="*_tl" type="tlong" indexed="true" stored="false" />
<dynamicField name="*_td" type="tdouble" indexed="true" stored="false" />
<dynamicField name="*_tdt" type="tdate" indexed="true" stored="false" />
<!-- Both field types required for geolocation searches. First stores the
lat and lon components for the "coordinate" FieldType. Second stores
the coordinate. -->
<dynamicField name="*_coordinate" type="tdouble" indexed="true" stored="false"/>
<dynamicField name="*_c" type="coordinate" indexed="true" stored="false"/>
</fields>
<uniqueKey>id</uniqueKey>
<types>
<!-- The StrField type is not analyzed, but indexed/stored verbatim. -->
<fieldType name="string" class="solr.StrField" sortMissingLast="true" />
<!-- boolean type: "true" or "false" -->
<fieldType name="boolean" class="solr.BoolField" sortMissingLast="true" />
<!-- Default numeric field types. For faster range queries, consider the
tint/tlong/tdouble types. -->
<fieldType name="int" class="solr.TrieIntField" precisionStep="0" positionIncrementGap="0" />
<fieldType name="long" class="solr.TrieLongField" precisionStep="0" positionIncrementGap="0" />
<fieldType name="double" class="solr.TrieDoubleField" precisionStep="0" positionIncrementGap="0" />
<!-- Numeric field types that index each value at various levels of precision
to accelerate range queries when the number of values between the range endpoints
is large. See the javadoc for NumericRangeQuery for internal implementation
details. Smaller precisionStep values (specified in bits) will lead to more
tokens indexed per value, slightly larger index size, and faster range queries.
A precisionStep of 0 disables indexing at different precision levels. -->
<fieldType name="tint" class="solr.TrieIntField" precisionStep="8" positionIncrementGap="0" />
<fieldType name="tlong" class="solr.TrieLongField" precisionStep="8" positionIncrementGap="0" />
<fieldType name="tdouble" class="solr.TrieDoubleField" precisionStep="8" positionIncrementGap="0" />
<!-- The format for this date field is of the form 1995-12-31T23:59:59Z,
and is a more restricted form of the canonical representation of dateTime
http://www.w3.org/TR/xmlschema-2/#dateTime The trailing "Z" designates UTC
time and is mandatory. Optional fractional seconds are allowed: 1995-12-31T23:59:59.999Z
All other components are mandatory. Expressions can also be used to denote
calculations that should be performed relative to "NOW" to determine the
value, ie... NOW/HOUR ... Round to the start of the current hour NOW-1DAY
... Exactly 1 day prior to now NOW/DAY+6MONTHS+3DAYS ... 6 months and 3 days
in the future from the start of the current day Consult the DateField javadocs
for more information. Note: For faster range queries, consider the tdate
type -->
<fieldType name="date" class="solr.TrieDateField" precisionStep="0" positionIncrementGap="0" />
<!-- A Trie based date field for faster date range queries and date faceting. -->
<fieldType name="tdate" class="solr.TrieDateField" precisionStep="6" positionIncrementGap="0" />
<!-- A general text field that has reasonable, generic cross-language defaults:
it tokenizes with StandardTokenizer and down cases. -->
<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory" />
<filter class="solr.LowerCaseFilterFactory" />
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory" />
<filter class="solr.LowerCaseFilterFactory" />
</analyzer>
</fieldType>
<!-- A specialized field for geospatial search. If indexed, this fieldType must not be multivalued. -->
<fieldType name="coordinate" class="solr.LatLonType" subFieldSuffix="_coordinate"/>
</types>
`
solrconfig.xml
<?xml version="1.0" encoding="UTF-8" ?>
<config>
<luceneMatchVersion>4.10.3</luceneMatchVersion>
<directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.StandardDirectoryFactory}" />
<updateHandler class="solr.DirectUpdateHandler2" />
<query>
<maxBooleanClauses>1024</maxBooleanClauses>
<filterCache class="solr.FastLRUCache" size="512" initialSize="512" autowarmCount="0" />
<queryResultCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="0" />
<documentCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="0" />
<cache name="perSegFilter" class="solr.search.LRUCache" size="10" initialSize="0" autowarmCount="10"
regenerator="solr.NoOpRegenerator" />
<enableLazyFieldLoading>true</enableLazyFieldLoading>
<queryResultWindowSize>20</queryResultWindowSize>
<queryResultMaxDocsCached>200</queryResultMaxDocsCached>
<listener event="newSearcher" class="solr.QuerySenderListener" />
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst>
<str name="q">static firstSearcher warming in solrconfig.xml</str>
</lst>
</arr>
</listener>
<useColdSearcher>false</useColdSearcher>
<maxWarmingSearchers>2</maxWarmingSearchers>
</query>
<requestDispatcher handleSelect="false">
<requestParsers enableRemoteStreaming="true" multipartUploadLimitInKB="2048000" formdataUploadLimitInKB="2048"
addHttpRequestToContext="false"/>
<httpCaching never304="true" />
</requestDispatcher>
<requestHandler name="/select" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<int name="rowsa">10</int>
<str name="df">name_t</str>
</lst>
</requestHandler>
<queryResponseWriter name="json" class="solr.JSONResponseWriter">
<str name="content-type">text/plain; charset=UTF-8</str>
</queryResponseWriter>
For example when I search for 2 states it gives me lot of random results, which does not even contain 2 states in the title.
However when I search for 2 states in phrase "2 States", I do get the relevant results"
I dont want to restrict every search into quotes, since user might search for some combination like "book by author" which certainly give 0 results if searched in phrase since it wont match the exact phrase.
How can I imporve my search so that I can list most relevant results on the top.
You can use the pf2 and pf3 parameters in the edismax handler to give boosts to documents where two (pf2) or three (pf3) of your terms are found after each other in the field.
defType=edismax&pf2=title^4
You also have the pf argument for the regular dismax handler, but that's built on the assumption that all the terms are close together. It might help, but pf2 or pf3 sounds better suited for what you need.
I have a search view created:
<record model="ir.ui.view" id="helpdesk_ticket_search_view">
<field name="name">helpdesk.ticket.search.view</field>
<field name="model">helpdesk.ticket</field>
<field name="arch" type="xml">
<search string="Owner Search">
<filter string="Owner" name="res_users" domain="[('res_users_id', '=', 'uid')]"/>
<field name="res_users_id"/>
<field name="name"/>
<field name="ticketnumber" />
<group string="Group by">
<filter string="Owner" name="res_users" domain="[]"
context="{'group_by':'res_users_id'}"/>
</group>
</search>
</field>
</record>
When I search by user, I don't need to type the full name out eg. instead of typing "user", I can just type "us" and get a list
However, with numbers if I am looking for "12345678" I have to type it all out and not just "123".
Is it possible to search for numbers just using part of the number? This is helpful to me with a helpdesk when i am looking for all ticket starting with 123.
Any advise?
I'm trying to configure solr 4.0-BETA with a WordDelimiterFilterFactory so I can query numbers containing hyphens.
Field value: "123456-1234" when adding to ssn.
Queries:
"123456-1234" <- Works (with hyphen)
"1234561234" <- Doesn't work (without hyphen)
According to the documentation (AFAIUI) it should match since the fieldtype has generateNumberParts and catenateNumbers.
From the documentation:
generateNumberParts="1" causes number subwords to be generated:
"500-42" => "500" "42"
catenateNumbers="1" causes maximum runs of
number parts to be catenated: "500-42" => "50042"
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.WordDelimiterFilterFactory
My fields:
<fields>
<field name="ssn" type="text_en_splitting" indexed="true" stored="false" multiValued="false" />
<field name="ssn_exact" type="string" indexed="true" stored="true" multiValued="false" />
</fields>
<copyField source="ssn" dest="ssn_exact" />
<copyField source="ssn" dest="text" />
The filter in text_en_splitting:
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
What am I missing here?
I created a similar field in my local schema and using the analysis tool which is under the Solr Admin. (http://localhost:8983/solr/#/collection1/analysis - Note this url assumes solr running on http://localhost:8983/ and your index is named collection1 - modify as necessary).
I tried running your value to index and the query against text_en_splitting selected in the Analyse FieldName/FieldType dropdown. You will see from the results that the value 1234561234 is never added as an index term for this field type.
However, if you use the text_en_splitting_tight FieldType, then the behavior you want is being produced as the hypen is removed and 1234561234 is a term being added to the index. So I would switch the field type as follows and reindex and you should be set to go.
<fields>
<field name="ssn" type="text_en_splitting_tight" indexed="true" stored="false" multiValued="false" />
<field name="ssn_exact" type="string" indexed="true" stored="true" multiValued="false" />
</fields>
<copyField source="ssn" dest="ssn_exact" />
<copyField source="ssn" dest="text" />
I would like to get an auto-suggest / auto-complete field in my application and I am able to get that on a string field, but faceting or querying is not "working" on a text field as on a string field, specially with spaced words.
For now my request is q=cleared_keywords:piso\%20e*&facet=on&facet.field=cleared_keywords&facet.sort=result_count&facet.mincount=1&version=2.2&start=0&rows=0&indent=on&facet.limit=10
and my schema is :
<fields>
<field name="id" type="integer" indexed="true" stored="true" required="true"/>
<field name="country" type="string" indexed="true" stored="true" required="true"/>
<field name="city_id" type="integer" indexed="true" stored="true" required="false"/>
<field name="ad_type" type="integer" indexed="true" stored="true" required="true"/>
<field name="keywords" type="text" indexed="true" stored="true" required="true"/>
<field name="result_count" type="sint" indexed="true" stored="true" required="true"/>
<field name="hash" type="integer" indexed="true" stored="true" required="true"/>
<field name="cleared_keywords" type="string" indexed="true" stored="true" required="false"/>
<field name="keywords_score" type="sfloat" indexed="true" stored="true" required="true"/>
<field name="sorted_keywords" type="string" indexed="true" stored="true" required="true"/>
<field name="links_to" type="integer" indexed="true" stored="true" multiValued="true"/>
<field name="keywordsAsSuggestion" type="string" indexed="true" stored="true" />
<dynamicField name="random*" type="rand" indexed="true" stored="true"/>
<copyField source="keywords" dest="keywordsAsSuggestion" />
</fields>
If I try same query on text (keywords) field it's not working because of text.
I don't understand how copyField is working, do I need to reload / recreate the index ?
I wanted to skip the "recreate index" step, but if I can't I'll just load all Solr Document and recreate new ones with a String field with the keywords text field values ... I just don't like that idea.
Regards,
Alexis
The analyzers and tokenizers defined for field type text is different from that of string in the default schema.xml. If you want to try providing phrases for auto suggest, then it would be better to define your own field type with necessary analyzers and tokenizers. This gives detailed information about them.