ElasticSearch - Query the data on a field that matches from first position - search

Question

I searched alot on this and tried numerous combinations. But failed in all attempts :(.
Here is my problem:
I created a jdbc-river in elastic search as below:
{
"type" : "jdbc",
"jdbc" : {
"driver" : "oracle.jdbc.driver.OracleDriver",
"url" : "jdbc:oracle:thin:#//ip:1521/db",
"user" : "user",
"password" : "pwd",
"sql" : "select f1, f2, f3 from table"
},
"index" : {
"index" : "subject2",
"type" : "name2",
"settings": {
"analysis": {
"analyzer": {
"my_analizer": {
"type": "custom",
"tokenizer": "my_pattern_tokenizer",
"filter": []
}
},
"tokenizer": {
"my_pattern_tokenizer": {
"type": "pattern",
"pattern": "$^"
}
},
"filter": []
}
}
},
"mappings":
{
"subject2":
{
"properties" : {
"f1" : {"index" : "not_analyzed", "store": "yes", "analyzer": "my_analizer", "search_analyzer": "keyword", "type": "string"},
"f2" : {"index" : "not_analyzed", "store": "yes", "analyzer": "my_analizer", "search_analyzer": "keyword", "type": "string"},
"f3" : {"index" : "not_analyzed", "store": "yes", "analyzer": "my_analizer", "search_analyzer": "keyword", "type": "string"}
}
}
}
}
I want to implement an auto-complete feature that matches the user entered value with the data in "f1" field say as of now but from the start.
Data in the f1 field is like
"Hardin County ABC"
"Country of XYZ"
"County of Blah blah"
"County of Blah second"
What is as per requirement is when user types "Coun" then result 2nd, 3rd and 4th should be returned by the elastic search and not the first. I read about "keyword" analyzer that makes the complete word to be token but I don't know not working in this case.
Also, if user types "County of B" then 3rd and 4th option should be returned by the elastic search.
Below is the format of my querying the result.
Option 1
{"from":0,"size":10, "query":{ "field" : { "f1" : "count*" } } }
Option 2
{"from":0,"size":10, "query":{ "span_first" : {
"match" : {
"span_term" : { "COMPANY" : "hardin" }
},
"end" : 1
} } }
Please tell me what wrong I am doing here? Thanks in advance.

Answer 1

Before I answer I want to point out you are defining an analyzer then setting index: not_analyzed which means the analyzer is not used. (If you use not_analyzed it is the same as using the keyword analyzer, the whole string, untouched, is one token.)
Also analyzer: my_analizer is a shortcut for index_analyzer: my_analizer and search_analyzer: my_analizer, so your mapping is a bit confusing to me...
Also the fields will be stored in the _source unless you turn this off, you don't need to store the fields separately unless you turn off the _source storing and need that field returned in the result set.
There are 2 ways I can think of doing this:
1. Use a match_phrase_prefix query - Easier and slow
Don't define any analyzers, you don't need them.
Mapping:
"subject2": {
"properties" : {
"f1" : { "type": "string" },
"f2" : { "type": "string" },
"f3" : { "type": "string" },
}
}
}
Query:
"match_phrase_prefix" : {
"f1" : {
"query" : "Count"
}
}
2. Use an edge_ngram token filter - Harder and faster
"settings": {
"analysis": {
"analyzer": {
"edge_autocomplete": {
"type": "custom",
"tokenizer": "keyword",
"filter": ["my_edge_ngram"]
}
},
"filter" : {
"my_edge_ngram" : {
"type": "edgeNGram",
"min_gram": 2,
"max_gram": 15
}
}
}
}
Mapping:
"subject2": {
"properties" : {
"f1" : { "type": "string", "index": "edge_autocomplete" },
"f2" : { "type": "string", "index": "edge_autocomplete" },
"f3" : { "type": "string", "index": "edge_autocomplete" },
}
}
}
Query:
"match" : {
"f1" : "Count",
"analyzer": "keyword"
}
Good luck!

Answer 2

Have you tried an ngram filter? It will tokenize strings of character-length "n". So, your mapping could look like:
{
"settings": {
"analysis": {
"analyzer": {
"autocomplete": {
"type": "custom",
"tokenizer": "standard",
"filter": ["standard", "lowercase", "kstem", "ngram"]
}
},
"filter" : {
"ngram" : {
"type": "ngram",
"min_gram": 2,
"max_gram": 15
}
}
}
},
"mappings": {
"subject2": {
"properties" : {
"f1" : {
"type": "multi_field",
"fields": {
"f1": {
"type": "string"
},
"autocomplete": {
"analyzer": "autocomplete",
"type": "string"
},
...
This will return the ngram "count" for the 2nd, 3rd, and 4th results, which should give you the desired outcome.
Note that making "f1" a multi_field field is not required. However, when you don't need the "autocomplete" analyzer, such as when returning "f1" in the search results, then it is less expensive to use the "f1" subfield. If you do use a "multi_field", you can access "f1" at "f1" (without dot notation), but to access "autocomplete" you need to use dot notation - so "f1.autocomplete".

Answer 3

Although, The solution we final implemented is a mix of approaches but still answer by "ramseykhalaf" is the closest match. +1 to him.
What I did when ever user enters a word with space fire a match-prefix query and get the closest match result to show.
{"from":0,"size":10, "query":{ "match" : { "f1" : {"query" : "MICROSOU", "type" : "phrase_prefix", "boost":2} } } }
As soon as user hits any character after space I change the mode of query to query field with regex and being multiple words in a field match is again very close to what user is looking for.
{"from":0,"size":10, "query":{ "query_string" : { "default_field":"f1","query" : "micro int*", "boost":2 } } }
In this way we got the closest solution to this requirement. I would be happy to get more optimize solution that suffice my above mentioned use cases.
Just to add one more thing - now the river I created is simple plain vanilla with fields as "not_analyzed" and analyzer as "keyword"