I've been doing well using Elasticsearch on "English" documents.
However, I got stuck on prefix query when using "Korean" words.
In details, a document contains word such as "한글" and I want to get the document using prefix query with search term not only "한" but also "ㅎ".
I could not do that using default settings.
I saw that it's related to icu_normalizer or nfd decomposition or something else.
But I could not totally understand the way I have to do to get the result "한글" using "ㅎ" search term.
Is there anyone can help me?
Thanks in advance.
Maybe this code helps you.
curl -XPUT '127.0.0.1:9200/test' -d '{
"settings" : {
"analysis": {
"tokenizer" : {
"autocomplete_tokenizer" : {
"type" : "edgeNGram",
"min_gram" : "1",
"max_gram" : "30",
"token_chars": ["letter", "digit"]
}
},
"char_filter" : {
"nfd_normalizer" : {
"type" : "icu_normalizer",
"name": "nfc",
"mode": "decompose"
}
},
"analyzer": {
"autocomplete_analyzer": {
"type": "custom",
"char_filter": ["nfd_normalizer"],
"tokenizer": "autocomplete_tokenizer"
}
}
}
}
}'
curl '127.0.0.1:9200/test/_analyze?pretty=1&analyzer=autocomplete_analyzer' -d '아버지가 방에 들어가신다. 태권-V'
Related
I need to translate the following SQL query to ES query:
SELECT *
FROM SKILL
WHERE SKILL.name LIKE 'text' and SKILL.type = 'hard'
I have tried the following using "elasticsearch" library for python3:
query = self.__es.search(index="skills",
body={"from" : skip, "size" : limit,
"query":
{"query_string":
{"query": 'text'}
})
and this worked well. But now, I don't know how to check that the field 'type' is equal to 'hard'.
How can I do that?
Thank you.
You have to use a bool query and in the "must" part put two queries, the full text one and a term one:
{
"query": {
"bool": [{
"match": {
"name": "this is a test"
}
}, {
"term": {
"type": "hard"
}
}]
}
}
Before this you have to store the type property as a keyword field.
I'm trying to implement relevance feedback for Elastic Search (Elastic.co).
I'm aware of boosting queries, which allow for the specification of postiive and negative terms, with the idea being to discount the negative terms, while not excluding them as would be the case in a boolean must_not.
However, I'm trying to achieve tiered boosting, of both positive and negative terms.
That is, I want to take a list of binned positive and negative terms and generate a query such that there are different positive and negative boost tiers, each containing their own query terms.
something like (pseudo query):
query{
{
terms: [very relevant terms]
pos_boost: 3
}
{
terms: [relevant terms]
pos_boost: 2
}
{
terms: [irrelevant terms]
neg_boost: 0.6
}
{
terms: [very irrelevant terms]
neg_boost: 0.3
}
}
My question is whether or not this can be achieved with nested boosting queries, or if I'm better off with multiple should clauses.
My concern is that I'm not sure if a boost of 0.2 in the should clause of a bool query still gives the document a positive increase in the score or not, as I want to discount the document, rather than provide any increase in score.
With boosting queries, the concern is that I can't control the degree to which positive terms are weighted.
Any help, or suggestions for other implementations, would be greatly appreciated. (What I really wanted to do was create a language model for relevant documents and use that to rank, but I don't see how that can easily be achieved in elastic.)
Seems that you can combine bool query and use boosting query clauses tweaking boost values.
POST so/boost/ {"text": "apple computers"}
POST so/boost/ {"text": "apple pie recipe"}
POST so/boost/ {"text": "apple tree garden"}
POST so/boost/ {"text": "apple iphone"}
POST so/boost/ {"text": "apple company"}
GET so/boost/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"text": "apple"
}
}
],
"should": [
{
"match": {
"text": {
"query": "pie",
"boost": 2
}
}
},
{
"match": {
"text": {
"query": "tree",
"boost": 2
}
}
},
{
"match": {
"text": {
"query": "iphone",
"boost": -0.5
}
}
}
]
}
}
}
Alternately, if you want to encode your language model into your collection at index-time, you can try the approach described here: Elasticsearch: Influence scoring with custom score field in document
To boost the elastic search document(priority based search query) based on custom/variable boost value at query time i.e. conditional boosting.
Java Coding example:
customerKeySearch = QueryBuilders.constantScoreQuery(QueryBuilders.termQuery(keys.type", "xxx"));
customerTypeSearch = QueryBuilders.constantScoreQuery(QueryBuilders.termQuery("keys.keyValues.value", "xxxx"));
keyValueQuery = QueryBuilders.boolQuery().must(customerKeySearch).must(customerTypeSearch).boost(2f);
customerKeySearch = QueryBuilders.constantScoreQuery(QueryBuilders.termQuery(keys.type", "xxx"));
customerTypeSearch = QueryBuilders.constantScoreQuery(QueryBuilders.termQuery("keys.keyValues.value", "xxxx"));
keyValueQuery = QueryBuilders.boolQuery().must(customerKeySearch).must(customerTypeSearch).boost(6f);
Description and search query:
elastic search has its internal score calculation technic so we need to disable this mechanism by setting disableCoord(true) property to true in java for BoleanQuery to apply custom boost effect.
Following Boolean query is running query for boosting the documents in elastic search index based on boost value.
{
"bool" : {
"should" : [ {
"bool" : {
"must" : [ {
"constant_score" : {
"query" : {
"term" : {
"keys.type" : "XXX"
}
}
}
}, {
"constant_score" : {
"query" : {
"term" : {
"keys.keyValues.value" : "XXXX"
}
}
}
} ],
"boost" : 2.0
}
}, {
"bool" : {
"must" : [ {
"constant_score" : {
"query" : {
"term" : {
"keys.type" : "XXX"
}
}
}
}, {
"constant_score" : {
"query" : {
"term" : {
"keys.keyValues.value" : "500072388315"
}
}
}
} ],
"boost" : 6.0
}
}, {
"bool" : {
"must" : [ {
"constant_score" : {
"query" : {
"term" : {
"keys.type" : "XXX"
}
}
}
}, {
"constant_score" : {
"query" : {
"term" : {
"keys.keyValues.value" : "XXXXXX"
}
}
}
} ],
"boost" : 10.0
}
} ],
"disable_coord" : true
}
}
Trying to get predictive drop down search ,How can i make search always starts from left to right
like in example "I_kimchy park" , "park"
If i search only "par" i have to get only park in return , but here i am getting both words , how to treat empty space as a character
POST /test1
{
"settings":{
"analysis":{
"analyzer":{
"autocomplete":{
"type":"custom",
"tokenizer":"standard",
"filter":[ "standard", "lowercase", "stop", "kstem", "edgeNgram" ,"whitespace"]
}
},
"filter":{
"ngram":{
"type":"edgeNgram",
"min_gram":2,
"max_gram":15,
"token_chars": [ "letter", "digit"]
}
}
}
}
}
PUT /test1/tweet/_mapping
{
"tweet" : {
"properties" : {
"user": {"type":"string", "index_analyzer" : "autocomplete","search_analyzer" : "autocomplete"}
}
}}
POST /test1/tweet/1
{"user" : "I_kimchy park"}
POST /test1/tweet/3
{ "user" : "park"}
GET /test1/tweet/_search
{
"query": {
"match_phrase_prefix": {
"user": "park"
}
}
}
That happens because your standard tokenizer splits your user field by white spaces. You can use Keyword Tokenizer in order to treat whole string as a single value (single token).
Please keep in mind that this change may affect other of your functionalities that use this field. You may have to add dedicated "not tokenized" user field for this purpose.
I'm trying to implement a match search in Elasticsearch, and I noticed that the behavior is different depending if I use _all or if a enter a specific string value as the field name of my query.
To give some context, I've created an index with the following settings:
{
"settings": {
"analysis": {
"analyzer": {
"default": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"standard",
"lowercase",
"stop",
"kstem",
"word_delimiter"
]
}
}
}
}
}
If I create a document like:
{
"name": "Hello.World"
}
And I execute a search using _all like:
curl -d '{"query": { "match" : { "_all" : "hello" } }}' http://localhost:9200/myindex/mytype/_search
It will correctly match the document (since I'm using the stop filter to split the words at the dot), but if I execute this query instead:
curl -d '{"query": { "match" : { "name" : "hello" } }}' http://localhost:9200/myindex/mytype/_search
Nothing is being returned instead. How is this possible?
Issue a GET for /myindex/mytype/_mapping and see if your index is configured the way you think it is. Meaning, see if that "name" field is not_analyzed, for example.
Even more, run the following query to see how name field is actually indexed:
{
"query": {
"match": {
"name": "hello"
}
},
"fielddata_fields": ["name"]
}
You should see something like this in the result:
"fields": {
"name": [
"hello",
"world"
]
}
If you don't, then you know something's wrong with your mapping for the name field.
How do I search for all unique values of a given field with Elasticsearch?
I have such a kind of query like select full_name from authors, so I can display the list to the users on a form.
You could make a terms facet on your 'full_name' field. But in order to do that properly you need to make sure you're not tokenizing it while indexing, otherwise every entry in the facet will be a different term that is part of the field content. You most likely need to configure it as 'not_analyzed' in your mapping. If you are also searching on it and you still want to tokenize it you can just index it in two different ways using multi field.
You also need to take into account that depending on the number of unique terms that are part of the full_name field, this operation can be expensive and require quite some memory.
For Elasticsearch 1.0 and later, you can leverage terms aggregation to do this,
query DSL:
{
"aggs": {
"NAME": {
"terms": {
"field": "",
"size": 10
}
}
}
}
A real example:
{
"aggs": {
"full_name": {
"terms": {
"field": "authors",
"size": 0
}
}
}
}
Then you can get all unique values of authors field.
size=0 means not limit the number of terms(this requires es to be 1.1.0 or later).
Response:
{
...
"aggregations" : {
"full_name" : {
"buckets" : [
{
"key" : "Ken",
"doc_count" : 10
},
{
"key" : "Jim Gray",
"doc_count" : 10
},
]
}
}
}
see Elasticsearch terms aggregations.
Intuition:
In SQL parlance:
Select distinct full_name from authors;
is equivalent to
Select full_name from authors group by full_name;
So, we can use the grouping/aggregate syntax in ElasticSearch to find distinct entries.
Assume the following is the structure stored in elastic search :
[{
"author": "Brian Kernighan"
},
{
"author": "Charles Dickens"
}]
What did not work: Plain aggregation
{
"aggs": {
"full_name": {
"terms": {
"field": "author"
}
}
}
}
I got the following error:
{
"error": {
"root_cause": [
{
"reason": "Fielddata is disabled on text fields by default...",
"type": "illegal_argument_exception"
}
]
}
}
What worked like a charm: Appending .keyword with the field
{
"aggs": {
"full_name": {
"terms": {
"field": "author.keyword"
}
}
}
}
And the sample output could be:
{
"aggregations": {
"full_name": {
"buckets": [
{
"doc_count": 372,
"key": "Charles Dickens"
},
{
"doc_count": 283,
"key": "Brian Kernighan"
}
],
"doc_count": 1000
}
}
}
Bonus tip:
Let us assume the field in question is nested as follows:
[{
"authors": [{
"details": [{
"name": "Brian Kernighan"
}]
}]
},
{
"authors": [{
"details": [{
"name": "Charles Dickens"
}]
}]
}
]
Now the correct query becomes:
{
"aggregations": {
"full_name": {
"aggregations": {
"author_details": {
"terms": {
"field": "authors.details.name"
}
}
},
"nested": {
"path": "authors.details"
}
}
},
"size": 0
}
Working for Elasticsearch 5.2.2
curl -XGET http://localhost:9200/articles/_search?pretty -d '
{
"aggs" : {
"whatever" : {
"terms" : { "field" : "yourfield", "size":10000 }
}
},
"size" : 0
}'
The "size":10000 means get (at most) 10000 unique values. Without this, if you have more than 10 unique values, only 10 values are returned.
The "size":0 means that in result, "hits" will contain no documents. By default, 10 documents are returned, which we don't need.
Reference: bucket terms aggregation
Also note, according to this page, facets have been replaced by aggregations in Elasticsearch 1.0, which are a superset of facets.
The existing answers did not work for me in Elasticsearch 5.X, for the following reasons:
I needed to tokenize my input while indexing.
"size": 0 failed to parse because "[size] must be greater than 0."
"Fielddata is disabled on text fields by default." This means by default you cannot search on the full_name field. However, an unanalyzed keyword field can be used for aggregations.
Solution 1: use the Scroll API. It works by keeping a search context and making multiple requests, each time returning subsequent batches of results. If you are using Python, the elasticsearch module has the scan() helper function to handle scrolling for you and return all results.
Solution 2: use the Search After API. It is similar to Scroll, but provides a live cursor instead of keeping a search context. Thus it is more efficient for real-time requests.