Here I have given my updated mapping
curl -X PUT localhost:9200/testing/listings/_mapping -d '{
"listings" : {
"properties" : {
"address" : {
"properties": {
"location": { "type" : "string",
"index" : "not_analyzed"
}
}
},
"suggest" : { "type" : "completion",
"index_analyzer" : "simple",
"search_analyzer" : "simple",
"payloads" : true
}
}
}
}'
my mapping is created index as follows
{
"testing": {
"mappings": {
"listings": {
"properties": {
"address": {
"properties": {
"city": {
"type": "string"
},
"line1": {
"type": "string"
},
"line2": {
"type": "string"
},
"line3": {
"type": "string"
},
"location": {
"type": "string",
"index": "not_analyzed"
},
"pincode": {
"type": "string"
}
}
},
"title": {
"type": "string"
}
}
}
}
}
}
but still my data is not matching.
my sample data is
{
"listings": {
"title": "testing 3",
"address": {
"line1": "3rd cross",
"line2": "6th main",
"line3": "",
"landmark": "",
"location": "k r puram",
"pincode": "",
"city": "Bangalore"
}
}
}
when I give the query as k r puram I am getting the matched results.
But when I am giving the query as r r puram or r k puram that time also I am getting the results which is belongs to k r puram.
In above query I am having listings only for k r puram others I don't have listings so other than k r puram it should give the empty results.
this is my query:
{
"query": {
"bool": {
"must": [
{
"match": {
"published": true
}
},
{
"match": {
"inActive": false
}
},
{
"range": {
"propertyDetailsCategory.build_up_area": {
"lte": 200
}
}
},
{
"match": {
"type": "commercial"
}
},
{
"match": {
"purpose": "rent"
}
},
{
"range": {
"commercialsCategory.exp_rent": {
"lte": 50000
}
}
},
{
"match": {
"address.location": "k r puram"
}
}
]
}
}
}
If the data is exactly "k r puram" and you're searching for exactly "k r puram" - then you shouldn't use an analyser.
When inserting data the default behaviour in Elasticsearch is to use the standard analyser.
To disable this use
"index": "not_analyzed"
in the mapping for the appropriate field.
if your mapping is as follows:
curl -XPOST http://localhost:9200/index/address/_mapping -d '
{"address": {
"properties": {
"city": {"type": "string"},
"line1": {"type": "string"},
"line2": {"type": "string"},
"line3": {"type": "string"},
"location": { "type": "string", "index": "not_analyzed"},
"pincode": {"type": "string"}
}}}'
then your data must match it, for example this doesn't match it:
curl -XPOST http://localhost:9200/index/address/ -d '
{"title":"testing",
"address":
{"line1":"#51",
"line2":"3rd cross",
"line3":"6th main",
"location":"k r puram",
"pincode":"560041"}}
This however does match (my modifications):
curl -XPOST http://localhost:9200/index/address/ -d '
{"line1":"#51",
"line2":"3rd cross",
"line3":"6th main",
"location":"k r puram",
"pincode":"560041"}'
And this query finds the document as expected:
curl -XGET http://localhost:9200/index/address/_search -d '
{
"query" :{"match" : {"location": "k r puram"}}
}'
if you can't change your data, then add the extra level to the mapping,e.g.:
curl -XPOST http://localhost:9200/index/address3/_mapping -d '{
"address3" : {
"properties" : {
"address" : {
"properties" : {
"city" : {
"type" : "string"
},
"line1" : {
"type" : "string"
},
"line2" : {
"type" : "string"
},
"location" : {
"type" : "string", "index": "not_analyzed"
}
}
},
"title" : {
"type" : "string"
}
}
}
}'
Again the query works well:
curl -XGET http://localhost:9200/index/address3/_search -d '
{
"query" :{"match" : {"address.location": "k r puram"}}
}'
Have you tried this? (use .raw sub field to match value on "not tokenized" value)
{"query":{
"bool":{
"must":[
{"match":{"published":true}},
{"match":{"inActive":false}},
{"range":{"propertyDetailsCategory.build_up_area":{"lte":200}}},
{"match":{"type":"commercial"}},
{"match":{"purpose":"rent"}},
{"range":{"commercialsCategory.exp_rent":{"lte":50000}}},
{"match":{"address.location.raw": "k r puram"}}
]
}
}
}
Try to use this query on your old mapping, it should work :)
Related
I created an index in elasticsearch 6.5.1 successfully loaded the data to that index. there is one field "submitted_date" which is the timestamp. below is the mapping like of this field.
"submitted_date": { "type": "date", "format":"yyyy-MM-dd HH:mm:ss.SSS" },
then I created the index pattern. I used the Time Filter field name as "submitted_date". after that, I tried to check the data in Discover tab, but data are not showing. there is a message saying that No results match your search criteria.
NOTE that I have changed the time in time range picker in every possible way which is on top of the right corner in kibana dashboard.
data appear in Dev Tools tab with elastic queries.
ps : I inserted the data using nodejs with elasticsearch official library, did not used logstash.
I followed this article, but it did not help me.
UPDATE : sample document
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 10480,
"max_score" : 1.0,
"hits" : [
{
"_index" : "test",
"_type" : "tests",
"_id" : "1214334",
"_score" : 1.0,
"_source" : {
"priority" : "4",
"submitted_date" : "2018-01-04T18:32:21.000Z",
"submitted_month" : 0,
"submitted_month_name" : "January",
"submitted_day" : 4,
"submitted_weekday" : "Tuesday",
"submitted_hour" : 18,
"submitted_year_month" : "2018-0",
"submitted_year_month_name" : "2018-January",
"date_key" : "20180104",
"year_month_key" : "201801",
"status" : "Closed"
}
}
]
}
}
Inspect request
{
"version": true,
"size": 500,
"sort": [
{
"_score": {
"order": "desc"
}
}
],
"_source": {
"excludes": []
},
"aggs": {
"2": {
"date_histogram": {
"field": "submitted_date",
"interval": "1d",
"time_zone": "Asia/Kolkata",
"min_doc_count": 1
}
}
},
"stored_fields": [
"*"
],
"script_fields": {},
"docvalue_fields": [
{
"field": "close_date",
"format": "date_time"
},
{
"field": "last_modified_date",
"format": "date_time"
},
{
"field": "last_resolved_date",
"format": "date_time"
},
{
"field": "submitted_date",
"format": "date_time"
},
{
"field": "time_to_resolve",
"format": "date_time"
}
],
"query": {
"bool": {
"must": [
{
"match_all": {}
},
{
"range": {
"submitted_date": {
"gte": 1514745000000,
"lte": 1543937620414,
"format": "epoch_millis"
}
}
}
],
"filter": [],
"should": [],
"must_not": []
}
},
"highlight": {
"pre_tags": [
"#kibana-highlighted-field#"
],
"post_tags": [
"#/kibana-highlighted-field#"
],
"fields": {
"*": {}
},
"fragment_size": 2147483647
}
}
Index pattern
function _putMapping() {
return client.indices.create({
index: process.env.ELASTICSEARCH_INDEX,
body: {
settings:{
index:{
"number_of_shards": 1,
"number_of_replicas": 5
},
"index.mapping.ignore_malformed" : true
},
mappings:{
tests:{
properties:{
"last_modified_date": { "type": "date" },
"last_resolved_date": { "type": "date" },
"time_to_resolve": { "type": "date" },
"submitted_date": { "type": "date", "format":"yyyy-MM-dd HH:mm:ss.SSS" },
"date_key": { "type": "integer" },
"priority": { "type": "long" },
"submitted_hour": { "type": "long" },
"submitted_month": { "type": "long" },
"submitted_year": { "type": "long" },
"submitted_year": { "type": "keyword" },
"submitted_year_month": { "type": "keyword" },
"submitted_year_month_name": { "type": "keyword" },
}
}
}
}
});
}
Your mYour submitted_date is coming like 2018-01-04T18:32:21.000Z but your mapping is set as yyyy-MM-dd HH:mm:ss.SSS.
You need to change it to "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'".
I am indexing an attachment field. The POST query in sense returns expected resultset.
My query is
POST /mydocs/_search
{
"query" : {
"bool" : {
"must" : [
{ "match" : { "file.content":"abc"} },
{ "match":{"otherDetails":"asd"}},
{ "match" : { "filePermissionInfo.accountValue" : "xyz"} }
]
}
}
}
I need to convert it to a c# Nest code. I tried converting it, but its not returning any result,even it contains data. If I remove the
m.Match(mt1 => mt1.Field(f1 => f1.File.Coontent).Query(queryTerm))
from the below experssion, it returns a result set. Is there any problem with the attachement field?
client.Search<IndexDocument>(s => s
.Index("mydocs")
.Query(q => q
.Bool(b => b
.Must(m =>
m.Match(mt1 => mt1.Field(f1 => f1.File.Coontent).Query(queryTerm)) &&
m.Match(mt2 => mt2.Field(f2 => f2.FilePermissionInfo.First().SecurityIdValue).Query(accountName)) &&
m.Match(mt3 => mt3.Field(f3 => f3.OtherDetails).Query(other))
)))
);
My mapping is
{
"mydocs": {
"mappings": {
"indexdocument": {
"properties": {
"docLocation": {
"type": "string",
"index": "not_analyzed",
"store": true
},
"documentType": {
"type": "string",
"store": true
},
"file": {
"type": "attachment",
"fields": {
"content": {
"type": "string",
"term_vector": "with_positions_offsets",
"analyzer": "full"
},
"author": {
"type": "string"
},
"title": {
"type": "string",
"term_vector": "with_positions_offsets",
"analyzer": "full"
},
"name": {
"type": "string"
},
"date": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"keywords": {
"type": "string"
},
"content_type": {
"type": "string"
},
"content_length": {
"type": "integer"
},
"language": {
"type": "string"
}
}
},
"filePermissionInfo": {
"properties": {
"fileSystemRights": {
"type": "string",
"store": true
},
"securityIdValue": {
"type": "string",
"store": true
}
}
},
"id": {
"type": "double",
"store": true
},
"lastModifiedDate": {
"type": "date",
"store": true,
"format": "strict_date_optional_time||epoch_millis"
},
"otherDetails": {
"type": "string"
},
"title": {
"type": "string",
"store": true,
"term_vector": "with_positions_offsets"
}
}
}
}
}
}
It looks like the query hasn't been translated to NEST correctly. In the query you have
"filePermissionInfo.accountValue"
but in the NEST query, you only have
f2 => f2.FilePermissionInfo
which would result only in filePermissionInfo. You need to change this to
f2 => f2.FilePermissionInfo.AccountValue
I have the following mapping
{
"cloth": {
"dynamic" : false,
"_source" : {"enabled" : false },
"properties": {
"name": {
"type": "string",
"index": "analyzed"
},
"variation": {
"type": "nested",
"properties": {
"size": {
"type": "string",
"index": "not_analyzed"
},
"color": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}
I am not able to figure out a way to retrieve the nested object fields using the fields query.
{
"fields" : ["name" , "variation.size", "variation.color"],
"query" : {
"nested" : {
"path" : "variation",
"query" : {
"bool" : {
"must" : [
{ "term" : { "variation.size" : "XXL" } },
{ "term" : { "variation.color" : "red" } }
]
}
}
}
}
}
The above query returns
"_id" : "1",
"_score" : 1.987628,
"fields" : {
"variation.size" : [ "XXL", "XL" ],
"variation.color" : [ "red", "black" ],
"name" : [ "Test shirt" ]
}
When I tried
"fields" : ["name" , "variation"]
I got the error
status: 400
reason: "ElasticsearchIllegalArgumentException[field [variation] isn't a leaf field]"
Which is as expected.
How can I get the variation object as it is?
Expected Result. I need to retrieve the variable object as a whole so that I can preserve the association of size and color. Like "red" with "XXL".
"variation" : { "XXL" , "red"}
Update: Source is disabled for this Index Type.
If you use Source Filtering it will return the nested objects as a whole, your query would be:
{
"_source": [
"name",
"variation"
],
"query": {
"nested": {
"path": "variation",
"query": {
"bool": {
"must": [
{
"term": {
"variation.size": "XXL"
}
},
{
"term": {
"variation.color": "red"
}
}
]
}
}
}
}
}
You should use this:
"script_fields": {
"variation": {
"script": {
"inline": "doc['variation.size'].value + ' ' + doc['variation.red'].value"
}
}
}
I use elasticsearch v. 5.1.1
I am new to Elasticsearch, and right now I am trying to figure out why my synonyms are not returning any results like I expect them to.
I created a custom filter and analyzer for my synonyms file and applied the analyzer to both the _all field and explicitly defined the specialty field to use it as well.
When I search for "specialty": "aids" without the analyzer/tokenizer, it gives me zero results as expected.
However, when I search for "specialty": "aids" with the analyzer/tokenizer, I expect it to give me the same results as searching for "speciality": "retrovirology", which should yields 3 results, but it comes back with nothing.
Is there something wrong with how I am approaching this?
Here are my settings and some sample data:
curl -XDELETE "http://localhost:9200/personsearch"
curl -XPUT "http://localhost:9200/personsearch" -d'
{
"settings": {
"index": {
"analysis": {
"analyzer": {
"XYZSynAnalyzer": {
"tokenizer": "standard",
"filter": [
"XYZSynFilter"
]
}
},
"filter": {
"XYZSynFilter": {
"type": "synonym",
"synonyms": [
"aids, retrovirology"
]
}
}
}
}
},
"mappings": {
"xyzemployee": {
"_all": {
"analyzer": "XYZSynAnalyzer"
},
"properties": {
"firstName": {
"type": "string"
},
"lastName": {
"type": "string"
},
"middleName": {
"type": "string",
"include_in_all": false,
"index": "not_analyzed"
},
"specialty": {
"type": "string",
"analyzer": "XYZSynAnalyzer"
}
}
}
}
}'
curl -XPUT "http://localhost:9200/personsearch/xyzemployee/1" -d'
{
"firstName": "Don",
"middleName": "W.",
"lastName": "White",
"specialty": "Adult Retrovirology"
}'
curl -XPUT "http://localhost:9200/personsearch/xyzemployee/2" -d'
{
"firstName": "Terrance",
"middleName": "G.",
"lastName": "Gartner",
"specialty": "Retrovirology"
}'
curl -XPUT "http://localhost:9200/personsearch/xyzemployee/3" -d'
{
"firstName": "Carter",
"middleName": "L.",
"lastName": "Taylor",
"specialty": "Pediatric Retrovirology"
}'
# Why is this returning nothing?
curl -XGET "http://localhost:9200/personsearch/xyzemployee/_search?pretty=true" -d'
{
"query": {
"match": {
"specialty": "retrovirology"
}
}
}'
You aren't lowercasing anywhere.
Try this:
{
"settings": {
"index": {
"analysis": {
"analyzer": {
"XYZSynAnalyzer": {
"tokenizer": "standard",
"filter": [
"lowercase", "XYZSynFilter"
]
}
},
"filter": {
"XYZSynFilter": {
"type": "synonym",
"synonyms": [
"aids, retrovirology"
]
}
}
}
}
}
Note: you may want to split your index analyzer and search analyzer, and choose only one of them to do the synonyms. Expanding them only during indexing will speed search results.
I have the following mapping:
curl -XPUT 'http://localhost:9200/bookstore/user/_mapping' -d '
{
"user": {
"properties": {
"user_id": { "type": "integer" },
"gender": { "type": "string", "index" : "not_analyzed" },
"age": { "type": "integer" },
"age_bracket": { "type": "string", "index" : "not_analyzed" },
"current_city": { "type": "string", "index" : "not_analyzed" },
"relationship_status": { "type": "string", "index" : "not_analyzed" },
"books" : {
"type": "nested",
"properties" : {
"b_oid": { "type": "string", "index" : "not_analyzed" },
"b_name": { "type": "string", "index" : "not_analyzed" },
"bc_id": { "type": "integer" },
"bc_name": { "type": "string", "index" : "not_analyzed" },
"bcl_name": { "type": "string", "index" : "not_analyzed" },
"b_id": { "type": "integer" }
}
}
}
}
}'
Now, I try to query for example for Users which have "gender": "Male", have bought book in a certain category "bcl_name": "Trivia" and show the "b_name" book titles. I somehow cannot get it to run.
I have the query
curl -XGET 'http://localhost:9200/bookstore/user/_search?pretty=1' -d '{
"size": 0,
"from": 0,
"query": {
"filtered": {
"query": {
"terms": {
"gender": [
"Male"
]
}
}
}
},
"facets": {
"CategoryFacet": {
"terms": {
"field": "books.b_name",
"size": 5,
"shard_size": 1000,
"order": "count"
},
"nested": "books",
"facet_filter": {
"terms": {
"books.bcl_name": [
"Trivia"
]
}
}
}
}
}'
which returns a result, but I'm not sure whether this is correct. I looked for some examples, and found this (http://www.spacevatican.org/2012/6/3/fun-with-elasticsearch-s-children-and-nested-documents/) for example. I'm able to rewrite my query like this:
curl -XGET 'http://localhost:9200/bookstore/user/_search?pretty=1' -d '{
"size": 0,
"from": 0,
"query": {
"filtered": {
"query": {
"terms": {
"gender": [
"Male"
]
}
},
"filter": {
"nested": {
"path": "books",
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"and": [
{
"term": {
"books.bcl_name": "Trivia"
}
}
]
}
}
}
}
}
}
},
"facets": {
"CategoryFacet": {
"terms": {
"field": "books.b_name",
"size": 5,
"shard_size": 1000,
"order": "count"
},
"nested": "books"
}
}
}'
which shows different results.
I, as a beginner, am a litte lost right now. Can someone please give me hint on how to solve this`? Thanks a lot in advance!
First query means:
Search for users whose gender : "Male"
But "CategoryFacet" includes the count of gender : "Male" AND
books.bcl_name : "Trivia"
So in result set you get all "Male" users, but your CategoryFacet gives you the count of "Male users AND whose books.bcl_name is Trivia".
In second query your "CategoryFacet" does not include extra filtering. It just returns the facets from the exact result set.