This is the query that throws the exception using ES 2.0:
bool query does not support filter
How to use Exists and Missing query?
Query:
{
"bool":{
"must":[
{
"bool":{
"should":[
{
"bool":{
"must":[
{
"range":{
"startDate":{
"lte":"2016-10-27T11:24:49.6616538+05:30"
}
}
}
],
"filter":[
{
"bool":{
"must_not":[
{
"exists":{
"field":"endDate"
}
}
]
}
}
]
}
}
]
}
}
]
}
}
Well first off, that error is generally from use on a 1.x version of Elasticsearch. (in that case you need a FilteredQuery)
Next it appears you have many levels of unnecessary nesting. Not sure if maybe you stripped other things out to make a simpler example. I have rewritten your query like this (and added the outer braces):
{
"query" : {
"bool" : {
"must" : [{
"range" : {
"startDate" : { "lte" : "2016-10-27T11:24:49.6616538+05:30" }
}
}
],
"filter" : [{
"bool" : {
"must_not" : [{
"exists" : { "field" : "endDate" }
}
]
}
}
]} }
}
Both your original query and my rewritten one work fine on my server (v2.3.1) so I'm guessing really you have ES 1.x ?
Also, if you are not leveraging the lucene scoring, and just want to return the documents (or apply your own sort) then you can drop the filter altogether and simplify it further:
{
"query" : {
"bool" : {
"must" : [{
"range" : {
"startDate" : { "lte" : "2016-10-27T11:24:49.6616538+05:30"}
}
}
],
"must_not" : [{
"exists" : { "field" : "endDate" }
}
]} }
}
Related
I have a collection with date stored as strings YYYY-mm-DD_HH:MM:SS.UUUZ like 2020-10-20_12:15:22.123+0100
My goal is to query on strings treating those as dates.
What am I doing:
I'm unwinding some header data on multiple documents:
{
"$unwind": {
"path": "$events",
"preserveNullAndEmptyArrays": true
}
}
and also
{
"$unwind": {
"path": "$events.hi2",
"preserveNullAndEmptyArrays": true
}
}
I'm adding a new field made with the string parsed as Date
{
"$addFields": {
"events.hi2.ConnectTimets": {
"$dateFromString": {
"dateString": "$events.hi2.ConnectTime",
"format": "%Y-%m-%d_%H:%M:%S.%L%Z"
}
}
}}
then on a $match stage I try to filter all records with date newer than 1 June 2020:
{
"$match":{
"events.hi2.ConnectTimets": {
"$gt": {"$dateFromString": {
"dateString": "2020-06-01",
"format": "%Y-%m-%d"
}
}
}
}
}
my result is Fetched 0 record(s) in 0ms
even though exists (at least a single document) in the database with a date matching the filter:
{
"_id" : ObjectId("5f438dfbf1feb13c4352e9f4"),
"timestamp" : NumberLong(1598262779045),
"attribute1" : [
"common"
],
"events" : [
{
"eventType" : "ty1",
"timestamp" : NumberLong(1598262779018),
"docId" : NumberLong(282578800148736),
"hi2" : {
"Priority" : 3,
"ClientId" : "client1",
"ConnectTime" : "2020-08-24_09:52:58.993+0000",
"Direction" : 1
}
},
{
"eventType" : "ty2",
"timestamp" : NumberLong(1598262781071),
"docId" : NumberLong(282578800148736),
"hi2" : {
"ref" : "bbbb"
}
}
]
}
When I espected something like
{
"_id" : ObjectId("5f438dfbf1feb13c4352e9f4"),
"timestamp" : NumberLong(1598262779045),
"attribute1" : [
"common"
],
"events" : [
{
"eventType" : "ty1",
"timestamp" : NumberLong(1598262779018),
"docId" : NumberLong(282578800148736),
"hi2" : {
"Priority" : 3,
"ClientId" : "client1",
"ConnectTime" : "2020-08-24_09:52:58.993+0000",
"Direction" : 1
}
}
}
Note_: the add field is ok because if i fire it without the match stage outputs a field with the string parsed as date
You should never store date/time values as string, use always proper Date objects.
Then the query is much simpler:
db.logging.aggregate([
{
$addFields: {
"events.hi2.ConnectTimets": {
$dateFromString: {
dateString: "$events.hi2.ConnectTime",
format: "%Y-%m-%d_%H:%M:%S.%L%Z"
}
}
}
},
{ $match: { "events.hi2.ConnectTimets": { $gte: ISODate("2020-06-01") } } }
])
When you have to work with date/time values then I recommend moment.js. Then you query could look like these:
{ $match: { "events.hi2.ConnectTimets": { $gte: moment("2020-06-01").toDate() } } }
{ $match: { "events.hi2.ConnectTimets": { $gte: moment("2020-06-01").tz('Europe/Zurich').toDate() } } }
{ $match: { "events.hi2.ConnectTimets": { $lte: moment.tz('Europe/Zurich').endOf('day').toDate() } } }
I'm working on ElasticSearch 6.5.3 and . Please find below my code:
from elasticsearch import Elasticsearch, helpers
def search(es_object, index_name, request):
toto = es_object.search(index=index_name, body=request)
return toto
fuzziness = 1
request1 = {
"query": {
"match" : {
"Family_name" : {
"query" : family_names,
"fuzziness": fuzziness,
}
}
},
"size" : size
}
request2 = {
"query": {
"match" : {
"First_name" : {
"query" : first_names,
"fuzziness": fuzziness,
}
}
},
"size" : size
}
result1 = search(es,ELASTICSEARCH_INDEX_NAME,request1)
result2 = search(es,ELASTICSEARCH_INDEX_NAME,request2)
I'd like to make a bool fuzzy query on firstname and family name. How can I do it please ?
I tried the following :
request = {
"bool": {
"must": [
{
"query": {
"match" : {
"First_name" : {
"query" : family_name,
"fuzziness": fuzziness,
}
}
},
"query": {
"match" : {
"Prénom" : {
"query" : firstname,
"fuzziness": fuzziness,
}
}
}
}
]
}
}
result = search(es,ELASTICSEARCH_INDEX_NAME,request)
I got the following error meaning that there is some problem in my query. It seems that I cannot combine two match queries having fuzziness simultaneously
RequestError: RequestError(400, 'parsing_exception', 'Unknown key
for a START_OBJECT in [bool].')
Your query is a bit wrong, you need to fix it like this:
{
"query": {
"bool": {
"must": [
{
"match": {
"First_name": {
"query": "family_name",
"fuzziness": fuzziness
}
}
},
{
"match": {
"Prénom": {
"query": "firstname",
"fuzziness": fuzziness
}
}
}
]
}
}
}
The reason is, that boolean query array already understand that it will be an object of query, so you don't need to specify it again
My mongo table contains contains collection 'Shops' and data like the below:
{
"ShopId" : 9999,
"products" : [
{
"productId" : "1234",
"productName" : "abcd",
},
{
"productId" : "5678",
"productName" : "abc",
},
{
"productId" : "2345",
"productName" : "def",
}
],
}
There will be several shops in the table having a list of products.
Requirement:
I want to fetch the records having shopId=9999 and products matches the string abc
My query
model.Shops.aggregate([{$match:{"ShopId":9999}},{$project:{products:{$filter:{input:'$products',cond: {'productName':/abc/ }}}}}])
Problem:
It is fetching the productname:defwith the other data that matches productname:abc.
You can't use regex search with the $filter operator. The only way to achieve this is to unwind products, filter document and then re-group them in an array
model.Shops.aggregate([
{
$match:{
"ShopId":9999
}
},
{
$unwind:"$products"
},
{
$match:{
"products.productName":/abc/
}
},
{
$group:{
_id:null,
products:{
$push:{
"productName":"$products.productName",
"productId":"$products.productId"
}
}
}
}
])
output:
{
"_id":null,
"products":[
{
"productName":"abcd",
"productId":"1234"
},
{
"productName":"abc",
"productId":"5678"
}
]
}
to use it with a variable, declare your regex like this:
var regex: /abc/;
and then use it directly in the query
$match:{
"products.productName": regex
}
Below code worked on mongoshell for me, and your code was giving error to me, that Missing 'as' parameter to $filter.
db.Shops.aggregate([
{$match:{"ShopId":9999}},
{$project:{
products:{$filter:{input:'$products',as:"product",cond: { $eq: [ "$$product.productName", "abc" ] }}}
}}
])
How to query :
{
"_id" : Object Id("58787242f7d06edbbb88f46e"),
"name" : "aah",
"values" : [
"1484196300685",
"10"
],
"attributes" : {
}
}
i need time where value=10 result 1484196300685
db.collection.find(
{ values: { $elemMatch: { $eq: "10" } } }
)
In Elasticsearch, I have this code that matches a thread and a query. It currently works fine for matching one thread, but I have an array of threads. And I want to get a hit if the field "thread" matches any of the threads in the array. For example, if I have an array of ['1', '2', '3'], I want to match if the "thread" field matches '1', '2', or '3', not just '1'. How do I do this?
client.search({
index: 'searchable-message',
body: {
query: {
bool: {
must: [
{
match: {
thread: '1' //<--WORKS FOR ONE, BUT NOT ARRAY
}
},
{
multi_match: {
query: req.query.q,
fields: ['message_text', 'stripped_text', 'links', 'documents.text_contents']
}
}
]
}
}
}
})
I think the best way to achieve this would be to use a different query method called terms.
Try changing you match query with a terms query. Here is the terms queries documentation.
Example:
{
"terms": {
"thread": ['1', '2', '3']
}
}
The bool query documentation also provides a nice example of the term query, which has a similar syntax with the terms query:
{
"bool" : {
"must" : {
"term" : { "user" : "kimchy" }
},
"filter": {
"term" : { "tag" : "tech" }
},
"must_not" : {
"range" : {
"age" : { "from" : 10, "to" : 20 }
}
},
"should" : [
{
"term" : { "tag" : "wow" }
},
{
"term" : { "tag" : "elasticsearch" }
}
]
}
}
Hope this helps :)