In elasticsearch, I search some tags and sort them from the most matching to the least matching. It's ok.
However, my problem is about order of equal matching situations.
For example:
I stored these tags: "tag1", "tag2", "tag3", "tag4", "tag5"
I stored these tags: "tag6", "tag1"
I stored these tags: "tag4", "tag3", "tag1"
I stored these tags: "tag2", "tag1", "tag5", "tag7"
My search query:
{
"query" : {
"bool" : {
"must" : [
{
"terms" : {
"my_field" : ["tag4", "tag6"],
minimum_should_match : 1
}
},
{"term" : {"my_cityId" : 1}},
{"term" : {"my_townId" : 8}}
]
}
},
"sort" : [
{"_score" : "desc"},
{"my_topTime" : "asc"}
],
"from" : 0,
"size" : 5
}
It returns:
"tag6", "tag1"
"tag1", "tag2", "tag3", "tag4", "tag5"
"tag4", "tag3", "tag1"
My search order is tag4 and then tag6. How can it be returned tag4 contained rows first and tag6 after?
I found a solution to my problem from Boosting Filtered Subsets
I used "function_score" for my filter query, and added "functions" and "score_mode". Beside this, I specified "weight" for each tag in "functions".
The query returned tag4s first and then tag6 contained rows:
"tag1", "tag2", "tag3", "tag4", "tag5"
"tag4", "tag3", "tag1"
"tag6", "tag1"
My new query:
{
"query" : {
"function_score" : {
"filter" : {
"query" : {
"bool" : {
"must" : [
{
"terms" : {
"my_field" : ["tag4", "tag6"],
minimum_should_match : 1
}
} ,
{"term" : {"my_cityId" : 1}},
{"term" : {"my_townId" : 8}}
]
}
}
},
"functions" : [
{"filter" : {"term" : { "my_field" : "tag4" }}, "weight" : 2},
{"filter" : { "term" : { "my_field" : "tag6" }}, "weight" : 1}
],
"score_mode": "sum"
}
},
"sort" : [
{"_score" : "desc"},
{"my_topTime" : "asc"}
],
"from" : 0,
"size" : 5
}
Related
FULL DISCLOSURE: I'm a MongoDB noob
I'm dealing with a legacy DB structure. A part of my MongoDB looks like this currently:
Events (_id, name (string), ...)
Orders (_id, eventId (as string), products (array of {prodIdentifier (string), quantity (number)}), customer_ID (string), signee (string), sign_time (date), ...)
Products (_id, prodIdentifier (string), price (number), sku (string), ...)
The relations are as follows:
Event 1..N Orders (via eventId)
Orders 1..N Products (via products array)
I need to query in a way that given an eventId, I return
Order ID Customer Name (can be a cascade request / premeditated
by frontend), Product SKU, Product Name, Quantity,
Value (quantity * price), Signee Name, Sign time
Mind that, my interface requires filters and sorts on all of the above fields along with limit and offset for pagination, to reduce query time, fast UI, etc.
I could use populate on orders, but how am I supposed to honor the limit and offset via mongoose then. I'm wondering if I should make a view, in which case how should I flatten it to send/receive a list that honors the limit and offset.
Or will it have to be a very manual, step-by-step build of the resulting list?
UPDATE:
Sample data in the DB:
Event Object:
{
"_id" : ObjectId("6218b9266487367ba1c20258"),
"name" : "XYZ",
"createdAt" : ISODate("2022-02-03T13:25:43.814+0000"),
"updatedAt" : ISODate("2022-02-14T09:34:47.819+0000"),
...
}
Order(s):
[
{
"_id" : ObjectId("613ae653d0112f6b49fdd437"),
"orderItems" : [
{
"quantity" : NumberInt(2),
"productCode" : "VEO001",
},
{
"quantity" : NumberInt(2),
"productCode" : "VEO002",
},
{
"quantity" : NumberInt(1),
"productCode" : "VEO003",
}
],
"orderCode" : "1000",
"customerCode" : "Customer 1",
"createdAt" : ISODate("2021-09-10T05:00:03.496+0000"),
"updatedAt" : ISODate("2022-02-08T10:06:42.255+0000"),
"eventId" : "6218b9266487367ba1c20258"
}
]
Products:
[
{
"_id" : ObjectId("604206685f25b8560a1cd48d"),
"Product name" : "ABC",
"createdAt" : ISODate("2021-03-05T10:22:32.085+0000"),
"tag" : "VEO001",
"updatedAt" : ISODate("2022-03-28T07:29:21.939+0000"),
"Product Price" : NumberInt(0),
"photo" : {
"_id" : ObjectId("6042071a5f25b8560a1cd4a9"),
"key" : "e8c9a085-4e8d-4ac4-84e9-bb0a83a59145",
"name" : "Screenshot 2021-03-05 at 11.24.50.png"
},
"name" : "ABC",
"_costprice" : NumberInt(12),
"_sku" : "SKUVEO001",
},
{
"_id" : ObjectId("604206685f25b8560a1cd48a"),
"Product name" : "DEF",
"createdAt" : ISODate("2021-03-05T10:22:32.085+0000"),
"tag" : "VEO002",
"updatedAt" : ISODate("2022-03-28T07:29:21.939+0000"),
"Product Price" : NumberInt(0),
"photo" : {
"_id" : ObjectId("6042071a5f25b8560a1cd4a9"),
"key" : "e8c9a085-4e8d-4ac4-84e9-bb0a83a59145",
"name" : "Screenshot 2021-03-05 at 11.24.50.png"
},
"name" : "DEF",
"_costprice" : NumberInt(13),
"_sku" : "SKUVEO002",
},
{
"_id" : ObjectId("604206685f25b8560a1cd48a"),
"Product name" : "GHI",
"createdAt" : ISODate("2021-03-05T10:22:32.085+0000"),
"tag" : "VEO003",
"updatedAt" : ISODate("2022-03-28T07:29:21.939+0000"),
"Product Price" : NumberInt(0),
"photo" : {
"_id" : ObjectId("6042071a5f25b8560a1cd4a9"),
"key" : "e8c9a085-4e8d-4ac4-84e9-bb0a83a59145",
"name" : "Screenshot 2021-03-05 at 11.24.50.png"
},
"name" : "GHI",
"_costprice" : NumberInt(13),
"_sku" : "SKUVEO003",
},
]
Expected output:
You can do something like:
db.orders.aggregate([
{$match: {eventId: "6218b9266487367ba1c20258"}},
{
$lookup: {
from: "products",
localField: "orderItems.productCode",
foreignField: "tag",
as: "orderItemsB"
}
},
{
"$addFields": {
"orderItems": {
"$map": {
"input": "$orderItemsB",
"in": {
"$mergeObjects": [
"$$this",
{
"$arrayElemAt": [
"$orderItems",
{"$indexOfArray": ["$orderItems.productCode", "$$this.tag"]}
]
}
]
}
}
},
orderItemsB: 0
}
},
{
$unset: "orderItemsB"
},
{
$lookup: {
from: "events",
let: {eventId: "$eventId"},
pipeline: [
{
$match: {$expr: {$eq: [{$toString: "$_id"}, "$$eventId"]}}
}
],
as: "event"
}
},
{
$set: {event: {"$arrayElemAt": ["$event", 0]}}
},
{$unwind: "$orderItems"}
])
As you can see on this playground example. This will give you a document for each product of the order with all the data.
I have MongoDB collection with multiple records. Each record has an array which contains objects with multiple fields.
I have collection like below:
[{
"name" : "Karthik Thurairaja"
"universities" : [
{
"name" : "Anna University",
"city" : "Chennai"
},
{
"name" : "Punjab University",
"city" : "Chandigarh"
},
{
"name" : "University of Delhi",
"city" : "New Delhi"
}
],
},
{
"name" : "Sathish Kumar"
"universities" : [
{
"name" : "Anna University",
"city" : "Chennai"
},
{
"name" : "University of Hyderabad",
"city" : "Hyderabad"
},
{
"name" : "University of Delhi",
"city" : "New Delhi"
}
],
}]
I need to find all the records universities city is equal to Chennai.
I have tried query like below:
Collection.find({ universities.city : "Chennai" }).exec(...);
You can use an $elemMatch query to achieve this.
Collection.find({ universities: { $elemMatch: { city: "Chennai" } } }).exec(...);
I have collection with documents like this :
{
"_id" : ObjectId("5c0685fd6afbd73b80f45338"),
"page_id" : "1234",
"category_list" : [
"football",
"sport"
],
"time_broadcast" : "09:13"
}
{
"_id" : ObjectId("5c0685fd6afbd7355f45338"),
"page_id" : "1234",
"category_list" : [
"sport",
"handball"
],
"time_broadcast" : "09:13"
}
{
"_id" : ObjectId("5c0694ec6afbd74af41ea4af"),
"page_id" : "123456",
"category_list" : [
"news",
"updates"
],
"time_broadcast" : "09:13"
}
....
now = datetime.datetime.now().time().strftime("%H:%M")
What i want is : when "time_broadcast" is equal to "now",i get list of distinct "category_list" of each "page_id".
Here is how the output should look like :
{
{
"page_id" : "1234",
"category_list" : ["football", "sport", "handball"]
},
{
"page_id" : "123456",
"category_list" : ["news", "updates"]
}
}
I have tried like this :
category_list = db.users.find({'time_broadcast': now}).distinct("category_list")
but this gives me as output list of distinct values but
of all "page_id" :
["football", "sport", "handball","news", "updates"]
not category_list by page_id .
Any help please ?
Thanks
you need to write an aggregate pipeline
$match - filter the documents by criteria
$group - group the documents by key field
$addToSet - aggregate the unique elements
$project - project in the required format
$reduce - reduce the array of array to array by $concatArrays
aggregate query
db.tt.aggregate([
{$match : {"time_broadcast" : "09:13"}},
{$group : {"_id" : "$page_id", "category_list" : {$addToSet : "$category_list"}}},
{$project : {"_id" : 0, "page_id" : "$_id", "category_list" : {$reduce : {input : "$category_list", initialValue : [], in: { $concatArrays : ["$$value", "$$this"] }}}}}
]).pretty()
result
{ "page_id" : "123456", "category_list" : [ "news", "updates" ] }
{
"page_id" : "1234",
"category_list" : [
"sport",
"handball",
"football",
"sport"
]
}
you can add $sort by page_id pipeline if required
I am newbie. But I try to learn the most logical ways to write the queries.
Assume I have collection which is as;
{
"id" : NumberInt(1),
"school" : [
{
"name" : "george",
"code" : "01"
},
{
"name" : "michelangelo",
"code" : "01"
}
],
"enrolledStudents" : [
{
"userName" : "elisabeth",
"code" : NumberInt(21)
}
]
}
{
"id" : NumberInt(2),
"school" : [
{
"name" : "leonarda da vinci",
"code" : "01"
}
],
"enrolledStudents" : [
{
"userName" : "michelangelo",
"code" : NumberInt(25)
}
]
}
I want to list occurence of a key with their corresponding code values.
As an example key : michelangelo
To find the occurence of the key, I wrote two differen aggregation queries as;
db.test.aggregate([
{$unwind: "$school"},
{$match : {"school.name" : "michelangelo"}},
{$project: {_id: "$id", "key" : "$school.name", "code" : "$school.code"}}
])
and
db.test.aggregate([
{$unwind: "$enrolledStudents"},
{$match : {"enrolledStudents.userName" : "michelangelo"}},
{$project: {_id: "$id", "key" : "$enrolledStudents.userName", "code" : "$enrolledStudents.code"}}
])
the result of these 2 queries return what I want as;
{ "_id" : 1, "key" : "michelangelo", "code" : "01" }
{ "_id" : 2, "key" : "michelangelo", "code" : 25 }
One of them to search in enrolledStudents, the other one is searching in school field.
Can these 2 queries reduced into more logical query? Or is this the only way to do it?
ps: I am aware that database structure is not logical, but I tried to simulate.
edit
I try to write a query with find.
db.test.find({$or: [{"enrolledStudents.userName" : "michelangelo"} , {"school.name" : "michelangelo"}]}).pretty()
but this returns the whole documents as;
{
"id" : 1,
"school" : [
{
"name" : "george",
"code" : "01"
},
{
"name" : "michelangelo",
"code" : "01"
}
],
"enrolledStudents" : [
{
"userName" : "elisabeth",
"code" : 21
}
]
}
{
"id" : 2,
"school" : [
{
"name" : "leonarda da vinci",
"code" : "01"
}
],
"enrolledStudents" : [
{
"userName" : "michelangelo",
"code" : 25
}
]
}
Mongo 3.4
$match - This stage will keep all the school array and enrolledStudents where there is atleast one embedded document matching both the query condition
$group - This stage will combine all the school and enrolledStudents array to 2d array for each _id in a group.
$project - This stage will $filter the merge array for matching query condition and $map the array to with new labels values array.
$unwind - This stage will flatten the array.
$addFields & $replaceRoot - This stages will add the id field and promote the values array to the top.
db.collection.aggregate([
{$match : {$or: [{"enrolledStudents.userName" : "michelangelo"} , {"school.name" : "michelangelo"}]}},
{$group: {_id: "$id", merge : {$push:{$setUnion:["$school", "$enrolledStudents"]}}}},
{$project: {
values: {
$map:
{
input: {
$filter: {
input: {"$arrayElemAt":["$merge",0]},
as: "onef",
cond: {
$or: [{
$eq: ["$$onef.userName", "michelangelo"]
}, {
$eq: ["$$onef.name", "michelangelo"]
}]
}
}
},
as: "onem",
in: {
key : { $ifNull: [ "$$onem.userName", "$$onem.name" ] },
code : "$$onem.code"}
}
}
}
},
{$unwind: "$values"},
{$addFields:{"values.id":"$_id"}},
{$replaceRoot: { newRoot:"$values"}}
])
Sample Response
{ "_id" : 2, "key" : "michelangelo", "code" : 25 }
{ "_id" : 1, "key" : "michelangelo", "code" : "01" }
Mongo <= 3.2
Replace last two stages of above aggregation with $project to format the response.
{$project: {"_id": 0 , id:"$_id", key:"$values.key", code:"$values.code"}}
Sample Response
{ "_id" : 2, "key" : "michelangelo", "code" : 25 }
{ "_id" : 1, "key" : "michelangelo", "code" : "01" }
You can use $redact instead of $group & match and add $project with $map to format the response.
$redact to go through a document level at a time and perform $$DESCEND and $$PRUNE on the matching criteria.
The only thing to note is usage of $ifNull in the first document level for id so that you can $$DESCEND to embedded document level for further processing.
db.collection.aggregate([
{
$redact: {
$cond: [{
$or: [{
$eq: ["$userName", "michelangelo"]
}, {
$eq: ["$name", "michelangelo"]
}, {
$ifNull: ["$id", false]
}]
}, "$$DESCEND", "$$PRUNE"]
}
},
{
$project: {
id:1,
values: {
$map:
{
input: {$setUnion:["$school", "$enrolledStudents"]},
as: "onem",
in: {
key : { $ifNull: [ "$$onem.userName", "$$onem.name" ] },
code : "$$onem.code"}
}
}
}
},
{$unwind: "$values"},
{$project: {_id:0,id:"$id", key:"$values.key", code:"$values.code"}}
])
/* 0 */
{
"_id" : ObjectId("5380c9e097632cee5b000007"),
"month" : "5",
"userid" : "53806aac12c75f4b51000001",
"__v" : 7,
"posts" : [{
"postid" : ObjectId("538185cae0c6b8666e000008"),
"ts" : ISODate("2014-05-25T05:55:22.976Z"),
"userid" : "53806aac12c75f4b51000001",
"name" : "BBB",
"text" : "b1",
}]
}
/* 1 */
{
"_id" : ObjectId("5380c80e97632cee5b000001"),
"month" : "5",
"userid" : "5380629ea3b31f864f000001",
"__v" : 24,
"posts" : [{
"postid" : ObjectId("538185b2e0c6b8666e000004"),
"ts" : ISODate("2014-05-25T05:54:58.703Z"),
"userid" : "5380629ea3b31f864f000001",
"name" : "AAA",
"text" : "a1",
}, {
"postid" : ObjectId("538185b7e0c6b8666e000006"),
"ts" : ISODate("2014-05-25T05:55:03.474Z"),
"userid" : "5380629ea3b31f864f000001",
"name" : "AAA",
"text" : "a2",
}, {
"postid" : ObjectId("538185d6e0c6b8666e00000a"),
"ts" : ISODate("2014-05-25T05:55:34.231Z"),
"userid" : "5380629ea3b31f864f000001",
"name" : "AAA",
"text" : "a3",
}]
}
This is my DATA.
I want to Sort This Data for 'Ts' ( Data ).
I want That Sorted List by 'posts.Ts' Like this..
name : AAA, text = a3
name : BBB, text = b1
name : AAA, text = a2
name : AAA, text = a1
but i Don't know How to query this. Please Talk To ME
This is my code in Node and mongoose.
db.collection('walls', function(err, collection) {
collection.find(function(err, data) {
collection.aggregate(
{$match: {userid:userid}},
{$project: {posts: 1,_id:0}},
{$sort:{'posts.ts':1}},
{$unwind: "$posts"}
)}
...
You are onto the right principles but you have pipeline stages the wrong way around. You need to $unwind the arrays before you sort:
db.collection.aggregate([
{ "$match": { "userId": userId" } },
{ "$project": { "_id": 0, "posts": 1 } },
{ "$unwind": "$posts" },
{ "$sort": { "posts.ts": 1, "posts.name": 1 } }
])