I have a collection called article_category which store all article_id belongs to the category with category_id with data format like so.
Collection 1: article_category
{
"article_id": 2015110920343902,
"all_category_id": [5,8,10]
}
Then I have other collection called article which store all my post
Collection 2: article
{
"title": "This is example rows in article collection"
"article_id": 2015110920343902,
},
{
"title": "Something change"
"article_id": 2015110920343903,
},
{
"title": "This is another rows",
"article_id": 2015110920343904,
}
Now I want to perform MongoDB query to find title with regex while category_id must equal to 8. Here is my query but is not work.
db.article.aggregate(
{
$match:
{
title:
{
$regex: /example/
}
}
},
{
$lookup:
{
from: "article_category",
pipeline: [
{ $match: { category_id: 8 } }
],
as: "article_category"
}
}
)
Above query only show the records which match by regex but not match by category_id.
Any idea?
First of all, it is all_category_id, not category_id. Secondly, you don't link articles - all documents will have exactly the same article_category array. Lastly, you probably want to filter out articles that don't have matched category. The conditional pipeline should look more like this:
db.article.aggregate([
{ $match: {
title: { $regex: /example/ }
} },
{ $lookup: {
from: "article_category",
let: {
article_id: "$article_id"
},
pipeline: [
{ $match: {
$expr: { $and: [
{ $in: [ 8, "$all_category_id" ] },
{ $eq: [ "$article_id", "$$article_id" ] }
] }
} }
],
as: "article_category"
} },
{ $match: {
$expr: { $gt: [
{ $size: "$article_category"},
0
] }
} }
] )
UPDATE:
If you don't match article_id, the $lookup will result with identical article_category array to all articles.
Let's say your article_category collection has another document:
{
"article_id": 0,
"all_category_id": [5,8,10]
}
With { $eq: [ "$article_id", "$$article_id" ] } in the pipeline the resulting article_category is
[
{
"article_id" : 2015110920343902,
"all_category_id" : [ 5, 8, 10 ]
}
]
without:
[
{
"article_id" : 2015110920343902,
"all_category_id" : [ 5, 8, 10 ]
},
{
"article_id": 0,
"all_category_id": [ 5, 8, 10 ]
}
]
If the later is what you need, it would be way simpler to make to find requests:
db.article.find({ title: { $regex: /example/ } })
and
db.article_category.find({ all_category_id: 8 })
You've couple of things incorrect here. category_id should be all_category_id. Use the join condition in $lookup and move the $match outside of $lookup stage with $unwind for optimized lookup.
Use $project with exclusion to drop the looked up field from final response.
Something like {$project:{article_category:0}}
Try
db.article.aggregate([
{"$match":{"title":{"$regex":/example/}}},
{"$lookup":{
"from":"article_category",
"localField":"article_id",
"foreignField":"article_id",
"as":"article_category"
}},
{"$unwind":"$article_category"},
{"$match":{"article_category.all_category_id":8}}
])
For uncorrelated subquery try
db.article.aggregate([
{"$match":{"title":{"$regex":/example/}}},
{"$lookup":{
"from":"article_category",
"pipeline":[{"$match":{"all_category_id":8}}],
"as":"categories"
}},
{"$match":{"categories":{"$ne":[]}}}
])
Related
I want to make a search function with mongoose, and I have to be able to make a research with multiple fields (with Mongoose, in NodeJS).
So, I do something like this :
const result = await myModel.find({
$or [{condition1: "value1"}, {condition2: "value2"}, etc...]
});
But, I want to sort the result by the number of condition the object returned have. Like :
If I have 2 conditions, I want to display first the objects respecting the 2 conditions, then the objects respecting the 1st condition, and finally the objects respecting the 2nd condition.
Do you guys know how I can do this? :)
Thanks in advance !
================EDIT================
This is the new search function :
/**
* Search function which returns users matching jobs and skills.
*
* #param {Array[String]} jobs
* #param {Array[String]} skills
* #return {Array[UserModel]} users
*/
async search(jobs, skills) {
// Normalized arrays of jobs, skills and fields (to use it in a mongoose request).
const jobSkills = [];
const associatedSkills = [];
const fields = [];
for (const job of jobs) {
jobSkills.push({
$cond: [
{
$eq: ["$jobSkills", job],
},
2,
0,
],
});
fields.push({
jobSkills: job,
});
}
for (const skill of skills) {
associatedSkills.push({
$cond: [
{
$eq: ["$associatedSkills", skill],
},
1,
0,
],
});
fields.push({
associatedSkills: skill,
});
}
// Request to find users matching jobs and skills.
const users = await UserModel.aggregate([
{
$match: {
$or: fields,
},
},
{
$addFields: {
sortField: {
$sum: jobSkills.concat(associatedSkills),
},
},
},
{
$sort: {
sortField: -1,
},
},
]);
return users;
}
Aggregation Log :
Aggregate {
_pipeline: [
{ '$match': [Object] },
{ '$addFields': [Object] },
{ '$sort': [Object] }
],
_model: Model { User },
options: {}
}
In general, a document either matches a query predicate or it doesn't. There isn't really a concept of one document matching "better" than another. So it looks like you'll want to generate a custom value in a new field and sort on that. This will need to be done via an aggregation.
So after the $match, we'll want an $addFields stage that effectively duplicates the query predicates. For each one it will be wrapped in a conditional statement ($cond) where we add 1 for a match or 0 otherwise, e.g.:
{
$cond: [
{
$eq: [
"$condition1",
"value1"
]
},
1,
0
]
}
Then there will be a $sum pulling them together to generate the final score to sort on.
Taken together, the aggregation will look something like this:
db.collection.aggregate([
{
$match: {
$or: [
{
condition1: "value1"
},
{
condition2: "value2"
}
]
}
},
{
$addFields: {
sortField: {
"$sum": [
{
$cond: [
{
$eq: [
"$condition1",
"value1"
]
},
1,
0
]
},
{
$cond: [
{
$eq: [
"$condition2",
"value2"
]
},
1,
0
]
}
]
}
}
},
{
$sort: {
"sortField": -1
}
}
])
Playground demonstration here
I'm working with a MongoDB collection that has a lot of duplicate keys. I regularly do aggregation queries to find out what those duplicates are, so that I can dig in and find out what is and isn't different about them.
Unfortunately the database is huge and duplicates are often intentional. What I'd like to do is to find the count of keys that have duplicates, instead of printing a result with thousands of lines of output. Is this possible?
(Side Note: I do all of my querying through the shell, so solutions that don't require external tools or a lot of code would be preferred, but I understand that's not always possible.)
Example Records:
[
ObjectId("622f2d94ecf6a5076c2e230b"),
ObjectId("622f329c6f10fe0490252611"),
ObjectId("623026366f10fe0490254341"),
ObjectId("623026de6f10fe0490254583"),
ObjectId("6234346adec0b842dcceb790"),
ObjectId("623434a86f10fe0490260db6"),
ObjectId("62382f91dab1e245d4e152f4"),
ObjectId("6238303b6f10fe0490265acf"),
ObjectId("623bf2af700224301c756394"),
ObjectId("623bf2f76f10fe04902729a4"),
ObjectId("623c5a1f282a052c3c0bbdfd"),
ObjectId("624bf013383df47699e6b141")
]
Here is the query that I've been using to find duplicates based on key:
db.getCollection('weldtestings').aggregate([
{
$match: {
weldId: {
$in: [
ObjectId("622f2d94ecf6a5076c2e230b"),
ObjectId("622f329c6f10fe0490252611"),
ObjectId("623026366f10fe0490254341"),
ObjectId("623026de6f10fe0490254583"),
ObjectId("6234346adec0b842dcceb790"),
ObjectId("623434a86f10fe0490260db6"),
ObjectId("62382f91dab1e245d4e152f4"),
ObjectId("6238303b6f10fe0490265acf"),
ObjectId("623bf2af700224301c756394"),
ObjectId("623bf2f76f10fe04902729a4"),
ObjectId("623c5a1f282a052c3c0bbdfd"),
ObjectId("624bf013383df47699e6b141")]
}
}
},
{
$facet: {
"NDEfailedDate": [
{
$match: { testResult: 'Failed' }
},
{
$group: {
_id: { $dateToString: { format: "%Y-%m-%d", date: "$testDate" } },
count: { $sum : 1 }
}
},
{ $sort: { _id: 1 } }
],
"NDEfailedCount": [
{
$match: { testResult: 'Failed' }
},
{
$group: {
_id: "$weldId",
data: { "$addToSet": "$testDate" }
}
},
{ $count: "totalCount" }
],
}
}
])
Which gives me an output of:
{
"NDEfailedDate" : [
{
"_id" : "2022-04-08",
"count" : 6.0
}
],
"NDEfailedCount" : [
{
"totalCount" : 5
}
]
}
The result I want to get instead:
"_id" : "2022-04-08",
"count" : 5
db.collection.aggregate([
{
$project: {
_id: {
$first: "$NDEfailedDate._id"
},
count: {
$first: "$NDEfailedCount.totalCount"
}
}
}
])
mongoplayground
I'm writing a query that gets data from "coll2" based on data that is inside "coll1".
Coll1 has the following data structure:
{
"_id": "asdf",
"name": "John",
"bags": [
{
"type": "typ1",
"size": "siz1"
},
{
"type": "typ2",
"size": "siz2"
}
]
}
Coll2 has the following data structure:
{
_id: "qwer",
coll1Name: "John",
types: ["typ1", "typ3"],
sizes: ["siz1", "siz4"]
}
{
_id: "zxcv",
coll1Name: "John",
types: ["typ2", "typ3"],
sizes: ["siz1", "siz2"]
}
{
_id: "fghj",
coll1Name: "John",
types: ["typ2", "typ3"],
sizes: ["siz1", "siz4"]
}
I want to get all the documents in coll2 that have the same Type+Size combo as in coll1 using the $lookup stage of the aggregation pipeline. I understand that this can be achieved by using the $lookup pipeline and $expr but I cant seem to figure out how to dynamically make a query to pass into the $match stage.
The output I would like to get for the above data would be:
{
_id: "qwer",
coll1Name: "John",
types: ["typ1", "typ3"],
sizes: ["siz1", "siz4"]
}
{
_id: "zxcv",
coll1Name: "John",
types: ["typ2", "typ3"],
sizes: ["siz1", "siz2"]
}
You can use $lookup to get the data from Col2. Then you need to check if there's any element in Col2 ($anyElemenTrue) that matches with Col1. $map and $in can be used here. Then you just need to $unwind and promote Col2 to root level using $replaceRoot
db.Col1.aggregate([
{
$lookup: {
from: "Col2",
localField: "name",
foreignField: "coll1Name",
as: "Col2"
}
},
{
$project: {
Col2: {
$filter: {
input: "$Col2",
as: "c2",
cond: {
$anyElementTrue: {
$map: {
input: "$bags",
as: "b",
in: {
$and: [
{ $in: [ "$$b.type", "$$c2.types" ] },
{ $in: [ "$$b.size", "$$c2.sizes" ] },
]
}
}
}
}
}
}
}
},
{
$unwind: "$Col2"
},
{
$replaceRoot: {
newRoot: "$Col2"
}
}
])
You are correct in your approach to use $lookup with the pipeline field to filter the input documents in the $match pipeline
The $expr expression should typically follow
"$expr": {
"$and": [
{ "$eq": [ "$name", "$$coll1_name" ] },
{ "$setEquals": [ "$bags.type", "$$types" ] },
{ "$setEquals": [ "$bags.size", "$$sizes" ] }
]
}
where the first match expression in the $and conditional { "$eq": [ "$name", "$$coll1_name" ] } checks to see if the name field in coll1 collection matches the coll1Name field in the input documents from coll2.
Of course the fields from coll2 should be defined in a variable in the pipeline with the let field for the $lookup pipeline to access them.
The other match filters are basically checking if the arrays are equal where "$bags.type" from coll1 resolves to an array of types i.e. [ "typ1", "typ3" ] for example.
On getting the output field from $lookup which happens to be an array, you can filter the documents in coll2 on that array field where there can be some empty lists as a resul of the above $lookup pipeline $match filter:
{ "$match": { "coll1Data.0": { "$exists": true } } }
Overall your aggregate pipeline operation would be as follows:
db.getCollection('coll2').aggregate([
{ "$lookup" : {
"from": "coll1",
"let": { "coll1_name": "$coll1Name", "types": "$types", "sizes": "$sizes" },
"pipeline": [
{ "$match": {
"$expr": {
"$and": [
{ "$eq": [ "$name", "$$coll1_name" ] },
{ "$setEquals": [ "$bags.type", "$$types" ] },
{ "$setEquals": [ "$bags.size", "$$sizes" ] }
]
}
} }
],
"as": "coll1Data"
} },
{ "$match": { "coll1Data.0": { "$exists": true } } },
{ "$project": { "coll1Data": 0 } }
])
After many many tries, I can't have a nice conditional aggregation of my collections.
I use two collections :
races which have a collection of reviews.
I need to obtain for my second pipeline only the reviews published.
I don't want to use a $project.
Is it possible to use only the $match ?
When I use localField, foreignField, it works perfect, but I need to filter only the published reviews.
I struggled so much on this, I don't understand why the let don't give me the foreignKey.
I tried : _id, $reviews, etc..
My $lookup looks like this :
{
$lookup: {
from: "reviews",
as: "reviews",
let: { reviewsId: "$_id" },
pipeline: [
{
$match: {
$expr: {
$and: [
// If I comment the next line, it give all the reviews to all the races
{ $eq: ["$_id", "$$reviewsId"] },
{ $eq: ["$is_published", true] }
]
}
}
}
]
// localField: "reviews",
// foreignField: "_id"
}
},
Example of a race :
{
"description":"Nice race",
"attendees":[
],
"reviews":[
{
"$oid":"5c363ddcfdab6f1d822d7761"
},
{
"$oid":"5cbc835926fa61bd4349a02a"
}
],
...
}
Example of a review :
{
"_id":"5c3630ac5d00d1dc26273dab",
"user_id":"5be89576a38d2b260bfc1bfe",
"user_pseudo":"gracias",
"is_published":true,
"likes":[],
"title":"Best race",
"__v":10,
...
}
I will become crazy soon :'(...
How to accomplish that ?
Your problem is this line:
{ $eq: ["$is_published", true] }
You are using this document _id field to match the reviews one.
The correct version looks like this:
(
[
{
"$unwind" : "$reviews"
},
{
"$lookup" : {
"from" : "reviews",
"as" : "reviews",
"let" : {
"reviewsId" : "$reviews"
},
"pipeline" : [
{
"$match" : {
"$expr" : {
"$and" : [
{
"$eq" : [
"$_id",
"$$reviewsId"
]
},
{ $eq: ["$is_published", true] }
]
}
}
}
]
}
}
],
);
and now if your want to restore the old structure add:
{
$group: {
_id: "$_id",
reviews: {$push: "$reviews"},
}
}
First you have to take correct field to get the data from the referenced collection i.e. reviews. And second you need to use $in aggregation operator as your reviews field is an array of ObjectIds.
db.getCollection('races').aggregate([
{ "$lookup": {
"from": "reviews",
"let": { "reviews": "$reviews" },
"pipeline": [
{ "$match": {
"$expr": { "$in": [ "$_id", "$$reviews" ] },
"is_published": true
}}
],
"as": "reviews"
}}
])
I have a saved a collection in my database and I want to filter it using companyId and cameras using ObjectId specific.
In the follow is the collection that a want to get.
{
"_id": ObjectID("5c3b584fa7e1b10155e6325f"),
"companyId": "5c3b5468a7e1b10155e9995b",
"name": "Place Test",
"cameras": {
"0": ObjectID("5c9149e3f054d00028cc9604"),
"1": ObjectID("5c9149e3f054d00028cc9605")
}
}
I'm trying to filter like:
const placeCollection = req.app.locals.db.collection('places')
const place = placeCollection.findOne({
companyId: req.body.companyId,
cameras: { $elemMatch: { $eq: new ObjectId(req.body.cameraId) } }
})
but not working with cameras filter, only with companyId.
Since the keys in cameras are dynamically generated you need $objectToArray operator to check if any value is equal to req.body.cameraId. You can take advantage of $anyElementTrue operator here:
db.col.aggregate([
{
$match: {
$expr: {
$and: [
{
$anyElementTrue: {
$map: {
input: { $objectToArray: "$cameras" },
in: { $eq: [ "$$this.v", new ObjectId(req.body.cameraId) ] }
}
}
},
{ $eq: [ "$companyId", req.body.companyId ] }
]
}
}
}
])
Mongo playground