After many many tries, I can't have a nice conditional aggregation of my collections.
I use two collections :
races which have a collection of reviews.
I need to obtain for my second pipeline only the reviews published.
I don't want to use a $project.
Is it possible to use only the $match ?
When I use localField, foreignField, it works perfect, but I need to filter only the published reviews.
I struggled so much on this, I don't understand why the let don't give me the foreignKey.
I tried : _id, $reviews, etc..
My $lookup looks like this :
{
$lookup: {
from: "reviews",
as: "reviews",
let: { reviewsId: "$_id" },
pipeline: [
{
$match: {
$expr: {
$and: [
// If I comment the next line, it give all the reviews to all the races
{ $eq: ["$_id", "$$reviewsId"] },
{ $eq: ["$is_published", true] }
]
}
}
}
]
// localField: "reviews",
// foreignField: "_id"
}
},
Example of a race :
{
"description":"Nice race",
"attendees":[
],
"reviews":[
{
"$oid":"5c363ddcfdab6f1d822d7761"
},
{
"$oid":"5cbc835926fa61bd4349a02a"
}
],
...
}
Example of a review :
{
"_id":"5c3630ac5d00d1dc26273dab",
"user_id":"5be89576a38d2b260bfc1bfe",
"user_pseudo":"gracias",
"is_published":true,
"likes":[],
"title":"Best race",
"__v":10,
...
}
I will become crazy soon :'(...
How to accomplish that ?
Your problem is this line:
{ $eq: ["$is_published", true] }
You are using this document _id field to match the reviews one.
The correct version looks like this:
(
[
{
"$unwind" : "$reviews"
},
{
"$lookup" : {
"from" : "reviews",
"as" : "reviews",
"let" : {
"reviewsId" : "$reviews"
},
"pipeline" : [
{
"$match" : {
"$expr" : {
"$and" : [
{
"$eq" : [
"$_id",
"$$reviewsId"
]
},
{ $eq: ["$is_published", true] }
]
}
}
}
]
}
}
],
);
and now if your want to restore the old structure add:
{
$group: {
_id: "$_id",
reviews: {$push: "$reviews"},
}
}
First you have to take correct field to get the data from the referenced collection i.e. reviews. And second you need to use $in aggregation operator as your reviews field is an array of ObjectIds.
db.getCollection('races').aggregate([
{ "$lookup": {
"from": "reviews",
"let": { "reviews": "$reviews" },
"pipeline": [
{ "$match": {
"$expr": { "$in": [ "$_id", "$$reviews" ] },
"is_published": true
}}
],
"as": "reviews"
}}
])
Related
{
$lookup: {
from: "Comment",
let: {
p_id: "$_id",
d_id: "$data_id",
},
pipeline: [
{
$match: {
$expr: {
$and: [
{
$eq: [
"$_id",
"$$p_id"
]
},
{
$eq: [
"$data_id",
"$$d_id"
]
}
]
}
}
}
],
as: "subComment"
}
}
https://mongoplayground.net/p/GbEgnVn3JSv
I am good at mongoplayground but tried to put there my thought
I want to fetch the comment of posts based on doc_id and post_id for mainComment query looks good to me but subcommand is not good. Please guide on this
Its simple as a post can have multiple comment need comment count base on Post.data._id which is equal to Comment.doc_id and Post._id is in Comment.post_id
Not sure what "mainComment" and "subComment" are, I believe you missed the dollar sign before them
{
$project: {
_id: 1,
main_comments_count: {
$size: "$mainComment"
},
sub_comments_count: {
$size: "$subComment"
},
}
}
Update
What you did wrong in the playground is that you used $data in the lookup.let stage. $data is a document and the field you actually want to lookup is $data._id.
sidenote: if you are looking up using just one field, you can simply use the localField and foreign in the lookup stage. Using let and pipeline is not necessary there.
db.setting.aggregate([
{
$lookup: {
from: "site",
"let": {
"pid": "$data._id" //here
},
"pipeline": [
{
"$match": {
"$expr": {
"$in": [
"$doc_id",
"$$pid"
]
}
}
}
],
"as": "subComment"
}
},
{
$addFields: {
countRecord: "$subComment"
}
}
])
i.e. this gives the same output
db.setting.aggregate([
{
$lookup: {
from: "site",
localField: "data._id",
foreignField: "doc_id",
as: "subComment"
}
},
{
$addFields: {
countRecord: "$subComment"
}
}
])
I have 2 collections. The documents looks like as follows. I have removed other properties for easy understanding:
Collection_A
{
"ref_id" : ObjectId("5e9561edf8beb57100dded8f"),
"features" : [
{
"_id" : ObjectId("5e9561edf8beb57100dded91"),
"k" : "foo",
"v" : "bar"
},
{
"_id" : ObjectId("5e9561edf8beb57100dded92"),
"k" : "foo2",
"v" : "bar2"
}
]
}
Collection_B
{
"ref_id" : ObjectId("5e9561edf8beb57100dded8f")
}
Using aggregate I am trying to find all documents in Collection_B where Collection_B.ref_id == Collection_A.ref_id and Collection_A.features == [{k:foo,v:bar},{k:foo2,v:bar2}]
Basically match supplied features array with $Collection_A.features. Aggregate should return document when all supplied features is present in $Collection_A.features.
After trying, this is the closest I have:
let aggregation_queries = [];
aggregation_queries.push({
$lookup: {
from: "Collection_A",
localField: "ref_id",
foreignField: "ref_id",
as: "Collection_A"
}
});
for(let i = 0; i< features.length; i++)
{
aggregation_queries.push({$match: { $expr: { $in : [features[i].k, "$Collection_A.features.k" ]}}});
}
let aggregateResult = Collection_BSchema.aggregate(aggregation_queries);
This only matches features.k but not features.v. I am trying to find a way to match both fetaures.k and features.v, something like $and: [{features[i].k, "$Collection_A.features.k"}, {features[i].v, "$Collection_A.features.v"}]
I have searched and tried a lot of approaches like $match with $all but doesn't seem to work because match doesn't support $all
for ex: "$match":{"$expr":{"$all":["$Collection_A.features",features]} which throws an error" Error: Unrecognized expression '$all'MongoError: Unrecognized expression".
Can someone please help with this or provide some guidance?
#whoami. This worked:
db.Collection_B.aggregate([
{
$lookup: {
from: "Collection_A",
let: {
refId: "$ref_id"
},
pipeline: [
{
$match: {
$expr: {
$eq: [
"$ref_id",
"$$refId"
]
}
}
},
{
"$match": {
"features": {
$all: [
{
"$elemMatch": {
"k": "foo",
"v": "bar"
}
},
{
"$elemMatch": {
"k": "foo2",
"v": "bar2"
}
}
]
}
}
}
],
as: "Collection_A"
}
}
])
There are couple of changes in your query, Since you're aggregating on Collection_B & wants to join Collection_B & Collection_A then in $lookup change this from: "Collection_B" to from: "Collection_A". Now you can use aggregation pipeline in $lookup which can accept multiple conditions before getting matched document from Collection_A to lookup result field Collection_A of Collection_B document :
Collection_BSchema.aggregate([
{
$lookup: {
from: "Collection_A",
let: { refId: "$ref_id" },
pipeline: [
{ $match: { $expr: { $eq: ["$ref_id", "$$refId"] } } },
{ $match: { features: { $elemMatch: { k: "foo", v: "bar" } } } },
],
as: "Collection_A",
},
},
]);
Test : MongoDB-Playground
Trying to figure out something from Mongo using mongoose in optimal way.
I have following documents
Regions
{
"_id" : ObjectId("5cf21263ff605c49cd6d8016"),
"name" : "Asia"
}
Countries can be part of multiple regions
{
"_id" : ObjectId("5d10a4ad80a93a1d7cd56cc6"),
"regions" : [
ObjectId("5d10a50080a93a1d7cd56cc7"),
ObjectId("5cf2126bff605c49cd6d8017")
],
"name" : "India"
}
Places belongs to one country
{
"_id" : ObjectId("5d11bb8180a93a1d7cd56d26"),
"name" : "Delhi",
"country" : ObjectId("5d136e7a4e480863a51c4056"),
}
Programs each in dayshows array represents one day. On a day show can cover multiple places.
{
"_id" : ObjectId("5d11cc9480a93a1d7cd56d31"),
"dayshows" : [
{
"_id" : ObjectId("5d11cc9480a93a1d7cd56d41"),
"places" : [
ObjectId("5d11bb8180a93a1d7cd56d26")
],
},
{
"_id" : ObjectId("5d11cc9480a93a1d7cd56d3c"),
"places" : [
ObjectId("5d11bb8180a93a1d7cd56d26"),
ObjectId("5d11bc7c80a93a1d7cd56d2e")
]
}
]
}
What am I trying to figure out?
For a given region, for each country in region which all places are covered and count of programs for each place. Using nodejs and mongoose.
Example
Input - Asia
Output
India
- Delhi (3)
- Mumbai (5)
Thailand
- Pattaya (2)
- Bangkok (5)
New to mongo.
You need to use $lookup to cross different collections.
Pipeline:
Stages 1-6 serves to get all related data.
(Optional) Stages 7-10 serves to transform aggregated data into key:pair object.
ASSUMPTION
Programs to visit 2 places counted as is (Place1: +1, Place2: +1)
You know how to execute MongoDB aggregation in node.js
db.Regions.aggregate([
{
$match: {
name: "Asia"
}
},
{
$lookup: {
from: "Countries",
let: {
region: "$_id"
},
pipeline: [
{
$match: {
$expr: {
$in: [
"$$region",
"$regions"
]
}
}
},
{
$lookup: {
from: "Places",
localField: "_id",
foreignField: "country",
as: "Places"
}
}
],
as: "Countries"
}
},
{
$unwind: "$Countries"
},
{
$unwind: "$Countries.Places"
},
{
$lookup: {
from: "Programs",
localField: "Countries.Places._id",
foreignField: "dayshows.places",
as: "Countries.Places.Programs"
}
},
{
$project: {
"name": 1,
"Countries.name": 1,
"Countries.Places.name": 1,
"Countries.Places.Programs": {
$size: "$Countries.Places.Programs"
}
}
},
{
$group: {
_id: {
name: "$name",
Countries: "$Countries.name"
},
Places: {
$push: {
k: "$Countries.Places.name",
v: "$Countries.Places.Programs"
}
}
}
},
{
$project: {
_id: 1,
Places: {
$arrayToObject: "$Places"
}
}
},
{
$group: {
_id: "$_id.name",
Countries: {
$push: {
k: "$_id.Countries",
v: "$Places"
}
}
}
},
{
$project: {
_id: 0,
name: "$_id",
Countries: {
$arrayToObject: "$Countries"
}
}
}
])
MongoPlayground
I'm writing a query that gets data from "coll2" based on data that is inside "coll1".
Coll1 has the following data structure:
{
"_id": "asdf",
"name": "John",
"bags": [
{
"type": "typ1",
"size": "siz1"
},
{
"type": "typ2",
"size": "siz2"
}
]
}
Coll2 has the following data structure:
{
_id: "qwer",
coll1Name: "John",
types: ["typ1", "typ3"],
sizes: ["siz1", "siz4"]
}
{
_id: "zxcv",
coll1Name: "John",
types: ["typ2", "typ3"],
sizes: ["siz1", "siz2"]
}
{
_id: "fghj",
coll1Name: "John",
types: ["typ2", "typ3"],
sizes: ["siz1", "siz4"]
}
I want to get all the documents in coll2 that have the same Type+Size combo as in coll1 using the $lookup stage of the aggregation pipeline. I understand that this can be achieved by using the $lookup pipeline and $expr but I cant seem to figure out how to dynamically make a query to pass into the $match stage.
The output I would like to get for the above data would be:
{
_id: "qwer",
coll1Name: "John",
types: ["typ1", "typ3"],
sizes: ["siz1", "siz4"]
}
{
_id: "zxcv",
coll1Name: "John",
types: ["typ2", "typ3"],
sizes: ["siz1", "siz2"]
}
You can use $lookup to get the data from Col2. Then you need to check if there's any element in Col2 ($anyElemenTrue) that matches with Col1. $map and $in can be used here. Then you just need to $unwind and promote Col2 to root level using $replaceRoot
db.Col1.aggregate([
{
$lookup: {
from: "Col2",
localField: "name",
foreignField: "coll1Name",
as: "Col2"
}
},
{
$project: {
Col2: {
$filter: {
input: "$Col2",
as: "c2",
cond: {
$anyElementTrue: {
$map: {
input: "$bags",
as: "b",
in: {
$and: [
{ $in: [ "$$b.type", "$$c2.types" ] },
{ $in: [ "$$b.size", "$$c2.sizes" ] },
]
}
}
}
}
}
}
}
},
{
$unwind: "$Col2"
},
{
$replaceRoot: {
newRoot: "$Col2"
}
}
])
You are correct in your approach to use $lookup with the pipeline field to filter the input documents in the $match pipeline
The $expr expression should typically follow
"$expr": {
"$and": [
{ "$eq": [ "$name", "$$coll1_name" ] },
{ "$setEquals": [ "$bags.type", "$$types" ] },
{ "$setEquals": [ "$bags.size", "$$sizes" ] }
]
}
where the first match expression in the $and conditional { "$eq": [ "$name", "$$coll1_name" ] } checks to see if the name field in coll1 collection matches the coll1Name field in the input documents from coll2.
Of course the fields from coll2 should be defined in a variable in the pipeline with the let field for the $lookup pipeline to access them.
The other match filters are basically checking if the arrays are equal where "$bags.type" from coll1 resolves to an array of types i.e. [ "typ1", "typ3" ] for example.
On getting the output field from $lookup which happens to be an array, you can filter the documents in coll2 on that array field where there can be some empty lists as a resul of the above $lookup pipeline $match filter:
{ "$match": { "coll1Data.0": { "$exists": true } } }
Overall your aggregate pipeline operation would be as follows:
db.getCollection('coll2').aggregate([
{ "$lookup" : {
"from": "coll1",
"let": { "coll1_name": "$coll1Name", "types": "$types", "sizes": "$sizes" },
"pipeline": [
{ "$match": {
"$expr": {
"$and": [
{ "$eq": [ "$name", "$$coll1_name" ] },
{ "$setEquals": [ "$bags.type", "$$types" ] },
{ "$setEquals": [ "$bags.size", "$$sizes" ] }
]
}
} }
],
"as": "coll1Data"
} },
{ "$match": { "coll1Data.0": { "$exists": true } } },
{ "$project": { "coll1Data": 0 } }
])
I have 3 arrays of ObjectIds I want to concatenate into a single array, and then sort by creation date. $setUnion does precisely what I want, but I'd like to try without using it.
Schema of object I want to sort:
var chirpSchema = new mongoose.Schema({
interactions: {
_liked : ["55035390d3e910505be02ce2"] // [{ type: $oid, ref: "interaction" }]
, _shared : ["507f191e810c19729de860ea", "507f191e810c19729de860ea"] // [{ type: $oid, ref: "interaction" }]
, _viewed : ["507f1f77bcf86cd799439011"] // [{ type: $oid, ref: "interaction" }]
}
});
Desired result: Concatenate _liked, _shared, and _viewed into a single array, and then sort them by creation date using aggregate pipeline. See below
["507f1f77bcf86cd799439011", "507f191e810c19729de860ea", "507f191e810c19729de860ea", "55035390d3e910505be02ce2"]
I know I'm suppose to use $push, $each, $group, and $unwind in some combination or other, but I'm having trouble piecing together the documenation to make this happen.
Update: Query
model_user.aggregate([
{ $match : { '_id' : { $in : following } } }
, { $project : { 'interactions' : 1 } }
, { $project : {
"combined": { $setUnion : [
"$interactions._liked"
, "$interactions._shared"
, "$interactions._viewed"
]}
}}
])
.exec(function (err, data) {
if (err) return next(err);
next(data); // Combined is returning null
})
If all the Object _id values are "unique" then $setUnion is your best option. It is of course not "ordered" in any way as it works with a "set", and that does not guarantee order. But you can always unwind and $sort.
[
{ "$project": {
"combined": { "$setUnion": [
{ "$ifNull": [ "$interactions._liked", [] ] },
{ "$ifNull": [ "$interactions._shared", [] ] },
{ "$ifNull", [ "$interactions._viewed", [] ] }
]}
}},
{ "$unwind": "$combined" },
{ "$sort": { "combined": 1 } },
{ "$group": {
"_id": "$_id",
"combined": { "$push": "$combined" }
}}
]
Of course again since this is a "set" of distinct values you can do the old way instead with $addToSet, after processing $unwind on each array:
[
{ "$unwind": "$interactions._liked" },
{ "$unwind": "$interactions._shared" },
{ "$unwind": "$interactions._viewed" },
{ "$project": {
"interactions": 1,
"type": { "$const": [ "liked", "shared", "viewed" ] }
}}
{ "$unwind": "$type" },
{ "$group": {
"_id": "$_id",
"combined": {
"$addToSet": {
"$cond": [
{ "$eq": [ "$type", "liked" ] },
"$interactions._liked",
{ "$cond": [
{ "$eq": [ "$type", "shared" ] },
"$interactions._shared",
"$interactions._viewed"
]}
]
}
}
}},
{ "$unwind": "$combined" },
{ "$sort": { "combined": 1 } },
{ "$group": {
"_id": "$_id",
"combined": { "$push": "$combined" }
}}
]
But still the same thing applies to ordering.
Future releases even have the ability to concatenate arrays without reducing to a "set":
[
{ "$project": {
"combined": { "$concatArrays": [
"$interactions._liked",
"$interactions._shared",
"$interactions._viewed"
]}
}},
{ "$unwind": "$combined" },
{ "$sort": { "combined": 1 } },
{ "$group": {
"_id": "$_id",
"combined": { "$push": "$combined" }
}}
]
But still there is no way to re-order the results without procesing $unwind and $sort.
You might therefore consider that unless you need this grouped across multiple documents, that the basic "contenate and sort" operation is best handled in client code. MongoDB has no way to do this "in place" on the array at present, so per document in client code is your best bet.
But if you do need to do this grouping over multiple documents, then the sort of approaches as shown here are for you.
Also note that "creation" here means creation of the ObjectId value itself and not other properties from your referenced objects. If you need those, then you perform a populate on the id values after the aggregation or query instead, and of course sort in client code.