I have the following documents
loanRequest (Writing just the keys that I want to project)
{
"_id": "5f2bf26783f65d33026ea592",
"lendingpartner": {
/* some keys here */
},
"loans": [
{
"loanid": 43809,
"loanamount": 761256,
"jewels": [
"5f2bf26783f65d33026ea593",
"5f2bf26783f65d33026ea594"
"5f2bf26783f65d33026ea595"
],
}
]
}
pledgedJewel
{
"_id": "5f2bf26783f65d33026ea593",
"netweight": 8.52,
"purity": 19,
}
What I want to achieve is
{
"_id": "5f2bf2b583f65d33026ea603",
"lendingpartner": {
/* some keys here */
},
"loans": [
{
"loanid": 40010,
"loanamount": 100000,
"jewels": [
{
"_id": "5f2bf26783f65d33026ea593",
"netweight": 8.52,
"purity": 19,
},
{
"_id": "5f2bf26783f65d33026ea594",
"netweight": 5.2,
"purity": 40,
},
{
"_id": "5f2bf26783f65d33026ea595",
"netweight": 4.52,
"purity": 39,
}
]
}
]
}
Since I want the jewel details to be populated inside the jewels array of each loan, $unwind would not help me. (I tried experimenting with it)
I thought I could run a $map on loans array and then run $lookup for each jewel of the loan(double map?), but could not come up with a workable solution.
That didn't seem to be the right approach anyway.
This is the best I could come up with (Far from my desired result). I'm using map to selectively pick keys from loans object.
const loanrequests = await db.collection('loanrequest').aggregate([
{ $match: { requester: ObjectID(user.id) } },
{
$project: {
lendingpartner: {
name: 1,
branchname: '$branch.branchname',
},
loans: {
$map: {
input: '$loans',
as: 'loan',
in: {
loanid: '$$loan.loanid',
loanamount: '$$loan.amount',
jewels: '$$loan.jewels',
},
},
},
},
},
/*
* I experimented with unwind here. Tried adding
* { $unwind: '$loans' },
* { $unwind: '$loans.jewels' }
* but it does not give me the result I need (as already said before)
*/
]).toArray();
I figure, I need to do the $lookup before the projection, but I'm it hard to write a workable solution due to 2 level nested structure of the document (First, the loans array and then loans.jewels)
I started working with mongodb aggregators today and while looking for answers, I stumbled upon a similar Question but it seemed more complex and hence harder for me to understand.
Thanks!
If there are not other things you are trying to achieve with aggregate you can use .populate in mongoose.
LoanReqests
.find(
{requester: user.id},
{name: 1, branch: 1, loans: 1} // Projection
)
.populate('loans.jewels');
If you have to use aggregate to do something not in your example, then $unwind is really your best bet, but then $grouping after the $lookup to get the output you desire. If this doesn't work for you, can you expand on what the issue with $unwind is? I am guessing it is to do with fields not listed in your question.
https://mongoplayground.net/p/O5pxWNy99J4
db.loanRequests.aggregate([
{
$project: {
name: 1,
loans: 1,
branch: "$branch.name"
}
},
{
$unwind: "$loans"
},
{
$lookup: {
localField: "loans.jewels",
foreignField: "_id",
from: "jewels",
as: "loans.jewels"
}
},
{
$group: {
_id: "$_id",
name: {
$first: "$name"
},
branch: {
$first: "$branch"
},
loans: {
$push: "$loans"
}
}
}
])
As mentioned by #GitGitBoom in the previous answer, $unwind followed by $group should have been the approach.
Ofcourse, prior to grouping (I think of it as "unspreading" the outcome of running unwind), I needed to run $lookup in order to populate loans.jewels
Here is the entire solution build on top of the previous answer.
const loanRequests = await db.collection('loanRequest').aggregate([
{ $match: { requester: ObjectID(user.id) } },
{
$project: {
lender: '$lendingpartner.name',
branch: '$lendingpartner.branch.branchname',
loans: 1,
},
},
{ $unwind: '$loans' },
{
$lookup: {
localField: 'loans.jewels',
from: 'pledgedJewel',
foreignField: '_id',
as: 'loans.jewels',
},
},
{
$group: {
_id: '$_id',
branch: { $first: '$branch' },
lender: { $first: '$lender' },
loans: { $push: '$loans' },
},
},
{
$project: {
_id: 1,
branch: 1,
lender: 1,
loans: 1,
},
},
]).toArray();
Issue with mismatch of types
Another issue was, my $lookup was not working due to mismatch of types. In loanRequest collection, on which I'm running the aggregate, the ids inside loans.jewels are of type string whereas the foreign field _id in pledgedJewel is an ObjectId
This can be solved by using $toObjectId or $toString (only supported in mongodb version >= 4.0)
{ $project: { jewelObjId: { $toObjectId: '$loans.jewels' } } }, // for mongodb >= 4.0
{
$lookup: {
localField: 'jewelObjId', // for mongodb >= 4.0
from: 'pledgedjewel',
foreignField: '_id',
as: 'loans.jewels',
},
},
But, I was running on a lower version of mongodb, hence these aggregations were not working for me. Only solution to this was to change the type of loans.jewels to ObjectId instead of keeping it as string which I did.
More on type mismatch
Need a workaround for lookup of a string to objectID foreignField
Mongodb Join on _id field from String to ObjectId
I'm trying to use $lookup aggregation in order to know how many documents in the other collection, so no need to to grab the whole documents, which's, in this case, will give me an exception because the size of the total document exceeded the 16MB limit.
the exception I got
MongoError: Total size of documents in visits matching pipeline's $lookup stage exceeds 104857600 bytes
the query
Link.aggregate(
[
{$match: {...query}},
{
$lookup: {
from:"visits",
localField: "_id",
foreignField: "linkId",
as: "visits"
}
},
{
$addFields: { totalVisits: { $size: "$visits" } },
},
{
$sort: {totalVisits: -1}
}
]
)
You can return $count from custom $lookup pipeline:
{
$lookup: {
from: "visits"
let: { id: "$_id" },
pipeline: [
{ $match: { $expr: { $eq: [ "$$id", "$linkId" ] } } },
{ $count: "total" }
],
as: "totalVisits"
}
}
Let say I have a schema of blog post which contain many keys and one of them is author (ObjectId). Now I have an another collection of Block users which contains two keys: userid (ObjectId) and userWhoHasBeenBlocked (ObjectId).
Now in aggregation I want to skip those collection which has a author equals to the userWhoHasBeenBlocked.
My Approch: First level I have a match query which chcecks the country from which the post has been made. Let say there is a key of a country.
After this I have a $lookup query for block user collection as
{ $match: { country: "usa" } },
{
$lookup:
{
from: "ublocks",
let: { whoHasBeenBlocked: "$author", currentUser: userid },
pipeline: [
{
$match:
{
$expr:
{
$and:
[
{ $eq: ["$blockedUser", "$$whoHasBeenBlocked"] },
]
}
}
},
],
as: "isBlocked"
},
}
},
{ $match: { "$author": { $ne: "$isBlocked.userId" } } }
}
after this I have $projection block. this is not working. How to skip a document within aggregation. I also have pagination after this.
I am aggregating a large data where i need to group the data according to their types and also i need to lookup the data from another collections.inside $group i want my lookup's data.
my code for aggregation goes like :
NotificationSchema.aggregate([{
$match: condition
}, {
$group: {
_id: "$type",
details: {
$push: "$$ROOT"
},
count: {
$sum: 1
}
}
}, {
$sort: {
_id: -1
}
}, {
$lookup: {
from: "vehicles",
localField: "details.device_id",
foreignField: "device_id",
as: "vehicle"
}
}], function(err, result) {
if (err) {
res.status(500);
return res.json({
result: err
});
}
console.log('res', result[0].details[0]);
res.json({
result: result
});
});
if i remove or comment the $group code i get the data with Vehicle array but using $group i get vehicle array empty, as i have only two types in records in the database, i get two empty array of vehicles. but i have 102 records so i need 102 arrays of vehicles how can i get such result.
what i am getting in console right now is
res [ { _id: 'Vehicle Delay Alert!',
details:
[ [Object],
....57 object...
[Object] ],
count: 57,
vehicle: [] },
and inside every object i dont find vehicle array so i wish to remove vehicle array from here and get a vehicle array that is generated from $lookup inside every object.
Any suggestions are highly appreciated.
You $lookup from details.device_id which comes from an array. To $lookup from a regular field, you can place $lookup after the $match :
NotificationSchema.aggregate([{
$match: condition
}, {
$lookup: {
from: "vehicles",
localField: "device_id",
foreignField: "device_id",
as: "vehicle"
}
}, {
$group: {
_id: "$type",
details: {
$push: "$$ROOT"
},
count: {
$sum: 1
}
}
}, {
$sort: {
_id: -1
}
}])
I have a collection with name post and I have one doc and its replicated doc but in replicated doc we have one field different ,some doc don't have replicated doc and that situation depends on an array field of the doc ,if that field have the userId of user then the replicated doc will exist otherwise it will not exist.
So what I want if the doc array have that id then get the replicated post and if not then original post
I have made the query but showing error I am using $exist in $cond ?
Post.aggregate([
{
$match: {
socomo_visibility: socomoId
}
},
{
$project: {
"post_stream_type": {
$cond: {
if: {
'following_users_list': {
$exist: [userId]
}
},
then: constants.POST_STREAM_TYPE.FOLLOW.value,
else: constants.POST_STREAM_TYPE.SOCIAL_CURRY_CHANNEL.value
}
}
}
}
]
You can check whether your array has some value in the boolean-expression in such way:
Do intersection of array and value using $setIntersection.
Check size of that intersection array using $size.
If the size is greater than 0 then value is present in the array. $gt will do this check.
Try the following code:
Post.aggregate([
{
$project: {
"post_stream_type": {
$cond: {
if: {$gt: [{$size: {$setIntersection: ["$following_users_list", [userId]] } }, 0] },
then: constants.POST_STREAM_TYPE.FOLLOW.value,
else: constants.POST_STREAM_TYPE.SOCIAL_CURRY_CHANNEL.value
}
}
}
}
])
okay finally I have done this without using the aggregation .
my answer for the query is
Post.find({
$or: [{
socomo_visibility: {
$elemMatch: {
$eq: socomoId
}
},
post_stream_type: constants.POST_STREAM_TYPE.SOCIAL_CURRY_CHANNEL.value,
following_users_list: {
$exists: true,
$nin: [userId]
}
},
{
socomo_visibility: {
$elemMatch: {
$eq: socomoId
}
},
post_stream_type: constants.POST_STREAM_TYPE.FOLLOW.value,
following_users_list: {
$elemMatch: {
$eq: userId
}
}
}]
})