I have the following documents
loanRequest (Writing just the keys that I want to project)
{
"_id": "5f2bf26783f65d33026ea592",
"lendingpartner": {
/* some keys here */
},
"loans": [
{
"loanid": 43809,
"loanamount": 761256,
"jewels": [
"5f2bf26783f65d33026ea593",
"5f2bf26783f65d33026ea594"
"5f2bf26783f65d33026ea595"
],
}
]
}
pledgedJewel
{
"_id": "5f2bf26783f65d33026ea593",
"netweight": 8.52,
"purity": 19,
}
What I want to achieve is
{
"_id": "5f2bf2b583f65d33026ea603",
"lendingpartner": {
/* some keys here */
},
"loans": [
{
"loanid": 40010,
"loanamount": 100000,
"jewels": [
{
"_id": "5f2bf26783f65d33026ea593",
"netweight": 8.52,
"purity": 19,
},
{
"_id": "5f2bf26783f65d33026ea594",
"netweight": 5.2,
"purity": 40,
},
{
"_id": "5f2bf26783f65d33026ea595",
"netweight": 4.52,
"purity": 39,
}
]
}
]
}
Since I want the jewel details to be populated inside the jewels array of each loan, $unwind would not help me. (I tried experimenting with it)
I thought I could run a $map on loans array and then run $lookup for each jewel of the loan(double map?), but could not come up with a workable solution.
That didn't seem to be the right approach anyway.
This is the best I could come up with (Far from my desired result). I'm using map to selectively pick keys from loans object.
const loanrequests = await db.collection('loanrequest').aggregate([
{ $match: { requester: ObjectID(user.id) } },
{
$project: {
lendingpartner: {
name: 1,
branchname: '$branch.branchname',
},
loans: {
$map: {
input: '$loans',
as: 'loan',
in: {
loanid: '$$loan.loanid',
loanamount: '$$loan.amount',
jewels: '$$loan.jewels',
},
},
},
},
},
/*
* I experimented with unwind here. Tried adding
* { $unwind: '$loans' },
* { $unwind: '$loans.jewels' }
* but it does not give me the result I need (as already said before)
*/
]).toArray();
I figure, I need to do the $lookup before the projection, but I'm it hard to write a workable solution due to 2 level nested structure of the document (First, the loans array and then loans.jewels)
I started working with mongodb aggregators today and while looking for answers, I stumbled upon a similar Question but it seemed more complex and hence harder for me to understand.
Thanks!
If there are not other things you are trying to achieve with aggregate you can use .populate in mongoose.
LoanReqests
.find(
{requester: user.id},
{name: 1, branch: 1, loans: 1} // Projection
)
.populate('loans.jewels');
If you have to use aggregate to do something not in your example, then $unwind is really your best bet, but then $grouping after the $lookup to get the output you desire. If this doesn't work for you, can you expand on what the issue with $unwind is? I am guessing it is to do with fields not listed in your question.
https://mongoplayground.net/p/O5pxWNy99J4
db.loanRequests.aggregate([
{
$project: {
name: 1,
loans: 1,
branch: "$branch.name"
}
},
{
$unwind: "$loans"
},
{
$lookup: {
localField: "loans.jewels",
foreignField: "_id",
from: "jewels",
as: "loans.jewels"
}
},
{
$group: {
_id: "$_id",
name: {
$first: "$name"
},
branch: {
$first: "$branch"
},
loans: {
$push: "$loans"
}
}
}
])
As mentioned by #GitGitBoom in the previous answer, $unwind followed by $group should have been the approach.
Ofcourse, prior to grouping (I think of it as "unspreading" the outcome of running unwind), I needed to run $lookup in order to populate loans.jewels
Here is the entire solution build on top of the previous answer.
const loanRequests = await db.collection('loanRequest').aggregate([
{ $match: { requester: ObjectID(user.id) } },
{
$project: {
lender: '$lendingpartner.name',
branch: '$lendingpartner.branch.branchname',
loans: 1,
},
},
{ $unwind: '$loans' },
{
$lookup: {
localField: 'loans.jewels',
from: 'pledgedJewel',
foreignField: '_id',
as: 'loans.jewels',
},
},
{
$group: {
_id: '$_id',
branch: { $first: '$branch' },
lender: { $first: '$lender' },
loans: { $push: '$loans' },
},
},
{
$project: {
_id: 1,
branch: 1,
lender: 1,
loans: 1,
},
},
]).toArray();
Issue with mismatch of types
Another issue was, my $lookup was not working due to mismatch of types. In loanRequest collection, on which I'm running the aggregate, the ids inside loans.jewels are of type string whereas the foreign field _id in pledgedJewel is an ObjectId
This can be solved by using $toObjectId or $toString (only supported in mongodb version >= 4.0)
{ $project: { jewelObjId: { $toObjectId: '$loans.jewels' } } }, // for mongodb >= 4.0
{
$lookup: {
localField: 'jewelObjId', // for mongodb >= 4.0
from: 'pledgedjewel',
foreignField: '_id',
as: 'loans.jewels',
},
},
But, I was running on a lower version of mongodb, hence these aggregations were not working for me. Only solution to this was to change the type of loans.jewels to ObjectId instead of keeping it as string which I did.
More on type mismatch
Need a workaround for lookup of a string to objectID foreignField
Mongodb Join on _id field from String to ObjectId
Related
I have a sample mongoose object that looks like this:
{
_id: 5fa849ad4f6be0382363809c,
ratings: {
ratedPersonId: 7,
rating: 7,
timeSpent: 30,
timestamp: 78,
userThreshold: 5
}
},
it contains an _id and a list of ratings which is a subdocument with the following features.
I have created an aggregation pipeline like this:
const ratedUser = await this.ratingModel
.aggregate([
{ $project: { ratings: 1 } },
{ $unwind: '$ratings' },
{
$match: {
$and: [{ 'ratings.ratedPersonId': userId }, { 'ratings.rating': { $gte: 5 } }],
},
},
])
.exec()
This works for the first condition ratings.ratedPersonId: userId
My problem is that my second condition is the rating should be greater than or equal to the userThreshold field in the same object.
whenever I type that in the query it returns nothing
$and: [{ 'ratings.ratedPersonId': userId }, { 'ratings.rating': { $gte: 'ratings.threshold'} }],
Demo - https://mongoplayground.net/p/AQMsJGkoFcu
Use $expr to compare the fields
Read aggregation-expressions
$expr can build query expressions that compare fields from the same document in a $match stage.
If the $match stage is part of a $lookup stage, $expr can compare fields using let variables. See Specify Multiple Join Conditions with $lookup for an example.
$expr only uses indexes on the from the collection for equality matches in a $match stage.
$expr does not support multikey indexes.
db.collection.aggregate([
{
$project: {
ratings: 1
}
},
{
$unwind: "$ratings"
},
{
$match: {
$and: [
{
"ratings.ratedPersonId": 7
},
{
$expr: {
$gte: [
"$ratings.rating",
"$ratings.userThreshold"
]
}
}
],
},
},
])
I have 3 collections: Categories, Exams and Results.
For each Category I have some exams. If one exam is done, I add its result on "Result" collection.
I have take all exams for each category.
The problem is that I can't count how many of them are done.
This is what I get right now:
{ category: "Test", total_exams: 10 }
This is what I want:
{ category: "Test", total_exams: 10, done_exam: 4 }
Below you can see what I have done until now:
https://mongoplayground.net/p/upSAPkYe4DR
Quick fixes:
second $lookup stage, let to pass examsCount._id,
inside pipeline $match stage match expression, $exam_id with $$examId
put user_id match condition outside expression
$count total documents in count
$addFieldsto getdoneExams.countthis will return count in zero index, get it using$firstor$arrayElemAt`
db.categories.aggregate([
{
$lookup: {
from: "exams",
localField: "_id",
foreignField: "categoryId",
as: "examsCount"
}
},
{
$lookup: {
from: "results",
let: { examId: "$examsCount._id" },
pipeline: [
{
$match: {
$expr: {
$and: [{ $in: ["$exam_id", "$$examId"] }]
},
user_id: 22222222
}
},
{ $count: "count" }
],
as: "doneExams"
}
},
{
$addFields: {
examsCount: { $size: "$examsCount" },
doneExams: { $first: "$doneExams.count" }
}
}
])
Playground
There are many approaches to get this result, i have just resolved your issue in your approach.
I have two collections which are of schema like
driver
_id:5f9c1d897ea5e246945cd73a
agent_name:"Ratnabh Kumar Rai"
agent_email:"ra2614#gmail.com"
agent_phone:"70****63331"
and
reviews
_id:5f9d54cb3a3ee10c6829c0a4
order_id:"5f9d40f096e4a506e8684aba"
user_id:"5f9bcb66f7a5bf4be0ad9973"
driver_id:"5f9c1d897ea5e246945cd73a"
rating:3
text:""
so i want to calculate the avg driver rating . I tried to use lookup aggregation so that i can get all details and then later calculate the sum...what i did was
let doc = await db
.collection("drivers")
.aggregate([
{
$project: {
_id: {
$toString: "$_id",
},
},
},
{
$lookup: {
from: "reviews",
localField: "_id",
foreignField: "driver_id",
as: "driver_info",
},
},
{
$project: {
agent_email: 1,
orderReview: "$driver_info",
},
},
])
.toArray();
But i am getting result as
{
_id: '5f9d63eb8737e82fbc193dd9',
orderReview: [ [Object], [Object], [Object], [Object], [Object] ]
}
which is partially correct as i also need to get details from my localfield collection that is drivers field, as of now you can see i am only getting id of driver in my projection i also did "agent_email:1" but not getting email
You're actually only projecting _id in the first pipeline and hence only _id is passed to further pipelines, If you need email in further pipelines you need to project it too
let doc = await db.collection("drivers").aggregate([
{
$project: {
_id: {
$toString: "$_id",
},
agent_email: "$agent_email"
},
},
{
$lookup: {
from: "reviews",
localField: "_id",
foreignField: "driver_id",
as: "driver_info",
},
},
{
$project: {
agent_email: 1,
orderReview: "$driver_info",
},
},
])
MongoDB PlayGround : https://mongoplayground.net/p/h7D-tYJ7sLU
[Update]
I realized that you're doing this for getting average and if you need it to be done in a single aggregate query, here it is how you can do it.
Using unwind operator you can flat the reviews array as objects and then group by _id and use the $avg aggregation operator
db.collection("drivers").aggregate([
{
$project: {
_id: {
$toString: "$_id",
},
agent_email: "$agent_email"
},
},
{
$lookup: {
from: "reviews",
localField: "_id",
foreignField: "driver_id",
as: "driver_info",
},
},
// Makes the driver info flat with other information
{
"$unwind": "$driver_info"
},
{
// Now you can group them
$group: {
_id: "$_id",
// Calculates avg on rating field
avg: {
"$avg": "$driver_info.rating"
},
// To make email field appear in next pipeline.
// You can do it for other fields too like Name, etc
agent_email: {
$first: "$agent_email"
}
}
},
{
$project: {
// select the fields you want to display
agent_email: 1,
avg: 1
},
},
])
MonogoDb playground Link
I am aggregating a large data where i need to group the data according to their types and also i need to lookup the data from another collections.inside $group i want my lookup's data.
my code for aggregation goes like :
NotificationSchema.aggregate([{
$match: condition
}, {
$group: {
_id: "$type",
details: {
$push: "$$ROOT"
},
count: {
$sum: 1
}
}
}, {
$sort: {
_id: -1
}
}, {
$lookup: {
from: "vehicles",
localField: "details.device_id",
foreignField: "device_id",
as: "vehicle"
}
}], function(err, result) {
if (err) {
res.status(500);
return res.json({
result: err
});
}
console.log('res', result[0].details[0]);
res.json({
result: result
});
});
if i remove or comment the $group code i get the data with Vehicle array but using $group i get vehicle array empty, as i have only two types in records in the database, i get two empty array of vehicles. but i have 102 records so i need 102 arrays of vehicles how can i get such result.
what i am getting in console right now is
res [ { _id: 'Vehicle Delay Alert!',
details:
[ [Object],
....57 object...
[Object] ],
count: 57,
vehicle: [] },
and inside every object i dont find vehicle array so i wish to remove vehicle array from here and get a vehicle array that is generated from $lookup inside every object.
Any suggestions are highly appreciated.
You $lookup from details.device_id which comes from an array. To $lookup from a regular field, you can place $lookup after the $match :
NotificationSchema.aggregate([{
$match: condition
}, {
$lookup: {
from: "vehicles",
localField: "device_id",
foreignField: "device_id",
as: "vehicle"
}
}, {
$group: {
_id: "$type",
details: {
$push: "$$ROOT"
},
count: {
$sum: 1
}
}
}, {
$sort: {
_id: -1
}
}])
i have this model:
var Chat = new Schema({
from: String,
to: String,
satopId: String,
createdAt: Date
});
var Chat = mongoose.model('Chat', Chat);
I want do a query to do a query that returns the max created at grouping by to and from field. I tried with:
Chat.aggregate([
{
$group: {
_id: '$to',
from: '$from',
createdAt: {
$max: '$createdAt'
}
}
},
{
$project: {
_id: 1,
createdAt: 1,
from: 1,
to: 1
}
}
], function(err, docs){
})
But this generates this error:
the group aggregate field 'from' must be defined as an expression
inside an object
I don't understand what does it mean. How can i solve it?
Thanks
Anything "outside" if the _id expression in a $group statement requires a "grouping operator" of some sort.
Assuming you are happy with the idea of the "sole" grouping key to be the "to" field in your documents then you probably want something like $first:
Chat.aggregate([
{ "$group": {
"_id": "$to",
"from": { "$first": "$from" },
"createdAt": { "$max": "$createdAt" }
}},
function (err, docs) {
// do something here
}
])
Otherwise if you want "distinct" values on "both" fields then they both belong as a composite value of the grouping key:
Chat.aggregate([
{ "$group": {
"_id": {
"to": "$to",
"from": "$from"
},
"createdAt": { "$max": "$createdAt" }
}},
function (err, docs) {
// do something here
}
])
That's basically how it works. Either it's part of the key or something that needs a grouping operator just like with SQL.
Also note that your $project assertion is not really required as $group already does what you are asking there.