Using $lookup and $group to aggregate data - node.js

I am aggregating a large data where i need to group the data according to their types and also i need to lookup the data from another collections.inside $group i want my lookup's data.
my code for aggregation goes like :
NotificationSchema.aggregate([{
$match: condition
}, {
$group: {
_id: "$type",
details: {
$push: "$$ROOT"
},
count: {
$sum: 1
}
}
}, {
$sort: {
_id: -1
}
}, {
$lookup: {
from: "vehicles",
localField: "details.device_id",
foreignField: "device_id",
as: "vehicle"
}
}], function(err, result) {
if (err) {
res.status(500);
return res.json({
result: err
});
}
console.log('res', result[0].details[0]);
res.json({
result: result
});
});
if i remove or comment the $group code i get the data with Vehicle array but using $group i get vehicle array empty, as i have only two types in records in the database, i get two empty array of vehicles. but i have 102 records so i need 102 arrays of vehicles how can i get such result.
what i am getting in console right now is
res [ { _id: 'Vehicle Delay Alert!',
details:
[ [Object],
....57 object...
[Object] ],
count: 57,
vehicle: [] },
and inside every object i dont find vehicle array so i wish to remove vehicle array from here and get a vehicle array that is generated from $lookup inside every object.
Any suggestions are highly appreciated.

You $lookup from details.device_id which comes from an array. To $lookup from a regular field, you can place $lookup after the $match :
NotificationSchema.aggregate([{
$match: condition
}, {
$lookup: {
from: "vehicles",
localField: "device_id",
foreignField: "device_id",
as: "vehicle"
}
}, {
$group: {
_id: "$type",
details: {
$push: "$$ROOT"
},
count: {
$sum: 1
}
}
}, {
$sort: {
_id: -1
}
}])

Related

How do I get how many exams have been done

I have 3 collections: Categories, Exams and Results.
For each Category I have some exams. If one exam is done, I add its result on "Result" collection.
I have take all exams for each category.
The problem is that I can't count how many of them are done.
This is what I get right now:
{ category: "Test", total_exams: 10 }
This is what I want:
{ category: "Test", total_exams: 10, done_exam: 4 }
Below you can see what I have done until now:
https://mongoplayground.net/p/upSAPkYe4DR
Quick fixes:
second $lookup stage, let to pass examsCount._id,
inside pipeline $match stage match expression, $exam_id with $$examId
put user_id match condition outside expression
$count total documents in count
$addFieldsto getdoneExams.countthis will return count in zero index, get it using$firstor$arrayElemAt`
db.categories.aggregate([
{
$lookup: {
from: "exams",
localField: "_id",
foreignField: "categoryId",
as: "examsCount"
}
},
{
$lookup: {
from: "results",
let: { examId: "$examsCount._id" },
pipeline: [
{
$match: {
$expr: {
$and: [{ $in: ["$exam_id", "$$examId"] }]
},
user_id: 22222222
}
},
{ $count: "count" }
],
as: "doneExams"
}
},
{
$addFields: {
examsCount: { $size: "$examsCount" },
doneExams: { $first: "$doneExams.count" }
}
}
])
Playground
There are many approaches to get this result, i have just resolved your issue in your approach.

Mongodb lookup aggregation not getting all field values

I have two collections which are of schema like
driver
_id:5f9c1d897ea5e246945cd73a
agent_name:"Ratnabh Kumar Rai"
agent_email:"ra2614#gmail.com"
agent_phone:"70****63331"
and
reviews
_id:5f9d54cb3a3ee10c6829c0a4
order_id:"5f9d40f096e4a506e8684aba"
user_id:"5f9bcb66f7a5bf4be0ad9973"
driver_id:"5f9c1d897ea5e246945cd73a"
rating:3
text:""
so i want to calculate the avg driver rating . I tried to use lookup aggregation so that i can get all details and then later calculate the sum...what i did was
let doc = await db
.collection("drivers")
.aggregate([
{
$project: {
_id: {
$toString: "$_id",
},
},
},
{
$lookup: {
from: "reviews",
localField: "_id",
foreignField: "driver_id",
as: "driver_info",
},
},
{
$project: {
agent_email: 1,
orderReview: "$driver_info",
},
},
])
.toArray();
But i am getting result as
{
_id: '5f9d63eb8737e82fbc193dd9',
orderReview: [ [Object], [Object], [Object], [Object], [Object] ]
}
which is partially correct as i also need to get details from my localfield collection that is drivers field, as of now you can see i am only getting id of driver in my projection i also did "agent_email:1" but not getting email
You're actually only projecting _id in the first pipeline and hence only _id is passed to further pipelines, If you need email in further pipelines you need to project it too
let doc = await db.collection("drivers").aggregate([
{
$project: {
_id: {
$toString: "$_id",
},
agent_email: "$agent_email"
},
},
{
$lookup: {
from: "reviews",
localField: "_id",
foreignField: "driver_id",
as: "driver_info",
},
},
{
$project: {
agent_email: 1,
orderReview: "$driver_info",
},
},
])
MongoDB PlayGround : https://mongoplayground.net/p/h7D-tYJ7sLU
[Update]
I realized that you're doing this for getting average and if you need it to be done in a single aggregate query, here it is how you can do it.
Using unwind operator you can flat the reviews array as objects and then group by _id and use the $avg aggregation operator
db.collection("drivers").aggregate([
{
$project: {
_id: {
$toString: "$_id",
},
agent_email: "$agent_email"
},
},
{
$lookup: {
from: "reviews",
localField: "_id",
foreignField: "driver_id",
as: "driver_info",
},
},
// Makes the driver info flat with other information
{
"$unwind": "$driver_info"
},
{
// Now you can group them
$group: {
_id: "$_id",
// Calculates avg on rating field
avg: {
"$avg": "$driver_info.rating"
},
// To make email field appear in next pipeline.
// You can do it for other fields too like Name, etc
agent_email: {
$first: "$agent_email"
}
}
},
{
$project: {
// select the fields you want to display
agent_email: 1,
avg: 1
},
},
])
MonogoDb playground Link

MongoError Total size of documents exceeds maximum document size

I am having issues with $aggregate with mongoose. I am using $lookup to get the document from the raps table, but as the size of the raps documents with matching userId is huge it fails every time.
errmsg: 'Total size of documents in raps matching { $match: { $and: [ { owner._id: { $eq: "ID" } }, {} ] } }
exceeds maximum document size',
code: 4568
I have tried with allowDiskUse and it didn't work also tried $out and adding it into temp table, didn't work either.
I have also tried using $unwind after the $lookup as mentioned in one of the answer, but it doesn't seem to work.
I have the following snippet.
userAccountModel.aggregate([
{
$match:
{
_id: userId
}
},
{
$lookup: {
from: "raps",
localField: "_id",
foreignField: "owner._id",
as: "rapsDocs"
}
},
{
$project : {
"likes": { $sum: "$rapsDocs.likes" }
}
}
]).allowDiskUse(true).exec(function(err, result){
})

Sort subdocument using mongoose (MongoDB)

I am trying to do something very simple but an new to MongoDB! I have a document called Device and a sub-document called Movement. I want to get the last two movement sub-documents out of Device ordered by last_seen (a date). Here is what I have along with the error I am getting:
Device.findOne({device_id: "1234"}, {movements: { $sort: {last_seen: -1}, $slice: 2 }}, function(err, device){
...
});
The Error:
MongoError: >1 field in obj: { $sort: { last_seen: -1 }, $slice: 2 }
You can use aggregate:
Device.aggregate(
{ $match: { device_id: "1234"}}, // query documents (can return more than one element)
{ $unwind: '$movements'}, //deconstruct the documents
{ $sort: { '$movements.last_seen': -1}},
{ $limit: 2 },
{ $group: { _id: '$device_id', movements: { $push: '$movements }}} //reconstruct the documents
function(err, devices){
//returns an array, probably with one elements depending on `$match` query
});

How to get documents with non unique array elements?

I have the following MongoDB documents:
{
_id: ObjectId('09de14821345dda65c471c99'),
items: [
_id: ObjectId('34de64871345dfa655471c99'),
_id: ObjectId('34de64871345dfa655471c91'),
_id: ObjectId('34de64871345dfa655471c99'),
]
},
{
_id: ObjectId('09de14821345dda65c471c98'),
items: [
_id: ObjectId('24de64871345dfa61271c10'),
_id: ObjectId('24de64871345dfa61271c11'),
_id: ObjectId('24de64871345dfa61271c11'),
]
},
{
_id: ObjectId('09de14821345dda65c471c07'),
items: [
_id: ObjectId('24de64871345dfa61271c05'),
_id: ObjectId('24de64871345dfa61271c06'),
_id: ObjectId('24de64871345dfa61271c07'),
]
}
I need to find all documents with repeated items array elements. So from the documents above I want to get the following result:
db.collection.documents.find({/** need query*/}).toArray(function (err, documents) {
console.dir(documents); // documents with id's 09de14821345dda65c471c99 and 09de14821345dda65c471c98
});
How could I do that?
In order to group and match results you will need to use the Aggregation Framework or Map/Reduce rather than a simple find() query.
Example data
Your example document include some errors: a few of the ObjectIDs are too short and the array elements should either be embedded documents ({_id: ObjectId(...)}) or simple values.
For test data I've used:
db.mydocs.insert([
{
_id: ObjectId('09de14821345dda65c471c99'),
items: [
ObjectId('34de64871345dfa655471c99'),
ObjectId('34de64871345dfa655471c91'),
ObjectId('34de64871345dfa655471c99')
]
},
{
_id: ObjectId('09de14821345dda65c471c98'),
items: [
ObjectId('24de64871345ddfa61271c10'),
ObjectId('24de64871345ddfa61271c11'),
ObjectId('24de64871345ddfa61271c11')
]
},
{
_id: ObjectId('09de14821345dda65c471c07'),
items: [
ObjectId('24de64871345ddfa61271c05'),
ObjectId('24de64871345ddfa61271c06'),
ObjectId('24de64871345ddfa61271c07')
]
}])
Aggregation query
Here is an aggregation query using the mongo shell:
db.mydocs.aggregate(
// Unpack items array into stream of documents
{ $unwind: "$items" },
// Group by original document _id and item
{ $group: {
_id: { _id: "$_id", item: "$items" },
count: { $sum: 1 }
}},
// Limit to duplicated array items (1 or more count per document _id)
{ $match: {
count: { $gt: 1 }
}},
// (Optional) clean up the result formatting
{ $project: {
_id: "$_id._id",
item: "$_id.item",
count: "$count"
}}
)
Sample results
{
"_id" : ObjectId("09de14821345dda65c471c98"),
"count" : 2,
"item" : ObjectId("24de64871345ddfa61271c11")
}
{
"_id" : ObjectId("09de14821345dda65c471c99"),
"count" : 2,
"item" : ObjectId("34de64871345dfa655471c99")
}

Resources