How to make aggregate + populate in Mongoose - node.js

I appreciate some help. I'm doing an api rest with express and mongodb (v3.4.4), using mongoose (v4.10.5). I need to do an aggregation operation, but I do not deal with it. I show you some code. The models (it has more properties, but I have left it simple):
const CategoryModel = mongoose.model('Category', new Schema({
slug: { type: String, unique: true, lowercase: true, index: true },
description: String
}));
const MyModel = mongoose.model('MyModel', new Schema({
category: { type: Schema.Types.ObjectId, ref: 'Category' },
other: [{ type: Schema.Types.ObjectId, ref: 'Other' }],
times_count: { type: Number, default: 0 }
}));
Important, I'm interested in populate category field of MyModel, not other field.
Suppose Category and MyModel has certain records well formed. The request:
MyModel.aggregate([
{
$group : {
_id : '$_id',
times: { $sum: '$times_count' }
}
},
{
$limit: 5
}
]).limit(5).exec().then((data) => {
console.log(data);
}).catch((err) => {
console.error(err);
});
data is correct, has 5 records, but not include category. Now, I try with:
MyModel.aggregate([
{
$group : {
_id : '$_id',
times: { $sum: '$times_count' }
}
},
{
$limit: 5
},
{
$lookup: {
from: 'Category', // I tried with 'Categories' and 'categories'
localField: 'category',
foreignField: '_id',
as: 'category'
}
},
{
$unwind: '$category'
}
]).limit(5).exec().then((data) => {
console.log(data);
}).catch((err) => {
console.error(err);
});
Now data is empty. I set mongoose.set('debug', true); and the operations they look right, inclusive the last operation aggregate, but data is empty...
I do not know if I explained well. Obviously there is something that I am not fully understanding. Thanks in advance.

I get the desired records in objs, the problem is that I only come with the _id and times properties, and I need to populate the category.
That's about right since you didn't explicitedly add the stage to join the other collection.
I've tried adding $project to the aggregation after the $ group but nothing.
In simple terms, $project is for including and excluding new fields using one collection, not joining.
You are looking for $lookup which is for joining one collection with another. When you join a new collection, each document will have a new array field containing the "joined" documents from the other collection.
In your case, your new array field will have one document from the other collection, so you probably want to $unwind also.
MyModel.aggregate([
{
$group : {
_id : '$_id',
times: { $sum: '$times_count' },
category: { $first: '$category' }
}
},
/*
{
$limit: 5
},
*/
{
$lookup: {
from: 'Categories',
localField: 'category',
foreignField: '_id',
as: 'category'
}
},
{
$unwind: '$category'
}
]).exec(...);
In terms of your initial problem, try uncommenting the 2nd stage above and not using limit(5) in your first example.

Related

Get trending groups using mongodb aggregation

I have two collections, Group and GroupChat. A group has a chatroom respectively.
// group.model
{
name: string,
desc: string
}
// group.chat.model
{
group: {type: Schema.Types.ObjectId, ref: 'Group'},
sender: ObjectId
message: String
}
Now I want to determine the top 4 trending groups based on these two criteria.
Highest number of followers in a group
most chats in a group.
I am trying to use MongoDB aggregate to do this because I know it is possible but I am currently stuck in coming up with the right pipeline query to achieve this. How do I go about this? Any help is appreciated.
Here is a snippet of my pipline (which is wrong)
const pipeline = [
{ $match: { name: { $exists: true } } },
{ $lookup: { from: 'groupchats', localField: '_id', foreignField: 'group', as: 'groupchat' } },
{ $unwind: { path: '$groupchat', preserveNullAndEmptyArrays: true } },
{ $sortByCount: '$groupchat.group' },
{ $sort: { createdAt: 1 } },
{ $limit: 5 },
];

Filter results using $match in MongoDB aggregate returning blank array

I have the following schema:
const UserQualificationSchema = new Schema(
{
user: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User',
},
qualification: {
type: mongoose.Schema.Types.ObjectId,
ref: 'Qualification',
},
expiry_date: {
type: Date
}
}
const QualificationSchema = new Schema(
{
fleet: {
type: [String], // Eg ["FleetA", "FleetB", "FleetC"]
required: true,
}
}
I am searching the UserQualifications with filters in a table, to search them by fleet, qualification or expiry date. I so far have the following aggregate:
db.UserQualifications.aggregate([{
{
$lookup: {
from: 'qualifications',
localField: 'qualification',
foreignField: '_id',
as: 'qualification',
},
},
{
$unwind: '$qualification',
},
{
$match: {
$and: [
'qualification.fleet': {
$in: ["Fleet A", "Fleet C"], // This works
},
expiry_date: {
$lt: req.body.expiry_date, // This works
},
qualification: { // Also tried 'qualification._id'
$in: ["6033e4129070031c07fbbf29"] // Adding this returns blank array
}
]
},
}
}])
Filtering by fleet, and expiry date both work, independently and in combination, however when adding by the qualification ID, it returns blank despite the ID's being sent in being valid.
Am i missing something here?
Looking at your schema I can infer that qualification in ObjectId and in the query you are passing only the string value of ObjectId. You can pass the ObjectId to get your expected output
db.UserQualifications.aggregate([
{
$lookup: {
from: "Qualifications",
localField: "qualification",
foreignField: "_id",
as: "qualification",
},
},
{
$unwind: "$qualification",
},
{
$match: {
"qualification.fleet": {
$in: [
"FleetA",
"FleetC"
],
},
expiry_date: {
$lt: 30 // some dummy value to make it work
},
"qualification._id": {
$in: [
// some dummy value to make it work
ObjectId("5a934e000102030405000000")
]
}
},
}
])
I have created a playground with some dummy data to test the query: Mongo Playground
Also, In $match stage there is no need to combine query explicitly in $and as by default behaviour will be same as $and only so I have remove that part in my query

How to find by referenced Object's property in Mongoose?

I have two models, made using Mongoose Schema.
Book {
title: String,
chapters: [{
type: Schema.Types.ObjectId,
ref: 'chapter'
}],
}
Chapter {
title: String,
status: String,
book: {
type: Schema.Types.ObjectId,
ref: 'book'
},
}
I want to find Books that have a chapter with "status":"unfinished". What is the most efficient way to achieve this? Since the Book model stores ObjectIds, how can I make the find query so that the filtered results will be fetched directly from the DB?
I think the most optimal way would be to denormalize your schema, as a book will have a limited amount of chapters and a chapter can belong to at most one book, we can store the schema like this
Book {
title: String,
chapters: [{
title: String,
status: String,
}],
}
with this schema, we can then create an index on 'chapters.status' and simply get the answer in a single query without the need of $lookup.
db.books.find({'chapters.status': 'unfinished'});
But in any case, you still need to go with the above schema, we always have an option for $lookup
db.book.aggregate([
{
$unwind: "$chapters",
},
{
$lookup: {
from: "chapter",
localField: "chapters",
foreignField: "_id",
as: "chapter",
},
},
{
$match: {
"chapter.status": "unfinished",
},
},
{
$group: {
_id: "$_id",
title: { $first: "$title" },
},
},
]);
You can always adjust the above query to your needs.
Example
You can try using aggregate(),
$lookup with pipeline, join Chapter collection
$match 2 conditions first match chapter _id in chaptersIds, second status is equal to unfinished
$match to match chapters not equal to empty array
$project to show or hide required fields
db.Book.aggregate([
{
"$lookup": {
from: "Chapter",
as: "chapters_list",
let: { chapterIds: "$chapters" },
pipeline: [
{
$match: {
$expr: {
$and: [
{ $in: ["$_id", "$$chapterIds"] },
{ $eq: ["$status", "unfinished"] }
]
}
}
}
]
}
},
{
$match: { chapters_list: { $ne: [] } }
},
// if you want chapters_list array then remove $project this part
{
$project: { chapters: 1, title: 1 }
}
])
Playground

Mongoose aggregation query skipping getters

Im have the following mongoose schema
var schema = new mongoose.Schema({
subtotal: {type: Number, required: true, default: 0},
owner: { type : Number, required : true, index: true},
products: [{type: mongoose.Schema.ObjectId, ref: 'Product'}],
}, {
timestamps: true
});
schema.path('subtotal').set(function(p){
return p * 100;
});
schema.path('subtotal').get(function(p){
return parseFloat((p/100).toFixed(2));
});
schema.set('toJSON', {getters: true, setters:true});
schema.set('toObject', {getters: true, setters:true});
Running a simple find query returns the subtotal value as expected (e.g. 11.99).
When I run an aggregation query the subtotal value is not divided by 100 as I expect it to be.
Order.aggregate([
{ $match: { $and: [
{status: 'COMPLETE'},
{premises: mongoose.Types.ObjectId(premisesId)},
{createdAt:{
$gte: startOfToday,
$lt: endOfToday
}}
]}},
{
$group: {
_id: {hour: {$hour: "$createdAt"}},
count: { $sum: 1 },
sales: { $sum: '$subtotal'}
}
},
{
$sort: {
"_id.hour": 1
}
}
]).exec(function(err, orders){
//do something
})
Is this expected behaviour of aggregation?
Is there a way that I can force the aggregation query to fire the getters.
Short answer: yes, that is the expected behavior.
Longer answer requires referencing mongoose's documentation for Model.aggregate:
Arguments are not cast to the model's schema because $project operators allow redefining the "shape" of the documents at any stage of the pipeline, which may leave documents in an incompatible format.
Essentially this means the magic of anything mongoose allows via using a schema is not applied when using aggregates (including getters). You will need to include that transform in the pipeline.

Add elements in nested document then retrieve the _id

I have the following collection definition:
// Includes
import mongoose from 'mongoose';
const Schema = mongoose.Schema;
// Create required sub schemas
const subSchema0 = new Schema({
value: String,
});
const subSchema = new Schema({
idWordsLibraryName: {
type: Schema.Types.ObjectId,
ref: 'WordsLibrary1_0',
},
type: String,
values: [
subSchema0,
],
});
const schema = new Schema({
version_: String,
idWordsLibraryName: {
type: Schema.Types.ObjectId,
ref: 'WordsLibrary1_0',
},
idsDads: [{
type: Schema.Types.ObjectId,
ref: 'LocationStructure1_0',
}],
params: [
subSchema,
],
});
Summary -> One document with nested parameters with nested values.
I have the following request that add some values into a particular parameter
this.findOneAndUpdate({
_id: data.idLocationStructure,
'params._id': data.idLocationStructureParameter,
}, {
$push: {
'params.$.values': {
$each: dataToPush,
},
},
}, {
new: true,
});
It works as expected.
What I want now is to get the _id of pushed elements, but without loading all values of the parameter.
I have tried to use the select option of findOneAndUpdate but it don't work using the projection:
this.findOneAndUpdate({
_id: data.idLocationStructure,
'params._id': data.idLocationStructureParameter,
}, {
$push: {
'params.$.values': {
$each: dataToPush,
},
},
}, {
new: true,
select: {
'params.$.values': 1,
},
});
It gets me:
{
"_id": "57273904135f829c3b0739dd",
"params": [
{},
{},
{},
{},
],
},
I have tried to perform a second request to get the _ids as well, but it don't work either:
this.find({
_id: data.idLocationStructure,
'params._id': data.idLocationStructureParameter,
}, {
_id: 1,
'params.$.values': {
$slice: -nbAdded,
},
});
If you have any idea of how retrieving the _id of the pushed values without loading all values of the parameter, you are very welcome :)
Well after tons of researches all over the web and stack overflow <3 I have found a solution, which is:
this.aggregate([{
$match: {
_id: new mongoose.Types.ObjectId(data.idLocationStructure),
},
},
{
$unwind: '$params',
}, {
$match: {
'params._id': new mongoose.Types.ObjectId(data.idLocationStructureParameter),
},
},
{
$unwind: '$params.values',
},
{
$sort: {
'params.values._id': -1
},
},
{
$limit: nbAdded,
},
{
$project: {
_id: '$params.values._id',
},
},
]);
If you experience the same problem, here is the explaination:
$match makes me taking the good high level document
$unwind makes me to go into the params array in the document we $match
$match makes me taking the good parameter
$unwind makes me to go into the values array
I $sort all values by _id DESC
I $limit to the number of values I added previsoulsy
I change the name of the _id (like an alias)
So I got as result an array that contains the last added values _ids

Resources