Related
I have the following documents
loanRequest (Writing just the keys that I want to project)
{
"_id": "5f2bf26783f65d33026ea592",
"lendingpartner": {
/* some keys here */
},
"loans": [
{
"loanid": 43809,
"loanamount": 761256,
"jewels": [
"5f2bf26783f65d33026ea593",
"5f2bf26783f65d33026ea594"
"5f2bf26783f65d33026ea595"
],
}
]
}
pledgedJewel
{
"_id": "5f2bf26783f65d33026ea593",
"netweight": 8.52,
"purity": 19,
}
What I want to achieve is
{
"_id": "5f2bf2b583f65d33026ea603",
"lendingpartner": {
/* some keys here */
},
"loans": [
{
"loanid": 40010,
"loanamount": 100000,
"jewels": [
{
"_id": "5f2bf26783f65d33026ea593",
"netweight": 8.52,
"purity": 19,
},
{
"_id": "5f2bf26783f65d33026ea594",
"netweight": 5.2,
"purity": 40,
},
{
"_id": "5f2bf26783f65d33026ea595",
"netweight": 4.52,
"purity": 39,
}
]
}
]
}
Since I want the jewel details to be populated inside the jewels array of each loan, $unwind would not help me. (I tried experimenting with it)
I thought I could run a $map on loans array and then run $lookup for each jewel of the loan(double map?), but could not come up with a workable solution.
That didn't seem to be the right approach anyway.
This is the best I could come up with (Far from my desired result). I'm using map to selectively pick keys from loans object.
const loanrequests = await db.collection('loanrequest').aggregate([
{ $match: { requester: ObjectID(user.id) } },
{
$project: {
lendingpartner: {
name: 1,
branchname: '$branch.branchname',
},
loans: {
$map: {
input: '$loans',
as: 'loan',
in: {
loanid: '$$loan.loanid',
loanamount: '$$loan.amount',
jewels: '$$loan.jewels',
},
},
},
},
},
/*
* I experimented with unwind here. Tried adding
* { $unwind: '$loans' },
* { $unwind: '$loans.jewels' }
* but it does not give me the result I need (as already said before)
*/
]).toArray();
I figure, I need to do the $lookup before the projection, but I'm it hard to write a workable solution due to 2 level nested structure of the document (First, the loans array and then loans.jewels)
I started working with mongodb aggregators today and while looking for answers, I stumbled upon a similar Question but it seemed more complex and hence harder for me to understand.
Thanks!
If there are not other things you are trying to achieve with aggregate you can use .populate in mongoose.
LoanReqests
.find(
{requester: user.id},
{name: 1, branch: 1, loans: 1} // Projection
)
.populate('loans.jewels');
If you have to use aggregate to do something not in your example, then $unwind is really your best bet, but then $grouping after the $lookup to get the output you desire. If this doesn't work for you, can you expand on what the issue with $unwind is? I am guessing it is to do with fields not listed in your question.
https://mongoplayground.net/p/O5pxWNy99J4
db.loanRequests.aggregate([
{
$project: {
name: 1,
loans: 1,
branch: "$branch.name"
}
},
{
$unwind: "$loans"
},
{
$lookup: {
localField: "loans.jewels",
foreignField: "_id",
from: "jewels",
as: "loans.jewels"
}
},
{
$group: {
_id: "$_id",
name: {
$first: "$name"
},
branch: {
$first: "$branch"
},
loans: {
$push: "$loans"
}
}
}
])
As mentioned by #GitGitBoom in the previous answer, $unwind followed by $group should have been the approach.
Ofcourse, prior to grouping (I think of it as "unspreading" the outcome of running unwind), I needed to run $lookup in order to populate loans.jewels
Here is the entire solution build on top of the previous answer.
const loanRequests = await db.collection('loanRequest').aggregate([
{ $match: { requester: ObjectID(user.id) } },
{
$project: {
lender: '$lendingpartner.name',
branch: '$lendingpartner.branch.branchname',
loans: 1,
},
},
{ $unwind: '$loans' },
{
$lookup: {
localField: 'loans.jewels',
from: 'pledgedJewel',
foreignField: '_id',
as: 'loans.jewels',
},
},
{
$group: {
_id: '$_id',
branch: { $first: '$branch' },
lender: { $first: '$lender' },
loans: { $push: '$loans' },
},
},
{
$project: {
_id: 1,
branch: 1,
lender: 1,
loans: 1,
},
},
]).toArray();
Issue with mismatch of types
Another issue was, my $lookup was not working due to mismatch of types. In loanRequest collection, on which I'm running the aggregate, the ids inside loans.jewels are of type string whereas the foreign field _id in pledgedJewel is an ObjectId
This can be solved by using $toObjectId or $toString (only supported in mongodb version >= 4.0)
{ $project: { jewelObjId: { $toObjectId: '$loans.jewels' } } }, // for mongodb >= 4.0
{
$lookup: {
localField: 'jewelObjId', // for mongodb >= 4.0
from: 'pledgedjewel',
foreignField: '_id',
as: 'loans.jewels',
},
},
But, I was running on a lower version of mongodb, hence these aggregations were not working for me. Only solution to this was to change the type of loans.jewels to ObjectId instead of keeping it as string which I did.
More on type mismatch
Need a workaround for lookup of a string to objectID foreignField
Mongodb Join on _id field from String to ObjectId
I have the following Mongoose schema:
let ExerciserSchema = new Schema({
username: {
type: String,
required: true
},
exercises: [{
desc: String,
duration: Number,
date: {
type: Date,
default: new Date()
}
}]
});
I want to search by username and limit the exercise results to a date range.
I tried this lookup function:
let user = await Exerciser.find(
{ "username": name },
{ "exercises.date": { "$gte": from }},
{ "exercises.date": { "$lte": to }}
).exec((err, data) => {
if (err) {
res.json({ Error: "Data not found" })
return done(err);
}
else {
res.json(data);
return done(null, data);
}
});
However, it's logging an error and not returning the data.
MongoError: Unsupported projection option: exercises.date: { $gte: new Date(1526342400000) }
I realize from that error it appears like my date is being searched for in milliseconds, but I console.log it right before I run the above function and it's in date mode, which is what I think I want: 2018-05-01T00:00:00.000Z
How can I make this work so that I can search by a date range given my Schema? I can change the format of the date in the Schema if necessary. I'd just like the simplest solution. Thanks for your help.
You're query is wrong. You were trying to write an AND condition, but you separated documents instead of putting everything into one. This means the "second" argument to Model.find() was interpreted as a a "projection of fields", hence the error:
MongoError: Unsupported projection option:
So it's not a "schema problem" but that you sent the wrong arguments to the Model.find() method
Also you need $elemMatch for multiple conditions on elements within an array:
// Use a try..catch block with async/await of Promises
try {
let user = await Exerciser.find({
"username": name,
"exercises": {
"$elemMatch": { "date": { "$gte": from, "$lte": to } }
}
});
// work with user
} catch(e) {
// handle any errors
}
Most importantly you don't await a callback. You either await the Promise like I am showing here or simply pass in the callback instead. Not both.
Exerciser.find({
"username": name,
"exercises": {
"$elemMatch": { "date": { "$gte": from, "$lte": to } }
}
}).exec((err,user) => {
// the rest
})
FYI, what you were attempting to do was this:
Exerciser.find({
"$and": [
{ "username": name },
{ "exercises.date": { "$gte": from }},
{ "exercises.date": { "$lte": to }}
]
)
But that is actually still incorrect since without the $elemMatch the $gte and $lte applies to ALL elements of the array and not just a single one. So the incorrect results would show if ANY array item was less than the date but not necessarily greater than.
For array elements the $elemMatch enforces the "between" of the two conditions.
I managed to get it. This answer matches by username, and filters exercises so they are between the dates with variable names to and from. This is what I had wanted.
let user = Exerciser.aggregate([
{ $match: { "username": id }},
{ $project: { // $project passes along the documents with the requested fields to the next stage in the pipeline
exercises: { $filter: {
input: "$exercises",
as: "exercise",
cond: { $and: [
{ $lte: [ "$$exercise.date", to ] },
{ $gte: [ "$$exercise.date", from ] },
]}
}},
username: 1, // include username in returned data
_id: 0
}}
])
Result:
[
{
"username": "scott",
"exercises": [
{
"desc": "Situps",
"duration": 5,
"_id": "5af4790fd9a9c80c11aac696",
"date": "2018-04-30T00:00:00.000Z"
},
{
"desc": "Situps",
"duration": 10,
"_id": "5afb3f03e12e38020d059e67",
"date": "2018-05-01T00:00:00.000Z"
},
{
"desc": "Pushups",
"duration": 8,
"_id": "5afc08aa9259ed008e7e0895",
"date": "2018-05-02T00:00:00.000Z"
}
]
}
]
Currently I have this schema.
var cartSchema = new Schema({
userPurchased: {type: Schema.Types.ObjectId, ref: 'users'},
products: [
{
product: {type: Schema.Types.ObjectId, ref: 'products'},
size: {type: String, required: true},
quantity: {type: Number, required: true},
subTotal: {type: Number, required: true}
}
],
totalPrice: {type: Number, default: 0, required: true}
});
Example of db record
{
"_id": {
"$oid": "586f4be94d3e481378469a08"
},
"userPurchased": {
"$oid": "586dca1f5f4a7810fc890f97"
},
"totalPrice": 0,
"products": [
{
"product": {
"$oid": "58466e8e734d1d2b0ceeae00"
},
"size": "m",
"quantity": 5,
"subTotal": 1995,
"_id": {
"$oid": "586f4be94d3e481378469a09"
}
},
{
"subTotal": 1197,
"quantity": 3,
"size": "m",
"product": {
"$oid": "58466e8e734d1d2b0ceeae01"
},
"_id": {
"$oid": "586f4ef64d3e481378469a0a"
}
}
],
"__v": 0
}
Is there any way to sum all the subTotal and put it in the total price field? Right now I am thinking about aggregate function but I doubt it will be the right approach in here. I guess I need an update query and sum method at the same time. Can anyone help me in here?
Using the aggregate() function, you can run the following pipeline which uses the $sum operator to get the desired results:
const results = await Cart.aggregate([
{ "$addFields": {
"totalPrice": {
"$sum": "$products.subTotal"
}
} },
]);
console.log(JSON.stringify(results, null, 4));
and the corresponding update operation follows:
db.carts.updateMany(
{ },
[
{ "$set": {
"totalPrice": {
"$sum": "$products.subTotal"
}
} },
]
)
Or if using MongoDB 3.2 and earlier versions, where $sum is available in the $group stage only, you can do
const pipeline = [
{ "$unwind": "$products" },
{
"$group": {
"_id": "$_id",
"products": { "$push": "$products" },
"userPurchased": { "$first": "$userPurchased" },
"totalPrice": { "$sum": "$products.subTotal" }
}
}
]
Cart.aggregate(pipeline)
.exec(function(err, results){
if (err) throw err;
console.log(JSON.stringify(results, null, 4));
})
In the above pipeline, the first step is the $unwind operator
{ "$unwind": "$products" }
which comes in quite handy when the data is stored as an array. When the unwind operator is applied on a list data field, it will generate a new record for each and every element of the list data field on which unwind is applied. It basically flattens the data.
This is a necessary operation for the next pipeline stage, the $group step where you group the flattened documents by the _id field, thus effectively regrouping the denormalised documents back to their original schema.
The $group pipeline operator is similar to the SQL's GROUP BY clause. In SQL, you can't use GROUP BY unless you use any of the aggregation functions. The same way, you have to use an aggregation function in MongoDB (called accumulators) as well. You can read more about the accumulators here.
In this $group operation, the logic to calculate the totalPrice and returning the original fields is through the accumulators. You get thetotalPrice by summing up each individual subTotal values per group with $sum as:
"totalPrice": { "$sum": "$products.subTotal }
The other expression
"userPurchased": { "$first": "$userPurchased" },
will return a userPurchased value from the first document for each group using $first. Thus effectively rebuilding the original document schema before the $unwind
One thing to note here is when executing a pipeline, MongoDB pipes operators into each other. "Pipe" here takes the Linux meaning: the output of an operator becomes the input of the following operator. The result of each operator is a new collection of documents. So Mongo executes the above pipeline as follows:
collection | $unwind | $group => result
As a side note, to help with understanding the pipeline or to debug it should you get unexpected results, run the aggregation with just the first pipeline operator. For example, run the aggregation in mongo shell as:
db.cart.aggregate([
{ "$unwind": "$products" }
])
Check the result to see if the products array is deconstructed properly. If that gives the expected result, add the next:
db.cart.aggregate([
{ "$unwind": "$products" },
{
"$group": {
"_id": "$_id",
"products": { "$push": "$products" },
"userPurchased": { "$first": "$userPurchased" },
"totalPrice": { "$sum": "$products.subTotal" }
}
}
])
Repeat the steps till you get to the final pipeline step.
If you want to update the field then you can add the $out pipeline stage as the last step. This will write the resulting documents of the aggregation pipeline to the same collection, thus technically updating the collection.
var pipeline = [
{ "$unwind": "$products" },
{
"$group": {
"_id": "$_id",
"products": { "$push": "$products" },
"userPurchased": { "$first": "$userPurchased" },
"totalPrice": { "$sum": "$products.subTotal" }
}
},
{ "$out": "cart" } // write the results to the same underlying mongo collection
]
UPDATE
To do both the update and query, you could then issue a find() call in the aggregate callback to get the updated json i.e.
Cart.aggregate(pipeline)
.exec(function(err, results){
if (err) throw err;
Cart.find().exec(function(err, docs){
if (err) return handleError(err);
console.log(JSON.stringify(docs, null, 4));
})
})
Using Promises, you could do this alternatively as
Cart.aggregate(pipeline).exec().then(function(res)
return Cart.find().exec();
).then(function(docs){
console.log(JSON.stringify(docs, null, 4));
});
I can't really say whether this approach is better than the aggregation, but in case you want to do it with virtuals:
cartSchema.virtual('totalPrice').get(function () {
return this.products.map(p => p.subTotal).reduce((a, b) => a + b);
});
But care:
If you use toJSON() or toObject() (or use JSON.stringify() on a mongoose document) mongoose will not include virtuals by default. Pass { virtuals: true } to either toObject() or toJSON()
I have an Inspection model in mongoose:
var InspectionSchema = new Schema({
business_id: {
type: String,
required: true
},
score: {
type: Number,
min: 0,
max: 100,
required: true
},
date: {
type: Number, // in format YYYYMMDD
required: true
},
description: String,
type: String
});
InspectionSchema.index({business_id: 1, date: 1}, {unique: true});
It is possible for there to be multiple inspections on the same Business (each Business is represented by a unique business_id). However, there is a limit of one inspection per business per day, which is why there is a unique index on business_id + date.
I also created a static method on the Inspection object which, given a list of business_ids, retrieves all of the inspections for the underlying businesses.
InspectionSchema.statics.getAllForBusinessIds = function(ids, callback) {
this.find({'business_id': {$in: ids}}, callback);
};
This function fetches all of the inspections for the requested businesses. However, I want to also create a function that fetches only the latest inspection per business_id.
InspectionSchema.statics.getLatestForBusinessIds = function(ids, callback) {
// query to get only the latest inspection per business_id in "ids"?
};
How might I go about implementing this?
You can use the .aggregate() method in order to get all the latest data in one request:
Inspection.aggregate(
[
{ "$sort": { "buiness_id": 1, "date": -1 } },
{ "$group": {
"_id": "$business_id",
"score": { "$first": "$score" },
"date": { "$first": "$date" },
"description": { "$first": "$description" },
"type": { "$first": "$type" }
}}
],
function(err,result) {
}
);
Just $sort then $group with the "business_id" as the grouping key. The $first gets the first results from the grouping boundary, where we already sorted by date within each id.
If you just want the date then do this using $max:
Inspection.aggregate(
[
{ "$group": {
"_id": "$business_id",
"date": { "$max": "$date" }
}}
],
function(err,result) {
}
);
Also see $match if you want to "pre-filter" the business id values or any other conditions when doing this.
try this:
Inpection.aggregate(
[
{ $match : { _id : { "$in" : ids} } },
{ $group: { "_id" : "$business_id", lastInspectionDate: { $last: "$date" } } }
],
function(err,result) {
}
);
My Mongoose Schema is as follows:
var DSchema = new mongoose.Schema({
original_y: {type: Number},,
new_y: {type: Number},,
date: {type: Date},
dummy: [dummyEmbeddedDocuments]
}, toObject: { virtuals: true }, toJSON: { virtuals: true}
});
DSchema.virtual('dateformatted').get(function () {
return moment(this.date).format('YYYY-MM-DD HH:mm:ss');
});
module.exports = mongoose.model('D', DSchema);
A document in my schema would be the following:
{
id:1,
original_y: 200,
new_y: 140,
date: 2015-05-03 00:00:00.000-18:30,
dummy: [
{id:1, storage:2, cost: 10},
{id:2, storage:0, cost: 20},
{id:3, storage:5, cost: 30},
]
}
My Query:
Item.aggregate([
{
"$match": {
"dummy.storage": {"$gt": 0}
}
},
{
"$unwind": "$dummy"
},
{
"$project": {
"original_y": 1,
"new_y": 1,
"dateformatted": 1,
"dummy.id": "$dummy.id",
"dummy.storage": "$dummy.storage",
"dummy.cost": "$dummy.cost",
"dummy.tallyAmount": {
"$divide": [
{ "$add": ["$new_y","$original_y"] },
"$dummy.cost"
]
}
}
},
{
"$group": {
"_id": "_$id",
"original_y": { "$first": "$original_y" },
"dateformatted": { "$first": "$dateformatted" },
"new_y": { "$first": "$new_y" },
"dummy": {
"$addToSet": "$dummy"
}
}
}
]).exec(callback);
This query however returns the VIRTUAL dateformatted attribute as NULL. Any thoughts as to why this is happening?
A couple notes in the docs touch on why this is so:
Arguments are not cast to the model's schema because $project operators allow redefining the "shape" of the documents at any stage
of the pipeline, which may leave documents in an incompatible format.
The documents returned are plain javascript objects, not mongoose documents (since any shape of document can be returned).
But it goes beyond this because the aggregate operation is performed server-side, where any client-side Mongoose concepts like virtuals do not exist.
The result is that you'll need to include the date field in your $project and $group stages and add your own dateformatted field to the results in code based on the date values.
This is an old question but I've come up with a useful hack to get back the virtuals and thought it might be useful for those searching for this problem.
You can easily convert the objects back to mongoose models:
documents = documents.map(d => {
return new Document(d);
});
var virtual = documents[0].virtualProperty;
the <field>: <1 or true> form is used to include an existing field which is not the case here since the dateformatted field doesn't exist and you have to create it using an expression, $dateToString can be used:
"$project": {
"original_y": 1,
"new_y": 1,
"dateformatted": { "$dateToString": { "format": "%Y-%m-%d %H:%M:%S", "date": "$date" } },
...
Another option is to use it with $addFields:
{
"$project": {
...
}
},
{
"$addFields": {
"dateformatted": { "$dateToString": {"format": "%Y-%m-%d %H:%M:%S", "date": "$date"} }
}
},
...
Here's a solution that works!
Aggregate queries return js objects which is not an instance of mongoose Document.
You may use Model.hydrate
const documents = docs.map(doc => myModel.hydrate(doc))