I'm looking for a way to use Mongoose's aggregation pipeline to sum data. I have a schema that looks like this:
var Object = new Schema ({
Object2 : { type : ObjectId, ref : 'Object2' },
Object3 : { type : ObjectId, ref : 'Object3' },
value : {},
unit : String
})
The 'value' field will typically be a number, but I am allowing the user to include text in the field that transforms the number. Thus, value is stored as a string, and then I convert it into a number. I can't just store it as a number because there will be no way to convert it back for the user's display.
I've been trying to perform a Mongoose $sum pipeline to sum these values, while still allowing me to convert the number. Here is what I've tried:
data.aggregate([
{ $match:
{
$and: [{Object2: mongoose.Types.ObjectId(id)}, {Object3: {$in: sessions.map(function(this){ return new mongoose.Types.ObjectId(this._id); })}}]
}
},
{ $project:
{
'value': calculateNumber('value')
}
},
{ $group:
{
_id: null,
value: { $sum: "$value"},
unit: { $first: "$unit"}
}
}
], function(err, result) {
sendCallback(err, result, callback);
return;
})
However, I get a value of 0. Not using the project stage gives a value of 0, and I can't figure out if there is any way to use the project stage to apply my function and convert the fields to Numbers, so that I can take advantage of $sum.
Any help, without just declaring value as a Number in my schema, would be greatly appreciated!
Related
I'm trying to make trivial SUM on mongoDB to count number of prices for single client.
My collection:
{"_id":"5d973c71dd93adfbda4c7272","name":"Faktura2019006","clientId":"5d9c87a6b9676069c8b5e15b","expiration":"2019-10-02T01:11:18.965Z","price":999999,"userId":"123"},
{"_id":"5d9e07e0b9676069c8b5e15d","name":"Faktura2019007","clientId":"5d9c87a6b9676069c8b5e15b","expiration":"2019-10-02T01:11:18.965Z","price":888,"userId":"123"}
What I tried:
// invoice.model.js
const mongoose = require("mongoose");
const InvoiceSchema = mongoose.Schema({
_id: String,
name: String,
client: String,
userId: String,
expiration: Date,
price: Number
});
module.exports = mongoose.model("Invoice", InvoiceSchema, "invoice");
and
// invoice.controller.js
const Invoice = require("../models/invoice.model.js");
exports.income = (req, res) => {
console.log("Counting Income");
Invoice.aggregate([
{
$match: {
userId: "123"
}
},
{
$group: {
total: { $sum: ["$price"] }
}
}
]);
};
What happen:
When I now open a browser and code above is being called, I get console log 'Counting Income' in terminal however in browser it's just loading forever and nothing happen.
Most likely I just miss some stupid minor thing but I'm trying to find it out for quite a long time without any success so any advise is welcome.
The reason that the controller never finishes is because you are not ending the response process (meaning, you need to use the res object and send something back to the caller).
In order to get the aggregate value, you also need to execute the pipeline (see this example).
Also, as someone pointed out in the comments, you need to add _id: null in your group to specify that you are not going to group by any specific field (see the second example here).
Finally, in the $sum operator, for what you're trying to do, you just need to remove the array brackets since you only want to sum on a single field (see a few examples down here).
Here is the modified code:
// invoice.controller.js
const Invoice = require("../models/invoice.model.js");
exports.income = (req, res) => {
console.log("Counting Income");
Invoice.aggregate([
{
$match: {
userId: "123"
}
},
{
$group: {
_id: null,
total: { $sum: "$price" }
}
}
]).then((response) => {
res.json(response);
});
};
Edit for your comment about when an empty array is returned.
If you want to always return the same type of object, I would control that in the controller. I'm not sure if there is a fancy way to do this with the aggregate pipeline in mongo, but this is what I would do.
Invoice.aggregate([
{
$match: {
userId: "123"
}
},
{
$group: {
_id: null,
total: { $sum: "$price" }
}
},
{
$project: {
_id: 0,
total: "$total"
}
}
]).then((response) => {
if (response.length === 0) {
res.json({ total: 0 });
} else {
// always return the first (and only) value
res.json(response[0]);
}
});
Here, if you find a userId of 123, then you would get this as the return:
{
"total": 1000887
}
But if you change the userId to, say, 1123 which doesn't exist in your db, the result will be:
{
"total": 0
}
This way, your client can always consume the same type of object.
Also, the reason I put the $project pipeline stage in there was to suppress the _id field (see here for more info).
I don't really know how to frame the question but what I have is the following schema in mongoose
new Schema({
gatewayId: { type: String, index: true },
timestamp: { type: Date, index: true },
curr_property:Number,
curr_property_cost:Number,
day_property:Number,
day_property_cost: Number,
curr_solar_generating: Number,
curr_solar_export:Number,
day_solar_generated:Number,
day_solar_export:Number,
curr_chan1:Number,
curr_chan2:Number,
curr_chan3:Number,
day_chan1:Number,
day_chan2:Number,
day_chan3:Number
},{
collection: 'owlelecmonitor'
});
and I want to be able to query all the documents in the collection but the data should be arranged inside the array in the following format
[ [{
gatewayId: 1,
timestamp: time
....
},
{
gatewayId: 1,
timestamp: time2
....
}],
[{
gatewayId: 2,
timestamp: time
....
},
{
gatewayId: 2,
timestamp: time2
....
}],
[{
gatewayId: 3,
timestamp: time
....
},
{
gatewayId: 3,
timestamp: time2
....
}]
];
Is there a way that I can do this in mongoose instead of retrieving the documents and processing them again ?
Yes, it's possible. Consider the following aggregation pipeline in mongo shell. This uses a single pipeline stream comprising of just the $group operator, grouping all the documents by gatewayId and creating another array field that holds all the grouped documents. This extra field uses the accumulator operator $push on the system variable $$ROOT which returns the root document, i.e. the top-level document, currently being processed in the aggregation pipeline stage.
With the cursor returned from the aggregate() method, you can then use its map() method to create the desired final array. The following mongo shell demonstration describes the above concept:
var result = db.owlelecmonitor.aggregate([
{
"$group": {
"_id": "$gatewayId",
"doc": {
"$push": "$$ROOT"
}
}
}
]).map(function (res){ return res.doc; });
printjson(result);
This will output to shell the desired result.
To implement this in Mongoose, use the following aggregation pipeline builder:
OwlelecMonitorModel
.aggregate()
.group({
"_id": "$gatewayId",
"doc": {
"$push": "$$ROOT"
}
})
.exec(function (err, result) {
var res = result.map(function (r){return r.doc;});
console.log(res);
});
I am using mongoose to perform CRUD operation on my db. This is how my model looks.
var EmployeeSchema = new Schema({
name: String,
description: {
type: String,
default: 'No description'
},
department: [],
lastUpdated: {
type: Date,
default: Date.now
}
});
The department can contains array of object like this.
[
{
"id" : "55ba28f680dec4383eeebf97",
"text" : "Sales",
"topParentId" : "55ba28f680dec4383eeebf8b",
"topParentText" : "XYZ"
},
{
"id" : "55ba28f680dec4383eeebf98",
"text" : "IT",
"topParentId" : "55ba28f680dec4383eeebf8b",
"topParentText" : "XYZ"
},
{
"id" : "55ba28f680dec4383eeebf94",
"text" : "Marketing",
"topParentId" : "55ba28f680dec4383eeebccc",
"topParentText" : "ABC"
}
]
Now I need to find all the employee where department.id = '55ba28f680dec4383eeebf94' and then I need to update the text of the object.
Employee.find({'department.id': '55ba28f680dec4383eeebf94'}, function(err, Employees) {
_.each(Employees, function (emp) {
_.each(emp.department, function (dept) {
if(dept.id === '55ba28f680dec4383eeebf94'){
dept.text = 'XXXXX'; // How to update the employee to save the updated text
}
});
});
});
What is the right way to save the employee with updated text for that department?
Iterating is code is not a "sharp" way to do this. It is better to use the MongoDB update operators, especially since there is no schema defined for the array items here, so no rules to worry about:
Employee.update(
{'department.id': '55ba28f680dec4383eeebf94'},
{ "$set": { "department.$.text": "XXXXX" },
function(err,numAffected) {
// handling in here
}
);
The $set is the important part, otherwise you overwrite the whole object. As is the positional $ operator in the statement, so only the matched ( queried item in the array ) index is updated.
Also see .find**AndUpdate() variants for a way to return the modified object.
I think you can use the update model:
Employee.update({department.id: '55ba28f680dec4383eeebf94'}, {department.text: 'XXXXX'}, {multi: true},
function(err, num) {
console.log("updated "+num);
}
);
First object is the query, what to find: {department.id: '55ba28f680dec4383eeebf94'}, the second one is the update, what to update: {department.text: 'XXXXX'} and the third one is the options to pass to the update, multi means update every records you find: {multi: true}
how can i combine match document's subdocument together as one and return it as an array of object ? i have tried $group but don't seem to work.
my query ( this return array of object in this case there are two )
User.find({
'business_details.business_location': {
$near: coords,
$maxDistance: maxDistance
},
'deal_details.deals_expired_date': {
$gte: new Date()
}
}, {
'deal_details': 1
}).limit(limit).exec(function(err, locations) {
if (err) {
return res.status(500).json(err)
}
console.log(locations)
the console.log(locations) result
// give me the result below
[{
_id: 55 c0b8c62fd875a93c8ff7ea, // first document
deal_details: [{
deals_location: '101.6833,3.1333',
deals_price: 12.12 // 1st deal
}, {
deals_location: '101.6833,3.1333',
deals_price: 34.3 // 2nd deal
}],
business_details: {}
}, {
_id: 55 a79898e0268bc40e62cd3a, // second document
deal_details: [{
deals_location: '101.6833,3.1333',
deals_price: 12.12 // 3rd deal
}, {
deals_location: '101.6833,3.1333',
deals_price: 34.78 // 4th deal
}, {
deals_location: '101.6833,3.1333',
deals_price: 34.32 // 5th deal
}],
business_details: {}
}]
what i wanted to do is to combine these both deal_details field together and return it as an array of object. It will contain 5 deals in one array of object instead of two separated array of objects.
i have try to do it in my backend (nodejs) by using concat or push, however when there's more than 2 match document i'm having problem to concat them together, is there any way to combine all match documents and return it as one ? like what i mentioned above ?
What you are probably missing here is the $unwind pipeline stage, which is what you typically use to "de-normalize" array content, particularly when your grouping operation intends to work across documents in your query result:
User.aggregate(
[
// Your basic query conditions
{ "$match": {
"business_details.business_location": {
"$near": coords,
"$maxDistance": maxDistance
},
"deal_details.deals_expired_date": {
"$gte": new Date()
}},
// Limit query results here
{ "$limit": limit },
// Unwind the array
{ "$unwind": "$deal_details" },
// Group on the common location
{ "$group": {
"_id": "$deal_details.deals_location",
"prices": {
"$push": "$deal_details.deals_price"
}
}}
],
function(err,results) {
if (err) throw err;
console.log(JSON.stringify(results,undefined,2));
}
);
Which gives output like:
{
"_id": "101.6833,3.1333",
"prices": [
12.12,
34.3,
12.12,
34.78,
34.32
]
}
Depending on how many documents actually match the grouping.
Alternately, you might want to look at the $geoNear pipeline stage, which gives a bit more control, especially when dealing with content in arrays.
Also beware that with "location" data in an array, only the "nearest" result is being considered here and not "all" of the array content. So other items in the array may not be actually "near" the queried point. That is more of a design consideration though as any query operation you do will need to consider this.
You can merge them with reduce:
locations = locations.reduce(function(prev, location){
previous = prev.concat(location.deal_details)
return previous
},[])
I have an article schema that has a subdocument comments which contains all the comments i got for this particular article.
What i want to do is select an article by id, populate its author field and also the author field in comments. Then sort the comments subdocument by date.
the article schema:
var articleSchema = new Schema({
title: { type: String, default: '', trim: true },
body: { type: String, default: '', trim: true },
author: { type: Schema.ObjectId, ref: 'User' },
comments: [{
body: { type: String, default: '' },
author: { type: Schema.ObjectId, ref: 'User' },
created_at: { type : Date, default : Date.now, get: getCreatedAtDate }
}],
tags: { type: [], get: getTags, set: setTags },
image: {
cdnUri: String,
files: []
},
created_at: { type : Date, default : Date.now, get: getCreatedAtDate }
});
static method on article schema: (i would love to sort the comments here, can i do that?)
load: function (id, cb) {
this.findOne({ _id: id })
.populate('author', 'email profile')
.populate('comments.author')
.exec(cb);
},
I have to sort it elsewhere:
exports.load = function (req, res, next, id) {
var User = require('../models/User');
Article.load(id, function (err, article) {
var sorted = article.toObject({ getters: true });
sorted.comments = _.sortBy(sorted.comments, 'created_at').reverse();
req.article = sorted;
next();
});
};
I call toObject to convert the document to javascript object, i can keep my getters / virtuals, but what about methods??
Anyways, i do the sorting logic on the plain object and done.
I am quite sure there is a lot better way of doing this, please let me know.
I could have written this out as a few things, but on consideration "getting the mongoose objects back" seems to be the main consideration.
So there are various things you "could" do. But since you are "populating references" into an Object and then wanting to alter the order of objects in an array there really is only one way to fix this once and for all.
Fix the data in order as you create it
If you want your "comments" array sorted by the date they are "created_at" this even breaks down into multiple possibilities:
It "should" have been added to in "insertion" order, so the "latest" is last as you note, but you can also "modify" this in recent ( past couple of years now ) versions of MongoDB with $position as a modifier to $push :
Article.update(
{ "_id": articleId },
{
"$push": { "comments": { "$each": [newComment], "$position": 0 } }
},
function(err,result) {
// other work in here
}
);
This "prepends" the array element to the existing array at the "first" (0) index so it is always at the front.
Failing using "positional" updates for logical reasons or just where you "want to be sure", then there has been around for an even "longer" time the $sort modifier to $push :
Article.update(
{ "_id": articleId },
{
"$push": {
"comments": {
"$each": [newComment],
"$sort": { "$created_at": -1 }
}
}
},
function(err,result) {
// other work in here
}
);
And that will "sort" on the property of the array elements documents that contains the specified value on each modification. You can even do:
Article.update(
{ },
{
"$push": {
"comments": {
"$each": [],
"$sort": { "$created_at": -1 }
}
}
},
{ "multi": true },
function(err,result) {
// other work in here
}
);
And that will sort every "comments" array in your entire collection by the specified field in one hit.
Other solutions are possible using either .aggregate() to sort the array and/or "re-casting" to mongoose objects after you have done that operation or after doing your own .sort() on the plain object.
Both of these really involve creating a separate model object and "schema" with the embedded items including the "referenced" information. So you could work upon those lines, but it seems to be unnecessary overhead when you could just sort the data to you "most needed" means in the first place.
The alternate is to make sure that fields like "virtuals" always "serialize" into an object format with .toObject() on call and just live with the fact that all the methods are gone now and work with the properties as presented.
The last is a "sane" approach, but if what you typically use is "created_at" order, then it makes much more sense to "store" your data that way with every operation so when you "retrieve" it, it stays in the order that you are going to use.
You could also use JavaScript's native Array sort method after you've retrieved and populated the results:
// Convert the mongoose doc into a 'vanilla' Array:
const articles = yourArticleDocs.toObject();
articles.comments.sort((a, b) => {
const aDate = new Date(a.updated_at);
const bDate = new Date(b.updated_at);
if (aDate < bDate) return -1;
if (aDate > bDate) return 1;
return 0;
});
As of the current release of MongoDB you must sort the array after database retrieval. But this is easy to do in one line using _.sortBy() from Lodash.
https://lodash.com/docs/4.17.15#sortBy
comments = _.sortBy(sorted.comments, 'created_at').reverse();