I have a collection which contains half million data, and i want an average for all months between the date i enter. Right now i am getting the data for whole year, but i want it seperated by single month i.e 12 data range for every single month.
Below is the aggregation pipeline i am using.
let filter = 'y';
const date = new Date();
let checkDate = moment().subtract(1.5, 'years')._d;
MeterData.aggregate([
{
$group: {
_id: "$meter_id",
// total: { $sum: 1 },
totalEnergy: filter !== 'a' ? {
$sum: {
$toDouble: {
$cond: {
if: {
$gte: [
"$date", checkDate
]
},
then: "$energy.Energy",
else: 0
}
}
}
} : { $sum: { $toDouble:
"$energy.Energy"
} }
},
}
]);
Here i am getting totalEnergy for all year, in totalEnergy field, but now i want totalEnergy plus monthly calculations for the year i enter.
Any idea on how to do that. ?
Below is a sample document from the collection.
{"_id":{"$oid":"5e557779ed588826d84cef11"},
"meter_id":"1001",
"date":{"$date":{"$numberLong":"1509474600000"}},
"parameter_name":"hvac","voltage":{"unit":"V"},
"current":{"unit":"AMP"},
"powerFactor":{"unit":"phi"},
"angle":{"unit":"degree"},
"activePower":{"unit":"kwh"},
"reactivePower":{"unit":"kwh"},
"apparentPower":{"unit":"kwh"},
"frequency":{"unit":"hz"},
"thd":{"unit":"percentage"},
"energy":{"Energy":"5.7"},
"power":{"unit":"watt"},
As per suggested by Ryan Gunner, i got my answer which i am pasting below, i just have one more problem.
[
{
meter_id: '1001',
month: '2017-10',
totalEnergy: 0,
averageEnergy: 0
} + 11 more months......
]
Now what i need is the total of the energy for 12 months. For example total of totalEnergy field for all 12 months in a single variable.
how about something like this?
var startDate = new ISODate('2020-04-01');
var endDate = new ISODate('2019-04-01');
db.collection.aggregate(
{
$match: {
$expr: {
$and: [
{ $gt: ['$date', endDate] },
{ $lt: ['$date', startDate] }]
}
}
},
{
$group: {
_id: {
meter: '$meter_id',
month: { $dateToString: { format: '%Y-%m', date: '$date' } }
},
totalEnergy: { $sum: { $toDouble: '$energy.Energy' } },
averageEnergy: { $avg: { $toDouble: '$energy.Energy' } }
}
},
{
$project: {
meter_id: '$_id.meter',
month: '$_id.month',
totalEnergy: '$totalEnergy',
averageEnergy: '$averageEnergy',
_id: 0
}
},
{
$sort: { meter_id: 1 }
}
{
$group: {
_id: null,
grandTotalEnergy: { $sum: '$totalEnergy' },
monthlyData: { $push: '$$ROOT' }
}
},
{ $project: { _id: 0 } }
)
update: added grandTotalEnergy field and pushed monthlyData to an array.
Related
there is a schema
const schemaDB = mongoose.Schema({
product:{
type: String,
},
DateOfInstallation: {
type: Date
},
standardWarranty: {
type: Number
},
extendedWarranty: {
type: Number
},
AMC:{
type: Number
}
})
i need to get the list of all products that are going to expire within 1 month from today.
total_years_to_Add = standardWarranty + extendedWarranty + AMC
date_exp = DateOfInstallation + addYears(total_years_to_Add)
date_diff = date_exp - today's Date
return data if date_diff less than month
plz help in debug the given mongodb aggregate query (implemented in mongoose)
const datas = await DBModel.aggregate([
{
$match: {
total: { $add: ['$standardWarranty','$extendedWarranty','$AMC']},
newdate: { $dateAdd: {
startDate: '$DateOfInstallation',
unit: "year",
amount: "$$total"
}},
$lte: [{$dateDiff:{
startDate: "$$newdate",
endDate: new Date(),
unit: "month"
}},1]
}},
{ $project:{
"DateOfInstallation":1, "standardWarranty":1, "extendedWarranty":1,"AMC":1
}},
{$sort:{
"DateOfInstallation": 1
}}
])
i know that $add cannot be used inside $match,
can anyone help to query the same
You are into the right direction. However you must use $expr:
db.collection.aggregate([
{
$match: {
$expr: {
$lte: [
{
$dateDiff: {
startDate: {
$dateAdd: {
startDate: "$DateOfInstallation",
unit: "year",
amount: {
$add: [
"$standardWarranty",
"$extendedWarranty",
"$AMC"
]
}
}
},
endDate: new Date(),
unit: "month"
}
}, 1
]
}
}
},
{
$project: {
"DateOfInstallation": 1,
"standardWarranty": 1,
"extendedWarranty": 1,
"AMC": 1
}
},
{ $sort: { "DateOfInstallation": 1 } }
])
or maybe a bit easier to understand:
db.collection.aggregate([
{
$set: {
total: {
$add: [
"$standardWarranty",
"$extendedWarranty",
"$AMC"
]
}
}
},
{
$set: {
newdate: {
$dateAdd: {
startDate: "$DateOfInstallation",
unit: "year",
amount: "$total"
}
}
}
},
{
$set: {
dateDiff: {
$dateDiff: {
startDate: "$newdate",
endDate: new Date(),
unit: "month"
}
}
}
},
{ $match: { "$dateDiff": { $lt: 1 } } },
{
$project: {
"DateOfInstallation": 1,
"standardWarranty": 1,
"extendedWarranty": 1,
"AMC": 1
}
},
{ $sort: { "DateOfInstallation": 1 } }
])
I'm using this query to get count of orders for 7 days of currently week. The result I get is something like this:
[
{ _id: '2021-01-31', orders: 3 },
{ _id: '2021-02-01', orders: 1 },
{ _id: '2021-02-02', orders: 2 },
{ _id: '2021-02-06', orders: 2 }
]
The problem is that if there was no order on specific day, the orders count should be 0 for that date. For example { _id: '2021-02-03', orders: 0 }, there was no order on 03-02-2021 it should be 0.
This is the query that I'm using:
let d = new Date()
d = new Date(Date.UTC(d.getFullYear(), d.getMonth(), d.getDate()));
d.setUTCDate(d.getUTCDate() + 4 - (d.getUTCDay()||7));
let yearStart = new Date(Date.UTC(d.getUTCFullYear(),0,1));
let weekNumber = Math.ceil(( ( (d - yearStart) / 86400000) + 1)/7);
const ordersweekly = await Order.aggregate([
{
"$set": { "date": { "$week": "$createdAt" } }
},
{
"$match": { "date": weekNumber }
},
{
"$set": { "date": { "$dateToString": { "format": "%Y-%m-%d", "date": "$createdAt" } } }
},
{
"$group": { "_id": "$date", "orders": { "$sum": 1 } }
},
{ "$sort": { "_id": 1 } }
])
I have the below code snippet which will retrieve Date and count from a MongoDb collection from a specific date. Example: Retrieve date, count from 05-05-2020.
||Date||Count||
|05-06-2020|4|
|05-07-2020|25| and so on.
i want to add another logic to retrieve aggregate sum of 7 days instead of individual dates. Appreciate any help.
mongoClient.db().collection(COLLECTION.AUDIT).aggregate([
{
$match: { created_at: { $gt: date } }
},
{
$group: {
_id: {
$dateToString: { format: "%Y-%m-%d", date: "$created_at" }
},
count: { $sum: 1 }
}
},
{
$sort: { "_id": -1 }
}
])
The simplest way to do what I think you're asking would be to transform your group operator to $week instead of $dateToString. Since a week is 7 days, this will group all the documents from the same week, and return a count of the documents, along with the number of the week. To get both results from 1 query, combine them into a facet. So:
mongoClient.db().collection(COLLECTION.AUDIT).aggregate([
{
$match: { created_at: { $gt: date } }
},
{
$facet: {
by_week: {
$group: {
_id: { $week: $created_at},
count: { $sum: 1 }
},
{ $sort: { "_id": -1 }}
},
by_day: {
$group: {
_id: {
$dateToString: { format: "%Y-%m-%d", date: "$created_at" }
},
count: { $sum: 1 }
}
},
{ $sort: { "_id": -1 }}
}
},
])
Here is a sample document.
{"_id":{"$oid":"5e557779ed588826d84cef11"},
"meter_id":"1001",
"date":{"$date":{"$numberLong":"1509474600000"}},
"parameter_name":"hvac","voltage":{"unit":"V"},
"current":{"unit":"AMP"},
"powerFactor":{"unit":"phi"},
"angle":{"unit":"degree"},
"activePower":{"unit":"kwh"},
"reactivePower":{"unit":"kwh"},
"apparentPower":{"unit":"kwh"},
"frequency":{"unit":"hz"},
"thd":{"unit":"percentage"},
"energy":{"Energy":"5.7"},
"power":{"unit":"watt"},
And there are around 100 000 documents. I want to filter out the documents whose date is greater than the date i specify and calculate the total energy i.e energy.Energy
I used the below aggregation, but it doesn't seem to be working
const endDate = new Date(12-12-2018)
MeterData.aggregate([
{
$group: {
_id: "$meter_id",
total: { $sum: 1 },
totalEnergy: { $sum: { $toDouble: "$energy.Energy"
} },
dateSum: {
$sum: {
$toDouble: {
$not: [{
$cond: {
if: {
$gte: [
"$date", endDate
]
},
then: "$energy.Energy",
else: 0
}
}
]
}
}
}
}
}
])
Would be this:
db.collection.aggregate([
{ $match: { date: { $gte: ISODate("2016-12-12") } } },
{
$group: {
_id: "$meter_id",
total: { $sum: 1 },
totalEnergy: { $sum: "$energy.Energy" }
}
}
])
See Mongo playground
I am trying to do a per-day aggregation in MongoDB. I already have an aggregation where I successfully group the data by day. However, I want to do the aggregation in such a way where days with no data show up, but empty. That is, they are empty bins.
Below is what I have so far. I have not been able to find anything in the MongoDB documentation or otherwise that suggests how to do aggregations and produce empty bins:
app.models.profile_view.aggregate(
{ $match: { user: req.user._id , 'viewing._type': 'user' } },
{ $project: {
day: {'$dayOfMonth': '$start'},month: {'$month':'$start'},year: {'$year':'$start'},
duration: '$duration'
} },
{ $group: {
_id: { day:'$day', month:'$month', year:'$year' },
count: { $sum: 1 },
avg_duration: { $avg: '$duration' }
} },
{ $project: { _id: 0, date: '$_id', count: 1, avg_duration: 1 }}
).exec().then(function(time_series) {
console.log(time_series)
return res.send(200, [{ key: 'user', values: time_series }])
}, function(err) {
console.log(err.stack)
return res.send(500, { error: err, code: 200, message: 'Failed to retrieve profile view data' })
})
I don't think you will be able to solve this problem using aggregation. When you use $group, mongo can only group based on the data you are providing it. In this case, how would mongo know which date values are missing or even what the range of acceptable dates is?
I think your best option would be to add the missing date values to the result of your aggregation.
Starting in Mongo 5.1, it's a perfect use case for the new $densify aggregation operator:
// { date: ISODate("2021-12-05") }
// { date: ISODate("2021-12-05") }
// { date: ISODate("2021-12-03") }
// { date: ISODate("2021-12-07") }
db.collection.aggregate([
{ $group: {
_id: { $dateTrunc: { date: "$date", unit: "day" } },
total: { $count: {} }
}},
// { _id: ISODate("2021-12-03"), total: 1 }
// { _id: ISODate("2021-12-05"), total: 2 }
// { _id: ISODate("2021-12-07"), total: 1 }
{ $densify: { field: "_id", range: { step: 1, unit: "day", bounds: "full" } } },
// { _id: ISODate("2021-12-03"), total: 1 }
// { _id: ISODate("2021-12-04") }
// { _id: ISODate("2021-12-05"), total: 2 }
// { _id: ISODate("2021-12-06") }
// { _id: ISODate("2021-12-07"), total: 1 }
{ $project: {
day: "$_id",
_id: 0,
total: { $cond: [ { $not: ["$total"] }, 0, "$total" ] }
}}
])
// { day: ISODate("2021-12-03"), total: 1 }
// { day: ISODate("2021-12-04"), total: 0 }
// { day: ISODate("2021-12-05"), total: 2 }
// { day: ISODate("2021-12-06"), total: 0 }
// { day: ISODate("2021-12-07"), total: 1 }
This:
$groups documents by day with their $count
$dateTrunc truncates your dates at the beginning of their day (the truncation unit).
$densifies documents ($densify) by creating new documents in a sequence of documents where certain values for a field (in our case field: "_id") are missing:
the step for our densification is 1 day: range: { step: 1, unit: "day" }
finally transforms ($project) fields:
renames _id to day
add the total field for new documents included during the densify stage ({ views: { $cond: [ { $not: ["$views"] }, 0, "$views" ] })