I have a collection with documents, of that structure
{
type: 'a',
date: '2014-01-04'
},
{
type: 'b',
date: '2014-01-04'
},
{
type: 'b',
date: '2014-01-04
},
{
type: 'c',
date: '2014-01-03'
},
{
type: 'a',
date: '2014-01-03'
}
I want to aggregate that data by date and type (group by date and count by type):
{
date: '2014-01-04': {
'a': 1,
'b': 2
},
date: '2014-01'03': {
'a': 1,
'c': 1
}
}
I have aggregate function, like this
db.items.aggregate([
{
$match: { user: user },
},
{
$group: { _id: {date: '$date'}, count: {$sum: 1}, services: {$push: '$type'}}
}
], function (err, results) {
But doing that I still need to reduce results by services.
Can this be done with one aggregation query?
You can of course group by more than one field:
{ $group: { _id: { date: '$date', services: '$services' } }
But that is not what you want it seems. You can not every easily convert data to keys, unless you can do that all by hand. The following query would be an option:
db.test.aggregate( [
{ $group: {
'_id' : { date: '$date' },
a: { $sum: {
$cond: [ { $eq: [ '$type', 'a' ] }, 1, 0 ]
} },
b: { $sum: {
$cond: [ { $eq: [ '$type', 'b' ] }, 1, 0 ]
} },
c: { $sum: {
$cond: [ { $eq: [ '$type', 'c' ] }, 1, 0 ]
} },
} },
{ $project: {
_id: 0,
date: '$_id.date',
a: '$a',
b: '$b',
c: '$c',
} }
] );
You will need to manually add a line for each new type.
By assuming you have fixed number of types, you can solve it as follows :
db.collection.aggregate(
{$group : {_id : "$date",
a:{$sum:{$cond:[{$eq:['$type','a']},1,0]}},
b:{$sum:{$cond:[{$eq:['$type','b']},1,0]}},
c:{$sum:{$cond:[{$eq:['$type','c']},1,0]}}
}},
{$project : {_id : 0, date : "$_id", a: "$a", b : "$b", c : "$c"}}
)
Related
I have multiple documents in a collection like this
[
{
_id: 123,
data: 1,
details: [
{
item: "a",
day: 1
},
{
item: "a",
day: 2
},
{
item: "a",
day: 3
},
{
item: "a",
day: 4
}
],
someMoreField: "xyz"
}
]
Now I want document with _id: 123 and details field should only contain day within range of 1 to 3. So the result will be like below.
{
_id: 123,
data: 1,
details: [
{
item: 'a',
day: 1,
},
{
item: 'a',
day: 2,
},
{
item: 'a',
day: 3,
},
],
someMoreField: 'xyz',
};
I tried to do this by aggregate query as:
db.collectionaggregate([
{
$match: {
_id: id,
'details.day': { $gt: 1, $lte: 3 },
},
},
{
$project: {
_id: 1,
details: {
$filter: {
input: '$details',
as: 'value',
cond: {
$and: [
{ $gt: ['$$value.date', 1] },
{ $lt: ['$$value.date', 3] },
],
},
},
},
},
},
])
But this gives me empty result. Could someone please guide me through this?
You are very close, you just need to change the $gt to $gte and $lt to $lte.
Another minor syntax error is you're accessing $$value.date but the schema you provided does not have that field, it seems you need to change it to $$value.day, like so:
db.collection.aggregate([
{
$match: {
_id: 123,
"details.day": {
$gt: 1,
$lte: 3
}
}
},
{
$project: {
_id: 1,
details: {
$filter: {
input: "$details",
as: "value",
cond: {
$and: [
{
$gte: [
"$$value.day",
1
]
},
{
$lte: [
"$$value.day",
3
]
},
],
},
},
},
},
},
])
Mongo Playground
I have a collection of Event containing a type that can be 0, 1, 2, 3, 4, 5 and a createdAt date. Each event is related to an other collection called RTS.
I want to gather for each Rts, the last event of each type.
Problem using my soluce :
The problem is, I have to describe each types one by one in order to make it work. Is there any solution to have dynamical key creation induced by the type value ?
Here is what I get now :
I sort the data
Group by idRTS whch contains the link to the second collection. For each type, push the values inside of a specific array.
Remove the null values from the types arrays.
Keep the first value only (the most updated).
Makes the data presentable.
[
{
$sort: {
idRTS: -1,
createdAt: -1
}
},
{
$group: {
_id: '$idRTS',
type0: {
$push: {
$cond: {
if: {
$eq: [
'$type', 0
]
},
then: '$$ROOT',
else: null
}
}
},
type5: {
$push: {
$cond: {
if: {
$eq: [
'$type', 5
]
},
then: '$$ROOT',
else: null
}
}
}
}
},
{
$project: {
_id: '$_id',
type0: {
'$filter': {
'input': '$type0',
'as': 'd',
'cond': {
'$ne': [
'$$d', null
]
}
}
},
type5: {
$filter: {
input: '$type5',
as: 'd',
cond: {
$ne: [
'$$d', null
]
}
}
}
}
},
{
$project: {
_id: '$_id',
type0: {
$arrayElemAt: [
'$type0', 0
]
},
type5: {
'$arrayElemAt': [
'$type5', 0
]
}
}
}
]
$match type in 0 or 5
$sort by idRTS and createdAt in descending order
$group by both idRTS and createdAt field and get first object, this will get first document of both type
$group by idRTS and make array of both types, in k(key) and v(value) format
$project to convert type array to object using $objectToArray
db.collection.aggregate([
{ $match: { type: { $in: [0, 5] } } },
{
$sort: {
idRTS: -1,
createdAt: -1
}
},
{
$group: {
_id: {
idRTS: "$idRTS",
type: "$type"
},
type: { $first: "$$ROOT" }
}
},
{
$group: {
_id: "$_id.idRTS",
type: {
$push: {
k: { $toString: "$type.type" },
v: "$type"
}
}
}
},
{ $project: { type: { $arrayToObject: "$type" } } }
])
Playground
Document
[
{
type: 1,//credit
amount: 60
},
{
type: 2,//debit
amount: 35
},
{
type: 3,//credit
amount: 25
},
{
type: 4,//debit
amount: 80
},
{
type: 5,//credit
amount: 70
},
]
Result
[
{
_id: {
Name: "Credition",
Type: [1, 3, 5]
},
Total_Amount: 155
},
{
_id: {
Name: "Debition",
Type: [2, 4]
},
Total_Amount: 115
},
]
In my schema, there are millions of logs records in which few are credited logs, few are debited logs.
I want to use MongoDB aggregate pipe and have to group like above for million records at a time
Yes you can do that first you need to add a new field transaction on the basis of the type of logs, then you can group the logs on the basis of that field.
Working example - https://mongoplayground.net/p/e4kqeKLIuIr
db.collection.aggregate([
{
$addFields: {
transaction: {
$cond: {
if: {
$in: [
"$type",
[
1,
3,
5
]
]
},
then: "Credition",
else: "Debition"
}
}
}
},
{
$group: {
_id: "$transaction",
Type: {
$addToSet: "$type"
},
Total_Amount: {
$sum: "$amount"
}
}
}
])
After this, you can also use $project operator to change the name or structure of the record, if needed
You can use the operator $cond during the grouping stage:
db.collection.aggregate([
{
$group: {
_id: {
$cond: [
{
$in: [ "$type", [1,3,5] ]
},
"Credition",
"Debition"
]
},
type: {
$addToSet: "$type"
},
amount: {
$sum: "$amount"
}
}
},
{
$project: {
_id: {
Name: "$_id",
Type: "$type"
},
Total_Amount: "$amount"
}
}
])
MongoPlayground
I'm trying to concatenate two nested arrays (using $concatArrays) into one new field. I'd like to sort the output of the concatenation (Model.timeline) by a property that exists in both sets of objects. I can't seem to get it working with $unwind. Here's the query without any sorting:
Model.aggregate([
{
$match: {
'id': id
}
},
{
$project: {
id: 1,
name: 1,
flagged: 1,
updatedAt: 1,
lastEvent: {
$arrayElemAt: ['$events', -1]
},
lastimage: {
$arrayElemAt: ['$images', -1]
},
timeline: {
$concatArrays: [
{ $filter: {
input: '$events',
as: 'event',
cond: { $and: [
{ $gte: ['$$event.timestamp', startAt] },
{ $lte: ['$$event.timestamp', endAt] }
]}
}},
{ $filter: {
input: '$images',
as: 'image',
cond: { $and: [
{ $gte: ['$$image.timestamp', startAt] },
{ $lte: ['$$image.timestamp', endAt] }
]}
}}
]
}
}
}
]);
Am I missing something obvious?
You need three pipeline stages after your match and project. First $unwind, then $sort and then re $group. Use the $first operator to retain all the fields.
{
$undwind : "$timeline",
},
{
$sort : {"your.sortable.field" : 1}
},
{
$group : {
_id : "$_id",
name : {$first : 1},
flagged : {$first : 1},
updatedAt : {$first : 1},
lastEvent : {$first : 1},
lastimage : {$first : 1},
timeline : {$push : "$timeline"}
}
}
Please note that this will work even when you have more than one document after the match phase. So basically this will sort the elements of an array within each document.
Your $match and $project aggregation stages worked after I substituted id with _id, and filled in the values for id, startAt and endAt like so:
db.items.aggregate([
{
$match: {
'_id': '58'
}
},
{
$project: {
'_id': 1,
name: 1,
flagged: 1,
updatedAt: 1,
lastEvent: {
$arrayElemAt: ['$events', -1]
},
lastimage: {
$arrayElemAt: ['$images', -1]
},
timeline: {
$concatArrays: [
{ $filter: {
input: '$events',
as: 'event',
cond: { $and: [
{ $gte: ['$$event.timestamp', ISODate("2016-01-19T20:15:31Z")] },
{ $lte: ['$$event.timestamp', ISODate("2016-12-01T20:15:31Z")] }
]}
}},
{ $filter: {
input: '$images',
as: 'image',
cond: { $and: [
{ $gte: ['$$image.timestamp', ISODate("2016-01-19T20:15:31Z")] },
{ $lte: ['$$image.timestamp', ISODate("2016-12-01T20:15:31Z")] }
]}
}}
]
}
}
}
]);
Let's say i have this 2 of huge documents:
[
{
_id: ....,
status: "A",
class: "DIP1A",
"created.user._id": ...,
"created.dt": ....,
"category": "private",
price: 100.00 //type double
},
{
_id: ....,
status: "A",
class: "DIP2A",
"created.user._id": ...
"created.dt": ...,
"category": "public",
price: 200.00 //type double
},
];
Query:
var pipeline = [
{
$match: {
"created.user._id": ....
}
},
{
$unwind: "$class"
},
{
$unwind: "$price"
},
{
$group: {
_id: "$class",
price: {
$sum: "$price"
},
count: {
$sum: 1
}
}
},
{
$project: {
_id: 0,
class: '$_id',
count: 1,
price: 1
}
}
];
db.myCollection.aggregate(pipeline);
Problem issue:
Query without calculate/$sum "$price", it's running really faster;
Indexes:
db.myCollection.ensureIndex({ 'created.user._id': -1 });
db.myCollection.ensureIndex({ 'created.user._id': -1, class: 1 });
db.myCollection.ensureIndex({ 'created.user._id': -1, price: 1});
Performance:
without $sum calc : 5 second with huge of records.
with $sum cals : 20 minutes with huge of records.
The one thing you really should do is move the $project stage to right after the $match stage (if the documents contain more data then stated in your question (huge documents)).
You want as little data as possible through the pipeline.
Also i see an $unwind on price and class but in your example they aren't array's. It could be a copy/paste issue ;-)
Like :
var pipeline = [
{
$match: {
"created.user._id": ....
}
},
{
$project: {
_id: 0,
class: '$_id',
count: 1,
price: 1
}
},
{
$unwind: "$class"
},
{
$unwind: "$price"
},
{
$group: {
_id: "$class",
price: {
$sum: "$price"
},
count: {
$sum: 1
}
}
},
];