i have simple schema like this
{
"productName": "pppppp"
"sku" : {
"carted" : [
{
"_id" : ObjectId("56c6d606c0987668109a21f7"),
"timestamp" : ISODate("2016-02-19T08:44:54.043+0000"),
"cartId" : "56c6c1fd60c4491c157e433d",
"qty" : NumberInt(2)
},
{
"_id" : ObjectId("56c6d653172fb54817ec2356"),
"timestamp" : ISODate("2016-02-19T08:46:11.902+0000"),
"cartId" : "56c6c1fd60c4491c157e433d",
"qty" : NumberInt(2)
},
{
"_id" : ObjectId("56c6d6a7172fb54817ec2358"),
"timestamp" : ISODate("2016-02-19T08:47:35.652+0000"),
"cartId" : "56c6c1fd60c4491c157e433d",
"qty" : NumberInt(2)
}
],
"qty" : NumberInt(14)
}
}
how the way to view the product "pppppp" and show the quantity to 20? the sku.quantity added with all available sku.carted.qty.
i want it looks like this
{
"productName": "pppppp"
"qty" : 20
}
Please try this one with $group, $sum and $add
> db.collection.aggregate([
{$unwind: '$sku.carted'},
// sum the `qty` in the carted array, put this result to `qt`
{$group: {
_id: {productName: '$productName', q: '$sku.qty'},
qt: {$sum: '$sku.carted.qty'}
}},
// add the `qt` and `sku.qty`
// and reshape the output result.
{$project: {
_id: 0,
productName: '$_id.productName',
qty: {$add: ['$_id.q', '$qt']}
}}
]);
Related
I'm practicing how to use MongoDB aggregation, but they seem to take a really long time (running time).
The problem seems to happen whenever I use $group. All other queries run just fine.
I have some 1.3 million dummy documents that need to perform two basic operations: get a count of the IP addresses and unique IP addresses.
My schema looks something like this:
{
"_id":"5da51af103eb566faee6b8b4",
"ip_address":"...",
"country":"CL",
"browser":{
"user_agent":...",
}
}
Running a basic $group query takes about 12s on average, which is much too slow.
I did a little research, and someone suggested creating an index on ip_addresses. That seems to have slowed it down because queries now take 13-15s.
I use MongoDB and the query I'm running looks like this:
visitorsModel.aggregate([
{
'$group': {
'_id': '$ip_address',
'count': {
'$sum': 1
}
}
}
]).allowDiskUse(true)
.exec(function (err, docs) {
if (err) throw err;
return res.send({
uniqueCount: docs.length
})
})
Any help is appreciated.
Edit: I forgot to mention, someone suggested it might be a hardware issue? I'm running the query on a core i5, 8GB RAM laptop if it helps.
Edit 2: The query plan:
{
"stages" : [
{
"$cursor" : {
"query" : {
},
"fields" : {
"ip_address" : 1,
"_id" : 0
},
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "metrics.visitors",
"indexFilterSet" : false,
"parsedQuery" : {
},
"winningPlan" : {
"stage" : "COLLSCAN",
"direction" : "forward"
},
"rejectedPlans" : [ ]
},
"executionStats" : {
"executionSuccess" : true,
"nReturned" : 1387324,
"executionTimeMillis" : 7671,
"totalKeysExamined" : 0,
"totalDocsExamined" : 1387324,
"executionStages" : {
"stage" : "COLLSCAN",
"nReturned" : 1387324,
"executionTimeMillisEstimate" : 9,
"works" : 1387326,
"advanced" : 1387324,
"needTime" : 1,
"needYield" : 0,
"saveState" : 10930,
"restoreState" : 10930,
"isEOF" : 1,
"invalidates" : 0,
"direction" : "forward",
"docsExamined" : 1387324
}
}
}
},
{
"$group" : {
"_id" : "$ip_address",
"count" : {
"$sum" : {
"$const" : 1
}
}
}
}
],
"ok" : 1
}
This is some info about using $group aggregation stage, if it uses indexes, and its limitations and what can be tried to overcome these.
1. The $group Stage Doesn't Use Index:
Mongodb Aggregation: Does $group use index?
2. $group Operator and Memory:
The $group stage has a limit of 100 megabytes of RAM. By default, if
the stage exceeds this limit, $group returns an error. To allow for
the handling of large datasets, set the allowDiskUse option to true.
This flag enables $group operations to write to temporary files.
See MongoDb docs on $group Operator and Memory
3. An Example Using $group and Count:
A collection called as cities:
{ "_id" : 1, "city" : "Bangalore", "country" : "India" }
{ "_id" : 2, "city" : "New York", "country" : "United States" }
{ "_id" : 3, "city" : "Canberra", "country" : "Australia" }
{ "_id" : 4, "city" : "Hyderabad", "country" : "India" }
{ "_id" : 5, "city" : "Chicago", "country" : "United States" }
{ "_id" : 6, "city" : "Amritsar", "country" : "India" }
{ "_id" : 7, "city" : "Ankara", "country" : "Turkey" }
{ "_id" : 8, "city" : "Sydney", "country" : "Australia" }
{ "_id" : 9, "city" : "Srinagar", "country" : "India" }
{ "_id" : 10, "city" : "San Francisco", "country" : "United States" }
Query the collection to count the cities by each country:
db.cities.aggregate( [
{ $group: { _id: "$country", cityCount: { $sum: 1 } } },
{ $project: { country: "$_id", _id: 0, cityCount: 1 } }
] )
The Result:
{ "cityCount" : 3, "country" : "United States" }
{ "cityCount" : 1, "country" : "Turkey" }
{ "cityCount" : 2, "country" : "Australia" }
{ "cityCount" : 4, "country" : "India" }
4. Using allowDiskUse Option:
db.cities.aggregate( [
{ $group: { _id: "$country", cityCount: { $sum: 1 } } },
{ $project: { country: "$_id", _id: 0, cityCount: 1 } }
], { allowDiskUse : true } )
Note, in this case it makes no difference in query performance or output. This is to show the usage only.
5. Some Options to Try (suggestions):
You can try a few things to get some result (for trial purposes only):
Use $limit stage and restrict the number of documents processed and
see what is the result. For example, you can try { $limit: 1000 }.
Note this stage needs to come before the $group stage.
You can also use the $match, $project stages before the $group
stage to control the shape and size of the input. This may
return a result (instead of an error).
[EDIT ADD]
Notes on Distinct and Count:
Using the same cities collection - to get unique countries and a count of them you can try using the aggregate stage $count along with $group as in the following two queries.
Distinct:
db.cities.aggregate( [
{ $match: { country: { $exists: true } } },
{ $group: { _id: "$country" } },
{ $project: { country: "$_id", _id: 0 } }
] )
The Result:
{ "country" : "United States" }
{ "country" : "Turkey" }
{ "country" : "India" }
{ "country" : "Australia" }
To get the above result as a single document with an array of unique values, use the $addToSetoperator:
db.cities.aggregate( [
{ $match: { country: { $exists: true } } },
{ $group: { _id: null, uniqueCountries: { $addToSet: "$country" } } },
{ $project: { _id: 0 } },
] )
The Result: { "uniqueCountries" : [ "United States", "Turkey", "India", "Australia" ] }
Count:
db.cities.aggregate( [
{ $match: { country: { $exists: true } } },
{ $group: { _id: "$country" } },
{ $project: { country: "$_id", _id: 0 } },
{ $count: "uniqueCountryCount" }
] )
The Result: { "uniqueCountryCount" : 4 }
In the above queries the $match stage is used to filter any documents with non-existing or null countryfield. The $project stage reshapes the result document(s).
MongoDB Query Language:
Note the two queries get similar results when using the MongoDB query language commands: db.collection.distinct("country") and db.cities.distinct("country").length (note the distinct returns an array).
You can create index
db.collectionname.createIndex( { ip_address: "text" } )
Try this, it is more faster.
I think it will help you.
I have records in my collection
{
"_id" : ObjectId("5c37a71c54956d08afb590ef"),
"user_id" : 45,
"result" : 9,
}
{
"_id" : ObjectId("5c37a7ad54956d08afb590f0"),
"user_id" : 1,
"result" : 3,
}
{
"_id" : ObjectId("5c37a80254956d08afb590f1"),
"user_id" : 45,
"result" : 10,
}
How to get distinct records with max values (result) for each user (user_id field is unique) ?
I expect result like this:
{
"_id" : ObjectId("5c37a80254956d08afb590f1"),
"user_id" : 45, //distinct user_id
"result" : 10, //max result for user
}
{
"_id" : ObjectId("5c37a7ad54956d08afb590f0"),
"user_id" : 1, //distinct user_id
"result" : 3, //max result for user
}
You can use below aggregation:
db.col.aggregate([
{
$sort: { result: -1 }
},
{
$group: {
_id: "$user_id",
result: { $first: "$result" },
o_id: { $first: "$_id" }
}
},
{
$project: {
_id: "$o_id",
user_id: "$_id",
result: 1
}
}
])
You need to use $sort first to be able to capture both _id and result from highest result document using $group and $first operators. Output:
{ "result" : 3, "_id" : ObjectId("5c37a7ad54956d08afb590f0"), "user_id" : 1 }
{ "result" : 10, "_id" : ObjectId("5c37a80254956d08afb590f1"), "user_id" : 45 }
I have a schema in which it has some fields..
i am not able to find query for this, i tried $group but was not able to find results
collection: tasks
{
"_id" : ObjectId("5a475ee4b342fa03e71192bd"),
"title" : "Some Title",
"assignedUsers" : [
{
"_id" : ObjectId("5a47386ee4788102e530f60d"),
"name" : "Sam",
"status" : "Unconfirmed"
},
{
"_id" : ObjectId("5a473878e4788102e530f60f"),
"name" : "Ricky",
"status" : "Rejected"
}
{
"_id" : ObjectId("5a47388be4788102e530f611"),
"name" : "Niel",
"status" : "Unconfirmed"
},
{
"_id" : ObjectId("5a47388be4788102e530f611"),
"name" : "ABC",
"status" : "Unconfirmed"
},
{
"_id" : ObjectId("5a473892e4788102e530f612"),
"name" : "Rocky",
"status" : "Rejected"
}
]
}
Result should contain
Unconfirmed=3
Rejected=2
Thanks
Use below query,
db.coll3.aggregate([{
$unwind: '$assignedUsers'
}, {
$group: {
_id: '$assignedUsers.status',
'count': {
$sum: 1
}
}
}
])
If you want to query against a particular document make sure, you use a $match as first stage and then use the other 2 $unwind and $group.
You would get result as
{ "_id" : "Rejected", "count" : 2 }
{ "_id" : "Unconfirmed", "count" : 3 }
Hope this helps.
I am newbie. But I try to learn the most logical ways to write the queries.
Assume I have collection which is as;
{
"id" : NumberInt(1),
"school" : [
{
"name" : "george",
"code" : "01"
},
{
"name" : "michelangelo",
"code" : "01"
}
],
"enrolledStudents" : [
{
"userName" : "elisabeth",
"code" : NumberInt(21)
}
]
}
{
"id" : NumberInt(2),
"school" : [
{
"name" : "leonarda da vinci",
"code" : "01"
}
],
"enrolledStudents" : [
{
"userName" : "michelangelo",
"code" : NumberInt(25)
}
]
}
I want to list occurence of a key with their corresponding code values.
As an example key : michelangelo
To find the occurence of the key, I wrote two differen aggregation queries as;
db.test.aggregate([
{$unwind: "$school"},
{$match : {"school.name" : "michelangelo"}},
{$project: {_id: "$id", "key" : "$school.name", "code" : "$school.code"}}
])
and
db.test.aggregate([
{$unwind: "$enrolledStudents"},
{$match : {"enrolledStudents.userName" : "michelangelo"}},
{$project: {_id: "$id", "key" : "$enrolledStudents.userName", "code" : "$enrolledStudents.code"}}
])
the result of these 2 queries return what I want as;
{ "_id" : 1, "key" : "michelangelo", "code" : "01" }
{ "_id" : 2, "key" : "michelangelo", "code" : 25 }
One of them to search in enrolledStudents, the other one is searching in school field.
Can these 2 queries reduced into more logical query? Or is this the only way to do it?
ps: I am aware that database structure is not logical, but I tried to simulate.
edit
I try to write a query with find.
db.test.find({$or: [{"enrolledStudents.userName" : "michelangelo"} , {"school.name" : "michelangelo"}]}).pretty()
but this returns the whole documents as;
{
"id" : 1,
"school" : [
{
"name" : "george",
"code" : "01"
},
{
"name" : "michelangelo",
"code" : "01"
}
],
"enrolledStudents" : [
{
"userName" : "elisabeth",
"code" : 21
}
]
}
{
"id" : 2,
"school" : [
{
"name" : "leonarda da vinci",
"code" : "01"
}
],
"enrolledStudents" : [
{
"userName" : "michelangelo",
"code" : 25
}
]
}
Mongo 3.4
$match - This stage will keep all the school array and enrolledStudents where there is atleast one embedded document matching both the query condition
$group - This stage will combine all the school and enrolledStudents array to 2d array for each _id in a group.
$project - This stage will $filter the merge array for matching query condition and $map the array to with new labels values array.
$unwind - This stage will flatten the array.
$addFields & $replaceRoot - This stages will add the id field and promote the values array to the top.
db.collection.aggregate([
{$match : {$or: [{"enrolledStudents.userName" : "michelangelo"} , {"school.name" : "michelangelo"}]}},
{$group: {_id: "$id", merge : {$push:{$setUnion:["$school", "$enrolledStudents"]}}}},
{$project: {
values: {
$map:
{
input: {
$filter: {
input: {"$arrayElemAt":["$merge",0]},
as: "onef",
cond: {
$or: [{
$eq: ["$$onef.userName", "michelangelo"]
}, {
$eq: ["$$onef.name", "michelangelo"]
}]
}
}
},
as: "onem",
in: {
key : { $ifNull: [ "$$onem.userName", "$$onem.name" ] },
code : "$$onem.code"}
}
}
}
},
{$unwind: "$values"},
{$addFields:{"values.id":"$_id"}},
{$replaceRoot: { newRoot:"$values"}}
])
Sample Response
{ "_id" : 2, "key" : "michelangelo", "code" : 25 }
{ "_id" : 1, "key" : "michelangelo", "code" : "01" }
Mongo <= 3.2
Replace last two stages of above aggregation with $project to format the response.
{$project: {"_id": 0 , id:"$_id", key:"$values.key", code:"$values.code"}}
Sample Response
{ "_id" : 2, "key" : "michelangelo", "code" : 25 }
{ "_id" : 1, "key" : "michelangelo", "code" : "01" }
You can use $redact instead of $group & match and add $project with $map to format the response.
$redact to go through a document level at a time and perform $$DESCEND and $$PRUNE on the matching criteria.
The only thing to note is usage of $ifNull in the first document level for id so that you can $$DESCEND to embedded document level for further processing.
db.collection.aggregate([
{
$redact: {
$cond: [{
$or: [{
$eq: ["$userName", "michelangelo"]
}, {
$eq: ["$name", "michelangelo"]
}, {
$ifNull: ["$id", false]
}]
}, "$$DESCEND", "$$PRUNE"]
}
},
{
$project: {
id:1,
values: {
$map:
{
input: {$setUnion:["$school", "$enrolledStudents"]},
as: "onem",
in: {
key : { $ifNull: [ "$$onem.userName", "$$onem.name" ] },
code : "$$onem.code"}
}
}
}
},
{$unwind: "$values"},
{$project: {_id:0,id:"$id", key:"$values.key", code:"$values.code"}}
])
I have a collection called "products" which has an array of "bids" objects.
I want to find out the Maximum bid for each product, for this I am aggregating Products on $max with $bids.bidamount field. However this is only giving me the largest bid amount. How do I project all the bid fields for the max aggregation.
Here is a sample document
{
"_id" : ObjectId("58109a5138fe12215cfdc064"),
"product_id" : 2,
"item_name" : "Auction Item1",
"item_description" : "Test",
"seller_name" : "ak#gmail.com",
"item_price" : "20",
"item_quantity" : 7,
"sale_type" : "Auction",
"posted_at" : "2016:10:26 04:58:09",
"expires_at" : "2016:10:30 04:58:09",
"bids" : [
{
"bid_id" : 1,
"bidder" : "ak#gmail.com",
"bid_amount" : 300,
"bit_time" : "2016:10:26 22:36:29"
},
{
"bid_id" : 2,
"bidder" : "ak#gmail.com",
"bid_amount" : 100,
"bit_time" : "2016:10:26 22:37:29"
}
],
"orders" : [
{
"buyer" : "ak#gmail.com",
"quantity" : "2"
},
{
"buyer" : "ak#gmail.com",
"quantity" : "3"
}
]
}
Here is my mongo query:
db.products.aggregate([
{
$project: {
bidMax: { $max: "$bids.bid_amount"}
}
}
])
which gives the following result:
{
"_id" : ObjectId("58109a5138fe12215cfdc064"),
"bidMax" : 300
}
db.products.aggregate([{$unwind:"$bids"},{$group:{_id:"$_id", sum:{$sum:"$bids.bid_amount"}}},{$project:{doc:"$$ROOT", _id:1, sum:1}, {$sort:{"sum":-1}},{$limit:1}]),
which return something like { "_id" : ObjectId("5811b667c50fb1ec88227860"), "sum" : 600, doc:{your document....} }
This should do it:
db.products.aggregate([{
$unwind: '$bids'
}, {
$group: {
_id: '$products_id',
maxBid: {
$max: '$bids.bid_amount'
}
}
}])
db.collectionName.aggregate(
[
{
$group:
{
_id: "$product_id",
maxBidAmount: { $max: "$bids.bid_amount" }
}
}
]
)
Hey use this query, you will get the result.