MongoDB apply iterator with additional query to all results - node.js

Is there anyway within mongo, via MapReduce or Aggregation to apply a second query based on the result set of the first?, such as an Aggregate within an aggregate, or new emit/query within MapReduce.
For example, I have a materialized path pattern of items (which also includes parentId), I can get all of the roots simply by:
db.collection.find({parentId: null}
.toArray(function(err, docs) {
});
What I want to do is determine if these docs have children, just a flag true/false. I can iterate through these docs using async each and check, but on large docs, this is not very performant at all and causes event loop delays, I can use eachSeries, but this is just slow.
Ideally, I'd like to be able to handle this all within Mongo. Any suggestions if that's possible?
Edit, Example collection:
{
_id: 1,
parentId: null,
name: 'A Root Node',
path: ''
}
{
_id: 2,
parentId: 1,
name: 'Child Node A',
path: ',1'
}
{
_id: 3,
parentId: 2,
name: 'Child Node B',
path: ',1,2'
}
{
_id: 4,
parentId: null,
name: 'Another Root Node',
path: ''
}
This basically represents two root nodes, where one root node ({_id: 1}) has two children (one being direct), example:
1
2
3
4
What I would like to do is do a query based on parentId so I can get the root nodes by using null or by passing a parentId I can get the children of that and determine if the result set from this, any of the items contain children, example response for where {parentId: null}:
[{
_id: 1,
parentId: null,
name: 'A Root Node',
path '',
hasChildren: true
},
{
_id: 4,
parentId: null,
name: 'Another Root Node',
path '',
hasChildren: false
}]

You could try creating an array of the parentIds from the materialized paths that you can then use in the aggregation pipeline to project the extra field/flag hasChildren.
This can be done by using the map() method on the cursor returned from the find() method. The following illustrates this:
var arr = db.collection.find({ "parentId": { "$ne": null } })
.map(function (e){ return e.path; })
.join('')
.split(',')
.filter(function (e){ return e; })
.map(function (e){ return parseInt(e); }),
parentsIds = _.uniq(arr); /* using lodash uniq method to return a unique array */
Armed with this array of parentIds, you can then use the aggregation framework in particular the $project pipeline which makes use of the the set operator $setIsSubset which takes two arrays and returns true when the first array is a subset of the second, including when the first array equals the second array, and false otherwise:
db.collection.aggregate([
{
"$match": {
"parentId": null
}
},
{
"$project": {
"parentId": 1,
"name": 1,
"path": 1,
"hasChildren": { "$setIsSubset": [ [ "$_id" ], parentIds ] }
}
}
], function (err, res) { console.log(res); });

Related

mongoDB projection on array in an object

I have this document structure in the collection:
{"_id":"890138075223711744",
"guildID":"854557773990854707",
"name":"test-lab",
"game": {
"usedWords":["akşam","elma","akım"]
}
}
What is the most efficient way to get its fields except the array (it can be really large), and at the same time, see if an item exists in the array ?
I tried this:
let query = {_id: channelID}
const options = { sort: { name: 1 }, projection: { name: 1, "game.usedWords": { $elemMatch: { word}}}}
mongoClient.db(db).collection("channels").findOne(query, options);
but I got the error: "$elemMatch can not be used on nested fields"
If I've understood correctly you can use this query:
Using positional operator $ you can return only the matched word.
db.collection.find({
"game.usedWords": "akşam"
},
{
"name": 1,
"game.usedWords.$": 1
})
Example here
The output is only name and the matched word (also _id which is returned by default)
[
{
"_id": "890138075223711744",
"game": {
"usedWords": [
"akşam"
]
},
"name": "test-lab"
}
]

mongoose: sort and paginating the field inside $project

$project: {
_id: 1,
edited: 1,
game: {
gta: {
totalUserNumber: {
$reduce: {
input: "$gta.users",
initialValue: 0,
in: { $add: [{ $size: "$$this" }, "$$value"] },
},
},
userList: "$gta.users", <----- paginating this
},
DOTA2: {
totalUserNumber: {
$reduce: {
input: "$dota2.users",
initialValue: 0,
in: { $add: [{ $size: "$$this" }, "$$value"] },
},
},
userList: "$dota2.users", <------ paginating this
},
},
.... More Games
},
I have this $project. I have paginated the list of games by using $facet,$sort, $skip and $limit after $project.
I am trying also trying to paginate each game's userList. I have done to get the total value in order to calculate the page number and more.
But, I am struggling to apply $sort and $limit inside the $project. So far, I have just returned the document and then paginated with the return value. However, I don't think this is very efficient and wondering if there is any way that I can paginate the field inside the $project.
Is there any way that I can apply $sort and $limit inside the $project, in order to apply pagination to the fields and return?
------ Edit ------
this is for paginating the field. Because, I am already paginating the document (game list), I could not find any way that I can paginate the field, because I could not find any way that I can apply $facet to the field.
e.g. document
[
gta: {
userID: ['aa', 'bb', 'cc' ......],
},
dota: {
userID: ['aa', 'bb', 'cc' ......],
}
....
]
I am using $facet to paginate the list of games (dota, gta, lol and more). However, I did not want to return all the userID. I had to return the entire document and then paginate the userID to replace the json doc.
Now, I can paginate the field inside the aggregate pipeline by using $function.
thanks to Mongodb sort inner array !
const _function = function (e) {
e // <---- will return userID array. You can do what you want to do.
return {
};
};
game
.collection("game")
.aggregate([
{},
{
$set: {
"game": {
$function: {
body: _function,
args: ["$userID"],
lang: "js",
},
},
},
},
])
.toArray();
By using $function multiple time, you will be able to paginate the field. I don' really know if this is faster or not tho. Plus, make sure you can use $function. I read that you can't use this if you are on the free tier at Atlas.
What you are looking for is the $slice Operator.
It requires three parameters.
"$slice": [<Array>, <start-N>, <No-Of.elements to fetch>]
userList: {"$slice": ["$dota2.users", 20, 10]} // <-- Will ignore first 20 elements in array and gets the next 10

MongoDB assymetrical return of data, first item in array returned in full, the rest with certain properties omitted?

I'm new to MongoDB and getting to grips with its syntax and capabilities. To achieve the functionality described in the title I believe I can create a promise that will run 2 simultaneous queries on the document - one to get the full content of one item in the array (or at least the data that is omitted in the other query, to re-add after), searched for by most recent date, the other to return the array minus specific properties. I have the following document:
{
_id : ObjectId('5rtgwr6gsrtbsr6hsfbsr6bdrfyb'),
uuid : 'something',
mainArray : [
{
id : 1,
title: 'A',
date: 05/06/2020,
array: ['lots','off','stuff']
},
{
id : 2,
title: 'B',
date: 28/05/2020,
array: ['even','more','stuff']
},
{
id : 3,
title: 'C',
date: 27/05/2020,
array: ['mountains','of','knowledge']
}
]
}
and I would like to return
{
uuid : 'something',
mainArray : [
{
id : 1,
title: 'A',
date: 05/06/2020,
array: ['lots','off','stuff']
},
{
id : 2,
title: 'B'
},
{
id : 3,
title: 'C'
}
]
}
How valid and performant is the promise approach versus constructing one query that would achieve this? I have no idea how to perform such 'combined-rule'/conditions in MongoDB, if anyone could give an example?
If your subdocument array you want to omit is not very large. I would just remove it at the application side. Doing processing in MongoDB means you choose to use the compute resources of MongoDB instead of your application. Generally your application is easier and cheaper to scale, so implementation at the application layer is preferable.
But in this exact case it's not too complex to implement it in MongoDB:
db.collection.aggregate([
{
$addFields: { // keep the first element somewhere
first: { $arrayElemAt: [ "$mainArray", 0] }
}
},
{
$project: { // remove the subdocument field
"mainArray.array": false
}
},
{
$addFields: { // join the first element with the rest of the transformed array
mainArray: {
$concatArrays: [
[ // first element
"$first"
],
{ // select elements from the transformed array except the first
$slice: ["$mainArray", 1, { $size: "$mainArray" }]
}
]
}
}
},
{
$project: { // remove the temporary first elemnt
"first": false
}
}
])
MongoDB Playground

Mongoose get document as an array of arrays based on a key

I don't really know how to frame the question but what I have is the following schema in mongoose
new Schema({
gatewayId: { type: String, index: true },
timestamp: { type: Date, index: true },
curr_property:Number,
curr_property_cost:Number,
day_property:Number,
day_property_cost: Number,
curr_solar_generating: Number,
curr_solar_export:Number,
day_solar_generated:Number,
day_solar_export:Number,
curr_chan1:Number,
curr_chan2:Number,
curr_chan3:Number,
day_chan1:Number,
day_chan2:Number,
day_chan3:Number
},{
collection: 'owlelecmonitor'
});
and I want to be able to query all the documents in the collection but the data should be arranged inside the array in the following format
[ [{
gatewayId: 1,
timestamp: time
....
},
{
gatewayId: 1,
timestamp: time2
....
}],
[{
gatewayId: 2,
timestamp: time
....
},
{
gatewayId: 2,
timestamp: time2
....
}],
[{
gatewayId: 3,
timestamp: time
....
},
{
gatewayId: 3,
timestamp: time2
....
}]
];
Is there a way that I can do this in mongoose instead of retrieving the documents and processing them again ?
Yes, it's possible. Consider the following aggregation pipeline in mongo shell. This uses a single pipeline stream comprising of just the $group operator, grouping all the documents by gatewayId and creating another array field that holds all the grouped documents. This extra field uses the accumulator operator $push on the system variable $$ROOT which returns the root document, i.e. the top-level document, currently being processed in the aggregation pipeline stage.
With the cursor returned from the aggregate() method, you can then use its map() method to create the desired final array. The following mongo shell demonstration describes the above concept:
var result = db.owlelecmonitor.aggregate([
{
"$group": {
"_id": "$gatewayId",
"doc": {
"$push": "$$ROOT"
}
}
}
]).map(function (res){ return res.doc; });
printjson(result);
This will output to shell the desired result.
To implement this in Mongoose, use the following aggregation pipeline builder:
OwlelecMonitorModel
.aggregate()
.group({
"_id": "$gatewayId",
"doc": {
"$push": "$$ROOT"
}
})
.exec(function (err, result) {
var res = result.map(function (r){return r.doc;});
console.log(res);
});

Execute query and grab result in two subsections mongodb

I'm using mongoose to deal with my database.
I have the following model:
var DeviceSchema = mongoose.Schema({
type: Number,
pushId: String
});
The type attribute can be either 0 or 1.
I want to execute a query that grab all documents and retrieve the result in the following format:
{
fstType: [
{
_id: "545d533e2c21b900000ad234",
type: 0,
pushId: "123"
},
{
_id: "545d533e2c21b900000ad235",
type: 0,
pushId: "124"
},
],
sndType: [
{
_id: "545d533e2c21b900000ad236",
type: 1,
pushId: "125"
},
{
_id: "545d533e2c21b900000ad237",
type: 1,
pushId: "126"
},
]
}
Is that possible? I want to do that in one single query.
Thanks.
Is that possible? I want to do that in one single query.
Yes. It is possible. You can achieve the desired result, through the following aggregation pipeline operations.
Sort by the type parameter in ascending order.
Group records together having the same type, construct an array of
documents for each group. After this stage, only two records will be
present, each with an attribute called items, which is an array of
records for each group.
Since our records are sorted by type, the first group will contain
records with type 0, and the second with type 1.
At last we merge the groups and give them each a name, based on their type.
var model = mongoose.model('collection',DeviceSchema);
model.aggregate([
{$sort:{"type":-1}},
{$group:{"_id":"$type",
"items":{$push:"$$ROOT"},
"type":{$first:"$type"}}},
{$project:{"items":{$cond:[{$eq:["$type",0]},
{"firstType":"$items"},
{"secondType":"$items"}]}}},
{$group:{"_id":null,
"firstType":{$first:"$items.firstType"},
"secondType":{$last:"$items.secondType"}}},
{$project:{"_id":0,"firstType":1,"secondType":1}}
], function (err, result) {
if (err) {
console.log(err);
return;
}
console.log(result);
});
o/p:
{ firstType:
[ { _id: '545d533e2c21b900000ad234', type: 0, pushId: '123' },
{ _id: '545d533e2c21b900000ad235', type: 0, pushId: '124' } ],
secondType:
[ { _id: '545d533e2c21b900000ad236', type: 1, pushId: '125' },
{ _id: '545d533e2c21b900000ad237', type: 1, pushId: '126' } ] }

Resources