Convert mongo query to aggregate query? - node.js

I am a newbie to MongoDB. I have a use case where the mongo query should be converted to aggregate query.
I have the following two collections:
items: {_id: "something", "name": "raj", "branch": "IT", subItems: ["subItemId","subItemId"]}
subItems: {_id: "something", "city": "hyd", list: ["adsf","adsf"]}
Query passed by the user is:
{
"query": { "name": "raj", "subItems.loc": "hyd" },
"sort": { "name": 1 },
"projection": { "_id": 0, "branch": 1, "subItems.loc": 1 }
}
I am able to create a dynamic query in a following way:
let itemConditions: any[] = [];
let subItemConditions: any[] = [];
let itemProjection: {[k:string]: any} = {};
let subItemProjection: {[k:string]: any} = {};
subItemConditions.push({$in: ["$_id", "$$subItems"]}); // Default condition to search files in the subItems collection
let isAnysubItemProj = false;
Object.keys(reqQuery).forEach(prop => {
let value = reqQuery[prop];
if(prop.includes('subItems.')) {
key = prop.split(".")[1];
if(key === '_id') value = new ObjectId(value);
subItemConditions.push({$eq: [`$${prop}`, value]});
return;
}
itemConditions.push({$eq: [`$${prop}`, value]});
});
if(config.projection)
Object.keys(config.projection).forEach(prop => {
if(prop.includes('subItems.')) {
isAnysubItemProj = true;
const key = prop.split(".")[1];
subItemProjection[key] = config.projection[prop];
return;
}
itemProjection[prop] = config.projection[prop];
});
if(isAnysubItemProj) itemProjection['subItems'] = 1;
let subItemPipeline: any[] = [];
subItemPipeline.push(
{ $match: {
$expr: {
$and: subItemConditions
}
}
});
if(Object.keys(subItemProjection).length)
subItemPipeline.push({$project: subItemProjection});
let query: any[] = [
{
$match: {
$expr : {
$and: itemConditions
}
}
},
{
$addFields: {
subItems: {
$map: {
input: "$subItems",
as: "id",
in: { $toObjectId: "$$id" }
}
}
}
},
{
$lookup: {
from: "subItems",
let: {subItems: "$subItems"},
pipeline: subItemPipeline,
as: "subItems"
}
}
];
if(config.sort && Object.keys(config.sort).length) query.push({$sort: config.sort});
if(Object.keys(itemProjection).length) query.push({$project: itemProjection});
const items = await collection.aggregate(query).toArray();
The above code will work only for the comparison of equality for items and subItems separately, but the user can send different types of queries like:
{
"query": { $or: [{"name": "raj"}, {subItems: {$gt: { $size: 3}}}], "subItems.city": "hyd" },
"sort": { "name": 1 },
"projection": { "_id": 0, "branch": 1, "subItems.loc": 1 }
}
{
"query": { $or: [{"name": "raj"}, {"subItems.city": {"$in" : ["hyd", "ncr"]}}], "subItems.list": {"$size": 2} },
"sort": { "name": 1 },
"projection": { "_id": 0, "branch": 1, "subItems.loc": 1 }
}
Is there any easy way to convert this normal MongoDB query into an aggregate query or is there any other approach to implement this...??
I am trying to modify the above dynamic query to work for any queries passed by the user but it is becoming difficult to handle all queries.
Is there any better approach to handle this situation like changing the query passed by the user or the way of handling it on the server-side or how I should change my code to support all types of queries passed by the user..??
Any help will be appreciated

If this is really your input
input = {
"query": { "name": "raj", "subItems.loc": "hyd" },
"sort": { "name": 1 },
"projection": { "_id": 0, "branch": 1, "subItems.loc": 1 }
}
Then the aggregation pipeline would be simply:
let pipeline = [
{$match: input.query},
{$sort: input.sort},
{$project: input.projection}
]
db.collection.aggregate(pipeline)

Related

Nodejs Mongoose | Fetch the array of values for a given field

From the query below
let fields = { 'local.email': 1 };
UserModel.find({ '_id': { $in: userIds } }).select(fields).setOptions({ lean: true });
Result which we get is
[
{
"_id": "54bf2d7415eaaa570c9ed5a0",
"local": {
"email": "neo#q.com"
}
},
{
"_id": "54bfb753e4c9406112267056",
"local": {
"email": "test#q.com"
}
}
]
Is is possible to modify query itself to get below result
["neo#q.com", "test#q.com"]
Thanks in advance
You could use aggregate to return a list of objects with the emails and the map them to an array of strings:
const emailObjs = await UserModel.aggregate([
{
$match: {
_id: {
$in: userIds
}
}
},
{
$project: {
"_id": 0,
"email": "$local.email"
}
}
]);
const emails = emailObjs.map(obj => obj.email)
Link to playground for the query.

MongoDB: Filter by element of object in a subdocument array in aggregate function

Due to some changes to the schema, I've had a to do refactoring that's broken what was a simple filter in an application, in this instance is isToRead while everything else continues to work.
The document in "Assets" that should be appearing is:
{
"_id": {
"$oid": "ID"
},
"userId": "ID",
"folderId": "ID",
"title": "Title",
"note": "<p><strong>Note.</strong></p>",
"typeOfAsset": "web",
"isFavourite": false,
"createdAt": {
"$date": {
"$numberLong": "1666702053399"
}
},
"updatedAt": {
"$date": {
"$numberLong": "1666702117855"
}
},
"isActive": 3,
"tags": [],
"traits": [
{
"$oid": "6357dae53298948a18a17c60"
}
]
"__v": 0
}
… and the reference document in "Assets_Traits" that I'm attempting to filter against should be:
{
"_id": {
"$oid": "6357dae53298948a18a17c60"
},
"userId": "ID",
"numberOfViews": 1,
"isToRead": true,
"__v": 0
}
I'll share the entire method, which includes the various attempts that — for whatever reason — won't work.
let tags = args.tags ? args.tags.split(',') : []
let tagsToMatch = []
if (tags.length > 0) {
tags.forEach(tag => {
tagsToMatch.push(new mongoose.Types.ObjectId(tag))
})
}
let search = {
...(args.phraseToSearch.length > 0 && {
$search: {
index: 'assets',
compound: {
must: [{
phrase: {
query: args.phraseToSearch,
path: 'title',
slop: 2,
score: { boost: { value: 3 } }
}
}],
should: [{
phrase: {
query: args.phraseToSearch,
path: 'note',
slop: 2
}
}]
}
}
})
}
let project = {
$project: {
_id: 0,
id: '$_id',
userId: 1,
folderId: 1,
title: 1,
note: 1,
typeOfAsset: 1,
isFavourite: 1,
createdAt: 1,
updatedAt: 1,
isActive: 1,
attributes: 1,
preferences: 1,
// ...(args.typeOfAttribute === 'isToRead' && {
// traits: {
// $filter: {
// input: "$traits",
// cond: { $eq: [ "$$this.isToRead", true ] }
// }
// }
// }),
tags: 1,
traits: 1,
score: {
$meta: 'searchScore'
}
}
}
let match = {
$match: {
userId: args.userId,
typeOfAsset: {
$in: args.typeOfAsset === 'all' ? MixinAssets().all : [args.typeOfAsset] // [ 'file', 'folder', 'message', 'note', 'social', 'web' ]
},
...(tagsToMatch.length > 0 && {
tags: {
$in: tagsToMatch
}
}),
...(args.typeOfAttribute === 'isToRead' && {
// $expr: {
// $allElementsTrue: [{
// $map: {
// input: '$traits',
// as: 't',
// in: {
// $and: [
// { $eq: [ "$$t.userId", args.userId ] },
// { $eq: [ "$$t.isToRead", true ] }
// ]
// }
// }
// }]
// }
// $expr: {
// $filter: {
// input: "$traits",
// cond: {
// $and: [
// { $eq: [ "$$this.userId", args.userId ] },
// { $eq: [ "$$this.isToRead", true ] }
// ]
// }
// }
// }
}),
isActive: 3
}
}
let lookup = {}
switch (args.typeOfAttribute) {
case 'areFavourites':
match.$match.isFavourite = true
break
case 'isToRead':
// match.$match.traits = {
// userId: args.userId,
// isToRead: true
// }
// match.$match.traits = {
// $elemMatch: {
// userId: args.userId,
// isToRead: true
// }
// }
// lookup = {
// $lookup: {
// from: 'assets_traits',
// let: { isToRead: '$isToRead' },
// pipeline: [{
// $match: {
// $expr: {
// $eq: [ '$isToRead', true ]
// }
// },
// }],
// localField: 'userId',
// foreignField: 'userId',
// as: 'traits'
// }
// }
break
case 'inTrash':
match.$match.isActive = 2
break
}
let skip = {
$skip: args.skip
}
let limit = {
$limit: args.first
}
let group = {
$group: {
_id: null,
count: { $sum: 1 }
}
}
let sort = {
$sort: {
[args.orderBy]: args.orderDirection === 'asc' ? 1 : -1
}
}
console.info('Queries:getAllAssetsForNarrative()', match.$match)
let allAssets = await Models.Assets.schema.aggregate(
(search.hasOwnProperty('$search')) // Order of criteria is critical to the functioning of the aggregate method.
? [search, project, match, sort, skip, limit]
: [match, project, sort, skip, limit]
// : [match, project, { $unwind: '$traits' }, { $match: { traits: { $elemMatch: { isToRead: true } } } }, sort, skip, limit]
)
let [ totalNumberOfAssets ] = await Models.Assets.schema.aggregate(
(search.hasOwnProperty('$search')) // Order of criteria is critical to the functioning of the aggregate method.
? [search, project, match, group]
: [match, project, group]
// : [match, project, { $unwind: '$traits' }, { $match: { traits: { $elemMatch: { isToRead: true } } } }, group]
)
await (() => {
if (args.phraseToSearch.length > 0) {
const SearchFactory = require('../services/search/search')
const Search = SearchFactory(Models)
Search.insertRecentSearch({
userId: args.userId,
phraseToSearch: args.phraseToSearch.toLowerCase()
})
}
})()
I removed lookup in the final two arrays for the aggregate function because it was becoming too complicated for to me understand what was happening.
Weird thing is, "Tags" match and it's also a reference, while "Assets_Traits" won't return or do anything.
The values for typeOfAsset are: [ 'file', 'folder', 'message', 'note', 'social', 'web' ]
While 'All Assets' is chosen, choosing 'To Read' performs a filter against all types of Assets, and additional filtering would happen when a specific type of Asset is chosen — as explained, this worked before the changes to the schema.
Also, ignore tags because those aren't in use here.
Thoughts appreciated!
You did not provide sample of your input (args) or the constants you use (for example MixinAssets().all which i'm suspecting is problematic).
I constructed my own input for the sake of this answer:
const args = {
typeOfAsset: 'isToRead',
typeOfAttribute: "isToRead",
tagsToMatch: ["tag1", "tag2"],
skip: 0,
first: 1,
orderBy: "_id",
orderDirection: "desc"
}
This produces the following pipeline (using your code):
db.Assets.aggregate([
{
"$match": {
"userId": "123",
"typeOfAsset": {
"$in": [
"isToRead"
]
},
"tags": {
"$in": [
"tag1",
"tag2"
]
},
"isActive": 3
}
},
{
"$project": {
"_id": 0,
"id": "$_id",
"userId": 1,
"folderId": 1,
"title": 1,
"note": 1,
"typeOfAsset": 1,
"isFavourite": 1,
"createdAt": 1,
"updatedAt": 1,
"isActive": 1,
"attributes": 1,
"preferences": 1,
"tags": 1,
"traits": 1,
"score": {
"$meta": "searchScore"
}
}
},
{
"$sort": {
"_id": -1
}
},
{
"$skip": 0
},
{
"$limit": 1
}
])
Which works, as you can see in this Mongo Playground sample.
So what is your issue? As I mentioned I suspect one issue is the MixinAssets().all if args.typeOfAsset === 'all' then you use that value, now if it's an array the match condition looks like this:
typeOfAsset: {
$in: [['web', '...', '...']]
}
This won't match anything as it's an array of arrays, if it's a constant value then again it won't match as the type in the db is different.
I will give one more tip, usually when you want to build a pagination system like this and need both the results and totalResultCount it's common practice to use $facet this way you don't have to execute the pipeline twice and you can improve perfomance, like so:
db.Assets.aggregate([
{
"$match": {
"userId": "123",
"typeOfAsset": {
"$in": [
"isToRead"
]
},
"tags": {
"$in": [
"tag1",
"tag2"
]
},
"isActive": 3
}
},
{
$facet: {
totalCount: [
{
$group: {
_id: null,
count: {
$sum: 1
}
}
}
],
results: [
{
"$project": {
"_id": 0,
"id": "$_id",
"userId": 1,
"folderId": 1,
"title": 1,
"note": 1,
"typeOfAsset": 1,
"isFavourite": 1,
"createdAt": 1,
"updatedAt": 1,
"isActive": 1,
"attributes": 1,
"preferences": 1,
"tags": 1,
"traits": 1,
"score": {
"$meta": "searchScore"
}
}
},
{
"$sort": {
"_id": -1
}
},
{
"$skip": 0
},
{
"$limit": 1
}
]
}
}
])
Mongo Playground

How to get particular details from nested object from MongoDB

I'm saving data for a NestJs based web app in MongoDB.
My MongoDB Data looks like this
"gameId": "1a2b3c4d5e"
"rounds": [
{
"matches": [
{
"match_id": "1111abc1111",
"team1": {
"team_id": "team8",
"score": 0
},
"team2": {
"team_id": "team2",
"score": 0
}
},
{
"match_id": "2222abc2222",
"team1": {
"team_id": "team6",
"score": 0
},
"team2": {
"team_id": "team5",
"score": 0
}
},
]
}
]
Here we have gameId for each game and inside each game, there are many rounds and many matches. Each match has match_id. How can I get a particular match info and edit it based on gameId & match_id?
(N.B: I'm willing to update score based on match_id)
I've tried something like this
const matchDetails = await this.gameModel.findOne({
gameId: gameId,
rounds: { $elemMatch: { match_id: match_id } },
});
But this doesn't work and returns null. How to do this correctly?
The problem is that you're applying the elemMatch on the rounds array, but it should be on rounds.matches. Changing your query to the following will fix the problem:
const matchDetails = await this.gameModel.findOne({
gameId: gameId,
"rounds.matches": { $elemMatch: { match_id: match_id } },
});
EDIT:
To only get a specific matching element, you can use a simple aggregation with $unwind and $filter:
db.collection.aggregate([
{
"$match": {
"gameId": gameId,
"rounds.matches": { $elemMatch: { match_id: match_id } }
}
},
{
"$unwind": "$rounds"
},
{
$project: {
match: {
$filter: {
input: "$rounds.matches",
as: "match",
cond: {
$eq: [
"$$match.match_id",
match_id
]
}
}
},
_id: 0
}
}
])
Example on mongoplayground.

Mongoose Aggregate match if an Array contains any value of another Array

I am trying to match by comparing the values inside the values of another array, and return it if any one of the values in the 1st array match any one value of the 2nd array. I have a user profile like this
{
"user": "bob"
"hobbies": [jogging]
},
{
"user": "bill"
"hobbies": [reading, drawing]
},
{
"user": "thomas"
"hobbies": [reading, cooking]
}
{
"user": "susan"
"hobbies": [coding, piano]
}
My mongoose search query as an example is this array [coding, reading] (but could include any hobby value) and i would like to have this as an output:
{
"user": "bill"
"hobbies": [reading, drawing]
},
{
"user": "thomas"
"hobbies": [reading, cooking]
}
{
"user": "susan"
"hobbies": [coding, piano]
}
I tried:
{"$match": {"$expr": {"$in": [searchArray.toString(), "$hobbies"]}}}
but this only works aslong as the search array has only one value in it.
const searchArray = ["reading", "coding"];
const orArray = searchArray.map((seachValue) => {
return {
hobies: searchValue,
}
});
collection.aggregate([{
$match: { $or: orArray }
}])
The query:
db.collection.aggregate([
{
$match: {
"$or": [
{
hobbies: "reading"
},
{
hobbies: "coding"
}
]
}
}
])
Or you can use another way, no need to handle arr:
db.collection.aggregate([
{
$match: {
hobbies: {
$in: [
"reading",
"coding"
]
}
}
}
])
Run here

how to use $count and $project together in node js mongoose aggregate

I am trying to get some filtered data and it's total count. I want to do both job within single query, so how can I do this. Below is my code.
var SubId = 1;
var TypeId = 1;
var lookup = {
$lookup:
{
from: 'sub_types',
localField: 'sub_id',
foreignField: 'sub_id',
as: 'sub_category'
}
};
var unwind = { $unwind: "$sub_category" };
var project = {
"ques_id": 1,
"ques_txt": 1,
"ans_txt": 1,
"ielts_sub_id": 1,
"ielts_tags_id": 1,
};
var match = {
"sub_category.type_id": parseInt(TypeId),
"sub_category.sub_id": parseInt(SubId),
"status": 1
};
ieltsmongoose.collection('ques').aggregate([
lookup, unwind,
{
$match: match
},
{
$project: project
}
]).limit(max_row).toArray(async function (error, Ques) {....});
Now I want to get count with this same query like
{
$count: "totalcount"
},
You can use another aggregate stage called group like this to achieve your goal:
{ "$group": {
"_id": null,
"count": { "$sum": 1},
"data": { "$push": "$$ROOT" }
}}
your aggregation would be like :
ieltsmongoose.collection('ques').aggregate([
lookup, unwind,
{
$match: match
},
{
$project: project
},
{ "$group": {
"_id": null,
"count": { "$sum": 1},
"data": { "$push": "$$ROOT" }
}
}
])
also you should use limit and offset as a aggregation stage with $limit and $skip

Resources