Related
I am using aggregation query to retrieve 20000 records. while retrieving it is taking much time. I will mention my query below, Please help me to improve the query performance.
Query:
[err, data] = await to(LeadsLog.aggregate([
{$lookup:{
from: "leads",
localField: "leadId",
foreignField: "_id",
as: "leadId"
}},
{$lookup:{
from: "company_contacts",
localField: "leadId.assignedTo",
foreignField: "_id",
as: "assignedTo"
}},
{
$unwind:{
path: "$leadId",
preserveNullAndEmptyArrays: true
}
},
{
$match:{"leadId.assignedTo":new mongoose.Types.ObjectId(userId),
"result":{$eq:null}}
},
{ '$facet' : {
metadata: [ { $count: "total" }, { $addFields: { page: 1 } } ],
data: [ { $skip: 0 }, { $limit: 20000 } ]
} }
] ));
LeadId:
{
"_id" : ObjectId("617a84b401c98424e00d1310"),
"status" : true,
"address" : "Howmif Trail",
"city" : "Kinawnet",
"state" : "LA",
"country" : "LA",
"pincode" : null,
"extraFormObject" : null,
"lead_name" : "Jayden",
"phone" : "(524) 387-4912",
"email" : "niligis#taptehe.cm",
"company" : ObjectId("6155c2758609663d10fff796"),
"createdBy" : ObjectId("6155c2758609663d10fff798"),
"createdAt" : ISODate("2021-10-28T11:08:40.433Z"),
"updatedAt" : ISODate("2021-10-30T04:43:49.490Z")
}
LeadLog:
{
"_id" : ObjectId("617a84bf01c98424e00daf52"),
"callLogId" : null,
"result" : null,
"assignedTo" : ObjectId("6155c2758609663d10fff798"),
"extraFormObject" : null,
"subResult" : null,
"apptDate" : null,
"nextcallDate" : ISODate("2021-10-28T11:02:50.516Z"),
"callDate" : null,
"leadId" : ObjectId("617a84b401c98424e00d1310"),
"company" : ObjectId("6155c2758609663d10fff796"),
"createdAt" : ISODate("2021-10-28T11:08:50.962Z"),
"updatedAt" : ISODate("2021-10-30T04:43:50.281Z")
}
Please help me with better solution. thank you.
There are a few simple tweaks that you can improve your existing query:
make intermediate result as small as possible; one of the common ways is pushing $match stages as early as possible
use Pipeline Coalescence Optimization as much as possible; one of the common tuples would be $lookup + $unwind combination
index the $match fields and $lookup fields
Based on the first 2 points, here is my suggested form of your query:
You can see result : {$eq: null} is pushed to first stage. The performance gain will depends on the selectivity of the clause.
the $lookup and $unwind leads are grouped together to utilize the coalescence optimization.
"leadId.assignedTo": new mongoose.Types.ObjectId(userId) is moved earlier to minimize intermediate result size
Don't forget to index the relevant $match fields and $lookup fields. From my personal experience, good usage of index will help most with the performance.
[err, data] = await to(LeadsLog.aggregate([
{
$match: {
"result": {
$eq: null
}
}
},
{
$lookup: {
from: "leads",
localField: "leadId",
foreignField: "_id",
as: "leadId"
}
},
{
$unwind: {
path: "$leadId",
preserveNullAndEmptyArrays: true
}
},
{
$match: {
"leadId.assignedTo": new mongoose.Types.ObjectId(userId)
}
},
{
$lookup: {
from: "company_contacts",
localField: "leadId.assignedTo",
foreignField: "_id",
as: "assignedTo"
}
},
{
"$facet": {
metadata: [
{
$count: "total"
},
{
$addFields: {
page: 1
}
}
],
data: [
{
$skip: 0
},
{
$limit: 20000
}
]
}
}
]));
I am trying to find documents in a collection, but filtered based on the value of an embedded ObjectID relation.
Mongoose schema is as follows:
const UserQualificationSchema = new Schema(
{
user: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User',
},
}
const UserSchema = new Schema(
{
fleet: {
type: [String], // Eg ["FleetA", "FleetB", "FleetC"]
required: true,
}
}
I need to find all UserQualifications where an item in the user's fleet equals a value in a filter array.
For example: Find all User Qualifications where user.fleet: {$in: ["FleetA", "FleetC"]}
I've looked at aggregations and querying inside .populate() but can't seem to get it to work.
Any ideas much appreciated.
Use aggregation query for your problem, I have created a query for you.
users.collection.json
/* 1 */
{
"_id" : ObjectId("61056c4a8cca27df3db2e4c8"),
"firstName" : "Rahul",
"lastName" : "soni",
"fleet" : [
"FleetA",
"FleetB",
"FleetC"
],
"createdAt" : ISODate("2021-07-31T15:29:14.918Z")
}
userqualifications.collection.json
/* 1 */
{
"_id" : ObjectId("61056c908cca27df3db2e4c9"),
"user" : ObjectId("61056c4a8cca27df3db2e4c8"),
"createdAt" : ISODate("2021-07-31T15:30:24.510Z")
}
aggregation query:
it will get the result only if a user has FleetA and FleetC.
if anyone is not matched then it will return 0 records
db.userqualifications.aggregate([{
"$lookup": {
"from": "users",
"localField": "user",
"foreignField": "_id",
"as": "user"
}
}, {
"$unwind": "$user"
}, {
"$match": {
"user.fleet": {
"$elemMatch": {
"$eq": "FleetA",
"$eq": "FleetC"
}
}
}
}])
Result:
/* 1 */
{
"_id" : ObjectId("61056c908cca27df3db2e4c9"),
"user" : {
"_id" : ObjectId("61056c4a8cca27df3db2e4c8"),
"firstName" : "Rahul",
"lastName" : "soni",
"fleet" : [
"FleetA",
"FleetB",
"FleetC"
],
"createdAt" : ISODate("2021-07-31T15:29:14.918Z")
},
"createdAt" : ISODate("2021-07-31T15:30:24.510Z")
}
if the goal is to get only UserQualifications then the following should be an efficient way as it can use an index on the fleet field of the User collection.
db.User.aggregate([
{
$match: {
fleet: { $in: ["FleetA", "FleetB"] }
}
},
{
$lookup: {
from: "UserQualification",
localField: "_id",
foreignField: "user",
as: "qualifications"
}
},
{
$unwind: "$qualifications"
},
{
$replaceWith: "$qualifications"
}
])
on the other hand if you start from the UserQualifications collection, you can't efficiently narrow down the records as you're filtering on something that it doesn't have the data for.
Thank you for the answer - it did achieve the results I was looking for - however I am now struggling to add a $match with $and to the aggregate to only return the qualifications where the user ID equals one inside a submitted array AND a given fleet.
I have the following aggregate:
db.UserQualifications.aggregate([{
{
$lookup: {
from: 'users',
localField: 'user',
foreignField: '_id',
as: 'user',
},
},
{
$unwind: '$user',
},
{
$match: {
$and: [
'user.fleet': {
$in: ["Fleet A", "Fleet C"], // This works on it's own
},
user: { // Also tried 'user._id'
$in: ["6033e4129070031c07fbbf29"] // Adding this returns blank array
}
]
},
}
}])
I have double checked that I am passing in the correct User ID's inside the arrays, but when I add this to the $and inside match, it only returns a blank array.
Is there another way to do this?
// Updated users collection
/* 1 */
{
"_id" : ObjectId("61056c4a8cca27df3db2e4c8"),
"firstName" : "Rahul",
"lastName" : "soni",
"fleet" : [
"Fleet A",
"Fleet B",
"Fleet C"
],
"createdAt" : ISODate("2021-07-31T15:29:14.918Z")
}
Query:
// userqualifications => this is my collection name, you can add your collection name here db.<YOUR>
db.userqualifications.aggregate([{
$lookup: {
from: 'users',
localField: 'user',
foreignField: '_id',
as: 'user',
},
},
{
$unwind: '$user',
},
{
$match: {
// $and operatory syntax [{}, {}]
$and: [{
'user.fleet': {
// "Fleet A", "Fleet C" (FleetA, FleetC) this is the my first options,
// I have changes according to your problem
$in: ["Fleet A", "Fleet C"], // This works on it's own
}
}, {
// Convert user id to ObjectId type (_bsonType)
"user._id": ObjectId("61056c4a8cca27df3db2e4c8")
}]
}
}
])
Result:
/* 1 */
{
"_id" : ObjectId("61056c908cca27df3db2e4c9"),
"user" : {
"_id" : ObjectId("61056c4a8cca27df3db2e4c8"),
"firstName" : "Rahul",
"lastName" : "soni",
"fleet" : [
"Fleet A",
"Fleet B",
"Fleet C"
],
"createdAt" : ISODate("2021-07-31T15:29:14.918Z")
},
"createdAt" : ISODate("2021-07-31T15:30:24.510Z")
}
Difference Image:
I have a query that is running fine, now i have requirement to filter some data that is inside array. I don't know how to do that. Below is my code. Please guide me where i am wrong.
Request data
[
'Online Casino/Betting',
'Other ',
'Prefer not to say ',
'Do you really care ? :) ',
'Spend time with friends'
]
Database Data
"interests" : [
{
"name" : "Computers/internet",
"_id" : ObjectId("60752406d8e7213e6b5306de"),
"id" : NumberInt(1)
},
{
"name" : "Astrology/Spiritualism",
"_id" : ObjectId("60752406d8e7213e6b5306df"),
"id" : NumberInt(3)
},
{
"name" : "Cars & motorbikes",
"_id" : ObjectId("60752406d8e7213e6b5306e0"),
"id" : NumberInt(2)
}
],
Query
if (filterData.interests != undefined && filterData.interests.length > 0) {
interests = {
interests: { $elemMatch: { $and: [{ name: filterData.interests }] } }
}
}
User.aggregate([
coordinatesCondition,
{
$match: {
$and: [
exerciseHabitsCondition,
interests
],
},
},
{
$sort: lastActivity,
},
{ $limit: skip + 12 },
{ $skip: skip },
{
$lookup: {
from: "favorites",
localField: "_id",
foreignField: "favorites.favoriteUserId",
as: "favUsers",
},
},
])
Any solution appreciated!
as per my understanding you want to match the result with interests in the req data.
I am sharing a simple update, that can work well for you.
if (filterData.interests != undefined && filterData.interests.length > 0) {
interestsQuery = {
'interests.name': { $in: filterData.interests } }
}
}
User.aggregate([
coordinatesCondition,
{
$match: {
$and: [
exerciseHabitsCondition,
interestsQuery
],
},
},
{
$sort: lastActivity,
},
])
I have a aggregate query that fetches result from 3 collections.
I am using mongoDb 3.4
One Sample doc from result.
{
"_id" : ObjectId("5ba1717ee4b00ce08ca47cfa"),
"name" : "captain jack",
"email" : "jack#gmail.com",
"mobile" : "9000000023",
"status" : "verified",
"courses" : [
{
"_id" : "13",
"name" : "Course (03)"
},{
"_id" : "12",
"name" : "Course (03)"
}
],
"examCompleted" : false,
"login" : "5ba1717ee4b00fe08ca47cfa",
"partnerMetaInfo" : {
"_id" : ObjectId("5ba1717ee4b00fe08ca47cfa"),
"costCode" : "5761",
"hub" : "CALCUTTA",
"location" : "Kolkata"
}
}
I am trying to bring partnerMetaInfo at root level.
I am also unable to filter courses._id using $match on _id == 13
This is my aggregate query :
db.getCollection("mainCollection").aggregate([
{
//Join two collection
$lookup:{
from: "Details",
localField: "username",
foreignField: "login",
as: "partnerData"
}
},{
//Limit fields
$project:{
"email":1,
"name":1,
"mobile":1,
"status" : 1,
"courses":"$partnerData.courses",
"examScore" : "$partnerData.examScore",
"examCompleted" : "$partnerData.examCompleted",
"login":"$partnerData.login"
}
},
{
//Join third collection
$lookup:{
from: "PartnerMetaInfo",
localField: "login",
foreignField: "partnerId",
as: "partnerMetaInfo"
}
},
//Remove from partnerData array and place at root level.
{
$unwind:
{
path: '$courses',
preserveNullAndEmptyArrays: true
}
},{
$unwind:
{
path: '$examScore',
preserveNullAndEmptyArrays: true
}
},{
$unwind:
{
path: '$examCompleted',
preserveNullAndEmptyArrays: true
}
},{
$unwind:
{
path: '$login',
preserveNullAndEmptyArrays: true
}
},//Bring $partnerMetaInfo array to root level.
{
$unwind:
{
path: '$partnerMetaInfo',
preserveNullAndEmptyArrays: true
}
},{
$limit:10
}
];
partnerMetaInfo after $unwind ends up as object. I want to flatten it and bring it at root level.
Can any body help me with this?
If all you want to get as a result is the content of your partnerMetaInfo field then you can simply add a $replaceRoot stage at the end of your pipeline like this:
{
$replaceRoot: { "newRoot": { $ifNull: [ "$partnerMetaInfo", {} ] } }
}
Otherwise, in case you want to simply move the fields inside the partnerMetaInfo field to the root then you would use $addFields:
{
$addFields: {
"partnerMetaInfoId" : "$partnerMetaInfo._id",
"costCode" : "$partnerMetaInfo.costCode",
"hub" : "$partnerMetaInfo.hub",
"location" : "$partnerMetaInfo.location"
}
}
If you have a dynamic number of fields or do not want to hardcode field names then you can use the following logic:
{
$replaceRoot: { // merge fields of and move them all the way up
"newRoot": { $mergeObjects: [ "$$ROOT", "$partnerMetaInfo" ] }
}
}, {
$project: { // remove the "partnerMetaInfo" field
"partnerMetaInfo": 0
}
}
Now consider the case , i have one document containing below collection like structure.
Below is the order collection
{
"_id" : ObjectId("5788fcd1d8159c2366dd5d93"),
"color" : "Blue",
"code" : "1",
"category_id" : ObjectId("5693d170a2191f9020b8c815"),
"description" : "julia tried",
"name" : "Order1",
"brand_id" : ObjectId("5b0e52f058b8287a446f9f05")
}
There is also a collection for Brand and Category. This is the
Category collection
{
"_id" : ObjectId("5693d170a2191f9020b8c815"),
"name" : "Category1",
"created_at" : ISODate("2016-01-11T20:32:17.832+0000"),
"updated_at" : ISODate("2016-01-11T20:32:17.832+0000"),
}
Brand Collection
{
"_id" : ObjectId("5b0e52f058b8287a446f9f05"),
"name" : "brand1",
"description" : "brand1",
"updated_at" : ISODate("2017-07-05T09:18:13.951+0000"),
"created_at" : ISODate("2017-07-05T09:18:13.951+0000"),
}
Now after aggregation applied, it should result in below format:
{
'brands': [
{
_id: '*******'
name: 'brand1',
categories: [
{
_id: '*****',
name: 'category_name1',
orders: [
{
_id: '*****',
title: 'order1'
}
]
}
]
}
]
}
You can try below aggregation:
db.brand.aggregate([
{
$lookup: {
from: "order",
localField: "_id",
foreignField: "brand_id",
as: "orders"
}
},
{
$unwind: "$orders"
},
{
$lookup: {
from: "category",
localField: "orders.category_id",
foreignField: "_id",
as: "categories"
}
},
{
$unwind: "$categories"
},
{
$group: {
_id: "$_id",
name: { $first: "$name" },
description: { $first: "$description" },
updated_at: { $first: "$updated_at" },
created_at: { $first: "$created_at" },
categories: { $addToSet: "$categories" },
orders: { $addToSet: "$orders" }
}
},
{
$addFields: {
categories: {
$map: {
input: "$categories",
as: "category",
in: {
$mergeObjects: [
"$$category", {
orders: [ {
$filter: {
input: "$orders",
as: "order",
cond: { $eq: [ "$$category._id", "$$order.category_id" ] }
}
} ]
} ]
}
}
}
}
},
{
$project: {
orders: 0
}
}
])
Basically you have to use $lookup twice to "merge" data from all these collections based on brand_id and category_id fields. Since you expect orders in categories in brands you can use $unwind for both arrays and then use $group to get following shape:
{
"_id" : ObjectId("5b0e52f058b8287a446f9f05"),
"name" : "brand1",
"description" : "brand1",
"updated_at" : ISODate("2017-07-05T09:18:13.951Z"),
"created_at" : ISODate("2017-07-05T09:18:13.951Z"),
"categories" : [
{
"_id" : ObjectId("5693d170a2191f9020b8c814"),
"name" : "Category1",
"created_at" : ISODate("2016-01-11T20:32:17.832Z"),
"updated_at" : ISODate("2016-01-11T20:32:17.832Z")
}
],
"orders" : [
{
"_id" : ObjectId("5788fcd1d8159c2366dd5d93"),
"color" : "Blue",
"code" : "1",
"category_id" : ObjectId("5693d170a2191f9020b8c814"),
"description" : "julia tried",
"name" : "Order1",
"brand_id" : ObjectId("5b0e52f058b8287a446f9f05")
}
]
}
Now you have brand1 with all its subcategories and all orders that should be placed in one of those categories. The only thing is how to "nest" orders in categories. One way to do that might be $map where you can merge each category with all orders that match that category (using $mergeObjects you don't have to specify all properties from categories object).
To match category with orders you can perform $filter on orders array.
Then you can drop orders since those are nested into categories so you don't need them anymore.
EDIT: 3.4 version
In MongoDB 3.4 you can't use $mergeObjects so you should specify all properties for `categories:
db.brand.aggregate([
{
$lookup: {
from: "order",
localField: "_id",
foreignField: "brand_id",
as: "orders"
}
},
{
$unwind: "$orders"
},
{
$lookup: {
from: "category",
localField: "orders.category_id",
foreignField: "_id",
as: "categories"
}
},
{
$unwind: "$categories"
},
{
$group: {
_id: "$_id",
name: { $first: "$name" },
description: { $first: "$description" },
updated_at: { $first: "$updated_at" },
created_at: { $first: "$created_at" },
categories: { $addToSet: "$categories" },
orders: { $addToSet: "$orders" }
}
},
{
$addFields: {
categories: {
$map: {
input: "$categories",
as: "category",
in: {
_id: "$$category._id",
name: "$$category.name",
created_at: "$$category.created_at",
updated_at: "$$category.updated_at",
orders: [
{
$filter: {
input: "$orders",
as: "order",
cond: { $eq: [ "$$category._id", "$$order.category_id" ] }
}
}
]
}
}
}
}
},
{
$project: {
orders: 0
}
}
])