i have total 3 collections:
1. users
2. posts
3. actions
My collections look like these:
users:
{
"_id" : ObjectId("57ee65fef5a0c032877725db"),
"fbId" : "EAAXZA4sZCZBKmgBAKe0JpJPrp7utWME6xbHT9yFD",
"name" : "Aftab",
"email" : "xxxe#hotmail.com",
"gender" : "male",
"__v" : NumberInt(0),
"updatedAt" : ISODate("2016-10-10T05:11:35.344+0000"),
"score" : NumberInt(90)
}
actions:
{
"_id" : ObjectId("57f7a0ba3a603627658afdd3"),
"updatedAt" : ISODate("2016-10-07T13:18:50.815+0000"),
"createdAt" : ISODate("2016-10-07T13:18:50.815+0000"),
"userId" : ObjectId("57ee65fef5a0c032877725db"),
"postId" : ObjectId("57f4b5e98899081203883a1b"),
"type" : "like"
}
{
"_id" : ObjectId("57f7a0ba3a603627658afdd4"),
"updatedAt" : ISODate("2016-10-07T13:18:50.815+0000"),
"createdAt" : ISODate("2016-10-07T13:18:50.815+0000"),
"userId" : ObjectId("57ee65fef5a0c032877725db"),
"postId" : ObjectId("57f4b5d58899081203883a1a"),
"type" : "dismiss"
}
posts:
{
"_id" : ObjectId("57f24593e272b5199e9351b9"),
"imgFileLocation" : "http://xxxx/buybye-platform/uploads/image-1475495315229",
"description" : "cool cool",
"title" : "Bad Image ",
"userId" : ObjectId("57f21e3d0b787d0f7ad76dd0"),
"__v" : NumberInt(0)
}
{
"_id" : ObjectId("57f4b5d58899081203883a1a"),
"imgFileLocation" : "http://xxx/buybye-platform/uploads/image-1475655125125",
"description" : "cool & cool",
"title" : "Good Image",
"userId" : ObjectId("57f21e3d0b787d0f7ad76dd0"),
"__v" : NumberInt(0)
}
user can create posts, and other users can perform actions on those posts
posts collection has a reference of userId
and actions collection has ref of userId(who performed that action), postId(on which post) and action-type(like/dislike/dismiss)
I need to query to get all actions performed on a specific user posts
I have able to get all posts against a user, which is pretty straight-forward and is an array. now I need to get all actions performed on every single post of this posts array.
If you want a solution that leverages the aggregation framework, you can use the $lookup stage that was introduced starting with MongoDB v3.2.
For example, if you want to return a result set containing the details of a post and an array of all the actions performed on that particular post, you can run the following aggregation query:
/*
* QUERY #1
*/
db.posts.aggregate([
{
$lookup: {
from: 'actions',
localField: '_id',
foreignField: 'postId',
as: 'post_actions'
}
}
]);
/*
* RESULT SET #1
*/
{
"_id" : ObjectId("57ff4512a134e614a7178c1d"),
"imgFileLocation" : "http://xxxx/buybye-platform/uploads/image-1475495315229",
"description" : "cool cool",
"title" : "Bad Image ",
"userId" : ObjectId("57f21e3d0b787d0f7ad76dd0"),
"__v" : 0,
"post_actions" : [
{
"_id" : ObjectId("57ff4563a134e614a7178c1e"),
"updatedAt" : ISODate("2016-10-07T13:18:50.815Z"),
"createdAt" : ISODate("2016-10-07T13:18:50.815Z"),
"userId" : ObjectId("57ee65fef5a0c032877725db"),
"postId" : ObjectId("57ff4512a134e614a7178c1d"),
"type" : "like"
},
{
"_id" : ObjectId("57ff4564a134e614a7178c1f"),
"updatedAt" : ISODate("2016-10-07T13:18:50.815Z"),
"createdAt" : ISODate("2016-10-07T13:18:50.815Z"),
"userId" : ObjectId("57ee65fef5a0c032877725db"),
"postId" : ObjectId("57ff4512a134e614a7178c1d"),
"type" : "share"
}
]
}
Otherwise, if you want to only retrieve the actions for a specific array of posts, you can add a $match stage in the aggregation pipeline:
const postIdsArray = [
ObjectId("57ff4512a134e614a7178c1d"),
ObjectId("57ee65fef5a0c032877725db")
];
/*
* QUERY #2
*/
db.posts.aggregate([
{
$match: {
_id: {
$in: postIdsArray
}
}
},
{
$lookup: {
from: 'actions',
localField: '_id',
foreignField: 'postId',
as: 'post_actions'
}
}
]);
Furthermore, if you would want to only retrieve the total count of actions performed on a post, you can add an $unwind stage and then $group all the results:
/*
* QUERY #3
*/
db.posts.aggregate([
{
$lookup: {
from: 'actions',
localField: '_id',
foreignField: 'postId',
as: 'post_actions'
}
},
{
$unwind: '$post_actions'
},
{
$group: {
_id: '$_id',
posts: { $sum: 1 }
}
}
]);
/*
* RESULT SET #3
*/
{ "_id" : ObjectId("57ff4512a134e614a7178c1d"), "posts" : 2 }
UPDATE #1
If you want to retrieve only the actions that are of a certain type (e.g.: like, share etc.), you can add an additional $match stage in your aggregation pipeline, after you $unwind the post_actions array retrieved in the $lookup stage.
For example, the first query would become:
/*
* UPDATED QUERY #1
*/
db.posts.aggregate([
{
$lookup: {
from: 'actions',
localField: '_id',
foreignField: 'postId',
as: 'post_actions'
}
},
{
$unwind: '$post_actions'
},
{
$match: {
"post_actions.type": 'like'
}
}
]);
The second query would become:
const postIdsArray = [
ObjectId("57ff4512a134e614a7178c1d"),
ObjectId("57ee65fef5a0c032877725db")
];
/*
* UPDATED QUERY #2
*/
db.posts.aggregate([
{
$match: {
_id: {
$in: postIdsArray
}
}
},
{
$lookup: {
from: 'actions',
localField: '_id',
foreignField: 'postId',
as: 'post_actions'
}
},
{
$unwind: '$post_actions'
},
{
$match: {
"post_actions.type": 'like'
}
}
]);
The third query would become:
/*
* UPDATED QUERY #3
*/
db.posts.aggregate([
{
$lookup: {
from: 'actions',
localField: '_id',
foreignField: 'postId',
as: 'post_actions'
}
},
{
$unwind: '$post_actions'
},
{
$match: {
"post_actions.type": 'like'
}
},
{
$group: {
_id: '$_id',
posts: { $sum: 1 }
}
}
]);
Related
I am using aggregation query to retrieve 20000 records. while retrieving it is taking much time. I will mention my query below, Please help me to improve the query performance.
Query:
[err, data] = await to(LeadsLog.aggregate([
{$lookup:{
from: "leads",
localField: "leadId",
foreignField: "_id",
as: "leadId"
}},
{$lookup:{
from: "company_contacts",
localField: "leadId.assignedTo",
foreignField: "_id",
as: "assignedTo"
}},
{
$unwind:{
path: "$leadId",
preserveNullAndEmptyArrays: true
}
},
{
$match:{"leadId.assignedTo":new mongoose.Types.ObjectId(userId),
"result":{$eq:null}}
},
{ '$facet' : {
metadata: [ { $count: "total" }, { $addFields: { page: 1 } } ],
data: [ { $skip: 0 }, { $limit: 20000 } ]
} }
] ));
LeadId:
{
"_id" : ObjectId("617a84b401c98424e00d1310"),
"status" : true,
"address" : "Howmif Trail",
"city" : "Kinawnet",
"state" : "LA",
"country" : "LA",
"pincode" : null,
"extraFormObject" : null,
"lead_name" : "Jayden",
"phone" : "(524) 387-4912",
"email" : "niligis#taptehe.cm",
"company" : ObjectId("6155c2758609663d10fff796"),
"createdBy" : ObjectId("6155c2758609663d10fff798"),
"createdAt" : ISODate("2021-10-28T11:08:40.433Z"),
"updatedAt" : ISODate("2021-10-30T04:43:49.490Z")
}
LeadLog:
{
"_id" : ObjectId("617a84bf01c98424e00daf52"),
"callLogId" : null,
"result" : null,
"assignedTo" : ObjectId("6155c2758609663d10fff798"),
"extraFormObject" : null,
"subResult" : null,
"apptDate" : null,
"nextcallDate" : ISODate("2021-10-28T11:02:50.516Z"),
"callDate" : null,
"leadId" : ObjectId("617a84b401c98424e00d1310"),
"company" : ObjectId("6155c2758609663d10fff796"),
"createdAt" : ISODate("2021-10-28T11:08:50.962Z"),
"updatedAt" : ISODate("2021-10-30T04:43:50.281Z")
}
Please help me with better solution. thank you.
There are a few simple tweaks that you can improve your existing query:
make intermediate result as small as possible; one of the common ways is pushing $match stages as early as possible
use Pipeline Coalescence Optimization as much as possible; one of the common tuples would be $lookup + $unwind combination
index the $match fields and $lookup fields
Based on the first 2 points, here is my suggested form of your query:
You can see result : {$eq: null} is pushed to first stage. The performance gain will depends on the selectivity of the clause.
the $lookup and $unwind leads are grouped together to utilize the coalescence optimization.
"leadId.assignedTo": new mongoose.Types.ObjectId(userId) is moved earlier to minimize intermediate result size
Don't forget to index the relevant $match fields and $lookup fields. From my personal experience, good usage of index will help most with the performance.
[err, data] = await to(LeadsLog.aggregate([
{
$match: {
"result": {
$eq: null
}
}
},
{
$lookup: {
from: "leads",
localField: "leadId",
foreignField: "_id",
as: "leadId"
}
},
{
$unwind: {
path: "$leadId",
preserveNullAndEmptyArrays: true
}
},
{
$match: {
"leadId.assignedTo": new mongoose.Types.ObjectId(userId)
}
},
{
$lookup: {
from: "company_contacts",
localField: "leadId.assignedTo",
foreignField: "_id",
as: "assignedTo"
}
},
{
"$facet": {
metadata: [
{
$count: "total"
},
{
$addFields: {
page: 1
}
}
],
data: [
{
$skip: 0
},
{
$limit: 20000
}
]
}
}
]));
How can I set this condition in my mongoose query?
Note that coinId is a populated field
signalModel
.find(
{
'coinId.Price': { $gt: 1000 }
},
(err, signals) => {
if (err) throw err
res.json(signals)
}
)
.populate('coinId')
Update:
I tried this query from the answers, but I didn't satisfied with the result since the coin field is an empty array.
let signals = await signalModel.aggregate([
{
$lookup: {
from: 'Coin',
localField: 'coinId',
foreignField: '_id',
as: 'coin'
}
},
{
$match: {
type: req.query.type ?? /^(buy|sell)$/i,
'coin.Price': { $gt: 1000 }
}
}
])
The result:
{
"_id": "616eb8d2c31a52512900f56b",
"coinId": "616e8c07ca518ef7fac3eea9",
"type": "buy",
"highEntryPoint": 60000,
"lowEntryPoint": 59500,
"stopLoss": 59000,
"takeProfit": 60500,
"lastPrice": 61998.1102423889,
"status": "waiting",
"__v": 0,
"coin": []
}
Update2
Input Example/Signal
{
"_id" : ObjectId("616eb8d2c31a52512900f56b"),
"coinId" : ObjectId("616e8c07ca518ef7fac3eea9"),
"type" : "buy",
"highEntryPoint" : 60000,
"lowEntryPoint" : 59500,
"stopLoss" : 59000,
"takeProfit" : 60500,
"lastPrice" : 61998.1102423889,
"status" : "waiting",
"__v" : 0
}
Input Example/Coin
{
"_id" : ObjectId("616e8c07ca518ef7fac3eea9"),
"Name" : "BTC",
"Price" : 61700.9584311177,
"__v" : 0
}
first use $lookup then add condition in aggregate like this :
signalModel.aggregate([
{$lookup:{
from: 'Coins' // coins collection name
localField: coindId,
foreignField: _id
as: coins
}},
{
$match:{'coins':{$ne: []}}
}
,{
$match :{"coins.Price":{$gt:1000}}
}
])
I am trying to find documents in a collection, but filtered based on the value of an embedded ObjectID relation.
Mongoose schema is as follows:
const UserQualificationSchema = new Schema(
{
user: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User',
},
}
const UserSchema = new Schema(
{
fleet: {
type: [String], // Eg ["FleetA", "FleetB", "FleetC"]
required: true,
}
}
I need to find all UserQualifications where an item in the user's fleet equals a value in a filter array.
For example: Find all User Qualifications where user.fleet: {$in: ["FleetA", "FleetC"]}
I've looked at aggregations and querying inside .populate() but can't seem to get it to work.
Any ideas much appreciated.
Use aggregation query for your problem, I have created a query for you.
users.collection.json
/* 1 */
{
"_id" : ObjectId("61056c4a8cca27df3db2e4c8"),
"firstName" : "Rahul",
"lastName" : "soni",
"fleet" : [
"FleetA",
"FleetB",
"FleetC"
],
"createdAt" : ISODate("2021-07-31T15:29:14.918Z")
}
userqualifications.collection.json
/* 1 */
{
"_id" : ObjectId("61056c908cca27df3db2e4c9"),
"user" : ObjectId("61056c4a8cca27df3db2e4c8"),
"createdAt" : ISODate("2021-07-31T15:30:24.510Z")
}
aggregation query:
it will get the result only if a user has FleetA and FleetC.
if anyone is not matched then it will return 0 records
db.userqualifications.aggregate([{
"$lookup": {
"from": "users",
"localField": "user",
"foreignField": "_id",
"as": "user"
}
}, {
"$unwind": "$user"
}, {
"$match": {
"user.fleet": {
"$elemMatch": {
"$eq": "FleetA",
"$eq": "FleetC"
}
}
}
}])
Result:
/* 1 */
{
"_id" : ObjectId("61056c908cca27df3db2e4c9"),
"user" : {
"_id" : ObjectId("61056c4a8cca27df3db2e4c8"),
"firstName" : "Rahul",
"lastName" : "soni",
"fleet" : [
"FleetA",
"FleetB",
"FleetC"
],
"createdAt" : ISODate("2021-07-31T15:29:14.918Z")
},
"createdAt" : ISODate("2021-07-31T15:30:24.510Z")
}
if the goal is to get only UserQualifications then the following should be an efficient way as it can use an index on the fleet field of the User collection.
db.User.aggregate([
{
$match: {
fleet: { $in: ["FleetA", "FleetB"] }
}
},
{
$lookup: {
from: "UserQualification",
localField: "_id",
foreignField: "user",
as: "qualifications"
}
},
{
$unwind: "$qualifications"
},
{
$replaceWith: "$qualifications"
}
])
on the other hand if you start from the UserQualifications collection, you can't efficiently narrow down the records as you're filtering on something that it doesn't have the data for.
Thank you for the answer - it did achieve the results I was looking for - however I am now struggling to add a $match with $and to the aggregate to only return the qualifications where the user ID equals one inside a submitted array AND a given fleet.
I have the following aggregate:
db.UserQualifications.aggregate([{
{
$lookup: {
from: 'users',
localField: 'user',
foreignField: '_id',
as: 'user',
},
},
{
$unwind: '$user',
},
{
$match: {
$and: [
'user.fleet': {
$in: ["Fleet A", "Fleet C"], // This works on it's own
},
user: { // Also tried 'user._id'
$in: ["6033e4129070031c07fbbf29"] // Adding this returns blank array
}
]
},
}
}])
I have double checked that I am passing in the correct User ID's inside the arrays, but when I add this to the $and inside match, it only returns a blank array.
Is there another way to do this?
// Updated users collection
/* 1 */
{
"_id" : ObjectId("61056c4a8cca27df3db2e4c8"),
"firstName" : "Rahul",
"lastName" : "soni",
"fleet" : [
"Fleet A",
"Fleet B",
"Fleet C"
],
"createdAt" : ISODate("2021-07-31T15:29:14.918Z")
}
Query:
// userqualifications => this is my collection name, you can add your collection name here db.<YOUR>
db.userqualifications.aggregate([{
$lookup: {
from: 'users',
localField: 'user',
foreignField: '_id',
as: 'user',
},
},
{
$unwind: '$user',
},
{
$match: {
// $and operatory syntax [{}, {}]
$and: [{
'user.fleet': {
// "Fleet A", "Fleet C" (FleetA, FleetC) this is the my first options,
// I have changes according to your problem
$in: ["Fleet A", "Fleet C"], // This works on it's own
}
}, {
// Convert user id to ObjectId type (_bsonType)
"user._id": ObjectId("61056c4a8cca27df3db2e4c8")
}]
}
}
])
Result:
/* 1 */
{
"_id" : ObjectId("61056c908cca27df3db2e4c9"),
"user" : {
"_id" : ObjectId("61056c4a8cca27df3db2e4c8"),
"firstName" : "Rahul",
"lastName" : "soni",
"fleet" : [
"Fleet A",
"Fleet B",
"Fleet C"
],
"createdAt" : ISODate("2021-07-31T15:29:14.918Z")
},
"createdAt" : ISODate("2021-07-31T15:30:24.510Z")
}
Difference Image:
I have a problem with getting a collection array with changed one field in each document.
My document is:
{postId: "123123", author: "someId"}
And I want to search by author id in other collection and replace author with author name instead of id
Is there any simple way to do that?
EDIT:
I've wrote something like this
{
$lookup: {
as: "USER",
foreignField: "created",
from: "Users",
localField: "userName"
}
}
Then i recive array of documents from Users collections. I wan't only one based on the value from the field of collection posts
EDIT:
Sample documents from collections:
POSTS:
{
_id: 'p7c3d',
author: 'p9jhkl',
content: "Hello",
createdAt: '12-12-2019',
tags: ['hello'];
}
USERS:
{
_id: 'p9jhkl',
name: 'John Smith'
}
And I wan't that "John Smith" to be in post author field instead of user id.
Edit:
Picture of actual documents
Assuming there is only 1-to-1 relation i.e; documents in POSTS can have only one matching document in USERS.
POSTS :
/* 1 */
{
"_id" : "p7c3d",
"author" : "p9jhkl",
"content" : "Hello",
"tags" : [
"hello"
]
}
/* 2 */
{
"_id" : "p7c3d11",
"author" : "p9jhkl11",
"content" : "Hello",
"tags" : [
"hello"
]
}
/* 3 */
{
"_id" : "p7c3d1122",
"author" : "p9jhkl1122",
"content" : "Hello",
"tags" : [
"hello"
]
}
/* 4 */
{
"_id" : "p7c3d No",
"author" : "p9jhkl No",
"content" : "Hello",
"tags" : [
"hello"
]
}
USERS :
/* 1 */
{
"_id" : "p9jhkl",
"name" : "John Smith"
}
/* 2 */
{
"_id" : "p9jhkl11",
"name" : "John Smith11"
}
/* 3 */
{
"_id" : "p9jhkl1122",
"name" : "John Smith1122"
}
Please try this :
db.POSTS.aggregate([{
$lookup:
{
from: "USERS",
localField: "author",
foreignField: "_id",
as: "userDetails"
}
},{$addFields : {author: { $arrayElemAt: [ '$userDetails.name', 0 ] }}},
{$project : { userDetails: 0}}])
Result :
/* 1 */
{
"_id" : "p7c3d",
"author" : "John Smith",
"content" : "Hello",
"tags" : [
"hello"
]
}
/* 2 */
{
"_id" : "p7c3d11",
"author" : "John Smith11",
"content" : "Hello",
"tags" : [
"hello"
]
}
/* 3 */
{
"_id" : "p7c3d1122",
"author" : "John Smith1122",
"content" : "Hello",
"tags" : [
"hello"
]
}
/* 4 */
{
"_id" : "p7c3d No",
"content" : "Hello",
"tags" : [
"hello"
]
}
Above query would completely remove the field author from documents of POSTS collection where there is no match with USERS documents (this would help to quickly remove POSTS docs from final o/p which has no match with USERS using {$match : {author: {$exists : true}}}), Just in case you need all documents from POSTS as is irrespective of whether there is a match or not, Check document 4 in results for clarification, Please try this :
db.POSTS.aggregate([{
$lookup:
{
from: "USERS",
localField: "author",
foreignField: "_id",
as: "userDetails"
}
},{$addFields :
{author: {
$cond: { if: { $ne: [ "$userDetails", [] ] },
then : { $arrayElemAt: [ '$userDetails.name', 0 ] },
else : '$author'}}}}, {$project : { userDetails: 0}}])
Result :
/* 1 */
{
"_id" : "p7c3d",
"author" : "John Smith",
"content" : "Hello",
"tags" : [
"hello"
]
}
/* 2 */
{
"_id" : "p7c3d11",
"author" : "John Smith11",
"content" : "Hello",
"tags" : [
"hello"
]
}
/* 3 */
{
"_id" : "p7c3d1122",
"author" : "John Smith1122",
"content" : "Hello",
"tags" : [
"hello"
]
}
/* 4 */
{
"_id" : "p7c3d No",
"author" : "p9jhkl No",
"content" : "Hello",
"tags" : [
"hello"
]
}
With actual data :
db.POSTS.aggregate([{ $addFields: { "createdBy": { "$toObjectId": "$createdBy" } } }, {
$lookup:
{
from: "USERS",
localField: "createdBy",
foreignField: "_id",
as: "userDetails"
}
}, {
$addFields:
{
createdBy: {
$cond: {
if: { $ne: ["$userDetails", []] },
then: { $arrayElemAt: ['$userDetails.name', 0] },
else: { "$toString": "$createdBy" }
}
}
}
}, { $project: { userDetails: 0 } }])
This can be done in three stages:
Match the author in posts to only get posts for an author
Join to authors using the created field
Project the fields you want, and only include the author's name in the result.
Example Below:
db.getCollection("posts").aggregate(
[
{
$match: {
created: ObjectId("5dd446cf3ccb4305e7fbec96")
}
},
{
$lookup: {from: "authors",
let: { id: "$created" },
pipeline: [
{
$match: {
$expr:
{
$eq: ["$$id","$_id"] }
}
},
{ $project: { _id: 0, name: 1 } },
],
as: "author"
}
},
{
$project: {
author: {
$arrayElemAt: ["$author.name", 0]
}
}
},
]
);
I have two collections (promotions,product) and product collection map to promotions its working fine.but I have doubt how to show particular columns in product collection.
promotional collection
{
"_id" : ObjectId("5cf7679a0b0bed2e7483b998"),
"group_name" : "Latest",
"products" :
[ObjectId("5cecc161e8c1e73478956333"),ObjectId("5cecc161e8c1e73478956334")]
}
product collection
{
"_id" : ObjectId("5cecc161e8c1e73478956333"),
"product_name" : "bourbon"
},
{
"_id" : ObjectId("5cecc161e8c1e73478956334"),
"product_name" : "bour"
}
mapping query
db.promotional.aggregate(
[
{
$lookup: {
from: "product",
localField: "products",
foreignField: "_id",
as: "products"
}
}
]
)
I tried to map product collection to promotional collection
I got Output
{
"_id" : ObjectId("5cf7679a0b0bed2e7483b998"),
"group_name" : "Latest",
"products" :
[
{
"_id" : ObjectId("5cecc161e8c1e73478956333"),
"product_name" : "bourbon"
},
{
"_id" : ObjectId("5cecc161e8c1e73478956334"),
"product_name" : "bour"
}
]
}
Expected output
{
"_id" : ObjectId("5cf7679a0b0bed2e7483b998"),
"group_name" : "Latest",
"products" :
[
{
"product_name" : "bourbon"
},
{
"product_name" : "bour"
}
]
}
You can exclude those columns using $project operator:
db.promotional.aggregate(
[
{
$lookup: {
from: "product",
localField: "products",
foreignField: "_id",
as: "products"
}
},
{
$project: {
"products._id": 0
}
}
]
)
db.promotional.aggregate([
{
$lookup: {
from: "product",
localField: "products",
foreignField: "_id",
as: "products"
}
},{$project :{products :{product_name : 1}}}
])
db.getCollection("promotional").aggregate(
// Pipeline
[
// Stage 1
{
$unwind: {
path: "$products"
}
},
// Stage 2
{
$lookup: {
from: "product",
localField: "products",
foreignField: "_id",
as: "products"
}
},
// Stage 3
{
$group: {
_id: {
_id: '$_id',
group_name: '$group_name'
},
products: {
$push: {
product_name: {
$arrayElemAt: ["$products.product_name", 0]
}
}
}
}
},
// Stage 4
{
$project: {
_id: '$_id._id',
group_name: '$_id.group_name',
products: 1
}
},
]
);