Related
I have a collection which comprises of three level array nesting as shown below
_id: ObjectID('abc'),
sections: [
{
sectionId: "sec0",
sectionName: "ABC",
contents: [
{
contentId: 0,
tasks: [
{
taskId: ObjectID('task1')
}
//May contain 1-100 tasks
],
contentDescription: "Content is etc",
}
]
}
]
Sections is an array of objects which contains an object each with sectionId, and contents array which is an array of objects comprising of contentId, contentDescription, and nested array of tasks which comprises of an object containing a taskId.
I am applying $lookup operator in order to join nested tasks array with tasks collection but I am facing a problem in document duplication as shown below.
_id: ObjectID('abc'),
sections: [
{
sectionId: "sec0",
sectionName: "ABC",
contents: [
{
contentId: 0,
tasks: [
{
//Task Document of ID 1
}
],
contentDescription: "Content is etc",
}
]
}
]
_id: ObjectID('abc'),
sections: [
{
sectionId: "sec0",
sectionName: "ABC",
contents: [
{
contentId: 0,
tasks: [
{
//Task Document of ID 2
}
],
contentDescription: "Content is etc",
}
]
}
]
Whereas the desired output is as follows
_id: ObjectID('abc'),
sections: [
{
sectionId: "sec0",
sectionName: "ABC",
contents: [
{
contentId: 0,
tasks: [
{
//Task Document of ID 1
},
{
//Task Document of ID 2
},
{
//Task Document of ID 3
}
],
contentDescription: "Content is etc",
}
]
}
]
In the collection, a sections array might contain multiple section object which might contain multiple contents and so on and so forth.
The schema in question is temporary as our company is currently migrating from an existing database to MongoDB, so architectural refactoring is not possible atm and I need to work with existing schema design from different database.
I tried the following way
const contents= await sections.aggregate([
{
$match: { _id: id},
},
{ $unwind: '$sections' },
{
$unwind: {
path: '$sections.contents',
preserveNullAndEmptyArrays: true,
},
},
{
$unwind: {
path: '$sections.contents.tasks',
preserveNullAndEmptyArrays: true,
},
},
{
$lookup: {
from: 'tasks',
let: { task_id: '$sections.contents.tasks.taskId' },
pipeline: [
{ $match: { $expr: { $eq: ['$_id', '$$task_id'] } } },
],
as: 'sections.contents.tasks',
},
},
{
$addFields: {
'sections.contents.tasks': {
$arrayElemAt: ['$sections.contents.tasks', 0],
},
},
},
{
$group: {
_id: '$_id',
exam: { $push: '$sections.contents.tasks' },
},
},
]);
And I am also unable to use $group aggregation operator like
$group: {
_id: '$_id',
sections: {
sectionId : { $first: '$sectionId' },
sectionName: { $first: '$sectionName' },
contents: {
contentId: { $first: '$contentId' },
task: { $push: $sections.contents.tasks }
}
},
},
Any help or directions will be appreciated, I also searched on SO, and found this but couldn't understand the following part
{"$group":{
"_id":{"_id":"$_id","mission_id":"$missions._id"},
"agent":{"$first":"$agent"},
"title":{"$first":"$missions.title"},
"clients":{"$push":"$missions.clients"}
}},
{"$group":{
"_id":"$_id._id",
"missions":{
"$push":{
"_id":"$_id.mission_id",
"title":"$title",
"clients":"$clients"
}
}
}}
So you're very close to the final solution, a good "rule" that's good to remember is if you unwind x times you need to group x to restore the original structure properly, like so:
db.collection.aggregate([
{
$match: {
_id: id
},
},
{
$unwind: "$sections"
},
{
$unwind: {
path: "$sections.contents",
preserveNullAndEmptyArrays: true,
},
},
{
$unwind: {
path: "$sections.contents.tasks",
preserveNullAndEmptyArrays: true,
},
},
{
$lookup: {
from: "tasks",
let: {
task_id: "$sections.contents.tasks.taskId"
},
pipeline: [
{
$match: {
$expr: {
$eq: [
"$_id",
"$$task_id"
]
}
}
},
],
as: "sections.contents.tasks",
},
},
{
$addFields: {
"sections.contents.tasks": {
$arrayElemAt: [
"$sections.contents.tasks",
0
],
},
},
},
{
$group: {
_id: {
contentId: "$sections.contents.contentId",
sectionId: "$sections.sectionId",
sectionName: "$sections.sectionName",
originalId: "$_id"
},
tasks: {
$push: "$sections.contents.tasks"
},
contentDescription: {
$first: "$sections.contents.contentDescription"
},
}
},
{
$group: {
_id: {
sectionId: "$_id.sectionId",
sectionName: "$_id.sectionName",
originalId: "$_id.originalId"
},
contents: {
$push: {
contentId: "$_id.contentId",
tasks: "$tasks",
contentDescription: "$contentDescription"
}
}
}
},
{
$group: {
_id: "$_id.originalId",
sections: {
$push: {
sectionId: "$_id.sectionId",
sectionName: "$_id.sectionName",
contents: "$contents"
}
}
}
}
])
Mongo Playground
However your pipeline could be made a little cleaner as it has 1 redundant $unwind stage that also adds a redundant $group stage. I won't post the entire fixed pipeline here as it's already a long post but feel free to check it out here: Mongo Playground fixed
I have the following collection (sectors):
[
{
sector: "IT",
organizations: [
{
org: "ACME",
owners: [
"Josh",
"Fred"
]
}
]
}
]
I also have another collection (owners):
[
{
name: "Josh",
age: 65,
male: true,
location: "LA"
}
]
I want the aggregation query to do the following:
For each sector document, go though each organization.
Find an owner document corresponding to index 0 of the owners array.
Add the { name, age, male } fields to the organization.
I want to get this result:
[
{
sector: "IT",
organizations: [
{
org: "ACME",
owners: [
"Josh",
"Fred"
],
name: "Josh",
age: 65,
male: true
}
]
}
]
I am writing this in Node.js. This is my current code:
await Sector.aggregate([
// Perhaps something with $lookup?
{ $match: query },
{ $skip: skip },
{ $limit: limit }
]);
I am totally new to aggregation with MongoDB. Can anyone tell me how it's done?
Thanks in advance.
You can use aggregation
$addFields and $arrayElementAt helps to get the first element by looking with $map
$unwind to deconstruct the array
$lookup to join collections
$group to reconstruct the array
Here is the code
db.collection1.aggregate([
{
$addFields: {
organizations: {
$map: {
input: "$organizations",
in: {
firstName: { "$arrayElemAt": [ "$$this.owners", 0 ] },
org: "$$this.org",
owners: "$$this.owners"
}
}
}
}
},
{ $unwind: "$organizations},
{
"$lookup": {
"from": "collection2",
"localField": "organizations.firstName",
"foreignField": "name",
"as": "join"
}
},
{ $addFields: { join: { "$arrayElemAt": [ "$join", 0 } } },
{
$addFields: {
"organizations.age": "$join.age",
"organizations.location": "$join.location",
"organizations.male": "$join.male",
"join": "$$REMOVE"
}
},
{
"$group": {
"_id": "$_id",
"organizations": { "$push": "$organizations" },
"sector": { $first: "$sector" }
}
}
])
Working Mongo playground
Given this Orders collection:
// Order documents
[
{
_id: "order_123",
items: [
{ _id: "item_123", type: "T-Shirt" },
{ _id: "item_234", type: "Hoodie" },
{ _id: "item_345", type: "Hat" },
],
refunds: [
{
_id: "refund_123",
items: ["item_123", "item_234"],
},
{
_id: "refund_234",
items: ["item_345"],
},
],
},
]
Is it possible to map refunds.items -> items._id, allowing us to filter by type?
This is how we currently get the refund sub-documents:
db.orders.aggregate([
{
$replaceRoot: {
newRoot: {
order: "$$ROOT",
refunds: "$$ROOT.refunds",
},
},
},
{
$unwind: "$refunds",
},
{
$project: {
order: "$order",
refund: "$refunds",
},
},
]);
Which gives us:
// Refund documents
[
{
refund: {
_id: "refund_123",
items: ["item_123", "item_234"],
},
order: { ... }, // The original order document
},
{
refund: {
_id: "refund_234",
items: ["item_345"],
},
order: { ... }, // The original order document
},
]
From here, we want to map up refund.items -> order.items._id to produce the following output:
[
{
_id: "refund_123",
items: [
{ _id: "item_123", type: "T-Shirt" },
{ _id: "item_234", type: "Hoodie" },
],
},
{
_id: "refund_234",
items: [
{ _id: "item_345", type: "Hat" }
],
},
]
Allowing us to filter refund documents by type.
You can do this using $unwind and $filter,
$unwind deconstruct array refunds
$project to show refund id in _id, and filter items that are in refunds.items array using $filter
db.orders.aggregate([
{ $unwind: "$refunds" },
{
$project: {
_id: "$refunds._id",
items: {
$filter: {
input: "$items",
cond: { $in: ["$$this._id", "$refunds.items"] }
}
}
}
}
])
Playground
I have 4 collections for store order data.
1. order => field => _id, order_no, cust_id, order_date
2. order_address => field => _id, order_id, cust_name, mobile, address
3. order_details => field => _id, order_id, product_id, seller_id, qty, price
4. order_payment => field => _id, payment_type, payment_status
in that order_details collection has n number of record for a number of product in one order.
in that how to get particular seller order from my data using aggregate in node.js from MongoDB database
i try this code but it's show ordre_details = [] but show order in my result:
var query = [
{ "$lookup": {
from: 'order_details',
let: { order_id: "$_id" },
pipeline: [
{ $match: { $expr: { $and: [{ $eq: [ "$order_id", "$$order_id" ] }, { $eq: [ "$seller_id", ObjectID(seller_id) ] }] } } },
{ $project: {
amount: 1,
cod_charge: 1,
shipping_charge: 1,
pid: 1,
product_attribute_id: 1,
qty: 1
} },
{ "$lookup": {
from: 'product',
let: { product_id: "$pid", product_attribute_id: '$product_attribute_id'},
pipeline: [
{ $match: { $expr: { $eq: [ "$_id", "$$product_id" ] } } },
{ $project: { _id: 1, name: 1, sku: 1 } },
{ "$lookup": {
from: 'product_image',
let: { product_attribute_id: '$$product_attribute_id' },
pipeline: [
{ $match: { $expr: { $eq: [ "$product_attribute_id", "$$product_attribute_id" ] } } },
{ $project: { _id: 0, image: 1, is_default: 1 } },
{ $sort : { is_default: -1 } },
{ $replaceRoot: { newRoot: {_id: "$_id", image: "$image" } } }
],
as: 'product_image'
} },
{ $replaceRoot: { newRoot: {
_id: "$_id",
name: "$name",
sku: "$sku",
image: { $arrayElemAt: [ "$product_image.image", 0 ] }
} } }
],
as: 'product'
} },
{ "$replaceRoot": { newRoot: {
_id: '$$ROOT._id',
pid: '$$ROOT.pid',
amount: '$$ROOT.amount',
cod_charge: '$$ROOT.cod_charge',
shipping_charge: '$$ROOT.shipping_charge',
product_attribute_id: "$$ROOT.product_attribute_id",
qty: "$$ROOT.qty",
product: { $arrayElemAt: [ "$product", 0 ] },
} } },
],
as: 'order_details'
} },
{ "$replaceRoot": {
newRoot: {
_id: "$_id",
order_no: "$order_no",
cust_id: "$cust_id",
order_date: "$order_date",
order_details: "$order_details"
}
} }
]
orderModel.order.aggregate(query, function(err, orderData){})
-room collection
_id: ObjectId("xxx")
bedspaces: Array
0:ObjectId("xx")
1:ObjectId("xx")
***
***
-bedspace collection
_id: ObjectId("xxxx");
number: 1
decks: Array
{
_id: ObjectId("xxx");
number: 1
status: "Vacant"
tenant: ObjectId("5c964ae7f5097e3020d1926c")
dueRent: 11
away: null
},
{
_id: ObjectId("xxx");
number: 2
status: "Vacant"
tenant: null
dueRent: 11
away: null
}
Under the decks array, is my tenant field, that have objectId, and i am going to lookup this object id, in the tenants, collection.
-tenant collection
_id: ObjectId("5c964ae7f5097e3020d1926c");
name: 'John Doe'
-expected output
/*room collection*/
_id: ObjectId("xxx")
bedspaces: [
{
_id: ObjectId("xxx")
number: 1
decks: [
{
_id: ObjectId("xxx")
number: 1
status: "Vacant"
tenant: {
name: 'John Doe'
}
dueRent: 11
away: null
},
{
_id: ObjectId("xxx");
number: 1
status: "Vacant"
tenant: null
dueRent: 11
away: null
}
]
}
]
There is also an instances, that deck array is equal to null.
In below aggregation it will only display the decks, that have tenant with object id, what i want is to display both the decks.
{
from: 'beds',
let: {bedspace: '$bedspaces'},
pipeline:[
{
$match: {
$expr: {
$in: ["$_id", "$$bedspace"]
}
}
},
{
$unwind: "$decks"
},
{
$lookup: {
from: 'tenants',
let: {tenant: "$decks.tenant"},
pipeline: [
{
$match: {
$expr: {
$eq: ["$_id", "$$tenant"]
}
}
}
],
as: "decks.tenant",
}
},
{
$unwind: "$decks.tenant"
},
{ $group: {
_id: "$_id",
decks: { $push: "$decks" },
number: {$first: "$number"}
}}
],
as: "bedspaces"
}
"how can i add condition on my second look up, to execute only if tenant is not null", so that i could retrieve both decks, or any work-around so i could achieved my desired result
Don't really have time for all the explanation right now (sorry),
Explanation
The basic issue here is that usage of $unwind is your problem and you don't need it. Use $map on the produced array content merging with the "decks" array instead. Then you can have nulls.
What you want to do here is have the values from the $lookup from your "tenants" collection transposed into the existing array within your "beds/bedspaces" collection for it's own existing "tenant" values which are the ObjectId references for the foreign collection.
The $lookup stage cannot do this by simply naming the field path within the "as" output where that path is already inside another array, and in fact the output of $lookup is always an array of results obtained from the foreign collection. You want singular values for each actual match, and of course you expect a null to be in place where nothing matches, and of course keeping the original document array of "decks" intact, but just including the foreign details where those were found.
Your code attempt seems partially aware of this point as you are using $unwind on the $lookup result on the ""tenants" collection into a "temporary array" ( but you put in in the existing path and that overwrites content ) and then attempting to "re-group" as an array through $group and $push. But the problem of course is the $lookup result does not apply to every array member within "decks", so you end up with less results than you want.
The real solution is not a "conditional $lookup", but instead to transpose the "temporary array" content from the result into the existing "decks" entries. You do this using $map to process the array members, and $arrayElemAt along with $indexOfArray in order to return the matching elements from the "temporary array" by the matching _id values to "tenant".
{ "$lookup": {
"from": Tenant.collection.name,
"let": { "tenant": "$decks.tenant" },
"pipeline": [
{ "$match": {
"$expr": { "$in": [ "$_id", "$$tenant" ] }
}}
],
"as": "tenant"
}},
{ "$addFields": {
"decks": {
"$map": {
"input": "$decks",
"in": {
"$mergeObjects": [
"$$this",
{
"tenant": {
"$cond": {
"if": {
"$eq": [
{ "$indexOfArray": ["$tenant._id", "$$this.tenant"] },
-1
]
},
"then": null,
"else": {
"$arrayElemAt": [
"$tenant",
{ "$indexOfArray": ["$tenant._id", "$$this.tenant"]}
]
}
}
}
}
Noting there we are using $mergeObjects inside the $map in order to keep the existing content of the "decks" array and only replace ( or "merge" ) an overwritten representation of "tenant" for each array member. You are using the expressive $lookup already and this like $mergeObjects is a MongoDB 3.6 feature.
Just for interest the same thing can be done by just specifying every field within the array. i.e:
"decks": {
"$map": {
"input": "$decks",
"in": {
"_id": "$$this._id",
"number": "$$this.number",
"tenant": {
// same expression
},
"__v": "$$this.__v" // just because it's mongoose
}
}
}
Much the same can be said for the $$REMOVE used in the $addFields which is also another MongoDB 3.6 feature. You can alternately just use $project and simply omit the unwanted fields:
{ "$project": {
"number": "$number",
"decks": {
"$map": { /* same expression */ }
},
"__v": "$__v"
// note we don't use the "tenant" temporary array
}}
But that's basically how it works. By taking the $lookup result and then transposing those results back into the original array within the document.
Example Listing
Also abstracting on your data from previous questions here, which is a bit better than what you posted in the question here. Runnable listing for demonstration:
const { Schema, Types: { ObjectId } } = mongoose = require('mongoose');
const uri = 'mongodb://localhost:27017/hotel';
const opts = { useNewUrlParser: true };
mongoose.set('useFindAndModify', false);
mongoose.set('useCreateIndexes', true);
mongoose.set('debug', true);
const tenantSchema = new Schema({
name: String,
age: Number
});
const deckSchema = new Schema({
number: Number,
tenant: { type: Schema.Types.ObjectId, ref: 'Tenant' }
});
const bedSchema = new Schema({
number: Number,
decks: [deckSchema]
});
const roomSchema = new Schema({
bedspaces: [{ type: Schema.Types.ObjectId, ref: 'Bed' }]
});
const Tenant = mongoose.model('Tenant', tenantSchema);
const Bed = mongoose.model('Bed', bedSchema);
const Room = mongoose.model('Room', roomSchema);
const log = data => console.log(JSON.stringify(data, undefined, 2));
(async function() {
try {
const conn = await mongoose.connect(uri, opts);
// Clean data
await Promise.all(
Object.entries(conn.models).map(([k, m]) => m.deleteMany())
);
// Insert data
let [john, jane, bilbo ] = await Tenant.insertMany([
{
_id: ObjectId("5c964ae7f5097e3020d1926c"),
name: "john doe",
age: 11
},
{
_id: ObjectId("5c964b2531bc162fdce64f15"),
name: "jane doe",
age: 12
},
{
_id: ObjectId("5caa5454494558d863513b24"),
name: "bilbo",
age: 111
}
]);
let bedspaces = await Bed.insertMany([
{
_id: ObjectId("5c98d89c6bd5fc26a4c2851b"),
number: 1,
decks: [
{
number: 1,
tenant: john
},
{
number: 1,
tenant: jane
}
]
},
{
_id: ObjectId("5c98d89f6bd5fc26a4c28522"),
number: 2,
decks: [
{
number: 2,
tenant: bilbo
},
{
number: 3
}
]
}
]);
await Room.create({ bedspaces });
// Aggregate
let results = await Room.aggregate([
{ "$lookup": {
"from": Bed.collection.name,
"let": { "bedspaces": "$bedspaces" },
"pipeline": [
{ "$match": {
"$expr": { "$in": [ "$_id", "$$bedspaces" ] }
}},
{ "$lookup": {
"from": Tenant.collection.name,
"let": { "tenant": "$decks.tenant" },
"pipeline": [
{ "$match": {
"$expr": { "$in": [ "$_id", "$$tenant" ] }
}}
],
"as": "tenant"
}},
{ "$addFields": {
"decks": {
"$map": {
"input": "$decks",
"in": {
"$mergeObjects": [
"$$this",
{
"tenant": {
"$cond": {
"if": {
"$eq": [
{ "$indexOfArray": ["$tenant._id", "$$this.tenant"] },
-1
]
},
"then": null,
"else": {
"$arrayElemAt": [
"$tenant",
{ "$indexOfArray": ["$tenant._id", "$$this.tenant"]}
]
}
}
}
}
]
}
}
},
"tenant": "$$REMOVE"
}}
],
"as": "bedspaces"
}}
]);
log(results);
} catch (e) {
console.error(e)
} finally {
mongoose.disconnect();
}
})()
Returns:
Mongoose: tenants.deleteMany({}, {})
Mongoose: beds.deleteMany({}, {})
Mongoose: rooms.deleteMany({}, {})
Mongoose: tenants.insertMany([ { _id: 5c964ae7f5097e3020d1926c, name: 'john doe', age: 11, __v: 0 }, { _id: 5c964b2531bc162fdce64f15, name: 'jane doe', age: 12, __v: 0 }, { _id: 5caa5454494558d863513b24, name: 'bilbo', age: 111, __v: 0 } ], {})
Mongoose: beds.insertMany([ { _id: 5c98d89c6bd5fc26a4c2851b, number: 1, decks: [ { _id: 5caa5af6ed3dce1c3ed72cef, number: 1, tenant: 5c964ae7f5097e3020d1926c }, { _id: 5caa5af6ed3dce1c3ed72cee, number: 1, tenant: 5c964b2531bc162fdce64f15 } ], __v: 0 }, { _id: 5c98d89f6bd5fc26a4c28522, number: 2, decks: [ { _id: 5caa5af6ed3dce1c3ed72cf2, number: 2, tenant: 5caa5454494558d863513b24 }, { _id: 5caa5af6ed3dce1c3ed72cf1, number: 3 } ], __v: 0 } ], {})
Mongoose: rooms.insertOne({ bedspaces: [ ObjectId("5c98d89c6bd5fc26a4c2851b"), ObjectId("5c98d89f6bd5fc26a4c28522") ], _id: ObjectId("5caa5af6ed3dce1c3ed72cf3"), __v: 0 })
Mongoose: rooms.aggregate([ { '$lookup': { from: 'beds', let: { bedspaces: '$bedspaces' }, pipeline: [ { '$match': { '$expr': { '$in': [ '$_id', '$$bedspaces' ] } } }, { '$lookup': { from: 'tenants', let: { tenant: '$decks.tenant' }, pipeline: [ { '$match': { '$expr': { '$in': [ '$_id', '$$tenant' ] } } } ], as: 'tenant' } }, { '$addFields': { decks: { '$map': { input: '$decks', in: { '$mergeObjects': [ '$$this', { tenant: [Object] } ] } } }, tenant: '$$REMOVE' } } ], as: 'bedspaces' } } ], {})
[
{
"_id": "5caa5af6ed3dce1c3ed72cf3",
"bedspaces": [
{
"_id": "5c98d89c6bd5fc26a4c2851b",
"number": 1,
"decks": [
{
"_id": "5caa5af6ed3dce1c3ed72cef",
"number": 1,
"tenant": {
"_id": "5c964ae7f5097e3020d1926c",
"name": "john doe",
"age": 11,
"__v": 0
}
},
{
"_id": "5caa5af6ed3dce1c3ed72cee",
"number": 1,
"tenant": {
"_id": "5c964b2531bc162fdce64f15",
"name": "jane doe",
"age": 12,
"__v": 0
}
}
],
"__v": 0
},
{
"_id": "5c98d89f6bd5fc26a4c28522",
"number": 2,
"decks": [
{
"_id": "5caa5af6ed3dce1c3ed72cf2",
"number": 2,
"tenant": {
"_id": "5caa5454494558d863513b24",
"name": "bilbo",
"age": 111,
"__v": 0
}
},
{
"_id": "5caa5af6ed3dce1c3ed72cf1",
"number": 3,
"tenant": null
}
],
"__v": 0
}
],
"__v": 0
}
]
Shows the null on the second entry of the second entry in the bedspaces array as expected.