Populate Object In an Array - node.js

Question

Having trouble Populating my user.
The case:
var User = new mongoose.Schema({
name: {
type: String,
lowercase: true,
unique: true
},
portfolio:[
{
name: String,
formatType: { type: mongoose.Schema.Types.ObjectId, ref: 'FormatType' },
}
]
});
And this is my Mongoose command:
User.findById(req.payload.id)
.populate({
path:'portfolio',
populate:{
path: 'formatType',
model: 'FormatType'
}
})
.then(user => { ...
So what we have here is a model - inside of an Obect - inside of an array - inside of an entity.
Couldn't find an Answer online, would be very thankful~!

Answer 1

What you basically missed here is the "path" to the field you want to populate() is actually 'portfolio.formatType' and not just 'portfolio' as you have typed. Due to that mistake and the structure, you might have a few general misconceptions though.
Populate Correction
The basic correction merely needs the correct path, and you don't need the model argument since this is already implied in the schema:
User.findById(req.params.id).populate('portfolio.formatType');
It is however generally not a great idea to "mix" both "embedded" data and "referenced" data within arrays, and you should really be either embedding everything or simply referencing everything. It's also a little bit of an "anti-pattern" in general to keep an array of references in the document if your intention is referencing, since your reason should be not to cause the document to grow beyond the 16MB BSON limit. And where that limit would never be reached by your data it's generally better to "embed fully". That's really a wider discussion, but something you should be aware of.
The next general point here is populate() itself is somewhat "old hat", and really not the "magical" thing most new users perceive it to be. To be clear populate() is NOT A JOIN, and all it is doing is executing another query to the server in order to return the "related" items, then merge that content into the documents returned from the previous query.
$lookup Alternative
If you are looking for "joins", then really you probably wanted "embedding" as mentioned earlier. This is really the "MongoDB Way" of dealing with "relations" but keeping all "related" data together in the one document. The other means of a "join" where data is in separate collections is via the $lookup operator in modern releases.
This gets a bit more complex due to your "mixed" content array form, but can generally be represented as:
// Aggregation pipeline don't "autocast" from schema
const { Types: { ObjectId } } = require("mongoose");
User.aggregate([
{ "$match": { _id: ObjectId(req.params.id) } },
{ "$lookup": {
"from": FormatType.collection.name,
"localField": "portfolio.formatType",
"foreignField": "_id",
"as": "formats"
}},
{ "$project": {
"name": 1,
"portfolio": {
"$map": {
"input": "$portfolio",
"in": {
"name": "$$this.name",
"formatType": {
"$arrayElemAt": [
"$formats",
{ "$indexOfArray": [ "$formats._id", "$$this.formatType" ] }
]
}
}
}
}
}}
]);
Or with the more expressive form of $lookup since MongoDB 3.6:
User.aggregate([
{ "$match": { _id: ObjectId(req.params.id) } },
{ "$lookup": {
"from": FormatType.collection.name,
"let": { "portfolio": "$portfolio" },
"as": "portfolio",
"pipeline": [
{ "$match": {
"$expr": {
"$in": [ "$_id", "$$portfolio.formatType" ]
}
}},
{ "$project": {
"_id": {
"$arrayElemAt": [
"$$portfolio._id",
{ "$indexOfArray": [ "$$portfolio.formatType", "$_id" ] }
]
},
"name": {
"$arrayElemAt": [
"$$portfolio.name",
{ "$indexOfArray": [ "$$portfolio.formatType", "$_id" ] }
]
},
"formatType": "$$ROOT",
}}
]
}}
]);
The two approaches work slightly differently, but both essentially work with the concept of returning the matching "related" entries and then "re-mapping" onto the existing array content in order to merge with the "name" properties "embedded" inside the array. That is actually the main complication that otherwise is a fairly straightforward method of retrieval.
It's pretty much the same process as what populate() actually does on the "client" but executed on the "server". So the comparisons are using the $indexOfArray operator to find where the matching ObjectId values are and then return a property from the array at that matched "index" via the $arrayElemAt operation.
The only difference is that in the MongoDB 3.6 compatible version, we do that "substitution" within the "foreign" content "before" the joined results are returned to the parent. In prior releases we return the whole matching foreign array and then "marry up" the two to form a singular "merged" array using $map.
Whilst these may initially look "more complex", the big advantage here is that these constitute a "single request" to the server with a "single response" and not issuing and receiving "multiple" requests as populate() does. This actually saves a lot of overhead in network traffic and greatly increases response time.
In addition, these are "real joins" so there is a lot more you can do which cannot be achieved with "multiple queries". For instance you can "sort" results on the "join" and only return the top results, where as using populate() needs to pull in "all parents" before it can even look for which "children" to return in result. The same goes for "filtering" conditions on the child "join" as well.
There is some more detail on this on Querying after populate in Mongoose about the general limitations and what you actually can even practically do to "automate" the generation of such "complex" aggregation pipeline statements where needed.
Demonstration
Another common problem with doing these "joins" and understanding referenced schema in general is that people often get the concepts wrong on where and when to store the references and how it all works. Therefore the following listings serve as demonstration of both the storage and retrieval of such data.
In a native Promises implementation for older NodeJS releases:
const { Schema } = mongoose = require('mongoose');
const uri = 'mongodb://localhost/usertest';
mongoose.Promise = global.Promise;
mongoose.set('debug',true);
const formatTypeSchema = new Schema({
name: String
});
const portfolioSchema = new Schema({
name: String,
formatType: { type: Schema.Types.ObjectId, ref: 'FormatType' }
});
const userSchema = new Schema({
name: String,
portfolio: [portfolioSchema]
});
const FormatType = mongoose.model('FormatType', formatTypeSchema);
const User = mongoose.model('User', userSchema);
const log = data => console.log(JSON.stringify(data, undefined, 2));
(function() {
mongoose.connect(uri).then(conn => {
let db = conn.connections[0].db;
return db.command({ buildInfo: 1 }).then(({ version }) => {
version = parseFloat(version.match(new RegExp(/(?:(?!-).)*/))[0]);
return Promise.all(Object.entries(conn.models).map(([k,m]) => m.remove()))
.then(() => FormatType.insertMany(
[ 'A', 'B', 'C' ].map(name => ({ name }))
)
.then(([A, B, C]) => User.insertMany(
[
{
name: 'User 1',
portfolio: [
{ name: 'Port A', formatType: A },
{ name: 'Port B', formatType: B }
]
},
{
name: 'User 2',
portfolio: [
{ name: 'Port C', formatType: C }
]
}
]
))
.then(() => User.find())
.then(users => log({ users }))
.then(() => User.findOne({ name: 'User 1' })
.populate('portfolio.formatType')
)
.then(user1 => log({ user1 }))
.then(() => User.aggregate([
{ "$match": { "name": "User 2" } },
{ "$lookup": {
"from": FormatType.collection.name,
"localField": "portfolio.formatType",
"foreignField": "_id",
"as": "formats"
}},
{ "$project": {
"name": 1,
"portfolio": {
"$map": {
"input": "$portfolio",
"in": {
"name": "$$this.name",
"formatType": {
"$arrayElemAt": [
"$formats",
{ "$indexOfArray": [ "$formats._id", "$$this.formatType" ] }
]
}
}
}
}
}}
]))
.then(user2 => log({ user2 }))
.then(() =>
( version >= 3.6 ) ?
User.aggregate([
{ "$lookup": {
"from": FormatType.collection.name,
"let": { "portfolio": "$portfolio" },
"as": "portfolio",
"pipeline": [
{ "$match": {
"$expr": {
"$in": [ "$_id", "$$portfolio.formatType" ]
}
}},
{ "$project": {
"_id": {
"$arrayElemAt": [
"$$portfolio._id",
{ "$indexOfArray": [ "$$portfolio.formatType", "$_id" ] }
]
},
"name": {
"$arrayElemAt": [
"$$portfolio.name",
{ "$indexOfArray": [ "$$portfolio.formatType", "$_id" ] }
]
},
"formatType": "$$ROOT",
}}
]
}}
]).then(users => log({ users })) : ''
);
})
.catch(e => console.error(e))
.then(() => mongoose.disconnect());
})()
And with async/await syntax for newer NodeJS releases, including current LTS v.8.x series:
const { Schema } = mongoose = require('mongoose');
const uri = 'mongodb://localhost/usertest';
mongoose.Promise = global.Promise;
mongoose.set('debug',true);
const formatTypeSchema = new Schema({
name: String
});
const portfolioSchema = new Schema({
name: String,
formatType: { type: Schema.Types.ObjectId, ref: 'FormatType' }
});
const userSchema = new Schema({
name: String,
portfolio: [portfolioSchema]
});
const FormatType = mongoose.model('FormatType', formatTypeSchema);
const User = mongoose.model('User', userSchema);
const log = data => console.log(JSON.stringify(data, undefined, 2));
(async function() {
try {
const conn = await mongoose.connect(uri);
let db = conn.connections[0].db;
let { version } = await db.command({ buildInfo: 1 });
version = parseFloat(version.match(new RegExp(/(?:(?!-).)*/))[0]);
log(version);
// Clean data
await Promise.all(Object.entries(conn.models).map(([k,m]) => m.remove()));
// Insert some things
let [ A, B, C ] = await FormatType.insertMany(
[ 'A', 'B', 'C' ].map(name => ({ name }))
);
await User.insertMany(
[
{
name: 'User 1',
portfolio: [
{ name: 'Port A', formatType: A },
{ name: 'Port B', formatType: B }
]
},
{
name: 'User 2',
portfolio: [
{ name: 'Port C', formatType: C }
]
}
]
);
// Show plain users
let users = await User.find();
log({ users });
// Get user with populate
let user1 = await User.findOne({ name: 'User 1' })
.populate('portfolio.formatType');
log({ user1 });
// Get user with $lookup
let user2 = await User.aggregate([
{ "$match": { "name": "User 2" } },
{ "$lookup": {
"from": FormatType.collection.name,
"localField": "portfolio.formatType",
"foreignField": "_id",
"as": "formats"
}},
{ "$project": {
"name": 1,
"portfolio": {
"$map": {
"input": "$portfolio",
"in": {
"name": "$$this.name",
"formatType": {
"$arrayElemAt": [
"$formats",
{ "$indexOfArray": [ "$formats._id", "$$this.formatType" ] }
]
}
}
}
}
}}
]);
log({ user2 });
// Expressive $lookup
if ( version >= 3.6 ) {
let users = await User.aggregate([
{ "$lookup": {
"from": FormatType.collection.name,
"let": { "portfolio": "$portfolio" },
"as": "portfolio",
"pipeline": [
{ "$match": {
"$expr": {
"$in": [ "$_id", "$$portfolio.formatType" ]
}
}},
{ "$project": {
"_id": {
"$arrayElemAt": [
"$$portfolio._id",
{ "$indexOfArray": [ "$$portfolio.formatType", "$_id" ] }
]
},
"name": {
"$arrayElemAt": [
"$$portfolio.name",
{ "$indexOfArray": [ "$$portfolio.formatType", "$_id" ] }
]
},
"formatType": "$$ROOT",
}}
]
}}
]);
log({ users })
}
mongoose.disconnect();
} catch(e) {
console.log(e)
} finally {
process.exit()
}
})()
The latter listing if commented on each stage to explain the parts, and you can at least see by comparison how both forms of syntax relate to each other.
Note that the "expressive" $lookup example only runs where the MongoDB server connected to actually supports the syntax.
And the "output" for those who cannot be bothered to run the code themselves:
Mongoose: formattypes.remove({}, {})
Mongoose: users.remove({}, {})
Mongoose: formattypes.insertMany([ { _id: 5b1601d8be9bf225554783f5, name: 'A', __v: 0 }, { _id: 5b1601d8be9bf225554783f6, name: 'B', __v: 0 }, { _id: 5b1601d8be9bf225554783f7, name: 'C', __v: 0 } ], {})
Mongoose: users.insertMany([ { _id: 5b1601d8be9bf225554783f8, name: 'User 1', portfolio: [ { _id: 5b1601d8be9bf225554783fa, name: 'Port A', formatType: 5b1601d8be9bf225554783f5 }, { _id: 5b1601d8be9bf225554783f9, name: 'Port B', formatType: 5b1601d8be9bf225554783f6 } ], __v: 0 }, { _id: 5b1601d8be9bf225554783fb, name: 'User 2', portfolio: [ { _id: 5b1601d8be9bf225554783fc, name: 'Port C', formatType: 5b1601d8be9bf225554783f7 } ], __v: 0 } ], {})
Mongoose: users.find({}, { fields: {} })
{
"users": [
{
"_id": "5b1601d8be9bf225554783f8",
"name": "User 1",
"portfolio": [
{
"_id": "5b1601d8be9bf225554783fa",
"name": "Port A",
"formatType": "5b1601d8be9bf225554783f5"
},
{
"_id": "5b1601d8be9bf225554783f9",
"name": "Port B",
"formatType": "5b1601d8be9bf225554783f6"
}
],
"__v": 0
},
{
"_id": "5b1601d8be9bf225554783fb",
"name": "User 2",
"portfolio": [
{
"_id": "5b1601d8be9bf225554783fc",
"name": "Port C",
"formatType": "5b1601d8be9bf225554783f7"
}
],
"__v": 0
}
]
}
Mongoose: users.findOne({ name: 'User 1' }, { fields: {} })
Mongoose: formattypes.find({ _id: { '$in': [ ObjectId("5b1601d8be9bf225554783f5"), ObjectId("5b1601d8be9bf225554783f6") ] } }, { fields: {} })
{
"user1": {
"_id": "5b1601d8be9bf225554783f8",
"name": "User 1",
"portfolio": [
{
"_id": "5b1601d8be9bf225554783fa",
"name": "Port A",
"formatType": {
"_id": "5b1601d8be9bf225554783f5",
"name": "A",
"__v": 0
}
},
{
"_id": "5b1601d8be9bf225554783f9",
"name": "Port B",
"formatType": {
"_id": "5b1601d8be9bf225554783f6",
"name": "B",
"__v": 0
}
}
],
"__v": 0
}
}
Mongoose: users.aggregate([ { '$match': { name: 'User 2' } }, { '$lookup': { from: 'formattypes', localField: 'portfolio.formatType', foreignField: '_id', as: 'formats' } }, { '$project': { name: 1, portfolio: { '$map': { input: '$portfolio', in: { name: '$$this.name', formatType: { '$arrayElemAt': [ '$formats', { '$indexOfArray': [ '$formats._id', '$$this.formatType' ] } ] } } } } } } ], {})
{
"user2": [
{
"_id": "5b1601d8be9bf225554783fb",
"name": "User 2",
"portfolio": [
{
"name": "Port C",
"formatType": {
"_id": "5b1601d8be9bf225554783f7",
"name": "C",
"__v": 0
}
}
]
}
]
}
Mongoose: users.aggregate([ { '$lookup': { from: 'formattypes', let: { portfolio: '$portfolio' }, as: 'portfolio', pipeline: [ { '$match': { '$expr': { '$in': [ '$_id', '$$portfolio.formatType' ] } } }, { '$project': { _id: { '$arrayElemAt': [ '$$portfolio._id', { '$indexOfArray': [ '$$portfolio.formatType', '$_id' ] } ] }, name: { '$arrayElemAt': [ '$$portfolio.name', { '$indexOfArray': [ '$$portfolio.formatType', '$_id' ] } ] }, formatType: '$$ROOT' } } ] } } ], {})
{
"users": [
{
"_id": "5b1601d8be9bf225554783f8",
"name": "User 1",
"portfolio": [
{
"_id": "5b1601d8be9bf225554783fa",
"name": "Port A",
"formatType": {
"_id": "5b1601d8be9bf225554783f5",
"name": "A",
"__v": 0
}
},
{
"_id": "5b1601d8be9bf225554783f9",
"name": "Port B",
"formatType": {
"_id": "5b1601d8be9bf225554783f6",
"name": "B",
"__v": 0
}
}
],
"__v": 0
},
{
"_id": "5b1601d8be9bf225554783fb",
"name": "User 2",
"portfolio": [
{
"_id": "5b1601d8be9bf225554783fc",
"name": "Port C",
"formatType": {
"_id": "5b1601d8be9bf225554783f7",
"name": "C",
"__v": 0
}
}
],
"__v": 0
}
]
}