I have these two models:
User.js
const UserSchema = new Schema({
profile: {
type: Schema.Types.ObjectId,
ref: "profiles",
},
following: [
{
type: Schema.Types.ObjectId,
ref: "users",
},
],
});
module.exports = User = mongoose.model("users", UserSchema);
Profile.js
const ProfileSchema = new Schema({
videoURL: {
type: String,
},
});
module.exports = Profile = mongoose.model("profiles", ProfileSchema);
Here's an example of a User document:
{
"following": [
{
"profile":{
"videoURL":"video_url_1"
}
},
{
"profile":{
"videoURL":"video_url_2"
}
},
{
"profile":{}
},
{
"profile":{
"videoURL":"video_url_3"
}
},
{
"profile":{
"videoURL":"video_url_4"
}
},
{
"profile":{
"videoURL":"video_url_5"
}
},
{
"profile":{}
},
{
"profile":{
"videoURL":"video_url_6"
}
}
]
}
I am trying to implement an infinite scroll of the videos of the users followed by the connected user.
This means, I will have to filter user.following.profile.videoURL
WHERE videoURL exists
Suppose, I will be loading two videos, by two videos:
Response 1: ["video_url_1","video_url_2"]
Response 2: ["video_url_3","video_url_4"]
Response 3: ["video_url_5","video_url_6"]
Usually, infinite scroll is easy because all I have to load the documents 2 by 2 by order of storage without filtering on any field.
Example: Displaying the followed users two by two in an infinite scroll
User.findById(user_id).populate({
path: "following",
options: {
skip: 2 * page,
limit: 2,
},
});
But, now I have to perform filtering on each followed_user.profile.video, and return two by two. And I don't see how I can perform BOTH the filtering and the infinite scroll at the same time.
NOTE: According to the documentation:
In general, there is no way to make populate() filter stories based on properties of the story's author. For example, the below query won't return any results, even though author is populated.
const story = await Story.
findOne({ 'author.name': 'Ian Fleming' }).
populate('author').
exec();
story; // null
So I suppose, there is no way for me to use populate to filter based user.followers, based on each user.follower.profile.videoURL
I am not sure it is possible with populate method, but you can try aggregation pipeline,
$match user_id condition
$lookup with aggregation pipeline in users collection for following
$match following id condition
$lookup with profile for following.profile
$match videoURL should exists
$project to show profile field and get first element using $arrayElemAt
$slice to do pagination in following
let page = 0;
let limit = 2;
let skip = limit * page;
User.aggregate([
{ $match: { _id: mongoose.Types.ObjectId(user_id) } },
{
$lookup: {
from: "users",
let: { following: "$following" },
pipeline: [
{ $match: { $expr: { $in: ["$_id", "$$following"] } } },
{
$lookup: {
from: "profiles",
localField: "profile",
foreignField: "_id",
as: "profile"
}
},
{ $match: { "profile.videoURL": { $exists: true } } },
{
$project: {
profile: { $arrayElemAt: ["$profile", 0] }
}
}
],
as: "following"
}
},
{
$addFields: {
following: {
$slice: ["$following", skip, limit]
}
}
}
])
Playground
Suggestion:
You can improve your schema design,
removing profile schema and add profile object in users collection, so you can achieve easily your requirement using populate method,
put match condition in following populate for videoURL exists
const UserSchema = new Schema({
profile: {
type: {
videoURL: {
type: String
}
}
},
following: [
{
type: Schema.Types.ObjectId,
ref: "users"
}
]
});
module.exports = User = mongoose.model("users", UserSchema);
User.findById(user_id).populate({
path: "following",
match: {
"profile.videoURL": { $ne: null }
},
options: {
skip: 2 * page,
limit: 2,
}
});
So what you want is table with infinite scroll and:
You can opt given ways to approach your problem :
Load data (first page) into grid.
Set filter on a col.
Load data again, this time using the filter.
I have a Model called Notes. It has a subdocument requests which holds various documents with values userId, reqType, accepted value( false by default) and noteId of the sender, request and note respectively. When the user hits a certain route I want to keep all the data to be as their previous values, just updating the accepted field to true.
The below code leads to no change in the data or a different iteration leads to erasing all the data other than accepted field and modifying it to true.
How should I do this?
const noteSchema = new mongoose.Schema(
requests: [
{
userId: mongoose.Schema.ObjectId,
noteId: mongoose.Schema.ObjectId,
reqType: String,
accepted: {
type: Boolean,
default: false,
},
},
],
}
)
const Note = mongoose.model('Note', noteSchema)
const note = await Note.findById(req.body.noteId)
await note.updateOne({
requests: {
$elemMatch: {
userId: req.body.userId,
reqType: req.body.reqType,
noteId: req.body.noteId,
},
$set: { "requests.$.accepted": true },
},
})
You do not need to retrieve the document and then update it. Just update it. Use this one:
await Note.updateOne(
{
"requests.userId": req.body.userId,
"requests.reqType": req.body.reqType,
"requests.noteId": req.body.noteId
},
{
$set:
{
"requests.$.accepted":true
}
}
);
I checked, it worked.
With first part mongoose will find the document, with $set it will be updated.
I have a document in mongoDB structured like that
_id: ObjectId("generatedByMongo"),
name: {
required: true,
type: String,
trim: true
},
last: {
required: true,
type: String,
trim: true
},
grades: [{
grade: {
_id: ObjectId(""),
grade: Number,
date: date
}
}]
And to server I send array of objects containing 3 fields
[
{studentId}, {gradeId}, {newGrade}
]
What I'm trying to accomplish is I want to find in within that user collection grade with given gradeId and update it's value to newGrade. As far as I tried to do that I have done this
router.patch('/students/updateGrade',async(req,res) => {
const studentId = req.body.updateGradeArray[0].studentId;
const gradeId = req.body.updateGradeArray[0].gradeId;
const newGrade = req.body.updateGradeArray[0].newGrade;
try {
const student = await Student.find({_id: studentId})
.select({'grades': {$elemMatch: {_id: gradeId}}});
} catch(e) {
console.log(e);
}
}
);
If you intend to update just grade.grade(the number value), try this:
Student.updateOne(
// Find a document with _id matching the studentId
{ "_id": studentId },
// Update the student grade
{ $set: { "grades.$[selectedGrade].grade": newGrade } },
{ arrayFilters: [{ "selectedGrade._id": gradeId }] },
)
Why this should work:
Since you are trying to update a student document, you should be using one of MongoDB update methods not find. In the query above, I'm using the updateOne method. Inside the updateOne, I am using a combination of $set and $[identifier] update operators to update the student grade.
I hope this helps✌🏾
I'm having trouble 'upserting' to my array. The code below creates duplicates in my answers array which I definitely do not want and by now it's apparent $push will not work. I have tried using the different methodologies I see on SO for a while now but none are working for me. With this web app, users are allows to view a question on the website and respond with a 'yes' or 'no' response and they are allowed to change(upsert) their response at any one time meaning a sort of upsert takes place on the db at different times. How do get around this?
var QuestionSchema = Schema ({
title :String,
admin :{type: String, ref: 'User'},
answers :[{type: Schema.Types.Mixed, ref: 'Answer'}]
});
var AnswerSchema = Schema ({
_question :{type: ObjectId, ref: 'Question'},
employee :{type: String, ref: 'User'},
response :String,
isAdmin :{type: Boolean, ref: 'User'}
})
var UserSchema = Schema({
username : String,
isAdmin : {type: Boolean, default: false}
});
module.exports = mongoose.model('Question', QuestionSchema);
module.exports = mongoose.model('Answer', AnswerSchema);
module.exports = mongoose.model('User', UserSchema);
Question.update(
{_id: req.body.id},
{$push: {answers: {_question: req.body.id,
employee: req.body.employee,
response: req.body.response, //this variable changes (yes/no/null)
isAdmin: req.body.isAdmin}}},
{safe: true, upsert: true},
function(err, model) {
}
);
As I see it you seem a little confused and it's reflected in your schema. You don't seem to fully grasp the differences between "embedded" and "referenced" since your schema is actually an invalid "mash" of the two techniques.
Probably best to walk you through both of them.
Embedded Model
So instead of the schema you have defined, you should in fact have something more like this:
var QuestionSchema = Schema ({
title :String,
admin :{type: String, ref: 'User'},
answers :[AnswerSchema]
});
var AnswerSchema = Schema ({
employee :{type: String, ref: 'User'},
response :String,
isAdmin :{type: Boolean, ref: 'User'}
})
mongoose.model('Question', questionSchema);
NOTE: Question is the only actual model here. The AnswerSchema is completely "embedded".
Note the clear definition of the "schema" where the "answers" property in Question is defined as an "array" of AnswerSchema. This is how you do embedding and keep control of the types within the object inside the array.
As for the update, there is a clear logic pattern but you are simply not enforcing it. All you need to do is "tell" the update that you do not want to "push" a new item if something for that "unique" "employee" in the array already exists.
Also. This is NOT and "upsert". Upsert implies "creating a new one", which is different to what you want. You want to "push" to the array of an "existing" Question. If you leave "upsert" on there, then something not found creates a new Question. Which is of course wrong here.
Question.update(
{
"_id": req.body.id,
"answers.employee": { "$ne": req.body.employee },
}
},
{ "$push": {
"answers": {
"employee": req.body.employee,
"response": req.body.response,
"isAdmin": req.body.isAdmin
}
}},
function(err, numAffected) {
});
That will look to check that the "unique" "employee" in the array members already and will only $push where it is not already there.
As a bonus, if you intend to allow the user to "change their answer" then we do this incantation with .bulkWrite():
Question.collection.bulkWrite(
[
{ "updateOne": {
"filter": {
"_id": req.body.id,
"answers.employee": req.body.employee,
},
"update": {
"$set": {
"answers.$.response": req.body.response,
}
}
}},
{ "updateOne": {
"filter": {
"_id": req.body.id,
"answers.employee": { "$ne": req.body.employee },
},
"update": {
"$push": {
"answers": {
"employee": req.body.employee,
"response": req.body.response,
"isAdmin": req.body.isAdmin
}
}
}
}}
],
function(err, writeResult) {
}
);
This effectively puts two updates in one. The first to attempt to alter an existing answer and $set the response at the matched position, and the second to attempt to add a new answer where one was not found on the question.
Referenced Model
With a "referenced" model you actually have the real members of the Answer within their own collection. So instead the schema is defined like this:
var QuestionSchema = Schema ({
title :String,
admin :{type: String, ref: 'User'},
answers :[{ type: Schema.Types.ObjectId, ref: 'Answer' }]
});
var AnswerSchema = Schema ({
_question :{type: ObjectId, ref: 'Question'},
employee :{type: String, ref: 'User'},
response :String,
isAdmin :{type: Boolean, ref: 'User'}
})
mongoose.model('Answer', answerSchema);
mongoose.model('Question', questionSchema);
N.B The other ref's here to User such as :
employee :{type: String, ref: 'User'},
isAdmin :{type: Boolean, ref: 'User'}
These are also really incorrect, and also should be of Schema.Type.ObjectId as they will "reference" the actual _id field of User. But this is actually outside of the scope of this question as asked, so if you still don't grasp that after this read, then Ask a New Question so someone can explain. On with the rest of the answer.
That's the "general" shape of the schema though, with the important thing being the "ref" to the 'Anwser' "model", which is by the registered name. You can optionally just use your "_question" field in modern mongoose versions with a "virtual", but I'm skipping over "Adavanced Usage" for now and keeping it simple with an array of "references" still in the Question model.
In this case, since the Answer model is actually in it's own "collection", then the operations actually become "upserts". Where we only want to "create" when there is no "employee" response to the given "_question" id.
Also demonstrating with a Promise chain instead:
Answer.update(
{ "_question": req.body.id, "employee": req.body.employee },
{
"$set": {
"response": req.body.reponse
},
"$setOnInsert": {
"isAdmin": req.body.isAdmin
}
},
{ "upsert": true }
).then(resp => {
if ( resp.hasOwnProperty("upserted") ) {
return Question.update(
{ "_id": req.body.id, "answers": { "$ne": resp.upserted[0]._id },
{ "$push": { "answers": resp.upserted[0]._id } }
).exec()
}
return;
}).then(resp => {
// either undefined where it was not an upsert or
// the update result from Question where it was
}).catch(err => {
// something
})
This is actually a simple statement since "when matched" we want to change the "response" data with the payload of the request, and really only when "upserting" or "creating/inserting" is when we actually change other data such as the "employee" ( which is always implied for create as part of the query expression ) and the "isAdmin" which clearly should not change with each update request we then explicitly use $setOnInsert so it only writes those two fields on an actual "create".
In the "Promise Chain" we actually look to see if the update request to Answer actually resulted in an "upsert", and when it does we want to append to the array of Question where it does not already exist. In much the same way as the "embedded" example, it's best to look to see if the array actually has the item before modifying with the "update". Alternately you could $addToSet here and just let the query match the Question by _id. To me though, that's a wasted write.
Summary
Those are your different approaches to how you handle this. Each has their own use cases for which you can see a general summary some other answers of mine in:
Mongoose populate vs object nesting which is an overview of the different approaches and reasons behind them
How to Model a “likes” voting system with MongoDB which gives a bit more detail on the "unique array upserts" technique for "embedded" models.
Not "required" reading, but it may help expand your insight into which approach is best for your particular case.
Working Example
Copy these and put them in a directory and do an npm install to install local dependencies. The code will run and create the collections in the database making the alterations.
Logging is turned on with mongoose.set(debug,true) so you should look at the console output and see what it does, along with the resulting collections where answers will be recorded to the related questions, and overwritten instead of "duplicating" where that was also the intent.
Change the connection string if you have to. But that is all you should change in this listing for it's purpose. Both approaches described in the answer are demonstrated.
package.json
{
"name": "colex",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC",
"dependencies": {
"async": "^2.4.1",
"mongodb": "^2.2.29",
"mongoose": "^4.10.7"
}
}
index.js
const async = require('async'),
mongoose = require('mongoose'),
Schema = mongoose.Schema,
ObjectId = require('mongodb').ObjectID
mongoose.Promise = global.Promise;
mongoose.set('debug',true);
mongoose.connect('mongodb://localhost/coltest');
const userSchema = new Schema({
username: String,
isAdmin: { type: Boolean, default: false }
});
const answerSchemaA = new Schema({
employee: { type: Schema.Types.ObjectId, ref: 'User' },
response: String,
});
const answerSchemaB = new Schema({
question: { type: Schema.Types.ObjectId, ref: 'QuestionB' },
employee: { type: Schema.Types.ObjectId, ref: 'User' },
response: String,
});
const questionSchemaA = new Schema({
title: String,
admin: { type: Schema.Types.ObjectId, ref: 'User' },
answers: [answerSchemaA]
});
const questionSchemaB = new Schema({
title: String,
admin: { type: Schema.Types.ObjectId, ref: 'User' },
answers: [{ type: Schema.Types.ObjectId, ref: 'AnswerB' }]
});
const User = mongoose.model('User', userSchema);
const AnswerB = mongoose.model('AnswerB', answerSchemaB);
const QuestionA = mongoose.model('QuestionA', questionSchemaA);
const QuestionB = mongoose.model('QuestionB', questionSchemaB);
async.series(
[
// Clear data
(callback) => async.each(mongoose.models,(model,callback) =>
model.remove({},callback),callback),
// Create some data
(callback) =>
async.each([
{
"model": "User",
"object": {
"_id": "594a322619ddbd437193c759",
"name": "Admin",
"isAdmin": true
}
},
{
"model": "User",
"object": {
"_id": "594a323919ddbd437193c75a",
"name": "Bill"
}
},
{
"model": "User",
"object": {
"_id": "594a327b19ddbd437193c75b",
"name": "Ted"
}
},
{
"model": "QuestionA",
"object": {
"_id": "594a32f719ddbd437193c75c",
"admin": "594a322619ddbd437193c759",
"title": "Question A Model"
}
},
{
"model": "QuestionB",
"object": {
"_id": "594a32f719ddbd437193c75c",
"admin": "594a322619ddbd437193c759",
"title": "Question B Model"
}
}
],(data,callback) => mongoose.model(data.model)
.create(data.object,callback),
callback
),
// Submit Answers for Users - Question A
(callback) =>
async.eachSeries(
[
{
"_id": "594a32f719ddbd437193c75c",
"employee": "594a323919ddbd437193c75a",
"response": "Bills Answer"
},
{
"_id": "594a32f719ddbd437193c75c",
"employee": "594a327b19ddbd437193c75b",
"response": "Teds Answer"
},
{
"_id": "594a32f719ddbd437193c75c",
"employee": "594a323919ddbd437193c75a",
"response": "Bills Changed Answer"
}
].map(d => ([
{ "updateOne": {
"filter": {
"_id": ObjectId(d._id),
"answers.employee": ObjectId(d.employee)
},
"update": {
"$set": { "answers.$.response": d.response }
}
}},
{ "updateOne": {
"filter": {
"_id": ObjectId(d._id),
"answers.employee": { "$ne": ObjectId(d.employee) }
},
"update": {
"$push": {
"answers": {
"employee": ObjectId(d.employee),
"response": d.response
}
}
}
}}
])),
(data,callback) => QuestionA.collection.bulkWrite(data,callback),
callback
),
// Submit Answers for Users - Question A
(callback) =>
async.eachSeries(
[
{
"_id": "594a32f719ddbd437193c75c",
"employee": "594a323919ddbd437193c75a",
"response": "Bills Answer"
},
{
"_id": "594a32f719ddbd437193c75c",
"employee": "594a327b19ddbd437193c75b",
"response": "Teds Anwser"
},
{
"_id": "594a32f719ddbd437193c75c",
"employee": "594a327b19ddbd437193c75b",
"response": "Ted Changed it"
}
],
(data,callback) => {
AnswerB.update(
{ "question": data._id, "employee": data.employee },
{ "$set": { "response": data.response } },
{ "upsert": true }
).then(resp => {
console.log(resp);
if (resp.hasOwnProperty("upserted")) {
return QuestionB.update(
{ "_id": data._id, "employee": { "$ne": data.employee } },
{ "$push": { "answers": resp.upserted[0]._id } }
).exec()
}
return;
}).then(() => callback(null))
.catch(err => callback(err))
},
callback
)
],
(err) => {
if (err) throw err;
mongoose.disconnect();
}
)
Here was my quick work around before Neill updated his answer (I used a $pull & $push). Works just as his but I'll mark his correct as I believe it's more efficient.
Question.update(
{_id: req.body.id},
{$pull: {answers: { employee: req.body.employee}}},
{safe: true, multi:true, upsert: true},
function(err, model) {
}
);
Question.update(
{_id: req.body.id},
{$push: {answers: {_question: req.body.id,
employee: req.body.employee,
response: req.body.response,
isAdmin: req.body.isAdmin}}},
{safe: true, upsert: true},
function(err, model) {
}
);
I have two Schema defined as below:
var WorksnapsTimeEntry = BaseSchema.extend({
student: {
type: Schema.ObjectId,
ref: 'Student'
},
timeEntries: {
type: Object
}
});
var StudentSchema = BaseSchema.extend({
firstName: {
type: String,
trim: true,
default: ''
// validate: [validateLocalStrategyProperty, 'Please fill in your first name']
},
lastName: {
type: String,
trim: true,
default: ''
// validate: [validateLocalStrategyProperty, 'Please fill in your last name']
},
displayName: {
type: String,
trim: true
},
municipality: {
type: String
}
});
And I would like to loop thru each student and show it's time entries. So far I have this code which is obviously not right as I still dont know how do I join WorksnapTimeEntry schema table.
Student.find({ status: 'student' })
.populate('student')
.exec(function (err, students) {
if (err) {
return res.status(400).send({
message: errorHandler.getErrorMessage(err)
});
}
_.forEach(students, function (student) {
// show student with his time entries....
});
res.json(students);
});
Any one knows how do I achieve such thing?
As of version 3.2, you can use $lookup in aggregation pipeline to perform left outer join.
Student.aggregate([{
$lookup: {
from: "worksnapsTimeEntries", // collection name in db
localField: "_id",
foreignField: "student",
as: "worksnapsTimeEntries"
}
}]).exec(function(err, students) {
// students contain WorksnapsTimeEntries
});
You don't want .populate() here but instead you want two queries, where the first matches the Student objects to get the _id values, and the second will use $in to match the respective WorksnapsTimeEntry items for those "students".
Using async.waterfall just to avoid some indentation creep:
async.waterfall(
[
function(callback) {
Student.find({ "status": "student" },{ "_id": 1 },callback);
},
function(students,callback) {
WorksnapsTimeEntry.find({
"student": { "$in": students.map(function(el) {
return el._id
})
},callback);
}
],
function(err,results) {
if (err) {
// do something
} else {
// results are the matching entries
}
}
)
If you really must, then you can .populate("student") on the second query to get populated items from the other table.
The reverse case is to query on WorksnapsTimeEntry and return "everything", then filter out any null results from .populate() with a "match" query option:
WorksnapsTimeEntry.find().populate({
"path": "student",
"match": { "status": "student" }
}).exec(function(err,entries) {
// Now client side filter un-matched results
entries = entries.filter(function(entry) {
return entry.student != null;
});
// Anything not populated by the query condition is now removed
});
So that is not a desirable action, since the "database" is not filtering what is likely the bulk of results.
Unless you have a good reason not to do so, then you probably "should" be "embedding" the data instead. That way the properties like "status" are already available on the collection and additional queries are not required.
If you are using a NoSQL solution like MongoDB you should be embracing it's concepts, rather than sticking to relational design principles. If you are consistently modelling relationally, then you might as well use a relational database, since you won't be getting any benefit from the solution that has other ways to handle that.
It is late but will help many developers.
Verified with
"mongodb": "^3.6.2",
"mongoose": "^5.10.8",
Join two collections in mongoose
ProductModel.find({} , (err,records)=>{
if(records)
//reurn records
else
// throw new Error('xyz')
})
.populate('category','name') //select only category name joined collection
//.populate('category') // Select all detail
.skip(0).limit(20)
//.sort(createdAt : '-1')
.exec()
ProductModel Schema
const CustomSchema = new Schema({
category:{
type: Schema.ObjectId,
ref: 'Category'
},
...
}, {timestamps:true}, {collection: 'products'});
module.exports = model('Product',CustomSchema)
Category model schema
const CustomSchema = new Schema({
name: { type: String, required:true },
...
}, {collection: 'categories'});
module.exports = model('Category',CustomSchema)