mongoose distinct and populate with documents - node.js

I have the following model:
var followerSchema = new Schema({
id_follower: {type: Schema.Types.ObjectId, ref: 'Users'},
id_post: {type: Schema.Types.ObjectId, ref: 'Posts'}
});
I want to be able to find all posts for a list of followers. When I use find, it returns me of course multiple times the same post as multiple users can follow the same post.
So I tried to use distinct, but I have the feeling the "populate" does not work afterwards.
Here is my code:
followerModel
.distinct('id_post',{id_follower:{$in:followerIds}})
.populate('id_post')
.sort({'id_post.creationDate':1})
.exec(function (err, postFollowers) {
console.log(postFollowers);
})
It only returns me the array of the posts, and it is not populated.
I am new to mongoDB, but according to the documentation of mongoose, the "distinct" method should return a query, just as the "find" method. And on a query you can execute the "populate" method, so I don't see what I am doing wrong.
I also tried to use the .distinct() method of the query, so then my code was like this:
followerModel
.find({id_follower:{$in:followerIds}})
.populate('id_post')
.distinct('id_post')
.sort({'id_post.creationDate':1})
.exec(function (err, postFollowers) {
console.log(postFollowers);
})
In that case it works, but as in the documentation of mongoose you need to provide a callback function when you use the distinct method on a query, and so in my logs I get errors all over. A workaround would be to have a dummy callback function, but I want to avoid that...
Does anybody has an idea why the first attempt is not working? And if the second approach is acceptable by providing a dummy callback?

Would this be the right way considering the current lack of support in mongoose?
followerModel
.find({id_follower:{$in:followerIds}})
.distinct('id_post',function(error,ids) {
Posts.find({'_id':{$in : ids}},function(err,result) {
console.log(result);
});
});

You can simply use aggregate to group and populate the collection.
now we have the desired result
db.<your collection name>.aggregate([
{
$match: {<match your fields here>}
},
{
$group: {_id: <your field to group the collection>}
},
{
$lookup: {
from: "<your collection of the poupulated field or referenced field>",
localField: "<give the id of the field which yout want to populate from the collection you matched above cases>",
foreignField: "_id", //this references the id of the document to match the localField id in the from collection
as: 'arrayName', //<some name to the returned document, this is a single document array>
}
},
{
$project: {
//you really don't want the whole populated fields, you can select the fields you want
<field name>:
<1 or 0>, // 1 to select and 0 to not select
//you can add multiple fields here
//to select the fields that just returned from the last stage we can use
"arrayName._id": <1 or 0>,
}
}
])
//at last you can return the data
.then((data) =>{
console.log(data);
});
we have distinct() by $group and
populate() by $lookup
and we also select() by $project

Related

MongoDB: How to perform a second match using the results (an array of ObjectIds) of the previous match in aggregation pipeline

I have a MongoDB collection called users with documents that look like:
{
_id: ObjectId('123'),
username: "abc",
avatar: "avatar/long-unique-random-string.jpg",
connections: [ObjectId('abc'), ObjectId('xyz'), ObjectId('lmn'), ObjectId('efg')]
}
This document belongs to the users collection.
What I want to do:
First, find one document from the users' collection that matches _id -> '123'.
Project the connections field received from step 1, which is an array of ObjectIds of other users within the same collection.
Find all documents of users from the array field projected in step 2.
Project and return an array of only the username and avatar of all those users from step 3.
While I know that I can do this in two separate queries. First using findOne which returns the friends array. Then, using find with the results of findOne to get all the corresponding usernames and avatars.
But, I would like to do this in one single query, using the aggregation pipeline.
What I want to know, is it even possible to do this in one query using aggregation?
If so, what would the query look like?
What, I currently have:
await usersCollection
.aggregate([
{ $match: { _id: new ObjectId(userId) } },
{ $project: { ids: "$connections" } },
{ $match: { _id: { $in: "ids" } } },
{
$project: {
username: "$username",
avatar: { $ifNull: ["$avatar", "$$REMOVE"] },
},
},
])
.toArray()
I know this is wrong because each aggregation stage receives the results from the previous stage. So, the second match cannot query on the entire users' collection, as far as I know.
I'm using MongoDB drivers for nodeJS. And I would like to avoid $lookup for possible solutions.

MongoDB - Mongoose with NodeJS find and join collection

I need to find and join another collection to get the businesses data from businesses collection and the profile description which is saved in the profiles collection. Latest version of nodejs and mongoose.
businesses = await Business.find({}, "business_id name industry")
.limit(limit * 1)
.skip((page - 1) * limit)
.exec();
That is the code, which I need later also for the pagination.
Now I found a solution with $Lookup in Mongoose. My code looks like
Business.aggregate([{
$lookup: {
from: "profiles", // collection name in db
localField: "business_id",
foreignField: "business_id",
as: "profile"
}
}]).exec(function(err, profile) {
console.log(profile[0]);
});
The Business and Profile is saved with the business_id in a field. So I can't work with _id from Mongoose. I never before work with mongoose and two collections.
The issue is now that the profile[0] is not connected to the correct business. So the profile is a another one as from the business find above.
I need to find the latest 10 Businesses and join to another collection and grap also the profile details. What I make wrong here, has anyone a example for this behauivor ?
Use https://mongoosejs.com/docs/populate.html
As in your case you don't have ObjectId here you can use populate-virtuals
So far you've only populated based on the _id field. However, that's sometimes not the right choice. In particular, arrays that grow without bound are a MongoDB anti-pattern. Using mongoose virtuals, you can define more sophisticated relationships between documents.
const BusinessSchema = new Schema({
name: String
});
BusinessSchema.virtual('profile', {
ref: 'Profile', // The model to use
localField: 'business_id', // Find people where `localField`
foreignField: 'business_id', // is equal to `foreignField`
// If `justOne` is true, 'members' will be a single doc as opposed to
// an array. `justOne` is false by default.
justOne: false,
options: { sort: { name: -1 }, limit: 5 } // Query options, see "bit.ly/mongoose-query-options"
});

MongoDb: Insert or update multiple documents with a unique index

I have a MongoDB collection with a unique index.
I am trying to insert or update an array of documents into that collection.
If there is NO existing document in the collection matching the unique index of the document, that new document should be inserted into the collection.
However, if there IS already a document in the collection with that unique index, it should be updated with the fields of the new document. Any fields that are NOT present in the new document should be left untouched.
This is what I have currently which is working for inserting (but NOT for updating).
const mongojs = require('mongojs');
const db = mongojs('mongodb://username:password#address.mlab.com:37230/database');
// items is an array of documents
db.items.insert(items, (err, task) => {
if (err) {
console.log(err);
}
})
I understand this is wrong and it currently gives this error:
E11000 duplicate key error index: database.items.$upc_1 dup key:
What is the proper query for this?
You can try using mongodb bulkWrite api:
var ops = []
items.forEach(item => {
ops.push(
{
updateOne: {
filter: { _id: unique_id },
update: {
$set: { fields_to_set_if_exists },
$setOnInsert: { fileds_to_insert_if_does_not_exist }
},
upsert: true
}
}
)
})
db.collections('collection_name').bulkWrite(ops, { ordered: false });
I don't believe that you can update an entire array of documents at the same time. Therefore, you would have to update each item in the array individually.
items.forEach((item) => {
db.items.update({_id: item._id}, item, {upsert: true}, (err, task) => {
if (err) {
console.log(err);
}
});
}
The {upsert: true} option will update if a record exists and insert if not.
What are you looking for is an upsert, not an insert. It can be done by the following code:
db.collection.update(
<query>,
<updates>,
{
upsert: <boolean>,
multi: <boolean>,
writeConcern: <document>,
collation: <document>
}
)
Query will search for a document using the parameters of the query, if it finds, it will update the fields mentioned in . If it doesn't find, it will insert a new document with the fields and values of .
The last object (with multiple fields), contains a field to say if an upsert is desired and one called "multi" to say if an update on multiple documents is desired.
For example:
db.items.update({name:"item1"},{$set:{price:20}},{upsert:true})
This will search for a document with the name "item1" and set its price to 20. If it doesn't find, it will create a new one with price 20.
One thing to be noticed though is:
If you don't use the tag $set on the fields, it will substitute the whole document.
Supposing you have a document like this:
{_id:"1234",name:"item1",price:10}
If you run the following two queries:
db.items.update({name:"item1"},{$set:{price:20}},...)
and
db.items.update({name:"item1"},{price:20},...)
it will yeld different results:
First one:
{_id:"1234",name:"item1",price:20}
Second one:
{_id:"1234",price:20}
If you don't call $set, it will change the whole document.
More information on the manual:
https://docs.mongodb.com/manual/reference/method/db.collection.update/
Hope my answer was helpful

Aggregate and flatten an array field in MongoDB

I have a Schema:
var ProjectSchema = new Schema({
name: {
type: String,
default: ''
},
topics: [{
type: Schema.ObjectId,
ref: 'Topic'
}],
user: {
type: Schema.ObjectId,
ref: 'User'
}
});
What I want to do is get an array with all topics from all projects. I cannot query Topic directly and get a full list because some topics are unassigned and they do not hold a reference back to a Project (for reasons of avoiding two way references). So I need to query Project and aggregate some how. I am doing something like:
Project.aggregate([{$project:{topics:1}}]);
But this is giving me an array of Project objects with the topics field. What I want is an array with topic objects.
How can I do this?
When dealing with arrays you typically want to use $unwind on the array members first and then $group to find the distinct entries:
Project.aggregate(
[
{ "$unwind": "$topics" },
{ "$group": { "_id": "$topics._id" } }
],
function(err,docs) {
}
)
But for this case, it is probably simplier to just use .distinct() which will do the same as above, but with just an array of results rather than documents:
Project.distinct("topics._id",function(err,topics) {
});
But wait there a minute because I know what you are really asking here. It's not the _id values you want but your Topic data has a property on it like "name".
Since your items are "referenced" and in another collection, you cannot do an aggregation pipeline or .distinct() operation on the property of a document in another collection. Put basically "MongoDB does not perform Joins" and mongoose .populate() is not a join, just something that "emulates" that with additional query(ies).
But you can of course just find the "distinct" values from "Project" and then fetch the information from "Topic". As in:
Project.distinct("topics._id",function(err,topics) {
Topic.find({ "_id": { "$in": topics } },function(err,topics) {
});
});
Which is handy because the .distinct() function already returned an array suitable for use with $in.

Mongoose $push cannot push object into correct document

I have a mongoose schema like this:
A = {
_id: Schema.Types.ObjectId,
arrayA:[{
_id,
nestedArray: [Schema.Types.ObjectId]
}],
arrayB: [Schema.Types.ObjectId]
}
I would like to push an Object Id into nestedArray in specific arrayA object AND
arrayB should contains an specific Object Id by following code:
A.update({'arrayA._id': arrayAId, arrayB: {$in: [arrayContainsSomeArrayBIds]}},
{$push: {'arrayA.$.nestedArray': nestedArrayId}}, function(err) {
});
However, the Object Id is pushed into nestedArray of the last object in arrayA.
If arrayB: {$in: [arrayContainsSomeArrayBIds]} is removed, the Object Id can be pushed into correct object in arrayA.
mongoose version: 3.8.21
Can anyone help me to find out the problem?
Currently it is not possible in MongoDB to update an a array element with the positional operator, when the query document contains references to other arrays apart from the one being updated.
The below code, contains reference to two arrays fields: arrayA and arrayB, when
the update is issued on arrayA. This is invalid and would lead to undesired behavior.
A.update({'arrayA._id': arrayAId, arrayB: {$in: [arrayContainsSomeArrayBIds]}},
{$push: {'arrayA.$.nestedArray': nestedArrayId}}, function(err) {
});
From the docs,
Only one array field may appear in the query document.
The query document should only contain a single condition on the array field
being projected.
Multiple conditions may override each other
internally and lead to undefined behavior.
Under these requirements,
the following query is incorrect:
db.collection.find( { <array>: <value>, <someOtherArray>: <value2> },
{ "<array>.$": 1 } )
The solution is to modify your code to fire two queries:
Get the _ids of the documents, which match our condition.
Then perform the update.
Sample Code flow:
A.find({'arrayA._id': arrayAId, arrayB: {$in: [arrayContainsSomeArrayBIds]}},
function(err,data){
data.forEach(function(doc){
A.update({'arrayA._id': arrayAId,
"_id":doc._id},
{$push: {'arrayA.$.nestedArray': nestedArrayId}},
function(err) {
});
})
});

Resources