I have a Schema:
var ProjectSchema = new Schema({
name: {
type: String,
default: ''
},
topics: [{
type: Schema.ObjectId,
ref: 'Topic'
}],
user: {
type: Schema.ObjectId,
ref: 'User'
}
});
What I want to do is get an array with all topics from all projects. I cannot query Topic directly and get a full list because some topics are unassigned and they do not hold a reference back to a Project (for reasons of avoiding two way references). So I need to query Project and aggregate some how. I am doing something like:
Project.aggregate([{$project:{topics:1}}]);
But this is giving me an array of Project objects with the topics field. What I want is an array with topic objects.
How can I do this?
When dealing with arrays you typically want to use $unwind on the array members first and then $group to find the distinct entries:
Project.aggregate(
[
{ "$unwind": "$topics" },
{ "$group": { "_id": "$topics._id" } }
],
function(err,docs) {
}
)
But for this case, it is probably simplier to just use .distinct() which will do the same as above, but with just an array of results rather than documents:
Project.distinct("topics._id",function(err,topics) {
});
But wait there a minute because I know what you are really asking here. It's not the _id values you want but your Topic data has a property on it like "name".
Since your items are "referenced" and in another collection, you cannot do an aggregation pipeline or .distinct() operation on the property of a document in another collection. Put basically "MongoDB does not perform Joins" and mongoose .populate() is not a join, just something that "emulates" that with additional query(ies).
But you can of course just find the "distinct" values from "Project" and then fetch the information from "Topic". As in:
Project.distinct("topics._id",function(err,topics) {
Topic.find({ "_id": { "$in": topics } },function(err,topics) {
});
});
Which is handy because the .distinct() function already returned an array suitable for use with $in.
Related
An update in the array of objects inside another array of objects.
mongodb field that I'm working on:
otherFields: values,
tasks: [
{
_id: mongodb.objectID(),
title: string,
items:[{
_id: mongodb.objectID(),
title: string,
completed: boolean //field need to be update.
}]
},
{}...
],
otherFields: value
sample mongodb document
I need to find the document using the task_id and the item_id and update a completed field in item of a task. Using the mongoose findOneAndUpdate method
const path = "tasks.$.items." + item_id + "completed";
collectionName.findOneAndUpdate(
{ _id: req.user._id, "tasks._id": taskID },
{ $set: { [path]: true }});
The above query doesn't work!!!
There is no need to use multiple query conditions, since you'd like to update a specific item that has an unique ID. Therefore you could use something along the lines:
collectionName.findOneAndUpdate(
{ 'tasks.items._id': itemID },
...
);
Keep in mind this structure is far away from optimized as it would basically look through the entire database...
Also now that I think of it, you'd also have issue with the update, as there are two nested arrays within the document. Read more here: How to Update Multiple Array Elements in mongodb
I have a MongoDB collection called users with documents that look like:
{
_id: ObjectId('123'),
username: "abc",
avatar: "avatar/long-unique-random-string.jpg",
connections: [ObjectId('abc'), ObjectId('xyz'), ObjectId('lmn'), ObjectId('efg')]
}
This document belongs to the users collection.
What I want to do:
First, find one document from the users' collection that matches _id -> '123'.
Project the connections field received from step 1, which is an array of ObjectIds of other users within the same collection.
Find all documents of users from the array field projected in step 2.
Project and return an array of only the username and avatar of all those users from step 3.
While I know that I can do this in two separate queries. First using findOne which returns the friends array. Then, using find with the results of findOne to get all the corresponding usernames and avatars.
But, I would like to do this in one single query, using the aggregation pipeline.
What I want to know, is it even possible to do this in one query using aggregation?
If so, what would the query look like?
What, I currently have:
await usersCollection
.aggregate([
{ $match: { _id: new ObjectId(userId) } },
{ $project: { ids: "$connections" } },
{ $match: { _id: { $in: "ids" } } },
{
$project: {
username: "$username",
avatar: { $ifNull: ["$avatar", "$$REMOVE"] },
},
},
])
.toArray()
I know this is wrong because each aggregation stage receives the results from the previous stage. So, the second match cannot query on the entire users' collection, as far as I know.
I'm using MongoDB drivers for nodeJS. And I would like to avoid $lookup for possible solutions.
i'm truing to fetch result from my mongodb server, query: get cars that in nearest agency
this what i have tried but getting result without sorting
let cars = await Cars.find({disponible: true})
.populate({
path: 'agency',
match: {
"location": {
$near: {
$geometry: {
coordinates: [ latitude , longitude ]
},
}
}
},
select: 'name'
})
.select('name agency');
// send result via api
res.status(200).json({cars})
my schemas
//Car Schema
const carSchema = new Schema({
name: { type: String, required: true},
agency: {type: Schema.Types.ObjectId, ref: 'agencies'},
}, { timestamps: true });
//Agency Schema
const agencySchema = new Schema({
name: { type: String, required: true},
location: {
type: {
type: String,
enum: ['Point'],
default: 'Point'
},
coordinates: {
type: [Number],
required: true
}
},
}, { timestamps: true });
i want to get cars with agency but sorted by the nearest agency
Theres a reason populate() cannot work
Using populate() you won't be able to do this, and for a number of reasons. The main reason being that all populate() is doing is essentially marrying up your foreign reference to results from another collection with given query parameters.
In fact with a $near query, the results could be quite weird, since you might not receive enough "near" results to actually marry up with all the parent references.
There's a bit more detail about the "foreign constraint" limitation with populate() in existing answers to Querying after populate in Mongoose and of course on the modern solution to this, which is $lookup.
Using $lookup and $geoNear
In fact, what you need is a $lookup along with a $geoNear, but you also must do the "join" the other way around to what you might expect. And thus from the Agency model you would do:
Agency.aggregate([
// First find "near" agencies, and project a distance field
{ "$geoNear": {
"near": {
"type": "Point",
"coordinates": [ longitude , latitude ]
},
"distanceField": "distance",
"spherical" true
}},
// Then marry these up to Cars - which can be many
{ "$lookup": {
"from": Car.collection.name,
"let": { "agencyId": "$_id" },
"pipeline": [
{ "$match": {
"disponible": true,
"$expr": { "$eq": [ "$$agencyId", "$agency" ] }
}}
],
"as": "cars"
}},
// Unwinding denormalizes that "many"
{ "$unwind": "$cars" },
// Group is "inverting" the result
{ "$group": {
"_id": "$cars._id",
"car": { "$first": "$cars" },
"agency": {
"$first": {
"$arrayToObject": {
"$filter": {
"input": { "$objectToArray": "$$ROOT" },
"cond": { "$ne": [ "$$this.k", "cars" ] }
}
}
}
}
}},
// Sort by distance, nearest is least
{ "$sort": { "agency.distance": 1 } },
// Reformat to expected output
{ "$replaceRoot": {
"newRoot": {
"$mergeObjects": [ "$car", { "agency": "$agency" } ]
}
}}
])
As stated the $geoNear part must come first. Bottom line is it basically needs to be the very first stage in an aggregation pipeline in order to use the mandatory index for such a query. Though it is true that given the form of $lookup shown here you "could" actually use a $near expression within the $lookup pipeline with a starting $match stage, it won't return what you expect since basically the constraint is already on the matching _id value. And it's really just the same problem populate() has in that regard.
And of course though $geoNear has a "query" constraint, you cannot use $expr within that option so this rules out that stage being used inside the $lookup pipeline again. And yes, still basically the same problem of conflicting constraints.
So this means you $geoNear from your Agency model instead. This pipeline stage has the additional thing it does which is it actually projects a "distanceField" into the result documents. So a new field within the documents ( called "distance" in the example ) will then indicate how far away from the queried point the matched document is. This is important for sorting later.
Of course you want this "joined" to the Car, so you want to do a $lookup. Note that since MongoDB has no knowledge of mongoose models the $lookup pipeline stage expects the "from" to be the actual collection name on the server. Mongoose models typically abstract this detail away from you ( though it's normally the plural of the model name, in lowercase ), but you can always access this from the .collection.name property on the model as shown.
The other arguments are the "let" in which you keep a reference to the _id of the current Agency document. This is used within the $expr of the $match in order to compare the local and foreign keys for the actual "joining" condition. The other constraints in the $match further filter down the matching "cars" to those criteria as well.
Now it's probably likely there are in fact many cars to each agency and that is one basic reason the model has been done like this in separate collections. Regardless of whether it's one to one or one to many, the $lookup result always produces an array. Basically we now want this array to "denormalize" and essentially "copy" the Agency detail for each found Car. This is where $unwind comes in. An added benefit is that when you $unwind the array of matching "cars", any empty array where the contraints did not match anything effectively removes the Agency from the possible results altogether.
Of course this is the the wrong way around from how you actually want the results, as it's really just "one car" with "one agency". This is where $group comes in and collects information "per car". Since this way around it is expected as "one to one", the $first operator is used as an accumulator.
There is a fancy expression in there with $objectToArray and $arrayToObject, but really all that is doing is removing the "cars" field from the "agency" content, just as the "$first": "$cars" is keeping that data separate.
Back to something closer to the desired output, the other main thing is to $sort the results so the "nearest" results are the ones listed first, just as the initial goal was all along. This is where you actually use the "distance" value which was added to the document in the original $geoNear stage.
At this point you are nearly there, and all that is needed is to reform the document into the expected output shape. The final $replaceRoot does this by taking the "car" value from the earlier $group output and promoting it to the top level object to return, and "merging" in the "agency" field to appear as part of the Car itself. Clearly $mergeObjects does the actual "merging".
That's it. It does work, but you may have spotted the problem that you don't actually get to say "near to this AND with this other constraint" technically as part of a single query. And a funny thing about "nearest" results is they do have an in-buit "limit" on results they should return.
And that is basically in the next topic to discuss.
Changing the Model
Whilst all the above is fine, it's still not really perfect and has a few problems. The most notable problem should be that it's quite complex and that "joins" in general are not good for performance.
The other considerable flaw is that as you might have gathered from the "query" parameter on the $geoNear stage, you are not really getting the equivalent of both conditions ( find nearest agency to AND car has disponible: true ) since on separate collections the initial "near" does not consider the other constraint.
Nor can this even be done from the original order just as was intended, and again comes back to the problem with populate() here.
So the real issue unfortunately is design. And it may be a difficult pill to swallow, but the current design which is extremely "relational" in nature is simply not a good fit for MongoDB in how it would handle this type of operation.
The core problem is the "join", and in order to make things work we basically need to get rid of it. And you do that in MongoDB design by embedding the document instead of keeping a reference in another collection:
const carSchema = new Schema({
name: { type: String, required: true},
agency: {
name: { type: String, required: true},
location: {
type: {
type: String,
enum: ['Point'],
default: 'Point'
},
coordinates: {
type: [Number],
required: true
}
}
}
}, { timestamps: true });
In short "MongoDB is NOT a relational database", and it also does not really "do joins" as the sort of itegral constraint over a join you are looking for simply is not supported.
Well, it's not supported by $lookup and the ways it will do things, but the official line has been and will always be that a "real join" in MongoDB is embedded detail. Which simply means "if it's meant to be a constraint on queries you want to do, then it belongs in the same document".
With that redesign the query simply becomes:
Car.find({
disponible: true,
"agency.location": {
$near: {
$geometry: {
coordinates: [ latitude , longitude ]
},
}
}
})
YES, that would mean that you likely duplicate a lot of information about an "agency" since the same data would likely be present on many cars. But the facts are that for this type of query usage, this is actually what MongoDB is expecting you to model as.
Conclusion
So the real choices here come down to which case suits your needs:
Accept that you are possibly returning less than the expected results due to "double filtering" though the use of a $geoNear and $lookup combination. Noting that $geoNear will only return 100 results by default, unless you change that. This can be an unreliable combination for "paged" results.
Restructure your data accepting the "duplication" of agency detail in order to get a proper "dual constraint" query since both criteria are in the same collection. It's more storage and maintenance, but it is more performant and completely reliable for "paged" results.
And of course if it's neither acceptable to use the aggregation approach shown or the restructure of data, then this can only show that MongoDB is probably not best suited to this type of problem, and you would be better off using an RDBMS where you decide you must keep normalized data as well as be able to query with both constraints in the same operation. Provided of course you can choose an RDBMS which actually supports the usage of such GeoSpatial queries along with "joins".
I have something like:
Schema Subdocument
name: String
data: Mixed
Schema Stuff
documents: [Subdocument]
Now, in my API there are two endpoints, one for the Subdocument and another for Stuff. When I want to get a Subdocument I need to contain the data field, but when I want to get Stuff, I want to show the name of those subdocuments, but I don't want to show the data field because is quite large and it won't be used.
So, to keep things clear, data is not private. It's just that I don't want it to be shown when I get it from Stuff
I tried by doing:
Stuff.findById(id)
.populate("documents")
.populate("-documents.data")
but that doesn't work... I'm getting the Stuffwith the Subdocumentcontaining the name and data. I feel like i'm missing to tell mongoose when I call populate("-documents.data") that documents is an array and I want to exclude the data field for each element in this array.
edit: Sorry the Schema I provided was not for my case. In my case it was not embedded, but a reference, like so:
Schema Subdocument
name: String
data: Mixed
Schema Stuff
documents: [{
type: Schema.Types.ObjectId,
ref: 'Subdocument'
}]
Assuming subDocument is not embedded and using as "ref" as you say populate is working but data part is not included:
Stuff.findById(id).populate( { "path" : "documents", "select" : "-data" })
Your documents have an "embedded" schema, so populate is not used here, it is used only for "referenced" schemas where the other objects are in another collection.
Fortunately with "embedded" there is an easy way using projection:
Stuff.findById(id,{ "documents.name": 1 },function(err,results) {
})
With results like
{ "documents": [{ "name": "this" },{ "name": "that" }] }
Or with .aggregate() and the $map operator:
Stuff.aggregate(
[
{ "$match": { "_id": ObjectID(id) } },
{ "$project": {
"documents": {
"$map": {
"$input": "$documents",
"as": "el",
"in": "$$el.name"
}
}
}}
],function(err,results) {
}
)
That will just tranform into an array of "only" the name "values", which is different to the last form.
{ "documents": ["this", "that"] }
Note, if using .aggregate() you need to properly cast the ObjectId as the autocasting from mongoose schema types does not work in aggregation pipeline stages.
As I wrote above, I have a schema which refers to another different schema. These are:
exports.Policies = new mongoose.Schema({
name: String,
description: String,
exploits: [ {type : mongoose.Schema.ObjectId, ref : 'exploit', required: true} ]
});
exports.exploit = new mongoose.Schema({
name: String,
type: String,
required: [String]
});
What I want to do is just know how many values have 'exploits' array as a field inside of the response, is this possible? In sql I have to write only "count(field)" and making a "group by", how might I do this?
An example:
{
"name" : "mmmmmm",
"description" : "jjjj",
"_id" : ObjectId("533721b91a985b883399cdc2"),
"n_exploits" : 2
}
Thanks in advance.
Actually there is just the $size operator if you have no need to filter array content.
Assuming that you have a model to your schema that is named "policies" then the following is achieved through use of .aggregate():
policies.aggregate([
{ "$unwind": "$exploits" }
{ "$group": {
"_id": "$_id",
"count": { "sum": 1 }
}}
],function(err,doc) {
});
So this works by unwinding the array and then counting the number of elements that are produced when you re-group.
In future releases there is a a new $size operator so you can skip those stages and just use project:
policies.aggregate([
{ "$project": {
"count": { "$size": "$exploits" }
}}
],function(err,doc) {
});
Or otherwise use that with $group and $sum to add up arrays across documents.
But for now you do the $unwind and $sum operations. Working inside the mongodb engine is typically done in "native" code and the JavaScript operators are not available. But also as native code these operators work very fast.
Also see the aggregation operator reference for other things you may wish to do.
Have you tried searching this before posting here? A simple google search with words "mongodb count group by" gives kind of a lot of results, which are not only telling you that you have too look into aggregation framework, but also give you some examples even on SO like this and this.