i'm truing to fetch result from my mongodb server, query: get cars that in nearest agency
this what i have tried but getting result without sorting
let cars = await Cars.find({disponible: true})
.populate({
path: 'agency',
match: {
"location": {
$near: {
$geometry: {
coordinates: [ latitude , longitude ]
},
}
}
},
select: 'name'
})
.select('name agency');
// send result via api
res.status(200).json({cars})
my schemas
//Car Schema
const carSchema = new Schema({
name: { type: String, required: true},
agency: {type: Schema.Types.ObjectId, ref: 'agencies'},
}, { timestamps: true });
//Agency Schema
const agencySchema = new Schema({
name: { type: String, required: true},
location: {
type: {
type: String,
enum: ['Point'],
default: 'Point'
},
coordinates: {
type: [Number],
required: true
}
},
}, { timestamps: true });
i want to get cars with agency but sorted by the nearest agency
Theres a reason populate() cannot work
Using populate() you won't be able to do this, and for a number of reasons. The main reason being that all populate() is doing is essentially marrying up your foreign reference to results from another collection with given query parameters.
In fact with a $near query, the results could be quite weird, since you might not receive enough "near" results to actually marry up with all the parent references.
There's a bit more detail about the "foreign constraint" limitation with populate() in existing answers to Querying after populate in Mongoose and of course on the modern solution to this, which is $lookup.
Using $lookup and $geoNear
In fact, what you need is a $lookup along with a $geoNear, but you also must do the "join" the other way around to what you might expect. And thus from the Agency model you would do:
Agency.aggregate([
// First find "near" agencies, and project a distance field
{ "$geoNear": {
"near": {
"type": "Point",
"coordinates": [ longitude , latitude ]
},
"distanceField": "distance",
"spherical" true
}},
// Then marry these up to Cars - which can be many
{ "$lookup": {
"from": Car.collection.name,
"let": { "agencyId": "$_id" },
"pipeline": [
{ "$match": {
"disponible": true,
"$expr": { "$eq": [ "$$agencyId", "$agency" ] }
}}
],
"as": "cars"
}},
// Unwinding denormalizes that "many"
{ "$unwind": "$cars" },
// Group is "inverting" the result
{ "$group": {
"_id": "$cars._id",
"car": { "$first": "$cars" },
"agency": {
"$first": {
"$arrayToObject": {
"$filter": {
"input": { "$objectToArray": "$$ROOT" },
"cond": { "$ne": [ "$$this.k", "cars" ] }
}
}
}
}
}},
// Sort by distance, nearest is least
{ "$sort": { "agency.distance": 1 } },
// Reformat to expected output
{ "$replaceRoot": {
"newRoot": {
"$mergeObjects": [ "$car", { "agency": "$agency" } ]
}
}}
])
As stated the $geoNear part must come first. Bottom line is it basically needs to be the very first stage in an aggregation pipeline in order to use the mandatory index for such a query. Though it is true that given the form of $lookup shown here you "could" actually use a $near expression within the $lookup pipeline with a starting $match stage, it won't return what you expect since basically the constraint is already on the matching _id value. And it's really just the same problem populate() has in that regard.
And of course though $geoNear has a "query" constraint, you cannot use $expr within that option so this rules out that stage being used inside the $lookup pipeline again. And yes, still basically the same problem of conflicting constraints.
So this means you $geoNear from your Agency model instead. This pipeline stage has the additional thing it does which is it actually projects a "distanceField" into the result documents. So a new field within the documents ( called "distance" in the example ) will then indicate how far away from the queried point the matched document is. This is important for sorting later.
Of course you want this "joined" to the Car, so you want to do a $lookup. Note that since MongoDB has no knowledge of mongoose models the $lookup pipeline stage expects the "from" to be the actual collection name on the server. Mongoose models typically abstract this detail away from you ( though it's normally the plural of the model name, in lowercase ), but you can always access this from the .collection.name property on the model as shown.
The other arguments are the "let" in which you keep a reference to the _id of the current Agency document. This is used within the $expr of the $match in order to compare the local and foreign keys for the actual "joining" condition. The other constraints in the $match further filter down the matching "cars" to those criteria as well.
Now it's probably likely there are in fact many cars to each agency and that is one basic reason the model has been done like this in separate collections. Regardless of whether it's one to one or one to many, the $lookup result always produces an array. Basically we now want this array to "denormalize" and essentially "copy" the Agency detail for each found Car. This is where $unwind comes in. An added benefit is that when you $unwind the array of matching "cars", any empty array where the contraints did not match anything effectively removes the Agency from the possible results altogether.
Of course this is the the wrong way around from how you actually want the results, as it's really just "one car" with "one agency". This is where $group comes in and collects information "per car". Since this way around it is expected as "one to one", the $first operator is used as an accumulator.
There is a fancy expression in there with $objectToArray and $arrayToObject, but really all that is doing is removing the "cars" field from the "agency" content, just as the "$first": "$cars" is keeping that data separate.
Back to something closer to the desired output, the other main thing is to $sort the results so the "nearest" results are the ones listed first, just as the initial goal was all along. This is where you actually use the "distance" value which was added to the document in the original $geoNear stage.
At this point you are nearly there, and all that is needed is to reform the document into the expected output shape. The final $replaceRoot does this by taking the "car" value from the earlier $group output and promoting it to the top level object to return, and "merging" in the "agency" field to appear as part of the Car itself. Clearly $mergeObjects does the actual "merging".
That's it. It does work, but you may have spotted the problem that you don't actually get to say "near to this AND with this other constraint" technically as part of a single query. And a funny thing about "nearest" results is they do have an in-buit "limit" on results they should return.
And that is basically in the next topic to discuss.
Changing the Model
Whilst all the above is fine, it's still not really perfect and has a few problems. The most notable problem should be that it's quite complex and that "joins" in general are not good for performance.
The other considerable flaw is that as you might have gathered from the "query" parameter on the $geoNear stage, you are not really getting the equivalent of both conditions ( find nearest agency to AND car has disponible: true ) since on separate collections the initial "near" does not consider the other constraint.
Nor can this even be done from the original order just as was intended, and again comes back to the problem with populate() here.
So the real issue unfortunately is design. And it may be a difficult pill to swallow, but the current design which is extremely "relational" in nature is simply not a good fit for MongoDB in how it would handle this type of operation.
The core problem is the "join", and in order to make things work we basically need to get rid of it. And you do that in MongoDB design by embedding the document instead of keeping a reference in another collection:
const carSchema = new Schema({
name: { type: String, required: true},
agency: {
name: { type: String, required: true},
location: {
type: {
type: String,
enum: ['Point'],
default: 'Point'
},
coordinates: {
type: [Number],
required: true
}
}
}
}, { timestamps: true });
In short "MongoDB is NOT a relational database", and it also does not really "do joins" as the sort of itegral constraint over a join you are looking for simply is not supported.
Well, it's not supported by $lookup and the ways it will do things, but the official line has been and will always be that a "real join" in MongoDB is embedded detail. Which simply means "if it's meant to be a constraint on queries you want to do, then it belongs in the same document".
With that redesign the query simply becomes:
Car.find({
disponible: true,
"agency.location": {
$near: {
$geometry: {
coordinates: [ latitude , longitude ]
},
}
}
})
YES, that would mean that you likely duplicate a lot of information about an "agency" since the same data would likely be present on many cars. But the facts are that for this type of query usage, this is actually what MongoDB is expecting you to model as.
Conclusion
So the real choices here come down to which case suits your needs:
Accept that you are possibly returning less than the expected results due to "double filtering" though the use of a $geoNear and $lookup combination. Noting that $geoNear will only return 100 results by default, unless you change that. This can be an unreliable combination for "paged" results.
Restructure your data accepting the "duplication" of agency detail in order to get a proper "dual constraint" query since both criteria are in the same collection. It's more storage and maintenance, but it is more performant and completely reliable for "paged" results.
And of course if it's neither acceptable to use the aggregation approach shown or the restructure of data, then this can only show that MongoDB is probably not best suited to this type of problem, and you would be better off using an RDBMS where you decide you must keep normalized data as well as be able to query with both constraints in the same operation. Provided of course you can choose an RDBMS which actually supports the usage of such GeoSpatial queries along with "joins".
Related
Here is the query:
const tags = await mongo
.collection("positive")
.aggregate<{ word: string; count: number }>([
{
$lookup: {
from: "search_history",
localField: "search_id",
foreignField: "search_id",
as: "history",
pipeline: [
{
$match: {
created_at: { $gt: prevSunday.toISOString() },
},
},
{
$group: {
_id: "$url",
},
},
],
},
},
{
$match: {
history: { $ne: [] },
},
},
{
$group: {
_id: "$word",
url: {
$addToSet: "$history._id",
},
},
},
{
$project: {
_id: 0,
word: "$_id",
count: {
$size: {
$reduce: {
input: "$url",
initialValue: [],
in: {
$concatArrays: ["$$value", "$$this"],
},
},
},
},
},
},
{
$sort: {
count: -1,
},
},
{
$limit: 50,
},
])
.toArray();
I think I need an index but not sure how or where to add.
Perhaps performance of this operation should be revisited after we confirm that it is satisfying the desired application logic that the approach itself is reasonable.
When it comes to performance, there is nothing that can be done to improve efficiency on the positive collection if the intention is to process every document. By definition, processing all documents requires a full collection scan.
To efficiently support the $lookup on the search_history collection, you may wish to confirm that an index on { search_id: 1, created_at: 1, url: 1 } exists. Providing the .explain("allPlansExecution") output would allow us to better understand the current performance characteristics.
Desired Logic
Updating the question to include details about the schemas and the purpose of the aggregation would be very helpful with respect to understanding the overall situation. Just looking at the aggregation, it appears to be doing the following:
For every single document in the positive collection, add a new field called history.
This new field is a list of url values from the search_history collection where the corresponding document has a matching search_id value and was created_at after last Sunday.
The aggregation then filters to only keep documents where the new history field has at least one entry.
The next stage then groups the results together by word. The $addToSet operator is used here, but it may be generating an array of arrays rather than de-duplicated urls.
The final 3 stages of the aggregation seem to be focused on calculating the number of urls and returning the top 50 results by word sorted on that size in descending order.
Is this what you want? In particular the following aspects may be worth confirming:
Is it your intention to process every document in the positive collection? This may be the case, but it's impossible to tell without any schema/use-case context.
Is the size calculation of the urls correct? It seems like you may need to use a $map when doing the $addToSet for the $group instead of using $reduce for the subsequent $project.
The best thing to do is to limit the number of documents passed to each stage.
Indexes are used by mongo in aggregations only in the first stage only if it's a match, using 1 index max.
So the best thing to do is to have a match on an indexed field that is very restrictive.
Moreover, please note that $limit, $skip and $sample are not panaceas because they still scan the entire collection.
A way to efficiently limit the number of documents selected on the first stage is to use a "pagination". You can make it work like this :
Once every X requests
Count the number of docs in the collection
Divide this in chunks of Yk max
Find the _ids of the docs at the place Y, 2Y, 3Y etc with skip and limit
Cache the results in redis/memcache (or as global variable if you really cannot do otherwise)
Every request
Get the current chunk to scan by reading the redis keys used and nbChunks
Get the _ids cached in redis used to delimit the next aggregation id:${used%nbChunks} and id:${(used%nbChunks)+1} respectively
Aggregate using $match with _id:{$gte: ObjectId(id0), $lt: ObjectId(id1)}) }
Increment used, if used > X then update chunks
Further optimisation
If using redis, supplement every key with ${cluster.worker.id}:to avoid hot keys.
Notes
The step 3) of the setup of chunks can be a really long and intensive process, so do it only when necessary, let's say every X~1k requests.
If you are scanning the last chunk, do not put the $lt
Once this process implemented, your job is to find the sweet spot of X and Y that suits your needs, constrained by a Y being large enough to retrieve max documents while being not too long and a X that keeps the chunks roughly equals as the collection has more and more documents.
This process is a bit long to implement but once it is, time complexity is ~O(Y) and not ~O(N). Indeed, the $match being the first stage and _id being a field that is indexed, this first stage is really fast and limits to max Y documents scanned.
Hope it help =) Make sure to ask more if needed =)
I'm querying a mongoose model called Service using the mongodb $nearSphere operator, to return results by distance from a given point:
Service.find({
$nearSphere: {
$geometry: {
type : "Point",
coordinates : [myLng, myLat]
}
}
})
.limit(10)
$nearSphere sorts by distance, but some of my services are "promoted", so I'd like to float them to the top of the list regardless of distance.
I would normally use the $sort operator for this, but I noticed a note in the mongodb documentation discouraging it.
Is it possible to bring some results to the top of the list, but otherwise preserve the existing distance sorting?
Ideally I'd prefer to do without:
making a second query
refactoring everything to use aggregations instead, because I'm worried about performance (Is this a legitimate concern? Should I be?)
If you don't want to sort based on distance, but on some other field, you should use $geoWithin instead of $nearSphere. This is the only reason why MongoDB documentation discourages to use $sort with $nearSphere. Since you'll be doing sorting two times, which is resource and time wastage when using both $nearSphere and $sort.
If you want to do sorting based on a combination of another field (say 'A') and distance both (when field 'A' is equal in multiple docs), then you need to use aggregation pipeline with $geoNear operator, it will add distance field to your results and you can do the sorting as the way you want.
Super exited to share this after great insights from the reply from Goel.
Just as he mention, to sort by multiple fields, in your case, you would need to aggregate that,with a geonear query which adds a distance field to your documentst that you would give a custom name and combine it with the other field you wish to combine with.Here is an example:
consider sample document(s):
[{
_id: xxxx,
location: {type: "Point", coordinates: [35.00, 1.00]},
createdAt: ISODate("2020-05-18T09:31:24Z"),
...other fields
}]
we can query and retrieve documents based on a particular point, sorted by both closeness and creation time as
{$geoNear:
{near: {type: "Point", coordinates: [35.268428,2.},
spherical: true, distanceField: 'distance' }},
{$sort: {distance: 1, createdAt: -1} },
{$limit: 5},
{$project:{distance1, createdAt: 1 } } ])
.pretty()
I have a Schema:
var ProjectSchema = new Schema({
name: {
type: String,
default: ''
},
topics: [{
type: Schema.ObjectId,
ref: 'Topic'
}],
user: {
type: Schema.ObjectId,
ref: 'User'
}
});
What I want to do is get an array with all topics from all projects. I cannot query Topic directly and get a full list because some topics are unassigned and they do not hold a reference back to a Project (for reasons of avoiding two way references). So I need to query Project and aggregate some how. I am doing something like:
Project.aggregate([{$project:{topics:1}}]);
But this is giving me an array of Project objects with the topics field. What I want is an array with topic objects.
How can I do this?
When dealing with arrays you typically want to use $unwind on the array members first and then $group to find the distinct entries:
Project.aggregate(
[
{ "$unwind": "$topics" },
{ "$group": { "_id": "$topics._id" } }
],
function(err,docs) {
}
)
But for this case, it is probably simplier to just use .distinct() which will do the same as above, but with just an array of results rather than documents:
Project.distinct("topics._id",function(err,topics) {
});
But wait there a minute because I know what you are really asking here. It's not the _id values you want but your Topic data has a property on it like "name".
Since your items are "referenced" and in another collection, you cannot do an aggregation pipeline or .distinct() operation on the property of a document in another collection. Put basically "MongoDB does not perform Joins" and mongoose .populate() is not a join, just something that "emulates" that with additional query(ies).
But you can of course just find the "distinct" values from "Project" and then fetch the information from "Topic". As in:
Project.distinct("topics._id",function(err,topics) {
Topic.find({ "_id": { "$in": topics } },function(err,topics) {
});
});
Which is handy because the .distinct() function already returned an array suitable for use with $in.
I'm learning Mongodb/Mongoose/Express and have come across a fairly complex query (relative to my current level of understanding anyway) that I'm not sure how best to approach. I have a collection - to keep it simple let's call it entities - with an embedded actions array:
name: String
actions: [{
name: String
date: Date
}]
What I'd like to do is to return an array of documents with each containing the most recent action (or most recent to a specified date), and the next action (based on the same date).
Would this be possible with one find() query, or would I need to break this down into multiple queries and merge the results to generate one result array? I'm looking for the most efficient route possible.
Provided that your "actions" are inserted with the "most recent" being the last entry in the list, and usually this will be the case unless you are specifically updating items and changing dates, then all you really want to do is "project" the last item of the array. This is what the $slice projection operation is for:
Model.find({},{ "actions": { "$slice": -1 } },function(err,docs) {
// contains an array with the last item
});
If indeed you are "updating" array items and changing dates, but you want to query for the most recent on a regular basis, then you are probably best off keeping the array ordered. You can do this with a few modifiers such as:
Model.update(
{
"_id": ObjectId("541f7bbb699e6dd5a7caf2d6"),
},
{
"$push": { "actions": { "$each": [], "$sort": { "date": 1 } } }
},
function(err,numAffected) {
}
);
Which is actually more of a trick that you can do with the $sort modifier to simply sort the existing array elements without adding or removing. In versions prior to 2.6 you need the $slice "update" modifier in here as well, but this could be set to a value larger than the expected array elements if you did not actually want to restrict the possible size, but that is probably a good idea.
Unfortunately, if you were "updating" via a $set statement, then you cannot do this "sorting" in a single update statement, as MongoDB will not allow both types of operations on the array at once. But if you can live with that, then this is a way to keep the array ordered so the first query form works.
If it just seems too hard to keep an array ordered by date, then you can in fact retrieve the largest value my means of the .aggregate() method. This allows greater manipulation of the documents than is available to basic queries, at a little more cost:
Model.aggregate([
// Unwind the array to de-normalize as documents
{ "$unwind": "$actions" },
// Sort the contents per document _id and inner date
{ "$sort": { "_id": 1, "actions.date": 1 } },
// Group back with the "last" element only
{ "$group": {
"_id": "$_id",
"name": { "$last": "$name" },
"actions": { "$last": "$actions" }
}}
],
function(err,docs) {
})
And that will "pull apart" the array using the $unwind operator, then process with a next stage to $sort the contents by "date". In the $group pipeline stage the "_id" means to use the original document key to "collect" on, and the $last operator picks the field values from the "last" document ( de-normalized ) on that grouping boundary.
So there are various things that you can do, but of course the best way is to keep your array ordered and use the basic projection operators to simply get the last item in the list.
As I wrote above, I have a schema which refers to another different schema. These are:
exports.Policies = new mongoose.Schema({
name: String,
description: String,
exploits: [ {type : mongoose.Schema.ObjectId, ref : 'exploit', required: true} ]
});
exports.exploit = new mongoose.Schema({
name: String,
type: String,
required: [String]
});
What I want to do is just know how many values have 'exploits' array as a field inside of the response, is this possible? In sql I have to write only "count(field)" and making a "group by", how might I do this?
An example:
{
"name" : "mmmmmm",
"description" : "jjjj",
"_id" : ObjectId("533721b91a985b883399cdc2"),
"n_exploits" : 2
}
Thanks in advance.
Actually there is just the $size operator if you have no need to filter array content.
Assuming that you have a model to your schema that is named "policies" then the following is achieved through use of .aggregate():
policies.aggregate([
{ "$unwind": "$exploits" }
{ "$group": {
"_id": "$_id",
"count": { "sum": 1 }
}}
],function(err,doc) {
});
So this works by unwinding the array and then counting the number of elements that are produced when you re-group.
In future releases there is a a new $size operator so you can skip those stages and just use project:
policies.aggregate([
{ "$project": {
"count": { "$size": "$exploits" }
}}
],function(err,doc) {
});
Or otherwise use that with $group and $sum to add up arrays across documents.
But for now you do the $unwind and $sum operations. Working inside the mongodb engine is typically done in "native" code and the JavaScript operators are not available. But also as native code these operators work very fast.
Also see the aggregation operator reference for other things you may wish to do.
Have you tried searching this before posting here? A simple google search with words "mongodb count group by" gives kind of a lot of results, which are not only telling you that you have too look into aggregation framework, but also give you some examples even on SO like this and this.