In MongoDB 2.6, the text-search is supposedly production ready and we can now use skip. I'd like to use text-search and skip for pagination in my, but I'm not yet sure how to implement it.
Right now, I'm using Mongoose and the Mongoose-text-search plugin, but I don't believe either of them support skip in MongoDB's text search, so I guess I'll need to use the native MongoClient...
My app connects to the database via Mongoose using:
//Bootstrap db connection
var db = mongoose.connect(config.db, function(e) {
Now, how can I use the native MongoClient to execute a full text search for my Products model, with a skip parameter. Here is what I had using Mongoose and Mongoose-text-search, but there is no way to add in skip:
Product = mongoose.model('Product')
var query = req.query.query;
var skip = req.query.skip;
var options = {
project: '-created', // do not include the `created` property
filter: filter, // casts queries based on schema
limit: 20,
language: 'english',
lean: true
};
Product.textSearch(query, options, function (err, response) {
});
The main difference introduced in 2.6 versions of MongoDB is that you can issue a "text search" query using the standard .find() interface so the old methods for textSearch would no longer need to be applied. This is basically how modifiers such as limit and skip can be applied.
But keep in mind that as of writing the current Mongoose dependency is for an earlier version of the MongoDB node driver that existed prior to the release of MongoDB 2.6. As Mongoose actually wraps the main methods and does some syntax checking of it's own, it is probably likely ( as in untried by me ) that using the Mongoose methods will currently fail.
So what you will need to do is get the underlying driver method for .find(), and also now use the $text operator instead:
Product.collection.find(
{ "$text": { "$search": "term" } },
{ "sort": { "score": { "$meta": "textScore" } }, "skip": 25, "limit": 25 },
function(err,docs) {
// processing here
});
Also noting that $text operator does not sort the results by "score" for relevance by default, but this is passed to the "sort" option using the new $meta operator, which is also introduced in MongoDB 2.6.
So alter your skip and limit values and you have paging on text search results and with a cursor. Just be wary of large data returns as skip and limit are not really efficient ways to move through a large cursor. Better to have another key where you can range match, even though counter-intuitive to "relevance matching".
So, text search facilities are a bit "better" but not "perfect". As always, if you really need more and/or more performance, look to an external solution.
Feel free to try a similar operation with the Mongoose implementation of .find() as well. But have my reservations from past experience that there is generally some masking and checking going on there, so hence the description of usage with the "native" node driver.
Related
What I am really trying to do is to make indexes for filtering and string matching of documents based on their property values.
I know that mongodb has built in operators such as $text that are very helpful with this sort of functionality.
I'm not sure how to access these operators while using mongoose, or if there are any methods i need to use to access them.
I want to use mongoose to still define schema and models but need the functionality of native mongodb.
Is this possible?
Below are my views, Please add if I miss anything or if something needs to be modified or well-explained :
1. You will still be able to use mongoDB's native functionalities on using Mongoose models.
2. Mongoose is a kind of wrapper on top of native mongoDB-driver.
3. It would be very useful if you want to have schema based collections/data.
4. Additionally it would provide few more features than native mongoDB's driver. You might see few syntax differences between those two.
5. Few examples like `.findByIdAndUpdate()` & `.populate()` are mongoose specific, which has equivalent functionalities available in mongoDB driver/mongoDB as well.
6. In general it's quiet common to define mongoose models and use those over mongoDB's functionality in coding(As in node.js - You would write all of your same DB queries on Mongoose models, queries that you execute in DB).
Point 2 :
Mongoose is an object document modeling (ODM) layer that sits on top of Node's MongoDB driver. If your coming from SQL, it's similar to an ORM for a relational database.
Point 3 :
In code if you're using mongoose models to implement your write queries, unless you define a field in model - it wouldn't be added to DB though you pass it in request. Additionally you can do multiple things like making a field unique/required etc.. it's kind of making your mongoDB data look like schema based. If your collections data is more like random data(newsfeed kind of thing where fields are not same for each document & you can't predict data) then you might not care of using mongoose.
Point 6 :
Let's say you use mongo shell or a client like mongo compass/robo3T and execute a query that's like this :
db.getCollection('yourCollection').find(
{
$text: {
$search: 'employeeName',
$diacriticSensitive: false
},
country: 'usa'
},
{
employee_id: 1,
name: 1
}
).sort({ score: { $meta: 'textScore' } });
you would do same on mongoose model(As yourCollectionModel is already defined) :
yourCollectionModel.find(
{
$text: {
$search: 'employeeName',
$diacriticSensitive: false
},
country: 'usa'
},
{
employee_id: 1,
name: 1
}
).sort({ score: { $meta: 'textScore' } });
You would see key functionality difference more on writes rather than reads while using mongoose, though all the above is not about performance - If you ask me, I can say you might see much performance gains using mongoose.
Ref : Mongoose Vs MongoDB Driver
I'm using an ajax request from the front end to load more comments to a post from the back-end which uses NodeJS and mongoose. I won't bore you with the front-end code and the route code, but here's the query code:
Post.findById(req.params.postId).populate({
path: type, //type will either contain "comments" or "answers"
populate: {
path: 'author',
model: 'User'
},
options: {
sort: sortBy, //sortyBy contains either "-date" or "-votes"
skip: parseInt(req.params.numberLoaded), //how many are already shown
limit: 25 //i only load this many new comments at a time.
}
}).exec(function(err, foundPost){
console.log("query executed"); //code takes too long to get to this line
if (err){
res.send("database error, please try again later");
} else {
res.send(foundPost[type]);
}
});
As was mentioned in the title, everything works fine, my problem is just that this is too slow, the request is taking about 1.5-2.5 seconds. surely mongoose has a method of doing this that takes less to load. I poked around the mongoose docs and stackoverflow, but didn't really find anything useful.
Using skip-and-limit approach with mongodb is slow in its nature because it normally needs to retrieve all documents, then sort them, and after that return the desired segment of the results.
What you need to do to make it faster is to define indexes on your collections.
According to MongoDB's official documents:
Indexes support the efficient execution of queries in MongoDB. Without indexes, MongoDB must perform a collection scan, i.e. scan every document in a collection, to select those documents that match the query statement. If an appropriate index exists for a query, MongoDB can use the index to limit the number of documents it must inspect.
-- https://docs.mongodb.com/manual/indexes/
Using indexes may cause increased collection size but they improve the efficiency a lot.
Indexes are commonly defined on fields which are frequently used in queries. In this case, you may want to define indexes on date and/or vote fields.
Read mongoose documentation to find out how to define indexes in your schemas:
http://mongoosejs.com/docs/guide.html#indexes
TLDR
Is there a way to limit queryByExample to a collection in NodeJS?
Problem faced
I have a complex query with some optional fields (i.e. sometimes some search fields will be omitted). So I need to create a query dynamically, e.g. in JSON. QueryByExample seems to be the right tool to use here as it gives me that flexibility to pass a JSON. However my problem is that I would like to limit my search to only one collection or directory.
e.g. I was hoping for something like
searchJSON = {
title: { $word: "test" },
description: { $word: "desc" }
};
//query
db.documents.query(qb.where(
qb.collection("collectionName"),
qb.byExample(searchJSON)
)).result()...
In this case searchJSON could have been built dynamically, for example maybe sometimes title may be omitted from the search.
This doesn't work because the query builder only allows queryByExample to be the only query. But I'd instead like to built a dynamic search query which is limited to a collection or directory.
At present, I think you would have to express the query with QueryBuilder instead of Query By Example using
qb.and([
qb.collection('collectionName'),
qb.word('title', 'test'),
qb.word('description', 'desc')
])
See http://docs.marklogic.com/jsdoc/queryBuilder.html#word
That said, it should be possible for the Node.js API to relax that restriction based on the fixes in MarkLogic 9.0-2
Please file an issue on https://github.com/marklogic/node-client-api
I don't see how to use query modifiers like $min/$max with mongoose in the docs. Is there a way to do so?
I'm not aware of any natural way to do that when using Mongoose, as Mongoose performs a lot of operations on your behalf to make things convenient that may conflict with some of these operations normally.
I've not tried this, but you may be able to do this:
Model.find( { $query: { <query> },
$max: { myField: 100 }, $min: { myField: 10 } }, function(err, models) {
// do something with results
});
When using the $query syntax and modifiers, you can't use other things like mongoose's sort, etc as they must be specified explicitly in the $query definition.
Alternatively, you could use one or more of the query operators such as $gte. $min and $max, as query modifiers, both force MongoDb to use an index where the other operators do not.
Model.find().gte("myField", 10).exec(function(err, models) {})
As pointed out in comments, you can also use setOptions (documentation). However, it still has the same restrictions that you can't use it and many of the other options.
In any case, I'd suggest you should consider using the other query operators such as $gte, etc. instead anyway and let the database query engine make a smart choice based on the query and available indexes what makes sense rather than forcing index use through $max/$min. By using either, even if there's a better choice for an index, they will be used.
It is necessary to make a selection on a certain condition, and sort the result on the previously mentioned field. How to do it?
As a driver for MongoDB using "Monk".
Assuming you have already got as far as getting a collection, then what you need is the find() method:
collection.find(query, options, callback);
You can use the query object to specify conditions, and the options object to sort. For detail on how to build the two objects see the mongodb native driver documentation.
So in your case specifically, something like this example might work. For the "condition" you mention, I'm using the condition that the "quantity" field is greater than 0, then sorting by quantity, largest first.
collection.find(
{quantity: {$gt: 0}},
{sort: {quantity: -1}},
function(err, docs) {
// docs is the sorted array of your results, process them here
});
Monk itself is fairly lightly documented, but it mostly calls through to mongoskin and then the native mongodb driver linked above, so the documentation for those is a good place to look.