It is necessary to make a selection on a certain condition, and sort the result on the previously mentioned field. How to do it?
As a driver for MongoDB using "Monk".
Assuming you have already got as far as getting a collection, then what you need is the find() method:
collection.find(query, options, callback);
You can use the query object to specify conditions, and the options object to sort. For detail on how to build the two objects see the mongodb native driver documentation.
So in your case specifically, something like this example might work. For the "condition" you mention, I'm using the condition that the "quantity" field is greater than 0, then sorting by quantity, largest first.
collection.find(
{quantity: {$gt: 0}},
{sort: {quantity: -1}},
function(err, docs) {
// docs is the sorted array of your results, process them here
});
Monk itself is fairly lightly documented, but it mostly calls through to mongoskin and then the native mongodb driver linked above, so the documentation for those is a good place to look.
Related
I'm using an ajax request from the front end to load more comments to a post from the back-end which uses NodeJS and mongoose. I won't bore you with the front-end code and the route code, but here's the query code:
Post.findById(req.params.postId).populate({
path: type, //type will either contain "comments" or "answers"
populate: {
path: 'author',
model: 'User'
},
options: {
sort: sortBy, //sortyBy contains either "-date" or "-votes"
skip: parseInt(req.params.numberLoaded), //how many are already shown
limit: 25 //i only load this many new comments at a time.
}
}).exec(function(err, foundPost){
console.log("query executed"); //code takes too long to get to this line
if (err){
res.send("database error, please try again later");
} else {
res.send(foundPost[type]);
}
});
As was mentioned in the title, everything works fine, my problem is just that this is too slow, the request is taking about 1.5-2.5 seconds. surely mongoose has a method of doing this that takes less to load. I poked around the mongoose docs and stackoverflow, but didn't really find anything useful.
Using skip-and-limit approach with mongodb is slow in its nature because it normally needs to retrieve all documents, then sort them, and after that return the desired segment of the results.
What you need to do to make it faster is to define indexes on your collections.
According to MongoDB's official documents:
Indexes support the efficient execution of queries in MongoDB. Without indexes, MongoDB must perform a collection scan, i.e. scan every document in a collection, to select those documents that match the query statement. If an appropriate index exists for a query, MongoDB can use the index to limit the number of documents it must inspect.
-- https://docs.mongodb.com/manual/indexes/
Using indexes may cause increased collection size but they improve the efficiency a lot.
Indexes are commonly defined on fields which are frequently used in queries. In this case, you may want to define indexes on date and/or vote fields.
Read mongoose documentation to find out how to define indexes in your schemas:
http://mongoosejs.com/docs/guide.html#indexes
I'm using Node.js + mongodb. I have few documents in my collection and i want to know does my collection have any document matched my condition. Of course i can simply use
myModel.find({ myField: someValue }) and check is anything comes or not. But i want to use solution like sql provides exists keyword? Help me, please
Edit: my bad. I forget to tell that "performance first".
MongoDB's $exists actually doesn't help you very much to find out if a certain document exists in your collection. It is used for example to give you all documents that have a specific field set.
MongoDB has no native support for an sql like exists. What you can use, however, is myModel.findOne({ myField: someValue }) and then check if it is null.
To enhance performance you can tell MongoDB to only load the object id via projection, like this:
myModel.findOne({ myField: someValue }, {_id: 1})
There is an exist mechanism in mongodb, I'll demonstrate a sample below.
For example below, I'm looking for records that have tomato.consensus fields and that it's empty, so I can delete them or avoid them. In case I was looking for "tomato.consensus": Dublin, I'd change Null to Dublin, to match that.
I hope this is helpful, if not fire away any questions
tomato
----consensus
db.movieDetails.updateMany({$and: [
{"tomato.consensus": {$exists: true} },
{"tomato.consensus": null} ] },
]})
I have a large MongoDB collection, containing more than 2GB of raw data and I use a very simple query to fetch a specific document from the collection by its Id. Document sizes currently range from 10KB to 4MB, and the Id field is defined as an index.
This is the query I'm using (with the mongojs module):
db.collection('categories').find({ id: category_id },
function(err, docs) {
callback(err, docs.length ? docs[0] : false);
}).limit(1);
When I execute this query using MongoDB shell or a GUI such as Robomongo it takes approximately 1ms to fetch the document, no matter what its physical size, but when I execute the exact same query on NodeJS the response time ranges from 2ms to 2s and more depending on the amount of data. I only measure the time it takes to receive a response and even in cases where NodeJS waits for more than 500ms the MongoDB profiler (.explain()) shows it took only a single millisecond to execute the query.
Now, I'm probably doing something wrong but I can't figure out what it is. I'm rather new to NodeJS but I had experience with MongoDB and PHP in the past and I never encountered such performance issues, so I tend to think I'm probably abusing NodeJS in some way.
I also tried profiling using SpyJS on WebStorm, I saw there are a lot of bson.deserialize calls which sums up quickly into a large stack, but I couldn't investigate farther because SpyJS always crashes at this point. Probably related but I still have no idea how to deal with it.
Please advise, any leads will be appreciated.
Edit:
This is the result of db.categories.getIndexes():
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "my_db.categories"
},
{
"v" : 1,
"key" : {
"id" : 1
},
"name" : "id_1",
"ns" : "my_db.categories"
}
]
I also tried using findOne which made no difference:
db.collection('categories').findOne({ id: category_id },
function(err, doc) {
callback(err, doc || false);
});
My guess is the .limit(1) is ignored because the callback is provided early. Once find sees a callback it's going to execute the query, and only after the query has been sent to mongo will the .limit modifier try to adjust the query but it's too late. Recode as such and see if that solves it:
db.collection('categories').find({ id: category_id }).limit(1).exec(
function(err, docs) {
callback(err, docs.length ? docs[0] : false);
});
Most likely you'll need to have a combination of normalized and denormalized data in your object. Sending 4MB across the wire at a time seems pretty heavy, and likely will cause problems for any browser that's going to be doing the parsing of the data.
Most likely you should store the top 100 products, the first page of products, or some smaller subset that makes sense for your application in the category. This may be the top alphabetically, most popular, newest, or some other app-specific metric you determine.
When you go about editing a category, you'll use the $push/$slice method to ensure you avoid unbounded array growth.
Then when you actually page through the results you'll do a separate query to the individual products table by category. (Index that.)
I've written about this before here:
https://stackoverflow.com/a/27286612/68567
I want to make a request like:
User.find().exec(function(){});
I know I can use toJSON in the model however I don't like this approach since sometimes I need different parameters. For instance if it's the logged in user I will return their email and other parameters. However if it the request fort he same data is made by a different user it would not include the email and a smaller subset of parameters.
I've also tried using:
User.find({}, {username:1}) ...
User.find({}, {fields: {username:1}});
But not having any luck. How can I specify the fields I need returned?
So actually found a weird workaround for this. the fields param WILL work as long as you pass other params with it such as limit or order:
User.find({}, {fields: {username:1}}).limit(1);
Note that this will NOT work with findOne or any of the singular returning types. This means in your result callback you will need to do user[1].
Of course the other option is to just scrub your data on the way out, which is a pain if you are using a large list of items. So if anything this works for large lists where you might actually set limit(20) and for single items you can just explicitly return paras until select() is available.
This is an update to the question, fields is no longer used in sails 11, please use select instead of fields.
Model.find({field: 'value'}, {select: ['id', 'name']})
.paginate({page: 1}, {limit: 10})
.exec(function(err, results) {
if(err) {
res.badRequest('reason');
}
res.json(results);
});
Waterline does not currently support any "select" syntax; it always returns all fields for a model. It's currently in development and may make it into the next release, but for now the best way to do what you want would be to use model class methods to make custom finders. For example, User.findUser(criteria, cb) could find a user give criteria, and then check whether it was the logged-in user before deciding which data to return in the callback.
In MongoDB 2.6, the text-search is supposedly production ready and we can now use skip. I'd like to use text-search and skip for pagination in my, but I'm not yet sure how to implement it.
Right now, I'm using Mongoose and the Mongoose-text-search plugin, but I don't believe either of them support skip in MongoDB's text search, so I guess I'll need to use the native MongoClient...
My app connects to the database via Mongoose using:
//Bootstrap db connection
var db = mongoose.connect(config.db, function(e) {
Now, how can I use the native MongoClient to execute a full text search for my Products model, with a skip parameter. Here is what I had using Mongoose and Mongoose-text-search, but there is no way to add in skip:
Product = mongoose.model('Product')
var query = req.query.query;
var skip = req.query.skip;
var options = {
project: '-created', // do not include the `created` property
filter: filter, // casts queries based on schema
limit: 20,
language: 'english',
lean: true
};
Product.textSearch(query, options, function (err, response) {
});
The main difference introduced in 2.6 versions of MongoDB is that you can issue a "text search" query using the standard .find() interface so the old methods for textSearch would no longer need to be applied. This is basically how modifiers such as limit and skip can be applied.
But keep in mind that as of writing the current Mongoose dependency is for an earlier version of the MongoDB node driver that existed prior to the release of MongoDB 2.6. As Mongoose actually wraps the main methods and does some syntax checking of it's own, it is probably likely ( as in untried by me ) that using the Mongoose methods will currently fail.
So what you will need to do is get the underlying driver method for .find(), and also now use the $text operator instead:
Product.collection.find(
{ "$text": { "$search": "term" } },
{ "sort": { "score": { "$meta": "textScore" } }, "skip": 25, "limit": 25 },
function(err,docs) {
// processing here
});
Also noting that $text operator does not sort the results by "score" for relevance by default, but this is passed to the "sort" option using the new $meta operator, which is also introduced in MongoDB 2.6.
So alter your skip and limit values and you have paging on text search results and with a cursor. Just be wary of large data returns as skip and limit are not really efficient ways to move through a large cursor. Better to have another key where you can range match, even though counter-intuitive to "relevance matching".
So, text search facilities are a bit "better" but not "perfect". As always, if you really need more and/or more performance, look to an external solution.
Feel free to try a similar operation with the Mongoose implementation of .find() as well. But have my reservations from past experience that there is generally some masking and checking going on there, so hence the description of usage with the "native" node driver.