I have something weird, I try just to sort on _id and have some paging. Hereunder you will see the query I execute:
var condition = { isArchived: false };
if(lastId) {
condition["_id"] = { $lt : lastId };
};
PostModel
.find(condition)
.sort({_id:-1})
.limit(10)
.exec(function (err, posts) {
if(err)
return callback(new customError.Database(err.toString()),null);
callback(null, posts);
})
What I see is that in 80% of the time the result is consistent, but sometimes the result is not the same (it does not vary much, but some objects are in a different order).
I use this technique with success on other models, but only with this query/collection I get this problem (not sure if the problem is in the query or on the collection...)
What can be the reason?
Related
I'd like to update some data in Mongoose by using array value that I've find before.
Company.findById(id_company,function(err, company) {
if(err){
console.log(err);
return res.status(500).send({message: "Error, check the console. (Update Company)"});
}
const Students = company.students;
User.find({'_id':{"$in" : Students}},function(err, users) {
console.log(Students);
// WANTED QUERY : Update company = null from Users where _id = Students[];
});
});
Students returns users._id in array with object inside, and I use that to find users object, and then I want to set null a field inside users object, that field named as "company". How I can do that? Thank you.
From what you posted (I took the liberty to use Promises but you can roughly achieve the same thing with callbacks), you can do something like:
User.find({'_id':{"$in" : Students}})
.then( users =>{
return Promise.all( users.map( user => {
user.company = null;
return user.save()
}) );
})
.then( () => {
console.log("yay");
})
.catch( e => {
console.log("failed");
});
Basically, what I'm doing here is making sure .all() user models returned by the .find() call are saved properly, by checking the Promised value returned for .save()ing each of them.
If one of these fails for some reasons, Promise.all() return a rejection you can catch afterhand.
However, in this case, each item will be mapped to a query to your database which is not good. A better strategy would be to use Model.update(), which will achieve the same, in, intrinsically, less database queries.
User.update({
'_id': {"$in": Students}
}, {
'company': <Whatever you want>
})
.then()
use .update but make sure you pass option {multi: true} something like:
User.update = function (query, {company: null}, {multi: true}, function(err, result ) { ... });
I'm trying to determine whether the document was found in my findOneAndUpdate operation. If it wasn't, I return a 404 not found error. I figured I'd use the "passRawValue" option Mongoose provides, and check for a raw value- if raw is undefined, I know the doc was not found.
However regardless whether the doc is found or not, my raw value is undefined. I've verified that the doc I'm trying to update is in the DB at the time of the query by running a simple "findOne" query just before the update. Where am I going wrong?
let updateItemById = (userId, itemId, params, cb) => {
//this finds and prints the document I'm testing with -- I know its in the DB
// Item.findOne({ "_id" : itemId, ownerId: userId }, (err, doc) => {
// if (doc) {
// console.log("This is the doc: ", doc);
// }
// });
Item.findOneAndUpdate({ "_id" : itemId, ownerId: userId },
{
$set: {
params
}
}, { runValidators: 1, passRawResult: true}, (err, doc, raw) => {
if (err) {
//winston.log
return cb(ErrorTypes.serverError(), false);
}
else if (raw) {
return cb(null, true);
}
else {
return cb(ErrorTypes.notFound(), false);
}
});
}
Hi I have a hunch that you are passing params that has a property that doesn't exist in the document in the database. In such case, nothing was modified, hence db doesn't return raw as the third parameter.
Update:
So I did some few tests of my own, and I see that if we pass option strict:false then your code should work as intended. So your options section will look like this
{ runValidators: 1, passRawResult: true, strict: false, new:true}
Explanation:
Mongoose has a strict option which by default is true. It makes sure that the values being updated is defined in the schema. So when we provide the option strict as false, as described in the [mongoose documentation] (http://mongoosejs.com/docs/api.html#query_Query-findOneAndUpdate) we can achieve updating document with new field.
I also added new:true option which will return you the updated document.
P.S.
I would like to add though, since our upsert is false, which means it won't insert new document when a match is not found, it will return null for doc, and you can simple check on that. Why are you checking on raw? Is there any particular reason for this?
I know it's been awhile but I had the same problem here so I decided to leave an answer that maybe can help other people.
I was able to check whether the findOneAndUpdate() method found a document or not by checking if the doc parameter was null on the callback function:
async Update(request: Request, response: Response) {
const productId = request.params.id;
const query = { _id: productId };
const options = { new: true };
try {
await Product.findOneAndUpdate(query, request.body, options, (err, doc, res) => {
if (doc === null)
return response.status(404).send({
error: 'Product not found'
})
return response.status(204).send();
});
}
catch (err) {
return response.status(400).send({
error: 'Product update failed'
});
}
}
All of the other solutions use skip() which is apparently bad for speed. Also I have seen people use nin which seems like it could be a bad solution if you have a bunch of ids that you are checking. Here is what I have so far
//Get the top 20 posts initially
exports.getPosts = (req, res) => {
db.post.find().sort({
numberOfLikes: -1
}).limit(20).exec(function(err, posts){
if(err){
return res.status(500).send();
}
res.json(posts);
});
};
//This is to enable pagination on posts
exports.getMorePosts = (req, res) => {
var query = req.query;
if(!query.hasOwnProperty('numberOfLikes') || !query.hasOwnProperty('postId')){
return res.status(400).send();
}
db.post.find({
numberOfLikes: {
$lte: query.numberOfLikes
},
_id: {
$ne: query.postId
}
}).sort('-numberOfLikes').limit(20).exec(function(err, posts){
if(err){
return res.status(500).send();
}
res.json(posts);
});
};
So I use the first function to get the first 20 posts and then the second to get 20 more posts and so on. The second function get passed the smallest number of likes from the original 20 and the last post's postId from the original 20. From there it gets 20 more.
I want to sort them by the number of likes but the problem is once it gets posts with 0 likes, it never knows when to end and it just keeps cycling back and fourth.
I have a json data which contains many objects. I want to limit the data for pagination and I need the total items count. Please help.
Model.find().skip((pageNumber-1)*limit).limit(limit).exec()
I want the count and skipped data in response.
You can use async library for running 2 queries at once. In your case you can run one query to get the number of documents and another for pagination.
Example with 'User' model:
var async = require('async');
var User = require('./models/user');
var countQuery = function(callback){
User.count({}, function(err, count){
if(err){ callback(err, null) }
else{
callback(null, count);
}
}
};
var retrieveQuery = function(callback){
User.find({}).skip((page-1)*PAGE_LIMIT)
.limit(PAGE_LIMIT)
.exec(function(err, doc){
if(err){ callback(err, null) }
else{
callback(null, doc);
}
}
};
async.parallel([countQuery, retrieveQuery], function(err, results){
//err contains the array of error of all the functions
//results contains an array of all the results
//results[0] will contain value of doc.length from countQuery function
//results[1] will contain doc of retrieveQuery function
//You can send the results as
res.json({users: results[1], pageLimit: PAGE_LIMIT, page: page, totalCount: results[0]});
});
async allows you to run a number of queries in parallel depending on the hardware you are using. This would be faster than using 2 independent queries to get count and get the required documents.
Hope this helps.
I have solved it with $facet and aggregate the following way in mongoose v3+:
const [{ paginatedResult, [{ totalCount }] }] = await Model.aggregate([{
$facet: {
paginatedResult: [
{ $match: query },
{ $skip: skip },
{ $limit: limit }
],
totalCount: [
{ $match: query },
{ $count: 'totalCount' }
]
}
}])
where the totalCount refers the total number of records matching the search query while the paginatedResult is only the paginated slice of them.
The problem with these solutions is that for every request you are doing two queries. This becomes problematic when you have a complex data structure and large data set as performance becomes an issue. Consider instead creating a special function that listens for the /resource?count=true or /resource/count GET methods and returns only the count.
You need to perform 2 queries to achieve that. One to get results and another to get total items amount with .count().
For example code you can watch at on of "paginator" for mongoose mongoose-paginate.
To performe only one query, you may use the find() method associated with promises and array slices. A small example would be:
getPaginated(query, skip, limit){
return this.model.find(query)
.lean()
.then((value)=>{
if (value.length === 0) return {userMessage: 'Document not found'};
const count = value.length;
//skip===0 must be handled
const start = parseInt(limit)*parseInt(skip - 1);
const end = start + parseInt(reqQuery.pagesize);
//slicing the array
value = value.slice(start,end);
//could return it another way...
value.push( { 'querySize': count });
return value;
})
.catch((reason)=>{
//...handling code
});
}
I'm trying to use this Sequelize ORM stuff for my project. I've integrated it as on example https://github.com/sequelize/express-example. So, cool - for now it's working with all relations and other goods. The problem is, that pm2 show's that my memory usage grows and never coming back.
This is my test script, that eats 100 Mb of RAM per launch. Have I missed something?
router.get('/test', hutils.authChecker, function(req, res, next) {
Project.findById(1,{ include : [Player]}).then(function(project) {
return Promise.denodeify(async.map)(project.Players, function(player, callback) {
Player.create({
project_id : 1,
name : 'iter_'+Math.floor(Math.random() * 1000000)+Math.floor(Math.random() * 1000000)
}).then(function(gamer) {
callback(null, gamer)
});
});
}).then(function(plrs) {
return Promise.denodeify(async.map)(plrs, function(guy, callback) {
guy.update({name : sqlRequest+'zzzzz'+Math.random()}).then(function(number) {
callback(null, number);
});
});
}).then(function(numbers) {
return Player.findAll({where : {name : {$like : '%zzzzz%'}}});
}).then(function(zets) {
return Promise.denodeify(async.map)(zets, function(zet, callback) {
zet.destroy().then(function(number) {
callback(null, number);
});
});
}).catch(function(err) {
next(err);
});
});
P.S. It`s make no sense, just to look how the ORM works. If it's matter, i have 1k players, for this project.
In queries that have results with a lot of rows, all of them will get loaded into memory before the callback.
So in this example, the query Project.findById(1,{ include : [Player]}) deserializes project 1 and all of its 1,000 players into JavaScript objects before returning them in the .then(function(project). Furthermore, the array of plrs, numbers and zets are all similarly stored in memory before they get returned thus increasing memory usage.
A way around this would be to get the database to do the heavy lifting. For example don't return each gamer that gets created and instead perform a update query on the db.
Player.update({
name : sqlRequest + 'zzzzz' + Math.random(),
}, {
where: {
createdAt: {
$gte: new Date() // or some other filter condition that identifies the records you need.
}
}
});
And then instead of destroying each zet, perform a delete query on the db.
Player.destroy({
where: {
name : {
$like : '%zzzzz%'
}
}
});