Memory leak, using Sequelize ORM for NodeJS - node.js

I'm trying to use this Sequelize ORM stuff for my project. I've integrated it as on example https://github.com/sequelize/express-example. So, cool - for now it's working with all relations and other goods. The problem is, that pm2 show's that my memory usage grows and never coming back.
This is my test script, that eats 100 Mb of RAM per launch. Have I missed something?
router.get('/test', hutils.authChecker, function(req, res, next) {
Project.findById(1,{ include : [Player]}).then(function(project) {
return Promise.denodeify(async.map)(project.Players, function(player, callback) {
Player.create({
project_id : 1,
name : 'iter_'+Math.floor(Math.random() * 1000000)+Math.floor(Math.random() * 1000000)
}).then(function(gamer) {
callback(null, gamer)
});
});
}).then(function(plrs) {
return Promise.denodeify(async.map)(plrs, function(guy, callback) {
guy.update({name : sqlRequest+'zzzzz'+Math.random()}).then(function(number) {
callback(null, number);
});
});
}).then(function(numbers) {
return Player.findAll({where : {name : {$like : '%zzzzz%'}}});
}).then(function(zets) {
return Promise.denodeify(async.map)(zets, function(zet, callback) {
zet.destroy().then(function(number) {
callback(null, number);
});
});
}).catch(function(err) {
next(err);
});
});
P.S. It`s make no sense, just to look how the ORM works. If it's matter, i have 1k players, for this project.

In queries that have results with a lot of rows, all of them will get loaded into memory before the callback.
So in this example, the query Project.findById(1,{ include : [Player]}) deserializes project 1 and all of its 1,000 players into JavaScript objects before returning them in the .then(function(project). Furthermore, the array of plrs, numbers and zets are all similarly stored in memory before they get returned thus increasing memory usage.
A way around this would be to get the database to do the heavy lifting. For example don't return each gamer that gets created and instead perform a update query on the db.
Player.update({
name : sqlRequest + 'zzzzz' + Math.random(),
}, {
where: {
createdAt: {
$gte: new Date() // or some other filter condition that identifies the records you need.
}
}
});
And then instead of destroying each zet, perform a delete query on the db.
Player.destroy({
where: {
name : {
$like : '%zzzzz%'
}
}
});

Related

Sails.js - Models & Controllers not working or throwing unexpected errors

I'm quite new to Sails.js, but not new to MVC. I feel like I'm really close, but just missing something somewhere.
I generated a controller with a single action "overview", and if I keep it "hello world"- easy, it works. The below code runs just fine:
// My controller
module.exports = {
overview : function(req, res) {
return res.send("<h1>Tossel<h1>");
},
}
However, when I try to get some data from the model and pass it to the controller, I have issues. I've tried it in two ways. I first defined a custom method on the Model and called it from my controller, like below:
// My model, e.g. Orders
module.exports = {
tableName: 'Orders',
attributes: {
id : {
type: 'integer',
primaryKey: true,
unique: true
},
client_id : {
type: 'integer'
}
},
getAllOrders: function(opts, cb) {
Orders.find().exec(function(err, data) {
cb(err, data);
});
}
};
// My controller
module.exports = {
overview : function(req, res) {
Orders. getAllOrders(null, function(err, data) {
return res.send(data);
});
},
}
From what I can see in the docs, that's pretty much how I should do it. I get the below error though. I get this when executing the command in the console, and by calling it from the controller:
TypeError: query.exec is not a function
I then tried doing everything in the controller:
// My controller
module.exports = {
overview : function(req, res) {
Orders.find().exec(function(err, data) {
return res.send(data);
});
},
}
This method didn't actually throw any errors, but it returned nothing. I'm pretty sure there is data in the database though. Since I wasn't 100% sure, I created an "add" action to the controller as below, to add data to the table. This results in no errors, but the page just hangs.
module.exports = {
overview : function(req, res) {
Orders.find().exec(function (err, data) {
return res.json(data);
});
},
add : function(req, res) {
var data = {
id: 1,
client_id: 1,
};
Orders.create(data).exec(function (err, data) {
return res.send(data);
});
}
};
Can someone please point me in the right direction here. I find the docs to be too simplistic. It's good at showing you the easy stuff, but that's all it shows.
Also, as a bonus, can someone please tell me what I need to do to be able to make changes and refresh the page to see them? At the moment I have to kill the server and bring it back up again to see the changes. This doesn't seem right, I'm must be missing something somewhere.
Edit
As per the below comment:
Node v8.10.0
Sails v0.12.14
I specify the table name because it is indeed different from the model name. I changed the actual name of the table and model in the code I posted.

How to limit my collection size to only one document?

I have a API, which allows a user to add a intervalValue - which will be used in my program to determine how often my program runs. I am using node js, express and mongodb in my project.
This is the API which allows a user to add a value:
router.post('/', function(req, res, next) {
intervalValue.create(req.body, function(err, post) {
if (err) {
console.log(err);
} else {
res.json(post);
}
});
});
And this is the schema for it:
var intervalValueSchema = new mongoose.Schema({
milliseconds: {
type: Number,
min: 15000
}
}, {
capped: {
size: 1024,
max: 1,
autoIndexId: true
}
});
From my understanding, this schema will only allow milliseconds value larger than 15000 milliseconds, and because it is capped, it will only allow one document in the collection.
THE AIM : add a interval value, and then only be able to modify that value - i.e. it will not be allowed to be deleted, and no more will be able to be added. Hence I need it to be limited to one document.
However with the current code, I am able to add multiple documents to this collection (even though when I do isCapped() I get true returned), and when I update the value I can insert a value less than 15000 - which should not be allowed.
This is the update API:
router.put('/updateValue/:id', function(req, res, next) {
intervalPoll.findByIdAndUpdate(req.params.id, req.body, {
new: true
}, function(err, post) {
if (err) {
console.log(err);
} else {
res.json(post);
}
});
});
What am I doing wrong - I am a beginner with mongo so any tips would be appreciated. Also, is there a better way to achieve my aim?
Thanks!
I wouldn't expose the ID of that value and I would not implement the PUT handler with :id (which is not convenient anyway, because user has to know the ID which once set will never change and be only one).
I would implement two endpoints:
router.get('/interval', ...
and
router.post('/interval', ...
# or:
router.put('/interval', ...
The get handler would search the database for any document and if present return its value. If there is no such document it would return some default value.
The post handler would first verify the value and then modify a document if it exists or insert it if it doesn't.
I think that in this case it would be much easier to check for that number in your handler then to fight with Mongoose to do what you need. This is a specific use case.
Udate
Here are some examples:
Schema can be:
mongoose.Schema({
milliseconds: Number
});
And the handlers would be something like this - let's say that your model is called Interval:
var defaultInterval = {milliseconds: 15000};
router.get('/interval', function(req, res, next) {
Interval.find().exec(function (err, intervals) {
if (err || intervals.length === 0) {
res.json(defaultInterval);
} else {
res.json({milliseconds: intervals[0].milliseconds});
}
});
});
router.put('/interval', function(req, res, next) {
var milliseconds = // get it from request
if (!milliseconds || milliseconds < 15000) {
// respond with error
} else {
Interval.findOneAndUpdate(
{}, {milliseconds: milliseconds}, {upsert: true},
function(err, interval){
if (err) {
// respond with error
} else {
// respond with success
}
}
);
}
});
This is not tested and you need to fill in the blanks but you get the idea.

Sorting on _id gives inconsistent results

I have something weird, I try just to sort on _id and have some paging. Hereunder you will see the query I execute:
var condition = { isArchived: false };
if(lastId) {
condition["_id"] = { $lt : lastId };
};
PostModel
.find(condition)
.sort({_id:-1})
.limit(10)
.exec(function (err, posts) {
if(err)
return callback(new customError.Database(err.toString()),null);
callback(null, posts);
})
What I see is that in 80% of the time the result is consistent, but sometimes the result is not the same (it does not vary much, but some objects are in a different order).
I use this technique with success on other models, but only with this query/collection I get this problem (not sure if the problem is in the query or on the collection...)
What can be the reason?

What is the best and fastest way to query lots of data with mongoose?

I want to do the following:
I have more then 10,000 users in my db.
I need to send all of them an event that will change a value in their document,
For example:
user:{
money:10,
skill : 5,
mood : 1
}
So this is what i want to avoid because it's a memory and a cpu hell:
User.find({}).exec(function(err,users){
users.forEach(function(user){
if(user.money < 10){
user.money += 5 * (some other params or something);
}
user.save();
});
);
Also i need to extract the id of each user who have money lower than 10 and send him a push.... so i cant use just "update","inc" or "set"
This code crashes my server, How can i make it better? should i use async? if yes how?
You can do that with a single update with the $inc operator and the {multi: true} option so that it's applied to all matching docs:
User.update(
{money: {$lt: 10}},
{$inc: {money: 5 * (some other params or something)}},
{multi: true},
function(err, num) { ... });
If your updates are such that each doc needs special handling based on its content, you can use a streaming approach to limit the number of docs in memory at any one time:
var stream = User.find({}).stream();
stream.on('data', function(user) {
stream.pause();
if(user.money < 10){
user.money += 5 * (some other params or something);
}
// More document-specific updates
...
user.save(function(err, doc) {
// The changes to this doc are complete, move on to the next one.
stream.resume();
});
}).on('error', function(err) {
console.error(err);
}).on('close', function() {
console.log('All done!');
});
what about Query Stream ? , in your example you are getting all users into process memory and then do operations , with stream u just could get singles doc and process it.

How to retrieve query data output from Dynamodb using Node.js

I am having difficulty to understand where is my query stored in which array variable. I know the "var params" contains all the query conditions and "db.query" will perform the actual query action. But how do I get my query result which match my search condition?
At the end, I want to pass my query output parameters (title, postedby) to html or jade for display so I need to understand where all the query data are stored at first place and how to retrieve them. Some sample code to demo this part would be greatly appreciated. Thanks
var params = {
TableName: tablename,
AttributesToGet: [
'title',
'postedBy'
],
Limit: 20,
ScanIndexForward: true,
KeyConditions: { /* required */
key : {
ComparisonOperator: 'EQ', /* required */
AttributeValueList: [
{
'N' : 'some value'
}
/* more items */
},
},
};
db.query(params, function(err, data) {
if (err) {
console.log('show error here');
} else {
console.log('Success');
}
});
I found out that all my query data is stored inside "data" from function(err, data). But i am having problem passing "data" to the following export function which later can be used as a local variable inside my webpage. Then I realized the "data" part is always empty {} inside export function even i am sure the above db.query can return the right result. So i believe somehow the export.list function is executed before the db.query happen so I am not getting any data assigned. How can I overcome this part? Thanks.
exports.list = function(req, res){
res.render('post', {title:'Title', posts : data });
console.log("Result: %j", data);
};
Note: Console.log("Result: ", data) is showing {}
I figured it out. Use the callback function due to the nature of javascript. Thanks.

Resources