I have learning application that only create users, manage sessions and delete users. The backend is in Node.js.
I have the query that retrieve the list of users
var userList = function ( page, callback ) {
user.find({}, function ( err, doc ) {
if (err) {
callback (err);
}
callback ( doc );
});
};
then I have the handler that send it back to the application
var userList = function ( req, res ) {
var page = req.param.page;
models.account.userList( page, function ( result ) {
res.json ( 200, result );
});
};
and on the client side I take the results and display them in a table on the browser.
What I want to accomplish, is to split the list into pages of... lets say... 10 users each. So using the parameter Page of the function userList I would expect to send the right answers:
Page1: results 0 to 9
page2: results 10 to 19
etc
I could do a loop to extract what I want, but I am sure that receive from the query the whole list of users to reprocess it on a loop to only extract 10 users is not the right approach, but I could not figure out what the right approach would be.
I have already googled, but did not find the answer what I wanted. Also quickly check MongoDB documentation.
As you can guess I am new at this.
Can you please help me guiding on the right approach for this simple task?
Thank you very much.
Using bduran code, I was able to do the pagination that I wanted. The code after the answer is:
var userList = function ( page, itemsPerPage, callback ) {
user
.find()
.skip( (page - 1) * itemsPerPage )
.limit ( itemsPerPage )
.exec ( function ( err, doc ) {
if (err) {
//do something if err;
}
callback ( doc );
});
};
The most easy and direct way to do this is with skip and query. You select pages of, for example, ten elements and then skip ten times that number per page. With this method you don't need to get all elements and if the query is sorted with and index is really fast:
user
.find()
.skip((page-1)*10)
.limit(10)
.exec(function(err, doc){
/*Rest of the logic here*/
});
If you need to sort the results in a non-natural order (with .sort) you will need to create an index in the database to speed up and optimize the queries. You can do it with .ensureIndex. In the terminal put the query you use to do and you are done.
db.user.ensureIndex({ createdAt: -1 })
Related
I have an object that looks like this:
let queries = [
{
name: "checkUsers",
query: 'select * from users where inactive = 1'
},
{
name: "checkSubscriptions",
query: 'select * from subscriptions where invalid = 1'
}
]
I am making an AWS Lambda function that will iterate these queries, and if any of them returns a value, I will send an email.
I have come up with this pseudo code:
for (let prop in queries) {
const result = await mysqlConnector.runQuery(prop.query).catch(async error => {
// handle error in query
});
if (result.length < 0){
// send email
}
}
return;
I am wondering is this ideal approach? I need to iterate all the object queries.
I don't see anything wrong with what you are trying to achieve but there are few changes you could do
Try to use Promise.all if you can. This will speed up the overall process as things will execute in parallel. It will depend on number of queries as well.
Try leverage executing multiple statements in one query. This way you will make one call and then you can add the logic to identify. Check here
Can anyone please suggest me how to implement pagination based on date? For example, users will push some data daily and they may want to see how many data records they've pushed on a certain day by paginating. They may not push the exact amount of data every day, so what is the most scalable way to implement this? Here is typical pagination that I implemented
router.get('/:page?', (req, res, next) => {
const perPage = 5;
let page = req.params.page || 1;
Upload.find({ user: req.user._id })
.populate('user text')
.skip((perPage * page) - perPage)
.limit(perPage)
.exec((err, uploads) => {
Upload.count().exec((err, count) => {
if (err) return next(err)
res.render('record', {
title: 'Records',
data: uploads,
current: page,
pages: Math.ceil(count / perPage)
});
})
})
});
well, so in this case we can go initially with the find query only.
later you can implement with aggregation.
so the idea is,
in each page we will show the data of each day, doesnot matter how many are there.
[surely not a good idea when you have a huge amount of data for that day or against your query]
here is what i have implemented for you:
router.get('/:page?', (req, res, next) => {
const page = req.params.page ? req.params.page : new Date().getDate();
const currentYear = new Date().getFullYear();
const currentMonth =new Date().getMonth();
Upload.find({
"createdAt": {
"$gte": new Date(currentYear, currentMonth, page),
"$lt": new Date(currentYear, currentMonth, page)}
})
.populate('user text')
.exec((err, uploads) => {
if (err) return next(err)
//uploads contains the result of date `page` of `currentMonth` month and year `currentYear`
res.render('record', {
title: 'Records',
data: uploads,
resultDay:page
});
})
});
few things to remember, in here, we cant use limit & skip as we are trying to query for a day.
so in this above if you pass the page param as 1 it will give the result of 1st day of the current month and current year.
in here we dont know how many page will be there, as because we dont know for which day how many docs are there in advance.
so, its expected in frontend you will show Next button and call this api. with the +1 value of current page value.
and accordingly you can show the previous button too.
to get count of all the docs of a sigle day you can pass the cindition part, which we are already using inside .find() in .count().
Hope this helps,
let me know if i've missed anything.
I am writing a Node.JS app with MongoDB. One of the things I need to implement is the listing of the objects. I've already implemented the pagination using the skip() and limit() functions, thanks to this answer:
myModel.find()
.skip(offset)
.limit(limit)
.exec(function (err, doc) {
// do something with objects
})
The thing is, I want my endpoint to return metadata, and one of the fields I need is a boolean representing if there are more objects that can be loaded.
The most simple way to implement this is just loading one more object (to determine if there are more objects to display) and not showing it to the user, something like that:
myModel.find()
.skip(offset)
.limit(limit + 1)
.exec(function (err, doc) {
var moreAvailable; // are there more documents?
if (doc.length > limit) {
moreAvailable = true;
doc.length = limit; // don't show the last object to the user
} else {
moreAvailable = false;
}
})
But I'm pretty sure that there should be more clever way to do this. Is it?
Use db.collection.find(<query>).count() https://docs.mongodb.com/manual/reference/method/cursor.count/
var total = db.collection.find(<query>).count();
var is_more = total > skip + limit;
I'm using nodejs with the module cradle to interact with the couchdb server, the question is to let me understanding the reduce process to improve the view query...
For example, I should get the user data from his ID with a view like this:
map: function (doc) { emit(null, doc); }
And in node.js (with cradle):
db.view('users/getUserByID', function (err, resp) {
var found = false;
resp.forEach(function (key, row, id) {
if (id == userID) {
found = true;
userData = row;
}
});
if (found) {
//good, works
}
});
As you can see, this is really bad for large amount of documents (users in the database), so I need to improve this view with a reduce but I don't know how because I don't understand of reduce works.. thank you
First of all, you're doing views wrong. View are indexes at first place and you shouldn't use them for full-scan operations - that's ineffective and wrong. Use power of Btree index with key, startkey and endkey query parameters and emit field you like to search for as key value.
In second, your example could be easily transformed to:
db.get(userID, function(err, body) {
if (!err) {
// found!
}
});
Since in your loop you're checking row's document id with your userID value. There is no need for that loop - you may request document by his ID directly.
In third, if your userID value isn't matches document's ID, your view should be:
function (doc) { emit(doc.userID, null); }
and your code will be looks like:
db.view('users/getUserByID', {key: userID}, function (err, resp) {
if (!err) {
// found!
}
});
Simple. Effective. Fast. If you need matched doc, use include_docs: true query parameter to fetch it.
This is likely a simple answer but I'm relatively new to asynchronous programming and I'm looking for somebody to point me in the right direction.
My question is this - What is the best way to go about finding or creating a number of documents from an array of names (I'm using Mongoose) and then returning an array of _id's?
So to be clear, I want to:
Given an array of names, find or create a document with each name
Return an array of the existing or newly created documents _ids
You can use async module and within it it's async.parallel() method -
https://github.com/caolan/async#quick-examples
async.parallel([
function(){ ... },
function(){ ... }
], callback);
Or you can use promises and then Q.all() to get the array of ids back -
https://github.com/kriskowal/q#combination
Q.all(arrayOfFindOps).then(function(rows) {
return _.pluck(rows, '_id')
}
If you don't want to use any of the above and do it with callbacks, then you have to keep track of the count of array length, keep adding the ids to an array and when your completion counter reaches array length, call another function with the array you made.
This code can be easily modified to meet your requirements. Call this function for each document that required to be created if doesn't exist.
function(req, reply) {
// return document. if not found, create docuemnt.
docModel.findOne( {'name': req.params.name}, function ( err , doc) {
if(err){
//handle error
}
if(doc===null){
//find failed, time to create.
doc = new docModel( {'name': req.params.name} );
doc.save(function(err){
if(err){
//handle error
}
});
}
return reply(user._id);
});
}