express mongoose pagination based on date (createdAt) - node.js

Can anyone please suggest me how to implement pagination based on date? For example, users will push some data daily and they may want to see how many data records they've pushed on a certain day by paginating. They may not push the exact amount of data every day, so what is the most scalable way to implement this? Here is typical pagination that I implemented
router.get('/:page?', (req, res, next) => {
const perPage = 5;
let page = req.params.page || 1;
Upload.find({ user: req.user._id })
.populate('user text')
.skip((perPage * page) - perPage)
.limit(perPage)
.exec((err, uploads) => {
Upload.count().exec((err, count) => {
if (err) return next(err)
res.render('record', {
title: 'Records',
data: uploads,
current: page,
pages: Math.ceil(count / perPage)
});
})
})
});

well, so in this case we can go initially with the find query only.
later you can implement with aggregation.
so the idea is,
in each page we will show the data of each day, doesnot matter how many are there.
[surely not a good idea when you have a huge amount of data for that day or against your query]
here is what i have implemented for you:
router.get('/:page?', (req, res, next) => {
const page = req.params.page ? req.params.page : new Date().getDate();
const currentYear = new Date().getFullYear();
const currentMonth =new Date().getMonth();
Upload.find({
"createdAt": {
"$gte": new Date(currentYear, currentMonth, page),
"$lt": new Date(currentYear, currentMonth, page)}
})
.populate('user text')
.exec((err, uploads) => {
if (err) return next(err)
//uploads contains the result of date `page` of `currentMonth` month and year `currentYear`
res.render('record', {
title: 'Records',
data: uploads,
resultDay:page
});
})
});
few things to remember, in here, we cant use limit & skip as we are trying to query for a day.
so in this above if you pass the page param as 1 it will give the result of 1st day of the current month and current year.
in here we dont know how many page will be there, as because we dont know for which day how many docs are there in advance.
so, its expected in frontend you will show Next button and call this api. with the +1 value of current page value.
and accordingly you can show the previous button too.
to get count of all the docs of a sigle day you can pass the cindition part, which we are already using inside .find() in .count().
Hope this helps,
let me know if i've missed anything.

Related

How do I copy entries from one collection to another using mongoose?

I'm trying to create a little task management site for a work project. The overall goal is here is that the tasks stay the same each month (their status can be updated and whatnot), and they need to be duplicated at the start of each new month so they can be displayed and sorted by on a table.
I already figured out how to schedule the task, I have the table I need set up. A little explanation before the code - the way I'm planning on doing this is having two different task collections - one I've called "assignments", will have the tasks that need to be duplicated (with their description, status and other necessary data) and another collection, which I called "tasks", will have the exact same data but with an additional "date" field. This is where the table will get it's data from, the date is just for sorting purposes.
This is what I have so far -
Index.js: gets all the assignments from the database, and sends the object over to the duplicate function.
router.get('/test', async function(req, res, next) {
let allTasks = await dbModule.getAllAssignments();
let result = await dbModule.duplicateTasks(allTasks);
res.json(result);
});
dbmodule.js:
getAllAssignments: () => {
allAssignments = Assignment.find({});
return allAssignments;
},
duplicateTasks: (allTasksToAdd) => {
try {
for (let i = 0; i < allTasksToAdd.length; i++) {
let newTask = new Task({
customername: allTasksToAdd.customername,
provname: allTasksToAdd.provname,
description: allTasksToAdd.description,
status: allTasksToAdd.status,
date: "07-2020"
})
newTask.save();
}
return "Done"
} catch (error) {
return "Error"
}
}
The issue arises when I try and actually duplicate the tasks. For testing purposes I've entered the date manually this time, but that's all that ends up being inserted - just the date, the rest of the data is skipped. I've heard of db.collection.copyTo(), but I'm not sure if it'll allow me to insert the field I need or if it's supported in mongoose. I know there's absolutely an easier way to do this but I can't quite figure it out. I'd love some input and suggestions if anyone has any.
Thanks.
The problem is that allTasksToAdd.customername (and the other fields your trying to access) will be undefined. You need to access the fields under the current index:
let newTask = new Task({
customername: allTasksToAdd[i].customername,
provname: allTasksToAdd[i].provname,
description: allTasksToAdd[i].description,
status: allTasksToAdd[i].status,
date: "07-2020"
})
Note that you can simplify this by using a for .. of loop instead:
for (const task of allTasksToAdd) {
const newTask = new Task({
customername: task.customername,
provname: task.provname,
description: task.description,
status: task.status,
date: "07-2020"
});
newTask.save();
}

Node & Mongoose - Loading correct collection after clicking a link

I need help!! I have this code:
router.get("/campgrounds", function(req,res) {
Campground.find({}, function(err, campgrounds) {
if(err) {
console.log(err);
} else {
res.render("campgrounds/index", {campgrounds: campgrounds});
}
});
});
Unfortunately I couldn't find any good examples online.
I wanted index to load different mongo collections after clicking on the link that leads to /campgrounds which I want to change to /:locations_id.
The main page will have 3 links to location1, location2 and location3 respectively.
Is there a way to load a different collection (location 1, 2 or 3) at /:locations_id depending on the link clicked before?
My idea was to use req.params.locations_id and maybe append some info on the clicked link and use it in an if statement in order to load the correct collection.
Thank you very much for your help and I apologize for the extremely conceptual question.
Hmmm, I guess you can pull a trick.
//Put every collection in an array
var collections = [
Campground,
Collection2,
Collection3
];
//When user visits: example.com/campgrounds/1
router.get("/campgrounds/:locationId", function(req,res) {
//Parse the index from the URL
var locId = parseInt(req.params.locationId);
//Make sure that we aren't blown up :)
if(isNaN(locId) || locId > 2) {
//Default collection's index - which is Campground in this example
locId = 0;
}
//Match the given collection by its index and return it
collections[locId].find({}, function(err, campgrounds) {
if(err) {
console.log(err);
} else {
res.render("campgrounds/index", {campgrounds: campgrounds});
}
});
});

Securing access to collection from the client's side

I have a meteor app prototype that works well, but is very insecure as of now: I needed to display a list of matching users to the currently logged-in user. For starters, I decided to publish all users, limiting the fields to what I would need to filter the user list on the client side.
Meteor.publish('users', function () {
return Meteor.users.find({}, {
fields: {
'profile.picture': 1,
'profile.likes': 1,
'profile.friends': 1,
'profile.type': 1
}
});
});
Then in my router, I would do a request to only show what I wanted on the client side:
Router.map(function() {
this.route('usersList', {
path: '/users',
waitOn: function () {
return Meteor.subscribe('users');
},
data: function () {
var user = Meteor.user();
return {
users: Meteor.users.find({ $and: [
{_id: {$ne : user._id}},
{'profile.type': user.profile.interest}
]})
};
}
});
});
In the code above, I query all users who are not the current user and whose type correspond the current user's interest. I also display a certain border on the photos of users who have my user in their "profile.friends" array, using this client helper:
Template.userItem.helpers({
getClass: function(userId) {
var user = Meteor.user();
var lookedup = Meteor.users.findOne({_id: userId});
if ($.inArray(user._id, lookedup.profile.friends) !== -1)
return "yes";
return "no";
}
});
Now this all worked great, but with this setup, every client can query every users and get their type, picture, list of friends and number of likes. If I was in an MVC, this info would only be accessible on server side. So I decided my next iteration to be a security one. I would move my query from the router file to the publications file. That's where trouble began...
Meteor.publish('users', function () {
var user = Meteor.users.findOne({_id: this.userId});
var interest = user.profile.interest;
// retrieve all users, with their friends for now
allUsers = Meteor.users.find({ $and: [
{'_id': {$ne: user._id}},
{'profile.type':interest}
]},
{ fields: {'profile.picture': 1, 'profile.friends': 1}}
);
return allUsers;
});
And in the router:
Router.map(function() {
this.route('usersList', {
path: '/users',
waitOn: function () {
return Meteor.subscribe('users');
},
data: function () {
var user = Meteor.user();
return {users: Meteor.users.find({_id: {$ne : user._id}})};
}
});
});
(note that I still need to exclude the current user from the router query since the current user is always fully published)
This works, but:
the user list does not get updated when I change the user interest and then do a Router.go('usersList'). Only when I refresh the browser, my list is updated according to the user's new interest. No idea why.
this solution still publishes the users' friends in order to display my matching borders. I wish to add a temporary field in my publish query, setting it to "yes" if the user is in the user's friends and "no" otherwise, but... no success so far. I read I could use aggregate to maybe achieve that but haven't managed to so far. Also, aggregate doesn't return a cursor which is what is expected from a publication.
This problem makes me doubt about the praises about meteor being suitable for secure apps... This would be so easy to achieve in Rails or others!
EDIT: As requested, here is the code I have so far for the transition of my "matching" check to the server:
Meteor.publish('users', function () {
var user = Meteor.users.findOne({_id: this.userId});
var interest = user.profile.interest;
// retrieve all users, with their friends for now
allUsers = Meteor.users.find({ $and: [
{'_id': {$ne: user._id}},
{'profile.type':interest}
]},
{ fields: {'profile.picture': 1, 'profile.friends': 1}}
);
// ------------- ADDED ---------------
allUsers.forEach(function (lookedup) {
if (_.contains(lookedup.profile.friends, user._id))
lookedup.profile.relation = "yes";
else
lookedup.profile.relation = "no";
lookedup.profile.friends = undefined;
return lookedup;
});
// ------------- END ---------------
return allUsers;
});
Obviously this code doesn't work at all, since I cannot modify cursor values in a foreach loop. But it gives an idea of what I want to achieve: give a way to the client to know if a friend is matched or not, without giving access to the friend lists of all users to the client. (and also, avoiding having to do one request per each user during display to ask the server if this specific user matches this specific one)
You can add a transform function and modify a cursor docs on the fly
meteor Collection.find

How to split mongodb query into result pages

I have learning application that only create users, manage sessions and delete users. The backend is in Node.js.
I have the query that retrieve the list of users
var userList = function ( page, callback ) {
user.find({}, function ( err, doc ) {
if (err) {
callback (err);
}
callback ( doc );
});
};
then I have the handler that send it back to the application
var userList = function ( req, res ) {
var page = req.param.page;
models.account.userList( page, function ( result ) {
res.json ( 200, result );
});
};
and on the client side I take the results and display them in a table on the browser.
What I want to accomplish, is to split the list into pages of... lets say... 10 users each. So using the parameter Page of the function userList I would expect to send the right answers:
Page1: results 0 to 9
page2: results 10 to 19
etc
I could do a loop to extract what I want, but I am sure that receive from the query the whole list of users to reprocess it on a loop to only extract 10 users is not the right approach, but I could not figure out what the right approach would be.
I have already googled, but did not find the answer what I wanted. Also quickly check MongoDB documentation.
As you can guess I am new at this.
Can you please help me guiding on the right approach for this simple task?
Thank you very much.
Using bduran code, I was able to do the pagination that I wanted. The code after the answer is:
var userList = function ( page, itemsPerPage, callback ) {
user
.find()
.skip( (page - 1) * itemsPerPage )
.limit ( itemsPerPage )
.exec ( function ( err, doc ) {
if (err) {
//do something if err;
}
callback ( doc );
});
};
The most easy and direct way to do this is with skip and query. You select pages of, for example, ten elements and then skip ten times that number per page. With this method you don't need to get all elements and if the query is sorted with and index is really fast:
user
.find()
.skip((page-1)*10)
.limit(10)
.exec(function(err, doc){
/*Rest of the logic here*/
});
If you need to sort the results in a non-natural order (with .sort) you will need to create an index in the database to speed up and optimize the queries. You can do it with .ensureIndex. In the terminal put the query you use to do and you are done.
db.user.ensureIndex({ createdAt: -1 })

How to make pagination with mongoose

I want to make a pagination feature in my Collection. How can find a documents with 'start' and 'limit' positions and get total document number in a single query?
You can't get both results in one query; the best you can do is to get them both using one Mongoose Query object:
var query = MyModel.find({});
query.count(function(err, count) {...});
query.skip(5).limit(10).exec('find', function(err, items) {...});
Use a flow control framework like async to cleanly execute them in parallel if needed.
You can use the plugin Mongoose Paginate:
$ npm install mongoose-paginate
After in your routes or controller, just add :
Model.paginate({}, { page: 3, limit: 10 }, function(err, result) {
// result.docs
// result.total
// result.limit - 10
// result.page - 3
// result.pages
});
If you plan to have a lot of pages, you should not use skip/limit, but rather calculate ranges.
See Scott's answer for a similar question: MongoDB - paging
UPDATE :
Using skip and limit is not good for pagination. Here is the discussion over it.
#Wes Freeman, Gave a good answer. I read the linked pose, you should use range query. n = 0; i = n+10; db.students.find({ "id" : { $gt: n, $lt: (n+i)} } );
OLD ANSWER (don't use this)
use something like
n = db.students.find().skip(n).limit(10);
//pass n dynamically, so for page 1 it should be 0 , page 2 it should be 10 etc
more documentation at http://www.mongodb.org/display/DOCS/Advanced+Queries
user.find({_id:{$nin:friends_id},_id:{$ne:userId}},function(err,user_details){
if (err)
res.send(err);
response ={
statusCode:200,
status:true,
values:user_details
}
res.json(response);
}).skip(10).limit(1);
I am using this function,
You can check if prev and next data is available or not.
async (req, res) => {
let { page = 1, size = 10 } = req.query
page = parseInt(page)
size = parseInt(size)
const query = {}
const totalData = await Model.find().estimatedDocumentCount()
const data = await Model.find(query).skip((page - 1) * size).limit(size).exec()
const pageNumber = Math.ceil(totalData / size)
const results = {
currentPage: page,
prevPage: page <= 1 ? null : page - 1,
nextPage: page >= pageNumber ? null : page + 1,
data
}
res.json(results) }
To know more estimatedDocumentCount

Resources