Solutions in ANY language will be highly appreciated.
I have a collection of messages like so:
"group_messages": {
"-FirebaseID": {
"group": "-FirebaseID"
"timestamp": ...
}
}
When I want to obtain messages using Firebase, it's quite simple I just do this:
firebase.database().ref('/group_messages").orderByChild("group").equalTo(...).on(...)
However, lets say I just want the last 50 messages, then I would have to orderByChild("timestamp") however Firebase does not allow you to run two .orderBy queries, which results in having to download all of the messages from the database.
So how would I need to structure my data so I can paginate based on timestamp and keep my messages seperate from the groups/ path, as if the messages are a child of the group then they will all be loaded with the group onto the client.
I would like to be able to load messages 1-50 in a group, then once the user scrolls far enough, load message 51-100. However every Firebase query I can think of involves pulling all of the data to the client upfront.
NOTE: I do not want to do client-side pagination. The reason we need
pagination is because we have too many messages and it's slowing down
our application waiting on them to all be loaded.
Related
I am stuck on implementing Pagination and I just need a bit of help, either some example code or even just a hint to help me proceed in the right direction.
I'm looking for some guidance on how to integrate Google Datastore database cursors with front-end UI pagination controls. I know how to build an angular pagination service, but that's with retrieving all the data at once and due to performance issues (5,000 records+) I want to use cursors to retrieve data in subsets.
NOTE: There's a similar question here, but I need more detail than this accepted answer provides. Node pagination in google datastore
QUESTION: How can I integrate the paginated datastore cursor queries with the front-end UI controls to allow the user to select the current page and control number of results displayed on each page?
I need to build a page that displays a large number of records with dynamic pagination. The user must be able to select the number of records display on each page.
Since there are several thousand records that might be returned at one time, I want to use cursors to retrieve subsets of data.
There is an example of how to paginate in the docs, but it is a pretty basic example and does not demonstrate how to integrate with front-end UI controls.
Can anyone provide a more detailed example and/or point me in the right direction on where to begin with this requirement? Unfortunately I haven't been able to find any detailed examples online.
https://googlecloudplatform.github.io/google-cloud-node/#/docs/datastore/1.1.0/datastore
Paginating Records
var express = require('express');
var app = express();
var NUM_RESULTS_PER_PAGE = 15;
app.get('/contacts', function(req, res) {
var query = datastore.createQuery('Contacts')
.limit(NUM_RESULTS_PER_PAGE);
if (req.query.nextPageCursor) {
query.start(req.query.nextPageCursor);
}
datastore.runQuery(query, function(err, entities, info) {
if (err) {
// Error handling omitted.
return;
}
// Respond to the front end with the contacts and the cursoring token
// from the query we just ran.
var frontEndResponse = {
contacts: entities
};
// Check if more results may exist.
if (info.moreResults !== datastore.NO_MORE_RESULTS) {
frontEndResponse.nextPageCursor = info.endCursor;
}
res.render('contacts', frontEndResponse);
});
});
One thing to keep in mind is the first item on the Limitations of cursors list:
A cursor can be used only by the same project that performed the original query, and only to continue the same query. It is not
possible to retrieve results using a cursor without setting up the
same query from which it was originally generated.
So you need to always be able to re-create the original query inside your handler, which means you need to pass around the equivalent of your NUM_RESULTS_PER_PAGE value from one request to another. You also need to reset the query every time that value changes - meaning you can't continue browsing from where you were after changing the number of results displayed per page.
Then, to be able to use the pagination, you also need to pass around the current cursor value from one request to another, updated at every request.
Now I'm not a NodeJS user, so I can tell exactly how this passing values around from one request to another would typically be implemented. In your code req.query.nextPageCursor and frontEndResponse.nextPageCursor appear to be intended for this, but I can't tell if that's OK or not. Maybe this is a less specific question easier to find an answer for.
In python webapp2, for example, I can store such values server-side in the user's session in one request and read them from the session in a subsequent request. Donno if this is of any help.
I'm trying to create RestAPI. There are a lot of documents in DB, let's call those - goods. Each good have some property.
Client will get those, and show to user. Something like: ...api/goods?filter value here.
It looks pretty easy, but it is a wrong way to send all goods which satisfied the filter. I need to separate those, and send only needed part of data.
The first idea was create a separate route, that will obtain filter and amount documents on page, and return page count. So, front-end can build pagination and then, using handler on each pagination tab, makes requests to server, and get needed data.
I've created something like basic-example(code is not good but the main idea is):
https://github.com/Gordienko-RU/Tiny-pagination
But I thought there are another, better implementation of it. In 'best practice' there was some note about sending pagination in header, but I couldn't figure it out.
So, I want to know, what is the best way to send data by parts, but also give to client information needed to build pagination pannel(amount of pages).
I've found some handy method. There will be one route. Client do a request for data on first page, but answer contains not only needed data, it also will be an object with information about pagination. Something like:
data: [...],
pagination: {
pageCount: ...
etc.
}
Maybe not a 'best practice' but good enough)
I'm building an activity feed application, where a user can like/comments on each activity feed. I went through GetStream.io documentation and looks like I'll have to send the activity with object ids.
{
id:"ef696c12-69ab-11e4-8080-80003644b625",
actor:"User:1",
object:"Comment:12",
started_at:"2014-11-11T15:06:16+01:00",
target:"Feed:100",
time:"2014-11-11T14:06:30.494",
verb:"add"
}
User:1 and Feed:12 are the objects in my application database? Does it mean that, while retrieving activities, I'll have to hit my database to retrieve the complete feeds?
Say the Feed:12 had few likes and comments earlier from other users. How do I get the complete set of likes/comments on user timeline feed?
What if I want to customize the view, say I want to show all users (image, name, the profile like etc) along with comment with timestamp similar to FB? Do I need to send these attributes as additional parameters for each feed?
Thanks,
Yes, when you fetch a feed from Stream and we give you back these references like user:1 or comment:12, we expect that you'd "enrich" those details from your database.
Typically what our users do is track the name of the model (eg, user) and the user_id (eg, 1). When you get the feed and put it into a hash map, you'll iterate over the activities, pull out all of the actor attributes, and do a single lookup like select * from user where id in (1,3,5,6,9,12) so that you're only hitting your database one time for all user objects or all comment objects or whatever. Then, replace those activities in your hash map so now you'd have actor: <object for User 9> and any other attributes you'd need for your UI presentment.
Then do the same for other references you pass in the activity, and so on.
Things we DON'T recommend are putting in string references for things that could change on your side. For example, if you had actor: "user:ian" instead of my user_id, if I ever change my username later then things probably wouldn't work properly on your side.
I am building an application that should be able to list several thousand articles with pagination/ infinite scrolling. The user should be able to filter/ sort this list, and currently I am experiencing performance problems when sorting articles.
I set up a very basic application to demonstrate the problem: http://meteor-paginated-subscription-example.meteor.com/ (see Github: https://github.com/lacco/meteor-paginated-subscription-example ). If you open the Firefox/ Chrome console and click on "created at"/ "priority" to initiate the sorting, you see that Template.articles.rendered is called several hundred times on one click. You also see that the table takes some time to be "final", during loading and rendering the order of the rows changes very often.
I am sure that I am doing some crap in my code, but I can't figure it out where the cause of my problems is. Can you help me out?
The collection.cursor, Models.Articles.find({}, ...), is a reactive data source. Any change to the items in that cursor will cause the template articles_table and the subtemplates inside `{{#each articles}} to rerender.
Your resort leads to changing your data subscription so the client calls the servers asking for new data. The flickering you see is caused by displaying the resorted intermediate steps as new data arrives and old data is deleted. Try checking if handle.ready() before returning anything from the helper Template.articles_table.articles so the table is only displayed when your dataset is complete.
Update: In response to additional function requests in the comments. If you store "previousSort" and also store "previousLimit" whenever the sort or limit changes you can display your old query with reactivity turned off while the client waits for for the subscription to become ready:
if ( handle.ready() ){
return Models.Articles.find({}, {sort: sort, limit: limit});
}
else {
return Models.Articles.find({}, {sort: previousSort, limit: previousLimit, reactive: false});
}
I'm doing a notification system for my website.
A notification systeme like facebook. Or stackoverflow.
I have 2 problems.
How store in db ? I can store ALL notifications in the user document ? or in a document apart (because i think monogdb is limited for size in a document ?) Or, store intelligently ? (using inc, or a value (see: true/false) in db, with query sophisticated)
How do for brought at the page ? For exemple, when i click in a link in my inbox for stackoverflow, i'm redirect to the page. But me, i have a system that is multipage for exemple: I have 100 friends. There are listed 30 per page. So when i click on the notification i can't redirect to the because it's impossible to know the good page (users can be removed).
Thank you very much !
And if you have another ideas, tell me. Thanks.
EDIT:
(sorry for my english, i'm french)
For the first problem, i realize that i have to wait the time comes to choose my structure. Because my notification is .. a little complicated, so advance to the feeling.
For the second, i solved the problem. I explain:
(I take the exemple of friends because it's easy to undestand.)
I stored my data like this:
{
friends: [
{_id: xxxxx, ts: xxxx},
{_id: xxxxx, ts: xxxx}
]
}
Imagine i display all friends: 30 per page.
The problems are:
when i want to display all friends i cant sort using mongo. (a little problem)
If i want to lead a user to this list (30 per page) at a special friend, always keeping the sort by ts. I can't know the page. The uniq solution is to take all document.
But: veryyy bad in performance.
So, i store like this:
{
friends: {
xxxx: {ts:xxx},
xxxx: {ts:xxx}
}
}
Know i can sort the document, with use skip and limit.
So if i want a portion, i do not need to take all documents.
To know the page, i just do the number of < or > to the ts, i have for exemple 11 friends who are > to the ts of the friends that i want, and do a count for all friends (ex: 50 friends) with 50 and 11, i can guess the page.
Is this solution is good ?
- i need a count
- a query to know the number of > or <
and i can take the page where is listed the friend, keeping the sort ts
You can don't understand why i use a count. I need because they are not store in the same docment.
2 EDIT:
The problem with this solution, is that i need to make query object and update object outside of the mongo query (ex: for do friends.xxxxxx: {$exists:true}
ps: And what advantages are to use ts instead of date for mongodb ?
I'm using ts but i think i will store date, and no ts.
3 EDIT
I will do like Sammaye. Store in separate document. Take a look at: http://mongly.com/Multiple-Collections-Versus-Embedded-Documents/#1 and http://openmymind.net/2012/1/30/MongoDB-Embedded-Documents-vs-Multiple-Collections/
#Stennie make a pretty complete answer.
However recently I did a similar thing in PHP for my website. The first thing to understand is whether you are doing a notification system or a wall (the two are very different), it seems unclear to me and I am not sure what you mean by:
How do for brought at the page ? For exemple, when i click in a link
in my inbox for stackoverflow, i'm redirect to the page. But me, i
have a system that is multipage for exemple: I have 100 friends. There
are listed 30 per page. So when i click on the notification i can't
redirect to the because it's impossible to know the good page (users
can be removed).
That is not very good English and is very confusing when I read it. If you can expand on that I am sure people can answer better.
For a notification system I found that a large collection of notification objects also worked. So I had a schema like:
{
_id: {},
to_user: ObjectId{},
user_id: ObjectId{}, // Originating user
custom_text: "has posted a new comment on your wall post",
read: false,
ts: MongoDate()
}
And this would literally be the document I have to produce notifications. Each time a user commits an action that generates a notification it writes a new row to the DB with to_user being populated each time with each user needing to be notified. As for multiple users commiting the same action I actually convert the user_id field in a list of OjbectId's so I can say:
Sam, Dan and Mike all commented on your wall post
I then query by ts storing the last ts that the user looked at in their row allowing me to do a range based query on the newest notifications each time. This works quite well for sharding and querying in my personal experience.
Hope it helps,
Whether to embed or link is a common question for data modelling in MongoDB. If your number of notifications is going to be unbounded, you are likely going to be better saving these in a separate collection.
The current 16Mb document limit actually isn't as much as an issue as some other considerations:
A performance issue you may encounter by including all notifications in a single document is that fast-growing documents may also need to be relocated in the database more frequently (see Padding Factor).
You may want to be applying multiple updates to a document (such as setting a "read" flag on notifications) in a very short period of time, which means more contention for updating the same document (see Atomic Operations).
In order to implement paging you can use limit() in combination with a range query or skip(). A range query (eg. based on an indexed notificationDate) will make more effective use of indexes and perform better than skip() as your collection grows.