firebase Starting point was already set - node.js

I use firebase admin and realtime database on node.js
Data look like
When I want to get data where batch = batch-7, I was doing
let batch = "batch-7";
let ref = admin.database().ref('qr/');
ref.orderByChild("batch").equalTo(batch).on('value', (snapshot) =>
{
res.json(Object.assign({}, snapshot.val()));
ref.off();
});
All was OK!
But now i should create pagination, i.e. I should receive data on 10 elements depending on the page.
I use this code:
let page = req.query.page;// num page
let batch = req.params.batch;// batch name
let ref = admin.database().ref('qr/');
ref.orderByChild("batch").startAt(+page*10).limitToFirst(10).equalTo(batch)
.on('value', (snapshot) =>
{
res.json(Object.assign({}, snapshot.val()));
ref.off();
});
But I have error:
Query.equalTo: Starting point was already set (by another call to startAt or equalTo)
How do I get data in the amount of N, starting at position M, where batch equal my batch

You can only call one startAt (and/or endAt) OR equalTo. Calling both is not possible, nor does it make a lot of sense.
You seem to have a general misunderstanding of how startAt works though, as you're passing in an offset. Firebase queries are not offset based, but work purely on the value, often also referred to as an anchor node.
So when you want to get the data for a second page, and you order by batch, you need to pass in the value of batch for the anchor node; first item that you want to be returned. This anchor node is typically the last item of the previous page, since you don't know the first item of the next page yet. And for this anchor node, you need to know the value of the item you order on (batch) and usually also its key (if/when there may be multiple nodes with the same value for batch).
It also means that you usually request one item more than you need, which is the anchor node.
So when you request the first page, you should track the key/batch of the last node:
var lastKey, lastValue;
ref.orderByChild("batch").equalTo(batch).limitToFirst(10).on('value', (snapshot) => {
snapshot.forEach((child) => {
lastKey = child.key;
lastValue = child.child('batch').value();
})
})
Then when you need the second page, you do a query like that:
ref.orderByChild("batch").start(lastValue, lastKey).endAt(lastValue+"\uf8ff").limitToFirst(11).on('value', (snapshot) => {
snapshot.forEach((child) => {
lastKey = child.key;
lastValue = child.child('batch').value();
})
})
There's one more trick above here: I use startAt instead of equalTo, so that we can get pagination working. But it then uses endAt to ensure we still end at the correct item, by using the last known Unicode character as the last batch value to return.
I'd also highly recommend checking out some of the previous questions on pagination with the Firebase Realtime Database.

Related

How to power a windowed virtual list with cursor based pagination?

Take a windowed virtual list with the capability of loading an arbitrary range of rows at any point in the list, such as in this following example.
The virtual list provides a callback that is called anytime the user scrolls to some rows that have not been fetched from the backend yet, and provides the start and stop indexes, so that, in an offset based pagination endpoint, I can fetch the required items without fetching any unnecessary data.
const loadMoreItems = (startIndex, stopIndex) => {
fetch(`/items?offset=${startIndex}&limit=${stopIndex - startIndex}`);
}
I'd like to replace my offset based pagination with a cursor based one, but I can't figure out how to reproduce the above logic with it.
The main issue is that I feel like I will need to download all the items before startIndex in order to receive the cursor needed to fetch the items between startIndex and stopIndex.
What's the correct way to approach this?
After some investigation I found what seems to be the way MongoDB approaches the problem:
https://docs.mongodb.com/manual/reference/method/cursor.skip/#mongodb-method-cursor.skip
Obviously he same approach can be adopted by any other backend implementation.
They provide a skip method that allows to skip an arbitrary amount of items after the provided cursor.
This means my sample endpoint would look like the following:
/items?cursor=${cursor}&skip=${skip}&limit=${stopIndex - startIndex}
I then need to figure out the cursor and the skip values.
The following code could work to find the closest available cursor, given I store them together with the items:
// Limit our search only to items before startIndex
const fragment = items.slice(0, startIndex);
// Find the closest cursor index
const cursorIndex = fragment.length - 1 - fragment.reverse().findIndex(item => item.cursor != null);
// Get the cursor
const cursor = items[cursorIndex];
And of course, I also have a way to know the skip value:
const skip = items.length - 1 - cursorIndex;

OrderBy and StartAt with two different fields firestore

In my app, I have comments that have a field value of threadCommentCount. I want to order the comments using orderBy threadCommentCount descending and then have pagination continue this using startAfter(lastThreadCommentCount). The problem is when threadCommentCount is 0, which is a lot of them, it will return and the same data every time since it starts at 0 everytime. Here is the query:
popularCommentsQuery = db
.collection('comments')
.where('postId', '==', postId)
.orderBy('threadCommentCount', 'desc')
.startAfter(startAfter)
.limit(15)
.get()
This will return the same comments everytime once threadComment count is 0. I'm unable to send the last document snapshot because im using cloud functions and I dont want to send the documentSnapshot in a get query parameter. I don;t really care how the comments are ordered after threadCommentCount is 0, I just need to not get any duplicates. Any help is great!
All Firestore queries have an implicit orderBy("__name__", direction) to resolve any ties between documents that have the same values for the other named orderBy fields. This makes the final sort order stable. But it also enables you to pass another argument to startAfter to provide the document ID of the anchor document that you wish to use for the purpose of pagination.
.startAfter(lastThreadCommentCount, lastDocumentId)
Between these two values, you should be able to uniquely identify the document in the result set to start the next page.
so, I was trying OrderBy and StartAfter with two different fields(time,key) in firestore to establish pagination in flastList.
Key point is that we can pass document snapshot to define the query cursor [reference]
Here is how I managed to do it.
step 1: get the document Id(which is auto generated by firebase) with where() [reference]
const docRef = firestore().collection('shots').where('key','==','custom_key')
const fbDocIdGeneratedByFirebase= await docRef.get().docs[0].id;
step 2: get document snapshot with firebase generated document Id (which we got in the 1st step)
const docRef2= firestore().collection('shots').doc(fbDocIdGeneratedByFirebase)
const snapshot = await docRef2.get();
step 3: pass the snapshot got in step 2 to startAfter() so that the cursor will point there [reference]
let additionalQuery = firestore().collection('shots')
.orderBy("time", "desc")
.startAfter(snapshot)
.limit(this.state.limit)
let documentSnapshots = await additionalQuery.get(); // you know what to do next
...
Can you Improve the solution??

How to limit .once('value) in firebase-admin node.js

How do I limit .once('value') in firebase-admin?
Code:
async function GetStuff(limit, page){
const data = await ref.limitToFirst(parseInt(limit)).once('value')
return data.val();
}
I wanted to create a page system, where a it sends request for a limited amount of data, and the user can change the page to get different data, but for some reason, I can't get it to work.
The code above only gets the first 20(when limit is 20), but how can I make it start at 20, so I can make this page feature.
I thought:
Code:
async function GetStuff(limit, page){
const data = await ref.startAt(limit*page).limitToFirst(parseInt(limit)).once('value')
return data.val();
}
You might want to review the relevant documentation. It looks like you're trying to pass the offset of a child to startAt, but that's not how startAt works. It accepts the actual value of the child to start at. Pagination by offset index is not supported.
The way you use startAt is typically passing the last sorted value retrieved by the prior query (or, if you don't want to retrieve that value again, 1 + that value, or a string that is lexically greater than the last string received. As such, some data sets might actually be difficult to paginate if they have the same sorted value repeated many times.

Paginating a mongoose mapReduce, for a ranking algorithm

I'm using a MongoDB mapReduce to code a ranking feed algorithm, it almost works but the latest thing to implement is the pagination. The map reduce supports the results limitation but how could I implement the offset (skipping) based e.g. on the latest viewed _id of the results, knowing that I'm using mongoose?
This is the procedure I wrote:
o = {};
o.map = function() {
//log10(likes+comments) / elapsed hours from the post creation
emit(Math.log(this.likes + this.comments + 1) / Math.LN10 / Math.abs((now - this.createdAt) / 6e7 + 1), this);
};
o.reduce = function(key, values) {
//sort the values, when they have the same score
values.sort(function(a, b) {
a.createdAt - b.createdAt;
});
//serialize the values, because mongoose does not support multiple returned values
return JSON.stringify(values);
};
o.scope = {now: new Date()};
o.limit = 15;
Posts.mapReduce(o, function(err, results) {
if (err) return console.log(err);
console.log(results);
});
Also, if the mapReduce it's not the way to go, do you suggest other on how to implement something like this?
What you need is a page delimiter which is not the id of the latest viewed as you say, but your sorting property. In this case, it seems to be the formula Math.log(this.likes + this.comments + 1) / Math.LN10 / Math.abs((now - this.createdAt) / 6e7 + 1).
So, in your mapReduce query needs to hold a where value of that formula above. Or specifically, 'formula >= . And also it needs to hold the value of createdAt at the last page, since you don't sort by that. (Assuming createdAt is unique). So yourqueryof mapReduce would saywhere: theFormulaExpression, createdAt: { $lt: lastCreatedAt }`
If you do allow multiple identical createdAt values, you have to play a little outside of the database itself.
So you just search by formula.
Ideally, that gives you one element with exactly that value, and the next ones sorted after that. So in reply to the module caller, remove this first element off the array (and make sure you actually ask for more results then you need because of this).
Now, since you allow for multiple similar values, you need another identifying prop, say, object id or created_at. Your consumer (caller of this module) will have to provide both (last value of the score, createdAt of the last object). Say you have a page split exactly in the middle - one or more objects is on the previous page, another set on the next
. You'd have to not simply remove the top value (because that same score is already served on the previous page), but possibly several of them from the top.
Then it goes really crazy, because potentially your whole page was already served - compare the _ids, look for the first one after the one your module caller has provided you with. Or look into the data and determine how many matching values like that are there, try to get at least as many more values from mapReduce then you have on your actual page size.
Aside from that, I would do this with aggregation instead, it should be much more preformant.

How to maintain counters with LinqToObjects?

I have the following c# code:
private XElement BuildXmlBlob(string id, Part part, out int counter)
{
// return some unique xml particular to the parameters passed
// remember to increment the counter also before returning.
}
Which is called by:
var counter = 0;
result.AddRange(from rec in listOfRecordings
from par in rec.Parts
let id = GetId("mods", rec.CKey + par.UniqueId)
select BuildXmlBlob(id, par, counter));
Above code samples are symbolic of what I am trying to achieve.
According to the Eric Lippert, the out keyword and linq does not mix. OK fair enough but can someone help me refactor the above so it does work? A colleague at work mentioned accumulator and aggregate functions but I am novice to Linq and my google searches were bearing any real fruit so I thought I would ask here :).
To Clarify:
I am counting the number of parts I might have which could be any number of them each time the code is called. So every time the BuildXmlBlob() method is called, the resulting xml produced will have a unique element in there denoting the 'partNumber'.
So if the counter is currently on 7, that means we are processing 7th part so far!! That means XML returned from BuildXmlBlob() will have the counter value embedded in there somewhere. That's why I need it somehow to be passed and incremented every time the BuildXmlBlob() is called per run through.
If you want to keep this purely in LINQ and you need to maintain a running count for use within your queries, the cleanest way to do so would be to make use of the Select() overloads that includes the index in the query to get the current index.
In this case, it would be cleaner to do a query which collects the inputs first, then use the overload to do the projection.
var inputs =
from recording in listOfRecordings
from part in recording.Parts
select new
{
Id = GetId("mods", recording.CKey + part.UniqueId),
Part = part,
};
result.AddRange(inputs.Select((x, i) => BuildXmlBlob(x.Id, x.Part, i)));
Then you wouldn't need to use the out/ref parameter.
XElement BuildXmlBlob(string id, Part part, int counter)
{
// implementation
}
Below is what I managed to figure out on my own:.
result.AddRange(listOfRecordings.SelectMany(rec => rec.Parts, (rec, par) => new {rec, par})
.Select(#t => new
{
#t,
Id = GetStructMapItemId("mods", #t.rec.CKey + #t.par.UniqueId)
})
.Select((#t, i) => BuildPartsDmdSec(#t.Id, #t.#t.par, i)));
I used resharper to convert it into a method chain which constructed the basics for what I needed and then i simply tacked on the select statement right at the end.

Resources