Nodejs & Mongo pagination random order - node.js

I am running an iOS app where I display a list of users that are currently online.
I have an API endpoint where I return 10 (or N) users randomly, so that you can keep scrolling and always see new users. Therefore I want to make sure I dont return a user that I already returned before.
I cannot use a cursor or a normal pagination as the users have to be returned randomly.
I tried 2 things, but I am sure there is a better way:
At first what I did was sending in the parameters of the request the IDs of the user that were already seen.
ex:
But if the user keeps scrolling and has gone through 200 profiles then the list is long and it doesnt look clean.
Then, in the database, I tried adding a field to each users "online_profiles_already_sent" where i would store an array of the IDs that were already sent to the user (I am using MongoDB)
I can't figure out how to do it in a better/cleaner way
EDIT:
I found a way to do it with MySQL, using RAND(seed)
but I can't figure out if there is a way to do the same thing with Mongo
PHP MySQL pagination with random ordering
Thank you :)

I think the only way that you will be able to guarentee that users see unique users every time is to store the list of users that have already been seen. Even in the RAND example that you linked to, there is a possibility of intersection with a previous user list because RAND won't necessarily exclude previously returned users.
Random Sampling
If you do want to go with random sampling, consider Random record from MongoDB which suggests using an an Aggregation and the $sample operator. The implementation would look something like this:
const {
MongoClient
} = require("mongodb");
const
DB_NAME = "weather",
COLLECTION_NAME = "readings",
MONGO_DOMAIN = "localhost",
MONGO_PORT = "32768",
MONGO_URL = `mongodb://${MONGO_DOMAIN}:${MONGO_PORT}`;
(async function () {
const client = await MongoClient.connect(MONGO_URL),
db = await client.db(DB_NAME),
collection = await db.collection(COLLECTION_NAME);
const randomDocs = await collection
.aggregate([{
$sample: {
size: 5
}
}])
.map(doc => {
return {
id: doc._id,
temperature: doc.main.temp
}
});
randomDocs.forEach(doc => console.log(`ID: ${doc.id} | Temperature: ${doc.temperature}`));
client.close();
}());
Cache of Previous Users
If you go with maintaining a list of previously viewed users, you could write an implementation using the $nin filter and store the _id of previously viewed users.
Here is an example using a weather database that I have returning entries 5 at a time until all have been printed:
const {
MongoClient
} = require("mongodb");
const
DB_NAME = "weather",
COLLECTION_NAME = "readings",
MONGO_DOMAIN = "localhost",
MONGO_PORT = "32768",
MONGO_URL = `mongodb://${MONGO_DOMAIN}:${MONGO_PORT}`;
(async function () {
const client = await MongoClient.connect(MONGO_URL),
db = await client.db(DB_NAME),
collection = await db.collection(COLLECTION_NAME);
let previousEntries = [], // Track ids of things we have seen
empty = false;
while (!empty) {
const findFilter = {};
if (previousEntries.length) {
findFilter._id = {
$nin: previousEntries
}
}
// Get items 5 at a time
const docs = await collection
.find(findFilter, {
limit: 5,
projection: {
main: 1
}
})
.map(doc => {
return {
id: doc._id,
temperature: doc.main.temp
}
})
.toArray();
// Keep track of already seen items
previousEntries = previousEntries.concat(docs.map(doc => doc.id));
// Are we still getting items?
console.log(docs.length);
empty = !docs.length;
// Print out the docs
docs.forEach(doc => console.log(`ID: ${doc.id} | Temperature: ${doc.temperature}`));
}
client.close();
}());

I have encountered the same issue and can suggest an alternate solution.
TL;DR: Grab all Object ID of the collections on first landing, randomized using NodeJS and used it later on.
Disadvantage: slow first landing if have million of records
Advantage: subsequent execution is probably quicker than the other solution
Let's get to the detail explain :)
For better explain, I will make the following assumption
Assumption:
Assume programming language used NodeJS
Solution works for other programming language as well
Assume you have 4 total objects in yor collections
Assume pagination limit is 2
Steps:
On first execution:
Grab all Object Ids
Note: I do have considered performance, this execution takes spit seconds for 10,000 size collections. If you are solving a million record issue then maybe used some form of partition logic first / used the other solution listed
db.getCollection('my_collection').find({}, {_id:1}).map(function(item){ return item._id; });
OR
db.getCollection('my_collection').find({}, {_id:1}).map(function(item){ return item._id.valueOf(); });
Result:
ObjectId("FirstObjectID"),
ObjectId("SecondObjectID"),
ObjectId("ThirdObjectID"),
ObjectId("ForthObjectID"),
Randomized the array retrive using NodeJS
Result:
ObjectId("ThirdObjectID"),
ObjectId("SecondObjectID"),
ObjectId("ForthObjectID"),
ObjectId("FirstObjectID"),
Stored this randomized array:
If this is a Server side script that randomized pagination for each user, consider storing in Cookie / Session
I suggest Cookie (with timeout expired linked to browser close) for scaling purpose
On each retrieval:
Retrieve the stored array
Grab the pagination item, (e.g. first 2 items)
Find the objects for those item using find $in
.
db.getCollection('my_collection')
.find({"_id" : {"$in" : [ObjectId("ThirdObjectID"), ObjectId("SecondObjectID")]}});
Using NodeJS, sort the retrieved object based on the retrived pagination item
There you go! A randomized MongoDB query for pagination :)

Related

How to get document one time and using it every where

I using firebase admin and use nodejs get data by phone number from firebase. When i get success , i want get only document one time and use it every time. It possible ?
Picture :
First i get document by phone look like.
Example:
const phone = await admin.firestore().collection('users').where('phone_number', '==', phone).get()
After that , i want using document phone every time in my code look like :
await handleLogic(phone)
Then
async function handleLogic(phone) {
//inside here i need call await admin.firestore().collection('users').where('phone_number', '==', somePhone).get() or re use phone in parameter ?
phone.ref.collection("subcollection").get()
.then(() => {
let data = {
created_at: timeNowFirebase(),
};
phone.ref.collection(subcollection).doc(someId)
.set(data);
}
I have question : Inside function handleLogic(phone), i need re call admin.firestore().collection('users').where('phone_number', '==', phone).get() or only use parameter phone and used phone.ref.collection(subcollection).doc(someId).set(data); . It will set subcollection into document my phone correct ?
Yes you can store the document ID (or even the DocumentReference as you are currently) in memory/cache instead of querying the document with phone number every time. The doc ID never changes and this seems to be a good way to prevent additional requests to the database.
// storing in memory for example
const phoneToUserId = {};
async function handleLogic(phone) {
if (!phoneToUserId[phone]) {
// run a query to get userID from phone number
const user = await admin.firestore().collection('users').where('phone_number', '==', somePhone).get()
phoneToUserId[phone] = user.docs[0].id
}
// get a reference to sub-collection
const subCol = admin.firestore().collection(`users/${phoneToUserId[phone]}/subcollection`)
// query data
}
However do note that you'll have to update that object whenever user updates their phone number or delete their document.

Mongodb, how to get all array's each matching first element in DB?

So I have a scenario that requires to select ALL element's first return.
What I am currently practicing is using a loop
const array_list = await Person.find({});
const results = [];
for (const element in array_list) {
const temp_result = await Transactions.find({_id: array_list[element]._id}).sort({$natural: -1 }).limit(1);
results.push(temp_results);
}
This works, but I figure out this may slow down the whole process if the database became larger in term of scale, is there anyway to faster this result?
You can simplify this to use a single db-query for finding the transaction documents by specifying the $in operator where you pass in the the _ids of the person array_list:
const elementIds = array_list.map(elem => elem._id);
await Transactions.find({_id: { $in: elementIds }}).sort({$natural: -1 }).limit(1);
Above solution still requires two db-reads - if you want to do it in a single query, you can do an aggregation-query where you $lookup the transaction documents for each person (see examples in the doc-page).

Mongodb How do I make this sorting of grabbing an individual players position function faster?

class User {
constructor(doc) {
this.username = doc.username;
this.kills = doc.kills;
this.deaths = doc.deaths;
}
}
const res = await Calls.getAllUsers();
const users = res.map((doc) => new User(doc));
const sorted = users.sort((a, b) => b.kills - a.kills);
const whereIam =
sorted.indexOf(users.find((u) => u.username === latest_user)) + 1;
Hi, everyone I am trying to sort out a players position ( kills)
They would do /stats and they would get a position based on the highest kills
This takes more than five minutes to determind a players position with 26,000 documents and 2,000 documents being made everyday, what are the steps to make this faster or to change?
You look to be getting all users and then doing the sort and search in memory. As your data set gets larger that will get much more memory intensive.
So first suggestion would be to get Mongo to be doing that for you in your query.
Adding to that, since Mongo will be doing the query, you'll want indexes in Mongo on the username and kills fields.
If there's a reason you want all the users in memory though, then you could consider a hash table to help with your user lookups, but there's no way around sorting time. Make sure your sort is using a reasonable algorithm (that's a whole subject on it's own, but Quicksort is the one that has the best average performance).
In this specfic scenario, you can follow these steps:
create index on { username: 1 }
create index on { kills: -1 }
query with find({ username: latest_user } to get this user's kills as latest_user_kills
query with countDocuments({ kills: { $gt: latest_user_kills } }) to get the count of all users whose kills is higher than this latest_user
the resulting count from step 4 is the position of this individual player

Is there a native way to get MongoDB results as object instead of array?

Checked couple of old similar questions but no luck in finding the right answers.
I have a mongo query that uses await/async to fetch results from the database, So the code looks like this :
(async () => {
//get connection object
const db = await getDatabaseConnection("admindb");
//get user data object
const configResponse = await db.collection("config").find({}).toArray();
//prints results
console.log("Successfully Fetched Configuration", configResponse);
})();
The above code works fine, just that it returns me an array of elements since I have used the .toArray() method.
Sample Result:
[
{
_id: "6028d30db7ea89f74df013d9",
tokenize: false,
configurations: { theme: {} }
}
]
Is there a NATIVE WAY (not looking forEach responses) to get result as an object since I will always have only one document returned. I have multiple queries that returns only one document, Hence accessing using the 0th index everytime did not seem the right way.
I am thinking if the findOne() will help you.
Check this out docs
await db.collection("config").findOne({});
Here are some examples

List the user based on a number of clicks Firebase

I'm using firebase for a listing user and I want to retrieve the user based on a number of clicks. I saved the number of clicks into the child key named click. Here you can look at the document structure. I am using the scroll load to list the user. 12 records should be in go as we scroll the record should from 12 to 24.
In short, The user has the most number of click should be on top with limit records.
Here is the little piece of code I am trying to use. Can Firebase do this?
return firebase
.database()
.ref("reachers_by_user_id")
.orderByChild("click")
.startAt(null)
.limitToFirst(12)
.once("value")
.then(snapshot => {
let reacherdata = [];
snapshot.forEach(snap => {
reacherdata.push(snap.val());
});
return reacherdata;
});
Thank you
If you want to retrieve the 12 users with the most clicks, you'd use this query:
firebase.database()
.ref("reachers_by_user_id")
.orderByChild("click")
.limitToLast(12)
.once("value")
.then(snapshot => {
snapshot.forEach(snap => {
console.log(snap.key+": "+snap.val().click);
});
});
If you run this code you'll see it prints (up to) the 12 highest click counts, in ascending order. This is because Firebase Realtime Database queries always sort nodes in ascending order. There is no way to get the result in descending order. For more on this, see:
firebase -> date order reverse
Sorting in descending order in Firebase database
Firebase Data Desc Sorting in Android
In your case, you can easily reverse the (up to) 12 results in the callback with:
.then(snapshot => {
let results = [];
snapshot.forEach(snap => {
results.push(snap);
});
results = results.reverse();
});
To get to the next twelve, or actually the previous 12, you will need to use endAt passing in:
The click value of the lowest value you already got.
The key of that snapshot, which is used to disambiguate in case there are multiple child nodes with the same click value.
So you'd get the top 12 scores with:
var lowestClickValue, lowestKey;
firebase.database()
.ref("reachers_by_user_id")
.orderByChild("click")
.limitToLast(12)
.once("value")
.then(snapshot => {
var isFirst = true;
snapshot.forEach(snap => {
console.log(snap.key+": "+snap.val().click);
if (isFirst) {
lowestKey = snap.key
lowestClickValue = snap.val().click
isFirst = false;
}
});
});
After running this code, the lowestClickValue and lowestKey variables hold the click and key for the lowest node on the current page. You then get the next page with:
firebase.database()
.ref("reachers_by_user_id")
.orderByChild("click")
.endAt(lowestClickValue, lowestKey)
.limitToLast(13)
Here we request 13 nodes, since there will be one node we already have. You'll need to exclude that node in your client-side code.
Note that pagination in Firebase is non-trivial, but quite consistent once you understand that the API is not based on offsets. I highly recommend studying some of the other questions on the pagination if you are still having problems.

Resources