I need to perform a query to get the oldest document in a sub collection.
I want to perform this query with few reads as possible.
DB description:
Based on Firebase.
Collection of devices. Each device holds a collection of call-backs. For a specific device I need to fetch the oldest call-back (call-backs has timestamp).
I think I know how to perform this query using the device Unique ID, But I want to do it by filtering by some field of the device, this field is also unique.
I was able to do it by querying the device with all of his call-backs but this will charge me for more reads then actually needed.
Query that works using ID:
admin
.firestore()
.collection("devices/{device_id}/callbacks")
.{order_by_timestamp}
.limit(1)
.get()
.then((data) => {
let callbacks = [];
data.forEach((doc) => {
callbacks.push(doc.data());
});
return res.json(callbacks);
})
.catch((err) => console.error(err));
If that field in devices collection is unique then you can fetch ID of that device first and then proceed with your existing logic as shown below:
async function getOldestCallback(thatFieldValue) {
const device = await admin.firestore().collection("devices").where("thatField", "==", thatFieldValue).get()
if (device.empty) return false;
const deviceId = device[0]["id"];
// existing function
}
This should incur 2 reads (1 for device document and 1 for oldest callback if it exist).
Additionally, since you are limiting number of docs to be returned to 1 then you can use [0] instead of using a forEach loop.
const callbacks = [ data[0].data() ]
Related
So here I Have multiple sub-collections(subjects) in different doc's(grades) and I want to get all the sub-collections(subjects) documents(questions) at once I tried to get them by using Collection group queries the only problem which I am facing in my code sometime it returning all the doc's(questions) but sometimes not what is the issue
this is what i have tried
const getAllQuestions = (request,response)=>{
const subjects = ['Maths','English']
const questionsArray = []
subjects.forEach((subject,index)=>{
db.collectionGroup(subject)
.get()
.then((querySnapshot)=>{
querySnapshot.forEach((doc) => {
questionsArray.push({...doc.data(),id:doc.id})
})
if(index==subjects.length-1){
response.status(200).json({
status:200,
data:questionsArray,
length:questionsArray.length
})
}
})
})
}
If you don't want to get the subcollections from all grades, but only from one of them, you should not use a collection group query but instead specify the entire path to the collection you want to query/read:
db.collection('quizQuesDb/Grade 5/'+subject)
.get()
If you want to perform a query across all collections of a certain name under a specific path, see: CollectionGroupQuery but limit search to subcollections under a particular document
I using firebase admin and use nodejs get data by phone number from firebase. When i get success , i want get only document one time and use it every time. It possible ?
Picture :
First i get document by phone look like.
Example:
const phone = await admin.firestore().collection('users').where('phone_number', '==', phone).get()
After that , i want using document phone every time in my code look like :
await handleLogic(phone)
Then
async function handleLogic(phone) {
//inside here i need call await admin.firestore().collection('users').where('phone_number', '==', somePhone).get() or re use phone in parameter ?
phone.ref.collection("subcollection").get()
.then(() => {
let data = {
created_at: timeNowFirebase(),
};
phone.ref.collection(subcollection).doc(someId)
.set(data);
}
I have question : Inside function handleLogic(phone), i need re call admin.firestore().collection('users').where('phone_number', '==', phone).get() or only use parameter phone and used phone.ref.collection(subcollection).doc(someId).set(data); . It will set subcollection into document my phone correct ?
Yes you can store the document ID (or even the DocumentReference as you are currently) in memory/cache instead of querying the document with phone number every time. The doc ID never changes and this seems to be a good way to prevent additional requests to the database.
// storing in memory for example
const phoneToUserId = {};
async function handleLogic(phone) {
if (!phoneToUserId[phone]) {
// run a query to get userID from phone number
const user = await admin.firestore().collection('users').where('phone_number', '==', somePhone).get()
phoneToUserId[phone] = user.docs[0].id
}
// get a reference to sub-collection
const subCol = admin.firestore().collection(`users/${phoneToUserId[phone]}/subcollection`)
// query data
}
However do note that you'll have to update that object whenever user updates their phone number or delete their document.
I have 2 models - Driver and User. Both of them rate each other, so while creating the API, how can I check whether a certain userId exists in db, and if it exists, I want to add to my driver's rating array such that a new object is added like this.
{userId:xyz/*Already Checked in the Db that it exists*/,rating:4}
It sounds like you need to use two requests.
const user = await Users.findOne({_id: userId});
if (user){
const driver = await Drivers.findOne({_id: driverId});
driver.ratings.push({userId, rating: 4})
await driver.save();
}
I have been working with the google cloud library, and I can successfully save data in DataStore, specifically from my particle electron device (Used their tutorial here https://docs.particle.io/tutorials/integrations/google-cloud-platform/)
The problem I am now having is retrieving the data again.
I am using this code, but it is not returning anything
function getData(){
var data = [];
const query = datastore.createQuery('ParticleEvent').order('created');
datastore.runQuery(query).then(results => {
const event = results[0];
console.log(results);
event.forEach(data => data.push(data.data));
});
console.log(data)
}
But each time it is returning empty specifically returning this :
[ [], { moreResults: 'NO_MORE_RESULTS', endCursor: 'CgA=' } ]
, and I can't figure out why because I have multiple entities saved in this Datastore.
Thanks
In the tutorial.js from the repo mentioned in the tutorial I see the ParticleEvent entities are created using this data:
var obj = {
gc_pub_sub_id: message.id,
device_id: message.attributes.device_id,
event: message.attributes.event,
data: message.data,
published_at: message.attributes.published_at
}
This means the entities don't have a created property. I suspect that ordering the query by such property name is the reason for which the query doesn't return results. From Datastore Queries (emphasis mine):
The results include all entities that have at least one value for
every property named in the filters and sort orders, and whose
property values meet all the specified filter criteria.
I'd try ordering the query by published_at instead, that appears to be the property with a meaning closest to created.
I am running an iOS app where I display a list of users that are currently online.
I have an API endpoint where I return 10 (or N) users randomly, so that you can keep scrolling and always see new users. Therefore I want to make sure I dont return a user that I already returned before.
I cannot use a cursor or a normal pagination as the users have to be returned randomly.
I tried 2 things, but I am sure there is a better way:
At first what I did was sending in the parameters of the request the IDs of the user that were already seen.
ex:
But if the user keeps scrolling and has gone through 200 profiles then the list is long and it doesnt look clean.
Then, in the database, I tried adding a field to each users "online_profiles_already_sent" where i would store an array of the IDs that were already sent to the user (I am using MongoDB)
I can't figure out how to do it in a better/cleaner way
EDIT:
I found a way to do it with MySQL, using RAND(seed)
but I can't figure out if there is a way to do the same thing with Mongo
PHP MySQL pagination with random ordering
Thank you :)
I think the only way that you will be able to guarentee that users see unique users every time is to store the list of users that have already been seen. Even in the RAND example that you linked to, there is a possibility of intersection with a previous user list because RAND won't necessarily exclude previously returned users.
Random Sampling
If you do want to go with random sampling, consider Random record from MongoDB which suggests using an an Aggregation and the $sample operator. The implementation would look something like this:
const {
MongoClient
} = require("mongodb");
const
DB_NAME = "weather",
COLLECTION_NAME = "readings",
MONGO_DOMAIN = "localhost",
MONGO_PORT = "32768",
MONGO_URL = `mongodb://${MONGO_DOMAIN}:${MONGO_PORT}`;
(async function () {
const client = await MongoClient.connect(MONGO_URL),
db = await client.db(DB_NAME),
collection = await db.collection(COLLECTION_NAME);
const randomDocs = await collection
.aggregate([{
$sample: {
size: 5
}
}])
.map(doc => {
return {
id: doc._id,
temperature: doc.main.temp
}
});
randomDocs.forEach(doc => console.log(`ID: ${doc.id} | Temperature: ${doc.temperature}`));
client.close();
}());
Cache of Previous Users
If you go with maintaining a list of previously viewed users, you could write an implementation using the $nin filter and store the _id of previously viewed users.
Here is an example using a weather database that I have returning entries 5 at a time until all have been printed:
const {
MongoClient
} = require("mongodb");
const
DB_NAME = "weather",
COLLECTION_NAME = "readings",
MONGO_DOMAIN = "localhost",
MONGO_PORT = "32768",
MONGO_URL = `mongodb://${MONGO_DOMAIN}:${MONGO_PORT}`;
(async function () {
const client = await MongoClient.connect(MONGO_URL),
db = await client.db(DB_NAME),
collection = await db.collection(COLLECTION_NAME);
let previousEntries = [], // Track ids of things we have seen
empty = false;
while (!empty) {
const findFilter = {};
if (previousEntries.length) {
findFilter._id = {
$nin: previousEntries
}
}
// Get items 5 at a time
const docs = await collection
.find(findFilter, {
limit: 5,
projection: {
main: 1
}
})
.map(doc => {
return {
id: doc._id,
temperature: doc.main.temp
}
})
.toArray();
// Keep track of already seen items
previousEntries = previousEntries.concat(docs.map(doc => doc.id));
// Are we still getting items?
console.log(docs.length);
empty = !docs.length;
// Print out the docs
docs.forEach(doc => console.log(`ID: ${doc.id} | Temperature: ${doc.temperature}`));
}
client.close();
}());
I have encountered the same issue and can suggest an alternate solution.
TL;DR: Grab all Object ID of the collections on first landing, randomized using NodeJS and used it later on.
Disadvantage: slow first landing if have million of records
Advantage: subsequent execution is probably quicker than the other solution
Let's get to the detail explain :)
For better explain, I will make the following assumption
Assumption:
Assume programming language used NodeJS
Solution works for other programming language as well
Assume you have 4 total objects in yor collections
Assume pagination limit is 2
Steps:
On first execution:
Grab all Object Ids
Note: I do have considered performance, this execution takes spit seconds for 10,000 size collections. If you are solving a million record issue then maybe used some form of partition logic first / used the other solution listed
db.getCollection('my_collection').find({}, {_id:1}).map(function(item){ return item._id; });
OR
db.getCollection('my_collection').find({}, {_id:1}).map(function(item){ return item._id.valueOf(); });
Result:
ObjectId("FirstObjectID"),
ObjectId("SecondObjectID"),
ObjectId("ThirdObjectID"),
ObjectId("ForthObjectID"),
Randomized the array retrive using NodeJS
Result:
ObjectId("ThirdObjectID"),
ObjectId("SecondObjectID"),
ObjectId("ForthObjectID"),
ObjectId("FirstObjectID"),
Stored this randomized array:
If this is a Server side script that randomized pagination for each user, consider storing in Cookie / Session
I suggest Cookie (with timeout expired linked to browser close) for scaling purpose
On each retrieval:
Retrieve the stored array
Grab the pagination item, (e.g. first 2 items)
Find the objects for those item using find $in
.
db.getCollection('my_collection')
.find({"_id" : {"$in" : [ObjectId("ThirdObjectID"), ObjectId("SecondObjectID")]}});
Using NodeJS, sort the retrieved object based on the retrived pagination item
There you go! A randomized MongoDB query for pagination :)