Google Cloud Functions Firestore Limitations - node.js

I have written a function which gets a Querysnapshot within all changed Documents of the past 24 hours in Firestore. I loop through this Querysnapshot to get the relevant informations. The informations out of this docs I want to save into maps which are unique for every user. Every user generates in average 10 documents a day. So every map gets written 10 times in average. Now I'm wondering if the whole thing is scalable or will hit the 500 writes per transaction limit given in Firebase as more users will use the app.
The limitation im speaking about is documented in Google documentation.
Furthermore Im pretty sure that my code is really slow. So im thankful for every optimization.
exports.setAnalyseData = functions.pubsub
.schedule('every 24 hours')
.onRun(async (context) => {
const date = new Date().toISOString();
const convertedDate = date.split('T');
//Get documents (that could be way more than 500)
const querySnapshot = await admin.firestore().collectionGroup('exercises').where('lastModified', '>=', `${convertedDate}`).get();
//iterate through documents
querySnapshot.forEach(async (doc) => {
//some calculations
//get document to store the calculated data
const oldRefPath = doc.ref.path.split('/trainings/');
const newRefPath = `${oldRefPath[0]}/exercises/`;
const document = await getDocumentSnapshotToSave(newRefPath, doc.data().exercise);
document.forEach(async (doc) => {
//check if value exists
const getDocument = await admin.firestore().doc(`${doc.ref.path}`).collection('AnalyseData').doc(`${year}`).get();
if (getDocument && getDocument.exists) {
await document.update({
//map filled with data which gets added to the exisiting map
})
} else {
await document.set({
//set document if it is not existing
}, {
merge: true
});
await document.update({
//update document after set
})
}
})
})
})

The code you have in your question does not use a transaction on Firestore, so is not tied to the limit you quote/link.
I'd still recommend putting a limit on your query through, and processing the documents in reasonable batches (a couple of hundred being reasonable) so that you don't put an unpredictable memory load on your code.

Related

Are "get" functions in Google cloud functions for firebase reading each document everytime?

I observed a huge amount of read on my firebase console and I was wondering if this might come from my "referral function".
This function works perfectly fine but I was wondering whether or not this function could end up with a crazy load of read in case of app scaling.
My question: does this function imply that every time a user comes in, it will account for a number of read equivalent to the number of users in my collection ?
Thus, as this function is an onUpdate, will it redo the job every time a document is updated ?
I would not mind some resources on the topic because I found it unclear on Firebase's website.
I hope my questions are clear!
Thank you very much!
export const onReferralInfoUpdate = functions.
firestore.document('users/{userUid}')
.onUpdate(async (change, context) => {
const before = change.before.data();
const after = change.after.data();
const currentUserUid = after["uid"];
if (before.godfather_code == after.godfather_code){
console.log('Text did not change')
return null
}
const godfatherUserSnapshot = await db.collection('users').where("referral_code", "==", after.godfather_code).get();
const godfather = godfatherUserSnapshot.docs[0].data();
const godfatherUid = godfather["uid"];
const userRef = db.collection('users').doc(after.uid);
const godfather_code = after.godfather_code
await userRef.update({godfather_code})
console.log(`the text before was >> ${before.godfather_code} << and after is ${after.godfather_code}` )
let batch = db.batch();
const updateGodfather = db.collection('users').doc(godfatherUid);
batch.update(updateGodfather, {
reward: admin.firestore.FieldValue.increment(100),
godChildUid: admin.firestore.FieldValue.arrayUnion(currentUserUid),
});
return batch.commit();
});
Yes, the where("referral_code", "==", after.godfather_code).get() will fetch all the documents matching the query every time onUpdate() function triggers and you'll be charged N reads (N = number of matched documents). The Admin SDK doesn't have any caching like Client SDKs.
Does this function imply that every time a user comes in, it will account for a number of read equivalent to the number of users in my collection ?
Not numbers of documents in the users collection, only the documents matching your query as mentioned.

Get and increment in Firebase Realtime

I'm making a Firebase function, that is supposed to get the value of a field in the Realtime Database, write the value in a Firestore Document and increment the original field. The problem is when the function gets called very frequently e.g. 500 times a second, it gets and writes the same value in a lot of documents, because many executions will get the same value before it gets incremented. Is there any way to get the value of a Realtime DB field and increment it at the same time or somehow prevent this issue?
Thank you in advance.
My code:
const { getFirestore } = require('firebase-admin/firestore');
const { getDatabase, ServerValue } = require('firebase-admin/database');
const rb = getDatabase();
const db = getFirestore();
exports.increment = functions.https.onCall(async (data, context) => {
rb.ref('count').get().then((snapshot)=>{
let value = snapshot.val();
db.collection("documents").doc(value.toString()).set({count:value});
rb.ref("count").set(ServerValue.increment(1))
})
});
Since you're using an auto-scaling infrastructure with Cloud Functions, it will spin up new instances if there are a lot of requests coming in. If you don't want to do that, it might be worth setting a maximum number of instances on your Cloud Function.

Firebase Firestore not returning documents

I was attempting to fetch all documents from a collection in a Node.js environment. The documentation advises the following:
import * as admin from "firebase-admin";
const db = admin.firestore();
const citiesRef = db.collection('cities');
const snapshot = await citiesRef.get();
console.log(snapshot.size);
snapshot.forEach(doc => {
console.log(doc.id, '=>', doc.data());
});
I have 20 documents in the 'cities' collection. However, the logging statement for the snapshot size comes back as 0.
Why is that?
Edit: I can write to the Firestore without issue. I can also get details of a single document, for example:
const city = citiesRef.doc("city-name").get();
console.log(city.id);
will log city-name to the console.
Ensure that Firebase has been initialized and verify the collection name matches your database exactly, hidden spaces and letter case can break the link to Firestore. One way to test this is to create a new document within the collection to validate the path.
db.collection('cities').doc("TEST").set({test:"value"}).catch(err => console.log(err));
This should result in a document in the correct path, and you can also catch it to see if there are any issues with Security Rules.
Update
To list all documents in a collection, you can do this with the admin sdk through a server environment such as the Cloud Functions using the listDocuments() method but this does not reduce the number of Reads.
const documentReferences = await admin.firestore()
.collection('someCollection')
.listDocuments()
const documentIds = documentReferences.map(it => it.id)
To reduce reads, you will want to aggregate the data in the parent document or in a dedicated collection, this would double the writes for any updates but crush read count to a minimal amount.

How to update a quantity in another document when creating a new document in the firebase firestore collection?

When I create a new document in the note collection, I want to update the quantity in the info document. What am I doing wrong?
exports.addNote = functions.region('europe-west1').firestore
.collection('users/{userId}/notes').onCreate((snap,context) => {
const uid = admin.user.uid.toString();
var t;
db.collection('users').doc('{userId}').collection('info').doc('info').get((querySnapshot) => {
querySnapshot.forEach((doc) => {
t = doc.get("countMutable").toString();
});
});
let data = {
countMutable: t+1;
};
db.collection("users").doc(uid).collection("info").doc("info").update({countMutable: data.get("countMutable")});
});
You have... a lot going on here. A few problems:
You can't trigger firestore functions on collections, you have to supply a document.
It isn't clear you're being consistent about how to treat the user id.
You aren't using promises properly (you need to chain them, and return them out of the function if you want them to execute properly).
I'm not clear about the relationship between the userId context parameter and the uid you are getting from the auth object. As far as I can tell, admin.user isn't actually part of the Admin SDK.
You risk multiple function calls doing an increment at the same time giving inconsistent results, since you aren't using a transaction or the increment operation. (Learn More Here)
The document won't be created if it doesn't already exist. Maybe this is ok?
In short, this all means you can do this a lot more simply.
This should do you though. I'm assuming that the uid you actually want is actually the one on the document that is triggering the update. If not, adjust as necessary.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
const db = admin.firestore();
exports.addNote = functions.firestore.document('users/{userId}/notes/{noteId}').onCreate((snap,context) => {
const uid = context.params.userId;
return db.collection("users").doc(uid).collection("info").doc("info").set({
countMutable: admin.firestore.FieldValue.increment(1)
}, { merge: true });
});
If you don't want to create the info document if it doesn't exist, and instead you want to get an error, you can use update instead of set:
return db.collection("users").doc(uid).collection("info").doc("info").update({
countMutable: admin.firestore.FieldValue.increment(1)
});

Datastore Query filters accumulating on subsequent calls

I am making a RESTFul API. I have the following endpoint
/users/getByState/stateId
I have several users in the database. 3 users are from say, Texas, 1 from New York.
When I call the endpoint the first time, either by
/user/getByState/tx
or by
/user/getByState/ny
I get a result, but calling immediately the endpoint using the other state id, returns an empty array and the message {"moreResults": "NO_MORE_RESULTS"}
Sending the query to the console log shows me that the first time the query only has one filter.. lets say
FILTER:{stateID:'tx'}
But the second time, instead of changing the filter option to 'ny' it instead adds another filter so now in the console log I see
FILTER:{stateId:'tx'}
FILTER:{stateId:'ny'}
Which will obviously always return an empty array because it will never find stateId='tx' AND stateId='ny'
I don't understand why the cloud datastore client is adding a filter to a CONST!!
If I call the endpoint 7 times, I see 7 filters. Only after I redeploy the filters "clear"
How do I clear the filters before running the query again? Have searched about this in the cloud datastore documentation but there is no information about filters concatenating after each call
Am I missing something? This is my code:
const Datastore = require('#google-cloud/datastore');
const datastore=Datastore();
const query=datastore.createQuery("user");
exports.get_user_by_state =(req,res,next) => {
const pageCursor = req.query.cursor;
const userState=req.params.stateId;
const selectQuery = query
.filter('stateId',userState)
console.log(selectQuery);
selectQuery.run({cache:false})
.then((results) => {
res.json(results);
})
.catch(err => res.status(500).json(err));
}
You need a new query object for each and every query. So, the query object should be created inside the block.
const Datastore = require('#google-cloud/datastore');
const datastore = Datastore();
exports.get_user_by_state =(req,res,next) => {
const pageCursor = req.query.cursor;
const query = datastore.createQuery("user");
const userState = req.params.stateId;
const selectQuery = query
.filter('stateId',userState)
console.log(selectQuery);
selectQuery.run({cache:false})
.then((results) => {
res.json(results);
})
.catch(err => res.status(500).json(err));
}

Resources