Firebase Cloud Function Increment Counter - node.js

Per Firebase cloud functions docs, "Events are delivered at least once, but a single event may result in multiple function invocations. Avoid depending on exactly-once mechanics, and write idempotent functions."
When looking at a firestore cloud function doc example below of a restaurant rating, they are using an increment counter to calculate the total number of ratings. What are some of the best ways to maintain the accuracy of this counter by using an idempotent function?
Is it reasonable to store the context.eventId in a subcollection document field and only execute the function if the new context.eventId is different?
function addRating(restaurantRef, rating) {
// Create a reference for a new rating, for use inside the transaction
var ratingRef = restaurantRef.collection('ratings').doc();
// In a transaction, add the new rating and update the aggregate totals
return db.runTransaction((transaction) => {
return transaction.get(restaurantRef).then((res) => {
if (!res.exists) {
throw "Document does not exist!";
}
// Compute new number of ratings
var newNumRatings = res.data().numRatings + 1;
// Compute new average rating
var oldRatingTotal = res.data().avgRating * res.data().numRatings;
var newAvgRating = (oldRatingTotal + rating) / newNumRatings;
// Commit to Firestore
transaction.update(restaurantRef, {
numRatings: newNumRatings,
avgRating: newAvgRating
});
transaction.set(ratingRef, { rating: rating });
});
});
}

Is it reasonable to store the context.eventId in a subcollection
document field and only execute the function if the new
context.eventId is different?
Yes, for your use case, using the Cloud Function eventId is the best solution to make you Cloud Function idempotent. I guess you have already watched this Firebase video.
In the Firebase doc from which you have taken the code in your question, you will find at the bottom, the similar code for a Cloud Function. I've adapted this code as follows, in order to check if a doc with ID = eventId exists in a dedicated ratingUpdateIds subcollection:
exports.aggregateRatings = functions.firestore
.document('restaurants/{restId}/ratings/{ratingId}')
.onWrite(async (change, context) => {
try {
// Get value of the newly added rating
const ratingVal = change.after.data().rating;
const ratingUpdateId = context.eventId;
// Get a reference to the restaurant
const restRef = db.collection('restaurants').doc(context.params.restId);
// Get a reference to the ratingUpdateId doc
const ratingUpdateIdRef = restRef.collection("ratingUpdateIds").doc(ratingUpdateId);
// Update aggregations in a transaction
await db.runTransaction(async (transaction) => {
const ratingUpdateIdDoc = await transaction.get(ratingUpdateIdRef);
if (ratingUpdateIdDoc.exists) {
// The CF is retried
throw "The CF is being retried";
}
const restDoc = await transaction.get(restRef);
// Compute new number of ratings
const newNumRatings = restDoc.data().numRatings + 1;
// Compute new average rating
const oldRatingTotal = restDoc.data().avgRating * restDoc.data().numRatings;
const newAvgRating = (oldRatingTotal + ratingVal) / newNumRatings;
// Update restaurant info and set ratingUpdateIdDoc
transaction
.update(restRef, {
avgRating: newAvgRating,
numRatings: newNumRatings
})
.set(ratingUpdateIdRef, { ratingUpdateId })
});
return null;
} catch (error) {
console.log(error);
return null;
}
});
PS: I made the assumption that the Cloud Function eventId can be used as a Firestore document ID. I didn't find any doc or info stating the opposite. In case using the eventId as an ID would be a problem, since you execute the transaction in a Cloud Function (and therefore use the Admin SDK), you could query the document based on a field value (where you store the eventId) instead of getting it through a Reference based on its ID.

Related

Trigger Firebase Cloud Function on inserting to Cloud Firestore document for the first time with given uid

I want to Trigger a Cloud Funtion on insert to Cloud Firestore document for the first time with given uid
The below code triggers on insert of newUserId
functions.firestore
.document(`teamProfile/{teamId}/teamMemberList/{newUserId}`)
.onCreate()
Requirement --
Document : Post
Fields : id, title, uid
id is the document id.
Here, the Cloud function should trigger if it's the first post by uid(user).
Edited response (following clarification in comments):
Data structure:
users/{userId}/posts/
◙ post18sfge89s
- title: My first post!
- t: 1572967518
◙ post2789sdjnf
- title: I like posting
- t: 1572967560
posts/
◙ post18sfge89s
- title: My first post!
- uid: someUid1
- comments/
◙ comment237492384
...
◙ comment234234235
...
◙ post2789sdjnf
- title: I like posting
- uid: someUid1
Cloud Function Code:
With the above structure, you will need two Cloud Functions to manage it – one for handling each new post (to copy information to the author's post list) and one for checking if it's the author's first post.
// Function: New post handler
exports.newPost = functions.firestore
.document('posts/{postId}')
.onCreate((postDocSnap, context) => {
// get relevant information
const postId = postDocSnap.id; // provided automatically
const postTitle = postDocSnap.get('title');
const userId = postDocSnap.get('uid');
const postedAt = postDocSnap.createTime; // provided automatically
// add to user's post list/index
return firestore.doc(`users/${userId}/posts/${postId}`)
.set({
title: postTitle,
t: postedAt
});
});
// Function: New post by user handler
exports.newPostByUser = functions.firestore
.document('users/{userId}/posts/{postId}')
.onCreate((postDocSnap, context) => {
// get references
const userPostsColRef = postDocSnap.ref.parent;
const userDocRef = userPostsCol.parent;
// get snapshot of user data
return userDocRef.get()
.then((userDocSnap) => {
// check if "First Post Event" has already taken place
if (userDocSnap.get('madeFirstPostEvent') != true) {
return getCollectionSize(userPostsColRef).then((length) => {
if (length == 1) {
return handleFirstPostByUser(userDocSnap, postDocSnap);
} else {
return; // could return "false" here
}
});
}
});
});
// Pseudo-function: First post by user handler
function handleFirstPostByUser(userDocSnap, postDocSnap) {
return new Promise(() => {
const postId = postDocSnap.id;
const postTitle = postDocSnap.get('title');
const userId = userDocSnap.id;
// do something
// mark event handled
return userDocSnap.ref.update({madeFirstPostEvent: true});
});
}
// returns a promise containing the length of the given collection
// note: doesn't filter out missing (deleted) documents
function getCollectionSize(colRef) {
return colRef.listDocuments()
.then(docRefArray => {
return docRefArray.length;
});
}
Original response (for posts that are private to each team):
Assumptions:
Checking for a user's first post in a specific team, not platform wide.
Unknown data structure – I have used what I think will work well with your existing structure.
Data Structure:
The teamContent/ collection is structured so that it can contain subcollections for different items such as posts, attachments, pictures, etc.
teamProfile/{teamId}/teamMemberList/{userId}/posts/
◙ post18sfge89s
- title: My first post!
- t: 1572967518
◙ post2789sdjnf
- title: I like posting
- t: 1572967560
teamContent/{teamId}/posts/
◙ post18sfge89s
- title: My first post!
- author: someUid1
- comments/
◙ comment237492384
...
◙ comment234234235
...
◙ post2789sdjnf
- title: I like posting
- author: someUid1
Cloud Function Code:
With the above structure, you will need two Cloud Functions to manage it – one for handling each new post (to copy information to the author's post list) and one for checking if it's the author's first post in that particular team.
// Function: New team post handler
exports.newPostInTeam = functions.firestore
.document('teamContent/{teamId}/posts/{postId}')
.onCreate((postDocSnap, context) => {
// get relevant information
const postId = postDocSnap.id; // provided automatically
const postTitle = postDocSnap.get('title');
const authorId = postDocSnap.get('author');
const postedAt = postDocSnap.createTime; // provided automatically
const teamId = context.params.teamId;
// add to user's post list/index
return firestore.doc(`teamProfile/${teamId}/teamMemberList/${authorId}/posts/${postId}`)
.set({
title: postTitle,
t: postedAt
});
});
// Function: New post by team member handler
exports.newPostByTeamMember = functions.firestore
.document('teamProfile/{teamId}/teamMemberList/{userId}/posts/{postId}')
.onCreate((postDocSnap, context) => {
// get references
const userPostsColRef = postDocSnap.ref.parent;
const userDocRef = userPostsCol.parent;
// get snapshot of user data
return userDocRef.get()
.then((userDocSnap) => {
// check if "First Post Event" has already taken place
if (userDocSnap.get('madeFirstPostEvent') != true) {
return getCollectionSize(userPostsColRef).then((length) => {
if (length == 1) {
return handleFirstPostInTeamEvent(userDocSnap, postDocSnap);
} else {
return; // could return "false" here
}
});
}
});
});
// Pseudo-function: First post by team member handler
function handleFirstPostInTeamEvent(userDocSnap, postDocSnap) {
return new Promise(() => {
const postId = postDocSnap.id;
const postTitle = postDocSnap.get('title');
const userId = userDocSnap.id;
// do something
// mark event handled
return userDocSnap.update({madeFirstPostEvent: true});
});
}
// returns a promise containing the length of the given collection
// note: doesn't filter out missing (deleted) documents
function getCollectionSize(colRef) {
return colRef.listDocuments()
.then(docRefArray => {
return docRefArray.length;
});
}
Notes:
Above code is not completely idempotent. If multiple posts from the same user are all uploaded at once, there is a possibility that the handleFirstPost* functions will be called multiple times.
Above code doesn't account for missing documents returned from listDocuments() in the getCollectionSize() function. This is not a concern in the above sample because the {userId}/posts collection doesn't have any subcollections. Be wary if you call it elsewhere though.
No error handling is included
Use of async/await syntax can make it cleaner
Above code was written without deploying it, bugs/typos may be present

Index messed up if I upload more than one file at once

I've got the following firebase function to run once a file is uploaded to firebase storage.
It basically gets its URL and saves a reference to it in firestore. I need to save them in a way so that I can query them randomly from my client. Indexes seem to be to best fit this requirement.
for the firestore reference I need the following things:
doc ids must go from 0 to n (n beeing the index of the last
document)
have a --stats-- doc keeping track of n (gets
incremented every time a document is uploaded)
To achieve this I've written the following node.js script:
const incrementIndex = admin.firestore.FieldValue.increment(1);
export const image_from_storage_to_firestore = functions.storage
.object()
.onFinalize(async object => {
const bucket = gcs.bucket(object.bucket);
const filePath = object.name;
const splittedPath = filePath!.split("/");
// se siamo nelle immagini
// path = emotions/$emotion/photos/$photographer/file.jpeg
if (splittedPath[0] === "emotions" && splittedPath[2] === "photos") {
const emotion = splittedPath[1];
const photographer = splittedPath[3];
const file = bucket.file(filePath!);
const indexRef = admin.firestore().collection("images")
.doc("emotions").collection(emotion).doc("--stats--");
const index = await indexRef.get().then((doc) => {
if (!doc.exists) {
return 0;
} else {
return doc.data()!.index;
}
});
if (index === 0) {
await admin.firestore().collection("images")
.doc("emotions")
.collection(emotion)
.doc("--stats--")
.set({index: 0});
}
console.log("(GOT INDEX): " + index);
let imageURL;
await file
.getSignedUrl({
action: "read",
expires: "03-09-2491"
})
.then(signedUrls => {
imageURL = signedUrls[0];
});
console.log("(GOT URL): " + imageURL);
var docRef = admin.firestore()
.collection("images")
.doc("emotions")
.collection(emotion)
.doc(String(index));
console.log("uploading...");
await indexRef.update({index: incrementIndex});
await docRef.set({ imageURL: imageURL, photographer: photographer });
console.log("finished");
return true;
}
return false;
});
Getting to the problem:
It works perfectly if I upload the files one by one.
It messes up the index if I upload more than one file at once, because two concurrent uploads will read the same index value from --stats-- and one will overwrite the other.
How would you solve this problem? would you use another approach instead of the indexed one?
You should use a Transaction in which you:
read the value of the index (from "--stats--" document),
write the new index and
write the value of the imageURL in the "emotion" doc.
See also the reference docs about transactions.
This way, if the index value is changed in the "--stats--" document while the Transaction is being executed, the Cloud Function can catch the Transaction failure and generates an error which finishes it.
In parallel, you will need to enable retries for this background Cloud Function, in order it is retried if the Transaction failed in a previous run.
See this documentation item https://firebase.google.com/docs/functions/retries, including the video from Doug Stevenson which is embedded in the doc.

Trigger a function when a new document is created is not working

I have tried an increment counter example (https://github.com/firebase/functions-samples/tree/master/child-count) with Cloud Functions which is referencing Realtime Database, but not Firebase Firestore.
I am following firestore document and tried some changes. Still facing issue and not able to run this code for firestore new document. I am writing cloud function first time, and I'm not sure I've written this right.
exports.countlikechange = functions.database.ref('/posts/{postid}/likes/{likeid}').onWrite(
async (change) => {
const collectionRef = change.after.ref.parent;
const countRef = collectionRef.parent.child('likes_count');
let increment;
if (change.after.exists() && !change.before.exists()) {
increment = 1;
} else if (!change.after.exists() && change.before.exists()) {
increment = -1;
} else {
return null;
}
// Return the promise from countRef.transaction() so our function
// waits for this async event to complete before it exits.
await countRef.transaction((current) => {
return (current || 0) + increment;
});
console.log('Counter updated.');
return null;
});
I want to update count in parent document.
You should review the documentation for Firestore triggers. What you're writing is a Realtime Database trigger, because the function declaration is functions.database. You'll want to use functions.firestore instead. You're also using Realtime Database APIs instead of Firestore APIs. Your final solution that uses Firestore will look almost completely different than what you have now.

Firestore cloud function asynchronous execution with promise

I have orders collection and products collection in my application. The user can have multiple products in their single order. What I want to do is calculating the amount of each product reading through products collection and then perform the further action. Below is what I got as of now.
exports.myfunc = functions.firestore.document('collection/{collid}')
.onCreate(event => {
let data = event.data.data();
const products = data.products;
const prices = [];
_.each(products, (data1, index) => {
const weight = data1.weight;
const isLess = data1.isLess;
firebaseAdmin.firestore().collection('collection').doc(data1.productId).onSnapshot(data2 => {
let amount = weight === '1/2' ? data2.data().price1 : data2.data().price1 * weight;
amount += isLess ? 50 : 0;
prices.push(amount);
});
});
//Do some task after _.each with new total
});
But am not able to achieve synchronous task here, so that I can store actual amount for the product against its order and calculate total to store in document.
Could anyone please tell me how I achieve the above-said scenarios? How I can work along with promise and then callback?
You can map the products array to promises, like this:
var productPromises = products.map(product => {
return new Promise((resolve, reject) => {
firebaseOperation()...onSnapshot(resolve)
})
})
Promise.all(productPromises).then(results => {
// process all results at once
})
First, don't use onSnapshot() with Cloud Functions. That attaches a listener that stay listening indefinitely, until you remove it. That's not what you want at all, because functions can't execute indefinitely.
Instead, use get(), which returns a promise when the fetch is complete.
Also, you could consider accumulating all the documents you want to access into an array and use getAll() (with the spread operator on the array) to fetch them all.

Google Cloud Datastore, how to query for more results

Straight and simple, I have the following function, using Google Cloud Datastore Node.js API:
fetchAll(query, result=[], queryCursor=null) {
this.debug(`datastoreService.fetchAll, queryCursor=${queryCursor}`);
if (queryCursor !== null) {
query.start(queryCursor);
}
return this.datastore.runQuery(query)
.then( (results) => {
result=result.concat(results[0]);
if (results[1].moreResults === _datastore.NO_MORE_RESULTS) {
return result;
} else {
this.debug(`results[1] = `, results[1]);
this.debug(`fetch next with queryCursor=${results[1].endCursor}`);
return this.fetchAll(query, result, results[1].endCursor);
}
});
}
The Datastore API object is in the variable this.datastore;
The goal of this function is to fetch all results for a given query, notwithstanding any limits on the number of items returned per single runQuery call.
I have not yet found out about any definite hard limits imposed by the Datastore API on this, and the documentation seems somewhat opaque on this point, but I only noticed that I always get
results[1] = { moreResults: 'MORE_RESULTS_AFTER_LIMIT' },
indicating that there are still more results to be fetched, and the results[1].endCursor remains stuck on constant value that is passed on again on each iteration.
So, given some simple query that I plug into this function, I just go on running the query iteratively, setting the query start cursor (by doing query.start(queryCursor);) to the endCursor obtained in the result of the previous query. And my hope is, obviously, to obtain the next bunch of results on each successive query in this iteration. But I always get the same value for results[1].endCursor. My question is: Why?
Conceptually, I cannot see a difference to this example given in the Google Documentation:
// By default, google-cloud-node will automatically paginate through all of
// the results that match a query. However, this sample implements manual
// pagination using limits and cursor tokens.
function runPageQuery (pageCursor) {
let query = datastore.createQuery('Task')
.limit(pageSize);
if (pageCursor) {
query = query.start(pageCursor);
}
return datastore.runQuery(query)
.then((results) => {
const entities = results[0];
const info = results[1];
if (info.moreResults !== Datastore.NO_MORE_RESULTS) {
// If there are more results to retrieve, the end cursor is
// automatically set on `info`. To get this value directly, access
// the `endCursor` property.
return runPageQuery(info.endCursor)
.then((results) => {
// Concatenate entities
results[0] = entities.concat(results[0]);
return results;
});
}
return [entities, info];
});
}
(except for the fact, that I don't specify a limit on the size of the query result by myself, which I have also tried, by setting it to 1000, which does not change anything.)
Why does my code run into this infinite loop, stuck on each step at the same "endCursor"? And how do I correct this?
Also, what is the hard limit on the number of results obtained per call of datastore.runQuery()? I have not found this information in the Google Datastore documentation thus far.
Thanks.
Looking at the API documentation for the Node.js client library for Datastore there is a section on that page titled "Paginating Records" that may help you. Here's a direct copy of the code snippet from the section:
var express = require('express');
var app = express();
var NUM_RESULTS_PER_PAGE = 15;
app.get('/contacts', function(req, res) {
var query = datastore.createQuery('Contacts')
.limit(NUM_RESULTS_PER_PAGE);
if (req.query.nextPageCursor) {
query.start(req.query.nextPageCursor);
}
datastore.runQuery(query, function(err, entities, info) {
if (err) {
// Error handling omitted.
return;
}
// Respond to the front end with the contacts and the cursoring token
// from the query we just ran.
var frontEndResponse = {
contacts: entities
};
// Check if more results may exist.
if (info.moreResults !== datastore.NO_MORE_RESULTS) {
frontEndResponse.nextPageCursor = info.endCursor;
}
res.render('contacts', frontEndResponse);
});
});
Maybe you can try using one of the other syntax options (instead of Promises). The runQuery method can take a callback function as an argument, and that callback's parameters include explicit references to the entities array and the info object (which has the endCursor as a property).
And there are limits and quotas imposed on calls to the Datastore API as well. Here are links to official documentation that address them in detail:
Limits
Quotas

Resources