I have a Firebase Cloud functions which is triggered by an update to some data in a Firebase Realtime Database. When the data is updated, I want to read the data, perform some calculations on that data, and then save the results of the calculations back to the Realtime Database. It looks like this:
exports.onUpdate = functions.database.ref("/some/path").onUpdate((change) => {
const values = change.after.val();
const newValues = performCalculations(value);
return change.after.ref.update(newValues);
});
My concern is that this may create an indefinite loop of updates. I saw a note on the Cloud Firestore Triggers that says:
"Any time you write to the same document that triggered a function,
you are at risk of creating an infinite loop. Use caution and ensure
that you safely exit the function when no change is needed."
So my first question is: Does this same problem apply to the Firebase Realtime Database?
If it does, what is the best way to prevent the infinite looping?
Should I be comparing before/after snapshots, the key/value pairs, etc.?
My idea so far:
exports.onUpdate = functions.database.ref("/some/path").onUpdate((change) => {
// Get old values
const beforeValues = change.before.val();
// Get current values
const afterValues = change.after.val();
// Something like this???
if (beforeValues === afterValues) return null;
const newValues = performCalculations(afterValues);
return change.after.ref.update(newValues);
});
Thanks
Does this same problem apply to the Firebase Realtime Database?
Yes, the chance of infinite loops occurs whenever you write back to the same location that triggered your Cloud Function to run, no matter what trigger type was used.
To prevent an infinite loop, you have to detect its condition in the code. You can:
either flag the node/document after processing it by writing a value into it, and check for that flag at the start of the Cloud Function.
or you can detect whether the Cloud Function code made any effective change/improvement to the data, and not write it back to the database when there was no change/improvement.
Either of these can work, and which one to use depends on your use-case. Your if (beforeValues === afterValues) return null is a form of the second approach, and can indeed work - but that depends on details about the data that you haven't shared.
Related
I am trying to build a logging mechanism, to log changes done to a record. I am currently logging previous and new record. However, as the site is very busy, I expect the logfile to grow seriously huge. To avoid this, I plan to only capture the modified fields only.
Is there a way to capture only the modifications done to a record (in REACT), so my {request.body} will have fewer fields?
My Server-side is build with NODE.JS and the client-side is REACT.
One approach you might want to consider is to add an onChange(universal) or onTextChanged(native) listener to the text field and store the form update in a local state/variables.
Finally, when a user makes an action (submit, etc.) you can send the updated data to the logging module.
The best way I found and works for me is …
on the api server-side, where I handle the update request, before hitting the database, I do a difference between the previous record and {request.body} using lodash and use the result to send to my update database function
var _ = require('lodash');
const difference = (object, base) => {
function changes(object, base) {
return _.transform(object, function (result, value, key) {
if (!_.isEqual(value, base[key])) {
result[key] = (_.isObject(value) && _.isObject(base[key])) ? changes(value, base[key]) : value;
}
});
}
return changes(object, base);
}
module.exports = difference
I saved the above code in a file named diff.js and included it in my server-side file.
It worked good.
Thanks for giving the idea...
I am using the Admin SDK for Node.js, working with firestore.
I am trying to handle many documents, and therefore wonders how the following function behaves.
export async function test() {
const collection = firestore.afs.collection(<some-path>);
const items = await collection.get();
// This method takes rather long time on a big collection.
}
test();
Does the get() method reads all method up in memory?
Reading all documents in a collection into a variable, as your code does, loads them all in memory. There really isn't anywhere else to put them.
If you don't want to load all document at once, you'll want to use queries to determine what specific documents to load.
I want to implement a function a by firebase cloud function.
In this function, when it received a request from client, I save the id and the score of client to a dictionary. In normally, I will save this data to firestore and response to client, but I want to save the writing operation to firestore, so I will update the data of client in dictionary and response to client. Checking every 15 minitues, I will save all data in dictionary to firestore and reset it.
This is the main flow I wrote:
var dic = {}; //save the temp data of client
var timeNow = new Date().getTime(); //get Time when function was deployed
exports.testSucCollectionQueryFunction = functions.https.onRequest( (request, response) => {
getDataInfoFromRequest(); //id, and score
dic[id] = score; //update data info
response.send(result); // only response that cloud had received request
if (currentTime - timeNow > 15 minutes) {
saveDatainDicToFireStore();
dic = {}; //reset Dictionary
}
}
I tested with the small concurrent connection from client (<5), it was still ok. But I dont know what happen with the 1000requests/second.
So, if Can you help me any ideas to understand about this problem?
Thanks for your help
So there are a few issues here.
First, response.send(result); should be the last line of code in the function. Once you call response you are sending a signal to the cloud function that you're function is complete. While the code after response might run, it isn't guaranteed. I'm referring to your if statement:
if (currentTime - timeNow > 15 minutes) {
saveDatainDicToFireStore();
dic = {}; //reset Dictionary
}
Next, cloud functions whether they are Firebase, AWS Lambda, or Azure Functions should be stateless. Depending on your workload their might be multiple containers created by the Firebase Function system and you do not have control over which instance you are caching data into. Plus, within that 15 minute period those instances might be shutdown due to inactivity and your dictionary will be lost completely.
If I were you I would redesign your architecture. I would either write to a temp location in the Firestore database that will act as a "caching" mechanism. Then setup a separate HTTP function that you will call every 15 minutes to aggregate that cached data and save it to your intended location.
Or simply write the data to that specific location right way instead of caching it at all. Finally, one other alternative is writing it to a file and saving that file in the Firebase Cloud Storage bucket for later processing, and then use that separate HTTP function from earlier to aggregate the data and write it to the Firestore.
It's a simple concept but yet finding the right answer is ridiculously stressing. Use Firebase Functions w/ Node.js to pull the "itemsExpires" (date) from the Firebase database. The parent node is the userId (random string) and each item is stored under a key node (another random string).. So, here's what the firebase database looks like:
firebase-database-name
+ 82hfijcbwjbjfbergjagbk_USERID
+ "My Stuff"
+ gnjfgsdkfjsdf_ITEMkey
-- "item name": whatever
-- "itemExpires": 05-01-2017
-- "itemType": whatever too
+ an3jeejwiag_ITEMkey
-- "item name": whatever
-- "itemExpires": 06-01-2017
-- "itemType": whatever too
+ zzzndjgabblsbl_ITEMkey
-- "item name": whatever
-- "itemExpires": 07-01-2017
-- "itemType": whatever too
I'm not asking for someone to write the code, a good reference will do but there are so many ways to call data and all I'm finding are the ways to Call using a structured tree and not one with random id's and keynumbers.
*** Basically, my goal here is to run a 3rd party cron job through Firebase Functions that runs through each item entry and checks the expiration date against today's date. This is the wall I'm against.
Bradley I am yet unclear as to what you want to do exactly. I suppose you intend on having multiple users (not just one as in the example) with multiple Items and compare the current date against the expiration date of all items for every user at a specified time (using cron). There are some considerations you should take into account here :
Do you really need cron ? ( or can you solve your problems more easily and natively with a javascript plain setInterval() ? )
How often are you going to check your entire database and how big is that database?
OK. So to explain, the first consideration is just a thought and the logic behind it should be pretty obvious. The second consideration takes some explaining. Since I believe your firebase data will NOT be static and will constantly change you need to find a way to get these changes inside you node script.
If you do not intend on calling your scheduled task too often and the database is not a mammoth (taking ages to load) you could just do the following :
const firebase = require('firebase-admin');
const serviceAccount = require('yourServiceAccountPath.json');
firebase.initializeApp({
credential: firebase.credential.cert(serviceAccount),
databaseURL: "youDatabaseURL"
});
setInteval(function(){ // or cron schedule
firebase.database().ref().once('value',function(snapshot){
let allYourUsers = snapshot.val();
// iterate through them all
// and do what you gotta do
});
},10000); // your interval in milliseconds
However this approach just loads once all your database each time you want to check all items. If you have other data in the database, consider adding users to a seperate path and load just that path. This approach is not recommended if your users are quite many and/or you want them to be checked very often. If such is the case and your data does not change very often you could consider this alternative:
Here you use the on function to update your data as it is edited and set the checking part seperate like so :
const firebase = require('firebase-admin');
const serviceAccount = require('yourServiceAccountPath.json');
firebase.initializeApp({
credential: firebase.credential.cert(serviceAccount),
databaseURL: "youDatabaseURL"
});
const databaseRef=firebase.database().ref();
let allYourUsers;
let allYourUsersStaticCopy;
databaseRef.on('value',function(snapshot){
allYourUsers = snapshot.val();
});
setInteval(function(){ // or cron schedule
if ( allYourUsers ) { // to ensure that data has loaded at least once
// (startup considerations)
allYourUsersStaticCopy = Object.assign({},allYourUsers);
// iterate through the static copy in order to avoid
// you data changing while you are accesing it
// and do what you gotta do
}
},10000); // your interval in milliseconds
The upside with the second piece of code is that your data is loaded every time there is a change and not every time your check runs. If however your data changes very often (additions,deletions and edits) this approach might not be optimum.
In the case that your script runs often enough, the database is big enough, and the changes are often enough to prevent any of the above to be efficient, you might want to consider the optimum solution : loading your users once and then attaching listeners for the child added,removed and changed to update your existing users object. Thus you receive only changes and not a whole snapshot. If such is the case you can read about the mentioned listeners in the firebase docs and I can squeeze in some extra code. Just let me know.
Hope this long post helps!
Assuming you have Firebase set up properly within your node project, you can do a one time read for your ITEMkey entries. Something like this:
var db = admin.database();
var ref = db.ref("82hfijcbwjbjfbergjagbk_USERID").child("My Stuff");
ref.once("value", function(snapshot) {
var contents = snapshot.val();
// Data returned here will be an object with all children
// nodes under "My Stuff". You can access it by calling
// snapshot.val() like I did above.
}
I just started developing nodejs. I'm confused to use async model. I believe there is a way to turn most of SYNC use cases into ASYNC way. Example, by SYNC, we load some data and wait until it returns then show them to user; by ASYNC, we load data and return, just tell the user data will be presented later. I can understand why ASYNC is used in this scenario.
But here I have a use case. I'm building an web app, allowing user to place a order (buying something). Before saving the order data into db, I want to put some user data together with order data (I'm using document NoSql db by the way). So I think by SYNC, after I get order data, I make a SYNC call to database and wait for its returned user data. After I get returned data, integrate them together and ingest into db.
I think there might be an issue if I make ASYNC call to db to query user data because user data may be returned after I save data to db. And that's not what I want.
So in this case, how can I do this thing ASYNCHRONOUSLY?
Couple of things here. First, if your application already has the user data (the user is already logged in), then this information should be stored in session so you don't have to access the DB. If you are allowing the user to register at the time of purchase, you would simply want to pass a callback function that handles saving the order into your call that saves the user data. Without knowing specifically what your code looks like, something like this is what you would be looking for.
function saveOrder(userData, orderData, callback) {
// save the user data to the DB
db.save(userData, function(rec) {
// if you need to add the user ID or something to the order...
orderData.userId = rec.id; // this would be dependent on your DB of choice
// save the order data to the DB
db.save(orderData, callback);
});
}
Sync code goes something like this. step by step - one after other. There can be ifs and loops (for) etc. all of us get it.
fetchUserDataFromDB();
integrateOrderDataAndUserData();
updateOrderData();
Think of async programming with nodejs as event driven. Like UI programming - code (function) is executed when an event occurs. E.g. On click event - framework calls back registered clickHandler.
nodejs async programming can also be thought on these lines. When db query (async) execution completes, your callback is called. When order data is updated, your callback is called. The above code goes something like this:
function nodejsOrderHandler(req,res)
{
var orderData;
db.queryAsync(..., onqueryasync);
function onqueryasync(userdata)
{
// integrate user data with order data
db.update(updateParams, onorderudpate);
}
function onorderupdate(e, r)
{
// handler error
write response.
}
}
javascript closure provides the way to keep state in variables across functions.
There is certainly much more to async programming and there are helper modules that help with basic constructs like chain, parallel, join etc as you write more involved async code. but this probably gives you a quick idea.