In my very simple app, I keep a running aggregate each time a new transaction is added/changed/deleted. I realised that I could simply loop through each transaction every time but this seems expensive. As such, I switch on the event type and apply a different logic depending if its a create/update/delete.
exports.aggregateTransactions = functions.firestore
.document('budgets/{budgetId}/transactions/{transactionId}')
.onWrite(async (change, context) => {
adjustment = 0;
if(change.after.data == undefined){
adjustment = -1 * change.before.data().amount;
}
if(change.before.data && change.after.data){
adjustment = change.after.data().amount - change.before.data().amount;
}
if(!change.before.data && change.after.data){
adjustment = change.after.data().amount;
}
const budgetRef = db.collection('budgets').doc(context.params.budgetId);
await db.runTransaction(async (transaction) => {
const budgetDoc = await transaction.get(budgetRef);
const newBalance = budgetDoc.data().balance + adjustment;
transaction.update(budgetRef, {
balance: newBalance
});
});
});
This feels like a lot of code each time just to work out what kind of event is happening. Is there a better way to handle this?
Most of the code looks fine to me. I'd just use a FieldValue.increment() operation for updating the budget, instead of a transaction:
const budgetRef = db.collection('budgets').doc(context.params.budgetId);
budgetRef.update({ balance: admin.firestore.FieldValue.increment(adjustment) });
For more on this operation, see the documentation on incrementing a numeric value.
Related
so I have a collection where I will have a lot of documents, but for now lets suppose it has about 100. (It had less than that when I was testing).
So, I need to get all the documents of the collection, put in a array, sort that and then send it to the websocket for the frontend. But the array is going with wrong data.
This is my code:
const emitSales = async (socket) => {
let salesArray = [];
const saleExists = (contract) => {
return salesArray.some(element => element.contract === contract);
}
const addSale = (contract) => {
const element = salesArray.find(e => e.contract === contract);
element.sales = element.sales+1;
}
const sales = await fiveMinSchema.find({}).lean();
if(sales) {
for await (x of sales) {
if(saleExists(x.contract)) {
addSale(x.contract);
continue;
}
const collection = await Collection.findOne({
contract: x.contract
});
let newsale = {
contract: x.contract,
title: collection.title,
description: collection.description,
image: collection.image,
sales: 1,
}
salesArray.push(newsale);
}
socket.emit("5min", salesArray.sort((a,b) => {
return b.sales-a.sales;
}).slice(0,10));
}
}
So, when I execute this function only once, for example, the array returns the correct values. But if I execute the function like 2 times in a row (like very fast), it starts returning the array with wrong data. (like mixing the data).
And as I using websocket, this function will execute like every 2 seconds (for example). How can I fix this problem? Like it seems to be executing more than one time simultaneously and mixing the data, idk..
I have a collection of teams containing around 80 000 documents. Every Monday I would like to reset the scores of every team using firebase cloud functions. This is my function:
exports.resetOrgScore = functions.runWith(runtimeOpts).pubsub.schedule("every monday 00:00").timeZone("Europe/Oslo").onRun(async (context) => {
let batch = admin.firestore().batch();
let count = 0;
let overallCount = 0;
const orgDocs = await admin.firestore().collection("teams").get();
orgDocs.forEach(async(doc) => {
batch.update(doc.ref, {score:0.0});
if (++count >= 500 || ++overallCount >= orgDocs.docs.length) {
await batch.commit();
batch = admin.firestore().batch();
count = 0;
}
});
});
I tried running the function in a smaller collection of 10 documents and it's working fine, but when running the function in the "teams" collection it returns "Cannot modify a WriteBatch that has been committed". I tried returning the promise like this(code below) but that doesn't fix the problem. Thanks in advance :)
return await batch.commit().then(function () {
batch = admin.firestore().batch();
count = 0;
return null;
});
There are three problems in your code:
You use async/await with forEach() which is not recommended: The problem is that the callback passed to forEach() is not being awaited, see more explanations here or here.
As detailed in the error you "Cannot modify a WriteBatch that has been committed". With await batch.commit(); batch = admin.firestore().batch(); it's exactly what you are doing.
As important, you don't return the promise returned by the asynchronous methods. See here for more details.
You'll find in the doc (see Node.js tab) a code which allows to delete, by recursively using a batch, all the docs of a collection. It's easy to adapt it to update the docs, as follows. Note that we use a dateUpdated flag to select the docs for each new batch: with the original code, the docs were deleted so no need for a flag...
const runtimeOpts = {
timeoutSeconds: 540,
memory: '1GB',
};
exports.resetOrgScore = functions
.runWith(runtimeOpts)
.pubsub
.schedule("every monday 00:00")
.timeZone("Europe/Oslo")
.onRun((context) => {
return new Promise((resolve, reject) => {
deleteQueryBatch(resolve).catch(reject);
});
});
async function deleteQueryBatch(resolve) {
const db = admin.firestore();
const snapshot = await db
.collection('teams')
.where('dateUpdated', '==', "20210302")
.orderBy('__name__')
.limit(499)
.get();
const batchSize = snapshot.size;
if (batchSize === 0) {
// When there are no documents left, we are done
resolve();
return;
}
// Delete documents in a batch
const batch = db.batch();
snapshot.docs.forEach((doc) => {
batch.update(doc.ref, { score:0.0, dateUpdated: "20210303" });
});
await batch.commit();
// Recurse on the next process tick, to avoid
// exploding the stack.
process.nextTick(() => {
deleteQueryBatch(resolve);
});
}
Note that the above Cloud Function is configured with the maximum value for the time out, i.e. 9 minutes.
If it appears that all your docs cannot be updated within 9 minutes, you will need to find another approach, for example using the Admin SDK from one of your server, or cutting the work into pieces and run the CF several times.
I am trying to update a field in my document in Firestore. The general location of the document would be "/games/{userId}/userGames/{gameId}. And in this game, there is a property called "status" which changes accordingly to the games start and end time.
As you can guess, the if the start time is bigger than the "now" timestamp and the status is "TO_BE_PLAYED", the game will begin and the status will be 1, "BEING_PLAYED". Also, if the end time is bigger than the "now" timestamp and the status is "BEING_PLAYED", the game will end, therefore the status will be 2, "PLAYED". I want to create a cloud function that is capable to do so.
However, even if the function logs output 'ok', the values are never updated. Unfortunately, I do not have that much experience in Javascript too.
THE CODE
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
const STATUS_PLAYED = 2;
const STATUS_BEING_PLAYED = 1;
const STATUS_TO_BE_PLAYED = 0;
exports.handleBeingPlayedGames = functions.runWith({memory: "2GB"}).pubsub.schedule('* * * * *')
.timeZone('Europe/Istanbul') // Users can choose timezone - default is America/Los_Angeles
.onRun(async () => {
// current time & stable
// was Timestamp.now();
const now = admin.firestore.Timestamp.fromDate( new Date());
const querySnapshot = await db.collection("games").get();
const promises = [];
querySnapshot.forEach( doc => {
const docRef = doc.ref;
console.log(docRef);
promises.push(docRef.collection("userGames").where("status", "==", STATUS_BEING_PLAYED).where("endtime", "<", now).get());
});
const snapshotArrays = await Promise.all(promises);
const promises1 = [];
snapshotArrays.forEach( snapArray => {
snapArray.forEach(snap => {
promises1.push(snap.ref.update({
"status": STATUS_PLAYED,
}));
});
});
return Promise.all(promises1);
});
exports.handleToBePlayedGames = functions.runWith({memory: "2GB"}).pubsub.schedule('* * * * *')
.onRun(async () => {
// current time & stable
// was Timestamp.now();
const now = admin.firestore.Timestamp.fromDate(new Date());
const querySnapshot = await db.collection("games").get();
const promises = [];
querySnapshot.forEach( async doc => {
const docData = await doc.ref.collection("userGames").where("status", "==", STATUS_TO_BE_PLAYED).where("startTime", ">", now).get();
promises.push(docData);
});
const snapshotArrays = await Promise.all(promises);
const promises1 = [];
snapshotArrays.forEach( snapArray => {
snapArray.forEach(snap => {
promises1.push(snap.ref.update({
"status": STATUS_BEING_PLAYED,
}));
});
});
return Promise.all(promises1);
});
Okay, so this answer goes to lurkers trying to solve this problem.
First I tried to solve this problem by brute force and not including much thinking and tried to acquire the value in subcollection. However, as I searched, I've found that denormalizing (flattening) data actually solves the problem a bit.
I created a new directory under /status/{gameId} with the properties
endTime, startTime, and status field and I actually did it on a single level by using promises. Sometimes denormalizing data can be your savior.
How can startTime be greater than now? Is it set by default to a date in the future?
My current assumption is that a game cannot set it's status to STATUS_BEING_PLAYED because of the inconsistency with startTime. Moreover, a game cannot have the status STATUS_PLAYED because it depends on having STATUS_BEING_PLAYED, which cannot have.
My recommendation would be to set the field startTime and endTime to null by default. If you do so you can check if a game has to be set to STATUS_BEING_PLAYED with this:
doc.ref.collection("userGames")
.where("status", "==", STATUS_TO_BE_PLAYED)
.where("startTime", "<", now)
.where("endTime", "==", null)
.get();
You could check if a game has to be on STATUS_PLAYED with this (exactly as you did):
docRef.collection("userGames")
.where("status", "==", STATUS_BEING_PLAYED)
.where("endtime", "<", now)
.get();
Now there's something that you should wonder, is this the best approach to change a game's status? You are querying the whole game library of a user every single minute as you know read operations are charged so this approach would imply meaningful charges. Maybe you should simply use update the game's status when the game is started and closed.
Also notice that the equals operation is ==, not =.
We're having an issue in our main data synchronization back-end function. Our client's mobile device is pushing changes daily, however last week they warned us some changes weren't updated in the main web app.
After some investigation in the logs, we found that there is indeed a single transaction that fails and rollback. However it appears that all the transactions before this one also rollback.
The code works this way. The data to synchronize is an array of "changesets", and each changset can update multiple tables at once. It's important that a changset be updated completely or not at all, so each is wrapped in a transaction. Then each transaction is executed one after the other. If a transaction fails, the others shouldn't be affected.
I suspect that all the transactions are actually combined somehow, possibly through the main db.task. Instead of just looping to execute the transactions, we're using a db.task to execute them in batch avoid update conflicts on the same tables.
Any advice how we could execute these transactions in batch and avoid this rollback issue?
Thanks, here's a snippet of the synchronization code:
// Begin task that will execute transactions one after the other
db.task(task => {
const transactions = [];
// Create a transaction for each changeset (propriete/fosse/inspection)
Object.values(data).forEach((change, index) => {
const logchange = { tx: index };
const c = {...change}; // Use a clone of the original change object
transactions.push(
task.tx(t => {
const queries = [];
// Propriete
if (Object.keys(c.propriete.params).length) {
const params = proprietes.parse(c.propriete.params);
const propriete = Object.assign({ idpropriete: c.propriete.id }, params);
logchange.propriete = { idpropriete: propriete.idpropriete };
queries.push(t.one(`SELECT ${Object.keys(params).join()} FROM propriete WHERE idpropriete = $1`, propriete.idpropriete).then(previous => {
logchange.propriete.previous = previous;
return t.result('UPDATE propriete SET' + qutil.setequal(params) + 'WHERE idpropriete = ${idpropriete}', propriete).then(result => {
logchange.propriete.new = params;
})
}));
}
else delete c.propriete;
// Fosse
if (Object.keys(c.fosse.params).length) {
const params = fosses.parse(c.fosse.params);
const fosse = Object.assign({ idfosse: c.fosse.id }, params);
logchange.fosse = { idfosse: fosse.idfosse };
queries.push(t.one(`SELECT ${Object.keys(params).join()} FROM fosse WHERE idfosse = $1`, fosse.idfosse).then(previous => {
logchange.fosse.previous = previous;
return t.result('UPDATE fosse SET' + qutil.setequal(params) + 'WHERE idfosse = ${idfosse}', fosse).then(result => {
logchange.fosse.new = params;
})
}));
}
else delete c.fosse;
// Inspection (rendezvous)
if (Object.keys(c.inspection.params).length) {
const params = rendezvous.parse(c.inspection.params);
const inspection = Object.assign({ idvisite: c.inspection.id }, params);
logchange.rendezvous = { idvisite: inspection.idvisite };
queries.push(t.one(`SELECT ${Object.keys(params).join()} FROM rendezvous WHERE idvisite = $1`, inspection.idvisite).then(previous => {
logchange.rendezvous.previous = previous;
return t.result('UPDATE rendezvous SET' + qutil.setequal(params) + 'WHERE idvisite = ${idvisite}', inspection).then(result => {
logchange.rendezvous.new = params;
})
}));
}
else delete change.inspection;
// Cheminees
c.cheminees = Object.values(c.cheminees).filter(cheminee => Object.keys(cheminee.params).length);
if (c.cheminees.length) {
logchange.cheminees = [];
c.cheminees.forEach(cheminee => {
const params = cheminees.parse(cheminee.params);
const ch = Object.assign({ idcheminee: cheminee.id }, params);
const logcheminee = { idcheminee: ch.idcheminee };
queries.push(t.one(`SELECT ${Object.keys(params).join()} FROM cheminee WHERE idcheminee = $1`, ch.idcheminee).then(previous => {
logcheminee.previous = previous;
return t.result('UPDATE cheminee SET' + qutil.setequal(params) + 'WHERE idcheminee = ${idcheminee}', ch).then(result => {
logcheminee.new = params;
logchange.cheminees.push(logcheminee);
})
}));
});
}
else delete c.cheminees;
// Lock from further changes on the mobile device
// Note: this change will be sent back to the mobile in part 2 of the synchronization
queries.push(t.result('UPDATE rendezvous SET timesync = now() WHERE idvisite = $1', [c.idvisite]));
console.log(`transaction#${++transactionCount}`);
return t.batch(queries).then(result => { // Transaction complete
logdata.transactions.push(logchange);
});
})
.catch(function (err) { // Transaction failed for this changeset, rollback
logdata.errors.push({ error: err, change: change }); // Provide error message and original change object to mobile device
console.error(JSON.stringify(logdata.errors));
})
);
});
console.log(`Total transactions: ${transactions.length}`);
return task.batch(transactions).then(result => { // All transactions complete
// Log everything that was uploaded from the mobile device
log.log(res, JSON.stringify(logdata));
});
I apologize, this is almost impossible to make a final good answer when the question is wrong on too many levels...
It's important that a change set be updated completely or not at all, so each is wrapped in a transaction.
If the change set requires data integrity, the whole thing must be one transaction, and not a set of transactions.
Then each transaction is executed one after the other. If a transaction fails, the others shouldn't be affected.
Again, data integrity is what a single transaction guarantees, you need to make it into one transaction, not multiple.
I suspect that all the transactions are actually combined somehow, possibly through the main db.task.
They are combined, and not through task, but through method tx.
Any advice how we could execute these transactions in batch and avoid this rollback issue?
By joining them into a single transaction.
You would use a single tx call at the top, and that's it, no tasks needed there. And in case the code underneath makes use of its own transactions, you can update it to allow conditional transactions.
Also, when building complex transactions, an app benefits a lot from using the repository patterns shown in pg-promise-demo. You can have methods inside repositories that support conditional transactions.
And you should redo your code to avoid horrible things it does, like manual query formatting. For example, never use things like SELECT ${Object.keys(params).join()}, that's a recipe for disaster. Use the proper query formatting that pg-promise gives you, like SQL Names in this case.
I have a Cloud Firestore trigger that takes care of adjusting the balance of a user's wallet in my app.
exports.onCreateTransaction = functions.firestore
.document('accounts/{accountId}/transactions/{transactionId}')
.onCreate(async (snap, context) => {
const { accountId, transactionId } = context.params;
const transaction = snap.data();
// See the implementation of alreadyTriggered in the next code block
const alreadyTriggered = await firestoreHelpers.triggers.alreadyTriggered(context);
if (alreadyTriggered) {
return null;
}
if (transaction.status === 'confirmed') {
const accountRef = firestore
.collection('accounts')
.doc(accountId);
const account = (await accountRef.get()).data();
const balance = transaction.type === 'deposit' ?
account.balance + transaction.amount :
account.balance - transaction.amount;
await accountRef.update({ balance });
}
return snap.ref.update({ id: transactionId });
});
As a trigger may actually be called more than once, I added this alreadyTriggered helper function:
const alreadyTriggered = (event) => {
return firestore.runTransaction(async transaction => {
const { eventId } = event;
const metaEventRef = firestore.doc(`metaEvents/${eventId}`);
const metaEvent = await transaction.get(metaEventRef);
if (metaEvent.exists) {
console.error(`Already triggered function for event: ${eventId}`);
return true;
} else {
await transaction.set(metaEventRef, event);
return false;
}
})
};
Most of the time everything works as expected. However, today I got a timeout error which caused data inconsistency in the database.
Function execution took 60005 ms, finished with status: 'timeout'
What was the reason behind this timeout? And how do I make sure that it never happens again, so that my transaction amounts are successfully reflected in the account balance?
That statement about more-than-once execution was a beta limitation, as stated. Cloud Functions is out of beta now. The current guarantee is at-least-once execution by default. you only get multiple possible events if you enable retries in the Cloud console. This is something you should do if you want to make sure your events are processed reliably.
The reason for the timeout may never be certain. There could be any number of reasons. Perhaps there was a hiccup in the network, or a brief amount of downtime somewhere in the system. Retries are supposed to help you recover from these temporary situations by delivering the event potentially many times, so your function can succeed.