I am using the Firebase Admin SDK with Cloud Functions. The function does multiple writes to several Firestore collections, which need to be consistent.
Now i am not sure how the Firestore operation behave if a valid operation like a write to a document fails (maybe through cosmic radiation or something which is similar unlikely).
Does the operation instantly return an error or is there some kind of retry or error correction mechanism?
Maybe this is a silly question and has nothing to do with the SDK itself.
First of all, if you have multiple documents to write that all must land at the same time, atomically, you should be using a batch or transaction in order to make that happen. If any document would fail to write for any reason, then nothing will happen for any of the documents referenced. If you instead choose to do several write operations, you would have to figure out to reliably roll back each change individually, which is going to be a lot of work.
If you do get an error, I don't believe there are any guarantees about the conditions of that error. You would likely want to retry on your own, unless you're able to determine that the error is not transient. To make retries reliable, you could enable the retry configuration on the function, allow the error to escape the function (don't catch the error), and let Cloud Functions invoke it again for you.
It will throw an error. If you notice every method use has a callback with succes or error.
If you are using something like await on node, you should then try/catch
If you have more than one operation and the procedure should be atomical and/or all or nothing, then use batches
https://firebase.google.com/docs/firestore/manage-data/transactions
When a Functions encounters if the error is not handle then the Function crash, you can modify the retries for Functions
https://firebase.google.com/docs/functions/retries
Related
If I'm using a firestore function triggered on firestore database events such as onUpdate() or onCreate(), for most writes I can check the change.after.updateTime field to see when the write occurred, which will be different than Timestamp.now(), the time the function is invoked.
However, when the document is deleted, triggering onWrite() or onDelete(), change.after.exists is false and change.after.updateTime is undefined. Is there any way to get access to the time that the database event occurred, rather than the time that the function is invoked?
Currently there is no information about when the exact transition occurs available for such a delete operation. You'd have to pass or get something out of band (i.e. writing it into a separate document, or somehow reading it from the logs), but I can imagine how that'd get pretty hairy quickly.
I recommend filing a feature request for this with the team.
Or am I missing something? Or has this already been addressed in their documentation
Yes, a change to a document that triggered the function could cause an infinite loop. This is noted in the documentation, which you will find as you scroll through it. It says:
Note: Any time you write to the same document that triggered a
function, you are at risk of creating an infinite loop. Use caution
and ensure that you safely exit the function when no change is needed.
Essentially what it's saying is it's the programmer's responsibility to detect this situation and bail out of the function early, given that the work was already done in a previous invocation.
In Java, I am used to try..catch, with finally to cleanup unused resources.
In Node.JS, I don't have that ability.
Odd errors can occur for example the database could shut down at any moment, any single table or file could be missing, etc.
With nested calls to db.query(..., function(err, results){..., it becomes tedious to call if(err) {send500(res); return} every time, especially if I have to cleanup resources, for example db.end() would definitely be appropriate.
How can one write code that makes async catch and finally blocks both be included?
I am already aware of the ability to restart the process, but I would like to use that as a last-resort only.
A full answer to this is pretty in depth, but it's a combination of:
consistently handling the error positional argument in callback functions. Doubling down here should be your first course of action.
You will see #izs refer to this as "boilerplate" because you need a lot of this whether you are doing callbacks or promises or flow control libraries. There is no great way to totally avoid this in node due to the async nature. However, you can minimize it by using things like helper functions, connect middleware, etc. For example, I have a helper callback function I use whenever I make a DB query and intend to send the results back as JSON for an API response. That function knows how to handle errors, not found, and how to send the response, so that reduces my boilerplate substantially.
use process.on('uncaughtExcepton') as per #izs's blog post
use try/catch for the occasional synchronous API that throws exceptions. Rare but some libraries do this.
consider using domains. Domains will get you closer to the java paradigm but so far I don't see that much talk about them which leads me to expect they are not widely adopted yet in the node community.
consider using cluster. While not directly related, it generally goes hand in hand with this type of production robustness.
some libraries have top-level error events. For example, if you are using mongoose to talk to mongodb and the connection suddenly dies, the connection object will emit an error event
Here's an example. The use case is a REST/JSON API backed by a database.
//shared error handling for all your REST GET requests
function normalREST(res, error, result) {
if (error) {
log.error("DB query failed", error);
res.status(500).send(error);
return;
}
if (!result) {
res.status(404).send();
return;
}
res.send(result); //handles arrays or objects OK
}
//Here's a route handler for /users/:id
function getUser(req, res) {
db.User.findById(req.params.id, normalREST.bind(null, res));
}
And I think my takeaway is that overall in JavaScript itself, error handling is basically woefully inadequte. In the browser, you refresh the page and get on with your life. In node, it's worse because you're trying to write a robust and long-lived server process. There is a completely epic issue comment on github that goes into great detail how things are just fundamentally broken. I wouldn't get your hopes up of ever having JavaScript code you can point at and say "Look, Ma, state-of-the-art error handling". That said, in practice if you follow the points I listed above, empirically you can write programs that are robust enough for production.
See also The 4 Keys to 100% Uptime with node.js.
I am running a transaction to update an item that needs to be stored in two keys. To accomplish this, I have setup a nested transaction as follows, and it seems to run as expected:
firebaseOOO.child('relationships/main').child(accountID).child(friendAccountID).transaction(function(data) {
data.prop = 'newval';
firebaseOOO.child('relationships/main').child(friendAccountID).child(accountID).transaction(function(data) {
return r;
});
return r;
});
Are there any gotchas or possible unexpected implications to this? I am most worried about getting stuck in some sort of transaction loop under load, where each transaction cancels the other out forcing them both to restart, or similar.
Is there a better way of doing this?
I am using the NodeJS client.
You probably don't want to start another transaction from within the callback to the first one. There is no guarantee as to how many times the function for your first transaction will run, particularly if there is a lot of contention at the location you are trying to update.
A better solution, which I believe you hit on in your other question, is to start the second transaction from the completion callback, after checking that the first one committed.
anybody know patterns to design idempotent operations to azure manipulation, specially the table storage? The more common approach is generate a id operation and cache it to verify new executions, but, if I have dozen of workers processing operations this approach will be more complicated. :-))
Thank's
Ok, so you haven't provided an example, as requested by knightpfhor and codingoutloud. That said, here's one very common way to deal with idempotent operations: Push your needed actions to a Windows Azure queue. Then, regardless of the number of worker role instances you have, only one instance may work on a specific queue item at a time. When a queue message is read from the queue, it becomes invisible for the amount of time you specify.
Now: a few things can happen during processing of that message:
You complete processing after your timeout period. When you go to delete the message, you get an exception.
You realize you're running out of time, so you increase the queue message timeout (today, you must call the REST API to do this; one day it'll be included in the SDK).
Something goes wrong, causing an exception in your code before you ever get to delete the message. Eventually, the message becomes visible in the queue again (after specified invisibility timeout period).
You complete processing before the timeout and successfully delete the message.
That deals with concurrency. For idempotency, that's up to you to ensure you can repeat an operation without side-effects. For example, you calculate someone's weekly pay, queue up a print job, and store the weekly pay in a Table row. For some reason, a failure occurs and you either don't ever delete the message or your code aborts before getting an opportunity to delete the message.
Fast-forward in time, and another worker instance (or maybe even the same one) re-reads this message. At this point, you should theoretically be able to simply re-perform the needed actions. If this isn't really possible in your case, you don't have an idempotent operation. However, there are a few mechanisms at your disposal to help you work around this:
Each queue message has a DequeueCount. You can use this to determine if the queue message has been processed before and, if so, take appropriate action (maybe examine the Table row for that employee, for example).
Maybe there are stages of your processing pipeline that can't be repeated. In that case: you now have the ability to modify the queue message contents while the queue message is still invisible to others and being processed by you. So, imagine appending something like |SalaryServiceCalled . Then a bit later, appending |PrintJobQueued and so on. Now, if you have a failure in your pipeline, you can figure out where you left off, the next time you read your message.
Hope that helps. Kinda shooting in the dark here, not knowing more about what you're trying to achieve.
EDIT: I guess I should mention that I don't see the connection between idempotency and Table Storage. I think that's more of a concurrency issue, as idempotency would need to be dealt with whether using Table Storage, SQL Azure, or any other storage container.
I believe you can use Reply log storage way to solve this problem