Firebase Storage file download stops working after a while - node.js

Question

I have a simple trigger set on file upload to Firebase. It reads uploaded file, process it and saves results to database. It works for a while, after that it crashes and stops working. Usually uploading function helps. Does anybody have an idea what might be the reason? Am I getting out of memory or ... ?
Here is the code :
const {Storage} = require('#google-cloud/storage');
const path = require('path');
const storage = new Storage();
exports.processLogs = functions
.region('europe-west1')
.storage
.object()
.onFinalize(async (object) => {
const filename = path.basename(object.name);
const bucket = storage.bucket(object.bucket);
try {
await bucket.file(filename).download(async (err, contents) => {
if (err) {
console.log('error', err);
return null
}
//Proces file and store into db
// (...)
bucket.file(filename).delete();
});
} catch(e){
console.log('error',e)
}
});
Error I am getting is :
Anonymous caller does not have storage.objects.get access to project-name.appspot.com/CrTwBuyNR2-log-2020-1-16-12-18.csv.'
thanks

Answer 1

As you are using Firebase, I recommend initialising the bucket from firebase-admin instead of #google-cloud/storage directly. This will sort out permissions so that security rules are skipped.
In your code you have also incorrectly mixed the callback and async/await APIs. Because this code is running in a Cloud Function, I recommend exclusively using Promises and async/await.
The code below is a rewrite with the following changes:
Code has been split into logical blocks
No callback API usage (see File#download)
Each block will log and throw errors separately for easier debugging
One-line log messages (i.e. no stack trace)
Leaves full error logging to Cloud Functions (makes finding erroneous runs easier)
const admin = require('firebase-admin');
const functions = require('firebase-functions');
admin.initializeApp();
exports.processLogs = functions.region('europe-west1').storage.object()
.onFinalize(async (object) => {
const bucketRef = admin.storage().bucket(object.bucket);
const fileRef = bucketRef.file(object.name);
console.log('Processing "' + object.id + '"...');
// 1) DOWNLOAD
let [contents] = await fileRef.download()
.catch((err) => {
console.log('DOWNLOAD FAILED: ', (err.code ? err.code + ': ' : '') + err.message);
throw err;
});
// 2) PARSE
let dataToUpload = {};
try {
// Transform file contents
dataToUpload = JSON.parse(contents);
} catch (err) {
console.log('PARSE FAILED: ', (err.code ? err.code + ': ' : '') + err.message);
throw err;
}
// 3) DATABASE SET
const dbRef = admin.database().ref('path/to/data');
await dbRef.set(dataToUpload)
.catch((err) => {
console.log('DATABASE SET FAILED: ', (err.code ? err.code + ': ' : '') + err.message);
throw err;
});
// 4) CLEANUP
await fileRef.delete()
.catch((err) => {
console.log('CLEANUP FAILED: ', (err.code ? err.code + ': ' : '') + err.message);
throw err;
});
// 5) LOG SUCCESS
console.log('SUCCEEDED');
});
The log messages above can also be bundled into a helper function if desired:
function logAndRethrowError(err, name) {
console.log((name || 'ERROR') + ': ', ((err.code ? err.code + ': ' : '') + err.message) || err);
throw err;
}
// Usage:
let [contents] = await fileRef.download()
.catch(err => logAndRethrowError(err, 'DOWNLOAD FAILED'));
try {
// ...
} catch (err) { logAndRethrowError(err, 'PARSE FAILED') }

Answer 2

This looks like a problem with Security Rules not being set properly? Please try this document or video.
For starters try this:
service firebase.storage {
match /b/{bucket}/o {
match /{allPaths=**} {
allow read, write: if true;
}
}
}
And if it proves successful set Storage Rules properly.