Firebase function always timeout on large files? - node.js

Question

I have created a firebase function that is triggered when a video is uploaded to the firebase storage and by using ffmpeg it add a watermark to it, it works fine with small video sizes but it always timeout in large ones. Any idea how I can overcome these limits
const functions = require('firebase-functions');
const { Storage, Bucket } = require('#google-cloud/storage');
const projectId = 'video-sharing-a57fa';
const admin = require('firebase-admin');
admin.initializeApp();
let gcs = new Storage({
projectId
});
const os = require('os');
const path = require('path');
const spawn = require('child-process-promise').spawn;
exports.addLogo = functions.runWith({ memory: '4GB', timeoutSeconds: 540 }).storage.object().onFinalize(async event => {
const bucket = event.bucket;
const contentType = event.contentType;
const filePath = event.name;
console.log('File change detected, function execution started');
if (path.basename(filePath).startsWith('resized-')) {
console.log('We already renamed that file!');
return;
}
const destBucket = gcs.bucket(bucket);
const tmpFilePath = path.join(os.tmpdir(), path.basename(filePath));
const metadata = { contentType: contentType };
const tmpLogoPath = path.join(os.tmpdir(), 'watermark.png');
await destBucket.file('watermark.png').download({
destination: tmpLogoPath
})
const newPath = path.join(os.tmpdir(), 'output.mp4')
return destBucket.file(filePath).download({
destination: tmpFilePath
}).then(() => {
console.log('entered spawn');
var str = "overlay=10:10"
return spawn('ffmpeg', ['-i', tmpFilePath, '-i', tmpLogoPath, '-filter_complex', str, newPath]);
}).then(() => {
console.log('chaning the name');
return destBucket.upload(newPath, {
destination: path.dirname(filePath) + '/resized-' + path.basename(filePath),
metadata: metadata
})
});
})

Answer 1

Cloud functions have a limited time for execution, it is limited to 9 mins max. More information here. Most likely the problem is that ffmpeg does not manage to add the watermark in time. Your actions should be:
Check the log of the function to confirm that this is exactly the error firebase functions:log --only <FUNCTION_NAME>
Consider different a different architecture option for processing really large files:
a. Limit the amount of data ffmpeg processes, e.g. with -ss 50 -t 10. In this scenario, there will the following architecture: a) one function that read files and put them into a queue, b) one function that reads the size of the file and puts the data into another queue, e.g. {name: "file1.mp4", start: 10, duration: 15}
b. Use an on-demand container such as Cloud Run
c. Use App Engine in case you are constantly processing some files