Running Background Process using FFMPEG on Google Cloud Run stopping in middle - node.js

Question

I have an external bash script that transcodes audio files using FFmpeg and then uploads the files to google cloud storage. I am using the google cloud run platform for this process but the process is stopping in the middle and not getting any clue from the logs. I am using the node js spawn command to execute the bash script
const createHLSVOD = spawn('/bin/bash', [script, file.path, file.destination, contentId, EPPO_MUSIC_HSL_URL, 'Content', speed]);
createHLSVOD.stdout.on('data', d => console.log(`stdout info: ${d}`));
createHLSVOD.stderr.on('data', d => console.log(`stderr error: ${d}`));
createHLSVOD.on('error', d => console.log(`error: ${d}`));
createHLSVOD.on('close', code => console.log(`child process ended with code ${code}`));
on cloud run beginning the process itself taking a lot of time but in my local machine transcoding and uploading is very fast. after some time transcoding logs are being stopped and no new logs appear. I have no clue what is happening
so what is happening here? why it is very slow in the first place and why the process is being stopped in middle without any error
node js script
Transcoding script
Dockerfile

Answer 1

The issue with spawn is that you create an asynchronous process that run in background. It's a problem, because Cloud Run allow CPU to the container only when a request is being processed. And in your case, you have that
A request arrive
A spawn is created
The spawned script run in background
An HTTP answer is sent on the request. Cloud Run throttles the CPU
Your spawned script continue to run
There is 2 consequences
Your script processing is very very long because the throttling limit the CPU under 5%
After a period with activity (i.e. request received by the instance), Cloud Run kill the unused instance to same resource on its side. it's about 15 minutes, but it's subject to change, it's Google Cloud internal sauce
I recommend you to wait the end of the spawned script or to use a synchronous call; such as execSync, instead of async spawn.

Answer 2

I think the best solution for this issue is to change the approach of the architecture. Instead of using the client to upload the file to Cloud Storage through Cloud Run I would suggest using the signed URL. Signed URL provides permission and time to make a request, allowing users without credentials to perform specific action on a resource, for example you can give the user write access to a bucket for a limited time so they can upload a file. From the client side, this is very similar to your current process, changing the Cloud Run URL for the signed URL.
You can check this link to read more about signed URLs, and here is a quick guide of how to generate it.
For example, you can create a signed URL to test with this command:
First create a private key
gcloud iam service-accounts keys create "file where you want to storage the key" \
--iam-account="name of the service account"#projectID.iam.gserviceaccount.com
Now you authenticate the service account
gcloud auth activate-service-account --key-file KEY_FILE_LOCATION/KEY_FILE_NAME
And now create the signed URL
gsutil signurl -m PUT -d 1h -c CONTENT_TYPE -u gs://BUCKET_NAME/OBJECT_NAME
After that, the client uploads the file using the signed URL to the bucket in Cloud Storage. I would use Pub/Sub notification so you can know that the file was uploaded without any problem, and with that notification you can use it to trigger other operations in Cloud Run.
Basically Pub/Sub notifications are sent when an object changes in a specific bucket, you can follow this guide that could help you configure that notification.
For example, to get notified about newly upload objects, you can configure it this way:
gsutil notification create -e OBJECT_FINALIZE -f json gs://<bucket name>
To use this notification to trigger a Cloud Run process you can see this link.