Read JSON file directly from google storage (using Cloud Functions) - node.js

Question

I created a function that extracts a specific attribute from a JSON file, but this file was together with the function in Cloud Functions. In this case, I was simply attaching the file and was able to refer to a specific attribute:
const jsonData = require('./data.json');
const result = jsonData.responses[0].fullTextAnnotation.text;
return result;
Ultimately, I want to read this file directly from cloud storage and here I have tried several solutions, but without success. How can I read a JSON file directly from google storage so that, as in the first case, I can read its attributes correctly?

Answer 1

As mentioned in the comment the Cloud Storage API allows you to do many things through API. Here's an example from documentation on how to download a file from Cloud Storage for your reference.
/**
* TODO(developer): Uncomment the following lines before running the sample.
*/
// The ID of your GCS bucket
// const bucketName = 'your-unique-bucket-name';
// The ID of your GCS file
// const fileName = 'your-file-name';
// The path to which the file should be downloaded
// const destFileName = '/local/path/to/file.txt';
// Imports the Google Cloud client library
const {Storage} = require('#google-cloud/storage');
// Creates a client
const storage = new Storage();
async function downloadFile() {
const options = {
destination: destFileName,
};
// Downloads the file
await storage.bucket(bucketName).file(fileName).download(options);
console.log(
`gs://${bucketName}/${fileName} downloaded to ${destFileName}.`
);
}
downloadFile().catch(console.error);

Answer 2

To clearly answer the question: you can't!
You need to download the file locally first, and then process it. You can't read it directly from GCS.
With Cloud Functions you can only store file in the /tmp directory, it's the only one writable. In addition, it's an in-memory file system, that means several things:
The size is limited by the memory set up to the Cloud Function. The memory space is shared between your app memory footprint and your file storage in /tmp (you won't be able to download a file of 10Gb for example)
The memory is lost when the instance goes down and
All the Cloud Functions instances have their own memory space. You can't share the files between all the Cloud Functions
The /tmp directory isn't cleaned between 2 functions invocation (on the same instance). Think to cleanup yourselves this directory.