I would like to upload files up to 1GB to Google Cloud Storage. I'm using Google App Engine Flexible. From what I understand, GAE has a 32MB limit on file uploads, which means I have to either upload directly to GCS or break the file into chunks.
This answer from several years ago suggests using the Blobstore API, however there doesn't seem to be an option for Node.js and the documentation also recommends using GCS instead of Blobstore to store files.
After doing some searching, it seems like using signed urls to upload directly to GCS may be the best option, but I'm having trouble finding any example code on how to do this. Is this the best way and are there any examples of how to do this using App Engine with Node.js?
Your best bet would be to use the Cloud Storage client library for Node.js to create a resumable upload.
Here's the official code example on how to create the session URI:
const {Storage} = require('#google-cloud/storage');
const storage = new Storage();
const myBucket = storage.bucket('my-bucket');
const file = myBucket.file('my-file');
file.createResumableUpload(function(err, uri) {
if (!err) {
// `uri` can be used to PUT data to.
}
});
//-
// If the callback is omitted, we'll return a Promise.
//-
file.createResumableUpload().then(function(data) {
const uri = data[0];
});
Edit: It seems you can nowadays use the createWriteStream method to perform an uplaod without having to worry about creation of a URL.
Related
I want to use write a cloud function to download a video file. I will do something like below to save the file into a mp4 format file.
const http = require('http');
const fs = require('fs');
const file = fs.createWriteStream("video.mp4");
const request = http.get("http://my-video-server.com/nice-day.mp4", function(response) {
response.pipe(file);
});
But I am wondering where is the file saved? Is it somewhere under /tmp on the firebase server? Can I specify the path for downloading the file? Also, if the video size is several GB, does firebase has any limit about how we use your server?
The reason I want to download a video file is I want to use the file to upload to YouTube via Youtube's API. They don't support upload via a link, it has to be a physical file.
The only writable path in Cloud Functions is /tmp. It's a memory-based filesystem, so you will only have a very limited amount of space available. I suggest reading the documentation on the Cloud Functions execution environment.
You can specify the exact file of the download using the API of the modules you're using for the download.
Cloud Functions is not a suitable product for work that requires large amount of disk space or memory.
well in my case I have a list of Url and I want to download each and every file from those urls and organise it in firebase storage bucket, my problem is I am unable to make folders in firebase storage bucket through nodejs javascript/typescript.
well firebase storage offers ref() and child method to upload files inside child folder (see this) but firebase only offers those method for firebase client libraries, it is not that we can not use client library in nodejs but they have made some namespaces hidden when you connect firebase client library in nodejs and storage is one of them (see this).
I am happy they have considered frontend and backend separately because of this very reason that front and backend have whole different scenario for security and use cases, so what they have really written to use in nodejs is firebase admin and I cannot see ref and child method in official documentation which they have said is this not any other way to name the file I am uploading nor any method for making folders to go child directories, when I upload a file from my computer it get saved in the bucket root with the same name as the filename it was in my computer, even though I can make folders from firebase console manually but it will not fulfill my requirement for sure there should must be any way to make folders in programmatically.
I also tried using google cloud storage library const {Storage} = require('#google-cloud/storage');
but it turned out firebase admin and gogole cloud library shares the same document and have same interface at least in upload file part.
well I have spent my day (well night too since it is 4:46am) trying different libraries and digging into their documents which I also found little unorganised and lack of code examples.
any help would be appreciated, my code snippet so far is following which is from their doc and uploading file correctly:
import "firebase/firestore"
admin.initializeApp({
credential: admin.credential.cert("./../path-to-service account-cert.json"),
databaseURL: 'gs://bilal-assistant-xxxxx.appspot.com'
});
const quran_bucket = admin.storage().bucket("quran-bucket");
quran_bucket.upload("./my_computer_path/fatiha.mp3", {
gzip: true,
metadata: {
cacheControl: 'public, max-age=31536000',
}
}).then(uploadResponse => {
console.log(` uploaded complete.`);
}).catch((reason: any) => {
console.log("reason: ", reason);
})
All I wanted is to save the audio file in folder bucket, not in bucket root
According to the API documentation, upload() takes an UploadOptions object as the second parameter. You will want to used the documented destination property of that object to specify the name of the file in Storage:
quran_bucket.upload("./my_computer_path/fatiha.mp3", {
destination: 'audio/juz30/fatiha.mp3',
gzip: true,
metadata: {
cacheControl: 'public, max-age=31536000',
}
})
You probably don't want to bother the gzip an mp3, as it's already compressed and won't compress much further.
I have created an App in Nodejs, this App involves users to upload some files such as profile pictures and some other media file, so these files are stored in certain folders here in my Web Application folder.
This works well locally now after deploying my App when the user do an upload these picture and other files it return an error, I suppose I should be maybe doing some configuration on Cloud Storage and let my App Engine Application be able to read and write to such folder using NodeJs. Please help me with how I can achieve this.
I use this following Code to create these files from Base64 data which is Uploaded using Rest API
var data = req.body.image; //base64 image data
const bucket = storage.bucket(process.env.GCLOUD_STORAGE_BUCKET); //decalre bucket
var blob = bucket.file('new_image.png');
writeScreenShot(blob, data);
blobStream.on('finish', () => {
res.send("Success");
});
blobStream.end(req.file.buffer);
function writeScreenShot(blob, data)
{
var strm = blob.createWriteStream();
strm.write(new Buffer(data, 'base64'));
strm.end();
}
This is how I implemented it so far, but it returns image that says it looks like we don't support this image type while file extension of this image is png, it seems like I have created this image wrongly now got affected the metadata of it. Thanks in advance
I want to create a HTTP trigger Firebase Function which basically gets the image url from request, downloads the image, stores the image in storage and then returns the new image url from storage. The code looks like this so far:
const functions = require('firebase-functions');
exports.mirror = functions.https.onRequest((req, res) => {
var url = req.query.url
//TODO: Download image from `url`.
//TODO: Store downloaded image in Firebase Storage.
//TODO: Return new image path in Firebase Store.
res.status(200).send(url)
});
How should I go on about solving this?
This could be done by using a mix of the techniques and libraries shown in these examples or documentation items. Just study these examples and documentation and you should be able to assembly all the pieces together.
https://github.com/request/request to download the image
The following Cloud Functions samples for image saving and retrieving
https://github.com/firebase/functions-samples/blob/master/generate-thumbnail/functions/index.js
https://github.com/firebase/functions-samples/blob/master/moderate-images/functions/index.js
https://cloud.google.com/nodejs/docs/reference/storage/1.6.x/
I am trying to use aws s3 image in lambda node js but it throws an error 'no such file or directory'. But I have made that image as public and all permissions are granted.
fs = require('fs');
exports.handler = function( event, context ) {
var img = fs.readFileSync('https://s3-us-west-2.amazonaws.com/php-7/pic_6.png');
res.writeHead(200, {'Content-Type': 'image/png' });
res.end(img, 'binary');
};
fs is node js file system core module. It is for writing and reading files on local machine. That is why it gives you that error.
There are multiple things wrong with your code.
fs is a core module used for file operations and can't be used to access S3.
You seem to be using express.js code in your example. In lambda, there is no built-in res defined(unless you define it yourself) that you can use to send response.
You need to use the methods on context or the new callback mechanism. The context methods are used on the older lambda node version(0.10.42). You should be using the newer node version(4.3.2 or 6.10) which return response using the callback parameter.
It seems like you are also using the API gateway, so assuming that, I'll give a few suggestions. If the client needs access to the S3 object, these are some of your options:
Read the image from S3 using the AWS sdk and return the image using the appropriate binary media type. AWS added support for binary data for API gateway recently. See this link OR
Send the public S3 URL to client in your json response. Consider whether the S3 objects need to be public. OR
Use the S3 sdk to generate pre-signed URLs that are valid for a configured duration back to the client.
I like the pre-signed URL approach. I think you should check that out. You might also want to check the AWS lambda documentation
To get a file from S3, you need to use the path that S3 give you. The base path is https://s3.amazonaws.com/{your-bucket-name}/{your-file-name}.
On your code, you must replace the next line:
var img = fs.readFileSync('https://s3.amazonaws.com/{your-bucket-name}/pic_6.png');
If don't have a bucket, you should to create one to give permissions.