I'm working on a project using Google Cloud Storage to allow users to upload media files into a predefined bucket using Node.js. I've been testing with small .jpg files. I also used gsutil to set bucket permissions to public.
At first, all files generated links that downloaded the file. Upon investigation of the docs, I learned that I could explicitly set the Content-Type of each file after upload using the gsutil CLI. When I used this procedure to set the filetype to 'image/jpeg', the link behavior changed to display the image in the browser. But this only worked if the link had not been previously clicked prior to updating the metadata with gsutil. I thought that this might be due to browser caching, but the behavior was duplicated in an incognito browser.
Using gsutil to set the mime type would be impractical at any rate, so I modified the code in my node server POST function to set the metadata at upload time using an npm module called mime. Here is the code:
app.post('/api/assets', multer.single('qqfile'), function (req, res, next) {
console.log(req.file);
if (!req.file) {
return ('400 - No file uploaded.');
}
// Create a new blob in the bucket and upload the file data.
var blob = bucket.file(req.file.originalname);
var blobStream = blob.createWriteStream();
var metadata = {
contentType: mime.lookup(req.file.originalname)
};
blobStream.on('error', function (err) {
return next(err);
});
blobStream.on('finish', function () {
blob.setMetadata(metadata, function(err, response){
console.log(response);
// The public URL can be used to directly access the file via HTTP.
var publicUrl = format(
'https://storage.googleapis.com/%s/%s',
bucket.name, blob.name);
res.status(200).send(
{
'success': true,
'publicUrl': publicUrl,
'mediaLink': response.mediaLink
});
});
});
blobStream.end(req.file.buffer);
});
This seems to work, from the standpoint that it does actually set the Content-Type on upload, and that is correctly reflected in the response object as well as the Cloud Storage console. The issue is that some of the links returned as publicUrl cause a file download, and others cause a browser load of the image. Ideally I would like to have both options available, but I am unable to see any difference in the stored files or their metadata.
What am I missing here?
Google Cloud Storage makes no assumptions about the content-type of uploaded objects. If you don't specify, GCS will simply assign a type of "application/octet-stream".
The command-line tool gsutil, however, is smarter, and will attach the right Content-Type to files being uploaded in most cases, JPEGs included.
Now, there are two reasons why your browser is likely to download images rather than display them. First, if the Content-Type is set to "application/octet-stream", most browsers will download the results as a file rather than display them. This was likely happening in your case.
The second reason is if the server responds with a 'Content-Disposition: attachment' header. This doesn't generally happen when you fetch GCS objects from the host "storage.googleapis.com" as you are doing above, but it can if you, for instance, explicitly specified a contentDisposition for the object that you've uploaded.
For this reason I suspect that some of your objects don't have an "image/jpeg" content type. You could go through and set them all with gsutil like so: gsutil -m setmeta 'Content-Type:image/jpeg' gs://myBucketName/**
Related
There is a site that contains data I want to parse through in my application. The JSON file is in a tar.gz. My code issues a request to that site, downloads the tar.gz file, extracts the JSON and then parses the information.
This is how the code looks so far but I have not added it into my backend yet.
const fs = require("fs");
const rp = require("request-promise");
const tar = require("tar");
(async function main() {
try {
const url = "https://statics.koreanbuilds.net/bulk/latest.tar.gz";
const arcName = "latest.tar.gz";
const response = await rp.get({ uri: url, encoding: null });
fs.writeFileSync(arcName, response, { encoding: null });
tar.x({ file: arcName, cwd: ".", sync: true });
let text = fs.readFileSync("latest1.json");
let fullText = JSON.parse(text);
let championsObj = {};
// Following logic that parses the json file
.......
} catch (err) {
console.error(err);
}
})();
I plan on storing my parsed JSON object into MongoDB. I also want to perform the above operation and update the JSON and tar.gz file every 24 hours.
I am worried that these operations have many consequences when deploying this project. This is my first time deploying a Full stack application and I am almost positive that having code that messes with the file structure of the overall project will cause some issues. But I just don't know what exactly I should be worried about and how to tackle it. I believe that there will be a problem with CORS but I am more worried about the application actually working and updating correctly. The entire application is being made with the MERN stack.
When you deploy your code on a VPS, saving and reading from the filesystem is completely fine. When you deploy to PaaS like Heroku, you have to keep in mind that the filesystem is ephemeral which means that you get a fresh new copy on each deploy. Files that are not part of the version control will disappear after a release. You can't rely on the filesystem for storage and you have to use an external service to store images/files (e.g.: AWS S3).
Having said that, your code will work on Heroku because you're saving and reading from the file right away. One thing I'd do is add a date/timestamp to the downloaded file name so you don't get an error on the second run when a file with that name already exists. You could also research the possibility of extracting the archive in memory so you don't have to use the filesystem at all.
Other than that you shouldn't be worried. CORS is not relevant in this context.
I have been using GCS to storage my images and also use the NodeJS package to upload these images to my bucket. I have noticed that if I frequently change an image, it either does one of the following:
It changes
It serves an old image
It doesn't change
This seems to happen pretty randomly despite setting all of the options properly and even cross-referencing that with GCS.
I upload my images like this:
const options = {
destination,
public: true,
resumable: false,
metadata: {
cacheControl: 'no-cache, max-age=0',
},
};
const file = await this.bucket.upload(tempImageLocation, options);
const { bucket, name, generation } = file[0].metadata;
const imageUrl = `https://storage.googleapis.com/${bucket}/${name}`;
I have debated whether to use the base URL you see there or use this one: https://storage.cloud.google.com.
I can't seem to figure out what I am doing wrong and how to always serve a fresh image. I have also tried ?ignoreCache=1 and other query parameters.
As per the official API documentation - accessible here - shows, you should not need the await. This might be affecting your upload sometime. If you want to use the await, you need to have your function to be async in the declaration, as showed in the second example from the documentation. Your code should look like this.
const bucketName = 'Name of a bucket, e.g. my-bucket';
const filename = 'Local file to upload, e.g. ./local/path/to/file.txt';
const {Storage} = require('#google-cloud/storage');
const storage = new Storage();
async function uploadFile() {
// Uploads a local file to the bucket
await storage.bucket(bucketName).upload(filename, {
// Support for HTTP requests made with `Accept-Encoding: gzip`
gzip: true,
// By setting the option `destination`, you can change the name of the
// object you are uploading to a bucket.
metadata: {
// Enable long-lived HTTP caching headers
// Use only if the contents of the file will never change
// (If the contents will change, use cacheControl: 'no-cache')
cacheControl: 'public, max-age=31536000',
},
});
console.log(`${filename} uploaded to ${bucketName}.`);
}
uploadFile().catch(console.error);
While this is untested, it should help you avoiding the issue with not uploading always the images.
Besides that, as explained in the official documentation of Editing Metada, you can change the way that metadata - which includes the cache control - is used and managed by your project. This way, you can change your cache configuration as well.
I also, would like to include the below link for a complete tutorial on how to send images to Cloud Storage with Node.js, in case you want to check a different approach.
Image Upload With Google Cloud Storage and Node.js
Let me know if the information helped you!
u can try change ?ignoreCache=1 to ?ignoreCache=0.
well in my case I have a list of Url and I want to download each and every file from those urls and organise it in firebase storage bucket, my problem is I am unable to make folders in firebase storage bucket through nodejs javascript/typescript.
well firebase storage offers ref() and child method to upload files inside child folder (see this) but firebase only offers those method for firebase client libraries, it is not that we can not use client library in nodejs but they have made some namespaces hidden when you connect firebase client library in nodejs and storage is one of them (see this).
I am happy they have considered frontend and backend separately because of this very reason that front and backend have whole different scenario for security and use cases, so what they have really written to use in nodejs is firebase admin and I cannot see ref and child method in official documentation which they have said is this not any other way to name the file I am uploading nor any method for making folders to go child directories, when I upload a file from my computer it get saved in the bucket root with the same name as the filename it was in my computer, even though I can make folders from firebase console manually but it will not fulfill my requirement for sure there should must be any way to make folders in programmatically.
I also tried using google cloud storage library const {Storage} = require('#google-cloud/storage');
but it turned out firebase admin and gogole cloud library shares the same document and have same interface at least in upload file part.
well I have spent my day (well night too since it is 4:46am) trying different libraries and digging into their documents which I also found little unorganised and lack of code examples.
any help would be appreciated, my code snippet so far is following which is from their doc and uploading file correctly:
import "firebase/firestore"
admin.initializeApp({
credential: admin.credential.cert("./../path-to-service account-cert.json"),
databaseURL: 'gs://bilal-assistant-xxxxx.appspot.com'
});
const quran_bucket = admin.storage().bucket("quran-bucket");
quran_bucket.upload("./my_computer_path/fatiha.mp3", {
gzip: true,
metadata: {
cacheControl: 'public, max-age=31536000',
}
}).then(uploadResponse => {
console.log(` uploaded complete.`);
}).catch((reason: any) => {
console.log("reason: ", reason);
})
All I wanted is to save the audio file in folder bucket, not in bucket root
According to the API documentation, upload() takes an UploadOptions object as the second parameter. You will want to used the documented destination property of that object to specify the name of the file in Storage:
quran_bucket.upload("./my_computer_path/fatiha.mp3", {
destination: 'audio/juz30/fatiha.mp3',
gzip: true,
metadata: {
cacheControl: 'public, max-age=31536000',
}
})
You probably don't want to bother the gzip an mp3, as it's already compressed and won't compress much further.
I'm working with NodeJS and using HAPI to create API for upload and download file. When uploading, I read the file information (filename, mime type and file content) and store it in database. File content is stored as base64 encoded string.
What I want to do is to create API, so when client hits it will be forced to download a file that is constructed based on the stored information using the code below
server.route({
method: 'GET',
path:'/file',
handler: function (request, reply) {
var fileData = // get file content;
var mime = // get file mime-type;
var fileBuffer = new Buffer(fileData, 'base64');
reply(fileBuffer)
.header('Content-disposition', 'attachment; filename=' + fileName)
.header('Content-type', mime).header('Content-length', fileBuffer.length).encoding('binary');
}
})
But this code looks like still not work, if I hit the API it will be loading process forever and no file downloaded. Anybody can give me suggestion on how to do it correctly?
UPDATE
the code is correct and works perfectly. the problem I had before is caused by another reason, incorrect encoding/decoding mechanism.
Check out the inert plugin for hapi which handles files, the repo is here
I'm quite new to node.js and would like to do the following:
user can upload one file
upload should be saved to amazon s3
file information should be saved to a database
script shouldn't be limited to specific file size
As I've never used S3 or done uploads before I might have some
wrong ideas - please correct me, if I'm wrong.
So in my opinion the original file name should be saved into the db and returned for download but the file on S3 should be renamed to my database entry id to prevent overwriting files. Next, should the files be streamed or something? I've never done this but it just seems not to be smart to cache files on the server to then push them to S3, does it?
Thanks for your help!
At first I recommend to look at knox module for NodeJS. It is from quite reliable source. https://github.com/LearnBoost/knox
I write a code below for Express module, but if you do not use it or use another framework, you should still understand basics. Take a look at CAPS_CAPTIONS in the code, you want to change them according to your needs / configuration. Please also read comments to understand pieces of code.
app.post('/YOUR_REQUEST_PATH', function(req, res, next){
var fs = require("fs")
var knox = require("knox")
var s3 = knox.createClient({
key: 'YOUR PUBLIC KEY HERE' // take it from AWS S3 configuration
, secret: 'YOUR SECRET KEY HERE' // take it from AWS S3 configuration
, bucket: 'YOUR BUCKET' // create a bucket on AWS S3 and put the name here. Configure it to your needs beforehand. Allow to upload (in AWS management console) and possibly view/download. This can be made via bucket policies.
})
fs.readFile(req.files.NAME_OF_FILE_FIELD.path, function(err, buf){ // read file submitted from the form on the fly
var s3req = s3.put("/ABSOLUTE/FOLDER/ON/BUCKET/FILE_NAME.EXTENSION", { // configure putting a file. Write an algorithm to name your file
'Content-Length': buf.length
, 'Content-Type': 'FILE_MIME_TYPE'
})
s3req.on('response', function(s3res){ // write code for response
if (200 == s3res.statusCode) {
// play with database here, use s3req and s3res variables here
} else {
// handle errors here
}
})
s3req.end(buf) // execute uploading
})
})