I'am working with NodeJs/ express to create a POST api endpoint to upload files to google cloud storage and returning a public URL like this :
https://storage.googleapis.com/[BUCKET_NAME]/[OBJECT_NAME]
When the upload is done, I get the URL and when I open it , the file is downloaded directly (image, pdf etc...)
Is there a way to view and open it in the browser ?
here is my upload function :
const uploadImage = (file) => new Promise((resolve, reject) => {
const { originalname, buffer } = file
const blob = bucket.file(originalname.replace(/ /g, "_"))
const blobStream = blob.createWriteStream({
resumable: false
})
blobStream.on('finish', () => {
const publicUrl = format(
`https://storage.googleapis.com/${bucket.name}/${blob.name}`
)
resolve(publicUrl)
})
.on('error', () => {
reject(`Unable to upload image, something went wrong`)
})
.end(buffer)
})
Thanks
If the URL looks like the code above, https://storage.googleapis.com/bucketName/objectName, a browser should be able to view it directly, so long as a few conditions are in place:
The object's contentType is set appropriately. If you don't specify a type, the default is application/octet-stream, and web browsers will probably decide to just download such an object rather than displaying it in some form.
The object's metadata does not override the contentDisposition. It's possible to force objects to be downloaded as attachments by setting that property to something like attachment; filename=foo.txt.
The object is either publicly viewable or appropriate credentials are passed in the GET request. This is not the default setting. When you upload the object, you'll need to explicitly note that the ACL should allow the group allUsers read permission. Alternately, you could set the default object ACL property of the bucket to include that permission.
In your case, the object is downloading successfully, so it's not the ACL issue, and if you don't know about the contentDisposition setting, then it's probably problem #1. Make sure you specify a reasonable content type for the object.
Example:
const blobStream = blob.createWriteStream({
resumable: false
contentType: "text/html"
})
Related
Goal: Try to download a pdf file from Amazon S3 to my local machine via a NodeJS/VueJS application without creating a file on the server's filesystem.
Server: NodeJs(v 18.9.0) Express (4.17.1)
Middleware function that retrieves the file from S3 and converts the stream into a base64 string and sends that string to the client:
const filename = 'lets_go_to_the_snackbar.pdf';
const s3 = new AWS.S3(some access parameters);
const params = {
Bucket: do_not_kick_this_bucket,
Key: `yellowbrickroad/${filename}`
}
try {
const data = await s3
.getObject(params)
.promise();
const byte_string = Buffer.from(data.Body).toString('base64');
res.send(byte_string);
} catch (err) {
console.log(err);
}
Client: VueJS( v 3.2.33)
Function in component receives byte string via an axios (v 0.26.1) GET call to the server. The code to download is as follows:
getPdfContent: async function (filename) {
const resp = await AxiosService.getPdf(filename) // Get request to server made here.
const uriContent = `data:application/pdf;base64,${resp.data}`
const link = document.createElement('a')
link.href = uriContent
link.download = filename
document.body.appendChild(link) // Also tried omitting this line along with...
link.click()
link.remove() // ...omitting this line
}
Expected Result(s):
Browser opens a window to allow a directory to be selected as the file's destination.
Directory Selected.
File is downloaded.
Ice cream and mooncakes are served.
Actual Results(s):
Browser opens a window to allow a directory to be selected as the file's destination
Directory Selected.
Receive Failed - Network Error message.
Lots of crying...
Browser: Chrome (Version 105.0.5195.125 (Official Build) (x86_64))
Read somewhere that Chrome will balk at files larger than 4MB, so I checked the S3 bucket and according to Amazon S3 the file size is a svelte 41.7KB.
After doing some reading, a possible solution was presented that I tried to implement. It involved making a change to the VueJs getPdfContent function as follows:
getPdfContent: async function (filename) {
const resp = await AxiosService.getPdf(filename) // Get request to server made here.
/**** This is the line that was changed ****/
const uriContent = window.URL.createObjectURL(new Blob([resp.data], { type: 'application/pdf' } ))
const link = document.createElement('a')
link.href = uriContent
link.download = filename
document.body.appendChild(link) // Also tried omitting this line along with...
link.click()
link.remove() // ...omitting this line
}
Actual Results(s) for updated code:
Browser opens a window to allow a directory to be selected as the file's destination
Directory Selected.
PDF file downloaded.
Trying to open the file produces the message:
The file “lets_go_to_the_snackbar.pdf” could not be opened.
It may be damaged or use a file format that Preview doesn’t recognize.
I am able to download the file directly from S3 using the AWS S3 console with no problems opening the file.
I've read through similar postings and tried implementing their solutions, but found no joy. I would be highly appreciative if someone can
Give me an idea of where I am going off the path towards reaching the goal
Point me towards the correct path.
Thank you in advance for your help.
After doing some more research I found the problem was how I was returning the data from the server back to the client. I did not need to modify the data received from the S3 service.
Server Code:
let filename = req.params.filename;
const params = {
Bucket: do_not_kick_this_bucket,
Key: `yellowbrickroad/${filename}`
}
try {
const data = await s3
.getObject(params)
.promise();
/* Here I did not modify the information returned */
res.send(data.Body);
res.end();
} catch (err) {
console.log(err);
}
On the client side my VueJS component receives a Blob object as the response
Client Code:
async getFile (filename) {
let response = await AuthenticationService.downloadFile(filename)
const uriContent = window.URL.createObjectURL(new Blob([response.data]))
const link = document.createElement('a')
link.setAttribute('href', uriContent)
link.setAttribute('download', filename)
document.body.appendChild(link)
link.click()
link.remove()
}
In the end the goal was achieved; a file on S3 can be downloaded directly to a user's local machine without the application storing a file on the server.
I would like to mention Sunpun Sandaruwan's answer which gave me the final clue I needed to reach my goal.
I was practicing on this tutorial
https://www.youtube.com/watch?v=NZElg91l_ms&t=1234s
It is working absolutely like a charm for me but the thing is I am storing images of products I am storing them in bucket and lets say I upload 4 images they all are uploaded.
but when I am displaying them i got access denied error as I am displaying the list and repeated request are maybe detecting it as a spam
This is how i am trying to fetch them on my react app
//rest of data is from mysql datbase (product name,price)
//100+ products
{ products.map((row)=>{
<div className="product-hero"><img src=`http://localhost:3909/images/${row.imgurl}`</div>
<div className="text-center">{row.productName}</div>
})
}
as it fetch 100+ products from db and 100 images from aws it fails
Sorry for such detailed question but in short how can i fetch all product images from my bucket
Note I am aware that i can get only one image per call so how can I get all images one by one in my scenario
//download code in my app.js
const { uploadFile, getFileStream } = require('./s3')
const app = express()
app.get('/images/:key', (req, res) => {
console.log(req.params)
const key = req.params.key
const readStream = getFileStream(key)
readStream.pipe(res)
})
//s3 file
// uploads a file to s3
function uploadFile(file) {
const fileStream = fs.createReadStream(file.path)
const uploadParams = {
Bucket: bucketName,
Body: fileStream,
Key: file.filename
}
return s3.upload(uploadParams).promise()
}
exports.uploadFile = uploadFile
// downloads a file from s3
function getFileStream(fileKey) {
const downloadParams = {
Key: fileKey,
Bucket: bucketName
}
return s3.getObject(downloadParams).createReadStream()
}
exports.getFileStream = getFileStream
It appears that your code is sending image requests to your back-end, which retrieves the objects from Amazon S3 and then serves the images in response to the request.
A much better method would be to have the URLs in the HTML page point directly to the images stored in Amazon S3. This would be highly scalable and will reduce the load on your web server.
This would require the images to be public so that the user's web browser can retrieve the images. The easiest way to do this would be to add a Bucket Policy that grants GetObject access to all users.
Alternatively, if you do not wish to make the bucket public, you can instead generate Amazon S3 pre-signed URLs, which are time-limited URLs that provides temporary access to a private object. Your back-end can calculate the pre-signed URL with a couple of lines of code, and the user's web browser will then be able to retrieve private objects from S3 for display on the page.
I did sililar S3 image handling while I handle my blog's image upload functionality, but I did not use getFileStream() to upload my image.
Because nothing should be done until the image file is fully processed, I used fs.readFile(path, callback) instead to read the data.
My way will generate Buffer Data, but AWS S3 is smart enough to know to intercept this as image. (I have only added suffix in my filename, I don't know how to apply image headers...)
This is my part of code for reference:
fs.readFile(imgPath, (err, data) => {
if (err) { throw err }
// Once file is read, upload to AWS S3
const objectParams = {
Bucket: 'yuyuichiu-personal',
Key: req.file.filename,
Body: data
}
S3.putObject(objectParams, (err, data) => {
// store image link and read image with link
}
}
I am creating a pdf using JSPDF on server-side, in NodeJS. Once done, I want to create a new folder for the user in Google Drive, upload the pdf to said folder, and also send it to the client-side (browser) for the user to view.
There are two problems that I'm encountering. Firstly, if I send the pdf in the response -via pdf.output()- the images don't display correctly. They are distorted, as though each row of pixels is offset by some amount. A vertical line "|" instead renders as a diagonal "\". An example is shown below.
Before
After
My workaround for this was to instead save it to the filesystem using doc.save() and then send it to the browser using fs.readFileSync(filepath).
However, I've discovered that when running remotely, I don't have file permissions to be saving the pdf and reading it. And after some research and tinkering, I'm thinking that I cannot change these permissions. This is the error I get:
Error: EROFS: read-only file system, open './temp/output.pdf'
at Object.openSync (fs.js:443:3)
at Object.writeFileSync (fs.js:1194:35)
at Object.v.save (/workspace/node_modules/jspdf/dist/jspdf.node.min.js:86:50626)
etc...
So I have this JSPDF object, and I believe I need to either, alter the permissions to allow writing/reading or take the jspdf object or, I guess, change it's format to one accepted by Google drive, such as a stream or buffer object?
The link below leads me to think these permissions can't be altered since it states: "These files are available in a read-only directory".
https://cloud.google.com/functions/docs/concepts/exec#file_system
I also have no idea 'where' the server filesystem is, or how to access it. Thus, I think the best course of action is to look at sending the pdf in different formats.
I've checked jsPDF documentation for types that pdf.output() can return. These include string, arraybuffer, window, blob, jsPDF.
https://rawgit.com/MrRio/jsPDF/master/docs/jsPDF.html#output
My simplified code is as follows:
const express = require('express');
const fs = require('fs');
const app = express();
const { jsPDF } = require('jspdf');
const credentials = require(credentialsFilepath);
const scopes = [scopes in here];
const auth = new google.auth.JWT(
credentials.client_email, null,
credentials.private_key, scopes
);
const drive = google.drive({version: 'v3', auth});
//=========================================================================
app.post('/submit', (req, res) => {
var pdf = new jsPDF();
// Set font, fontsize. Added some text, etc.
pdf.text('blah blah', 10, 10);
// Add image (signature) from canvas, which is passed as a dataURL
pdf.addImage(img, 'JPEG', 10, 10, 50, 20);
pdf.save('./temp/output.pdf');
drive.files.create({
resource: folderMetaData,
fields: 'id'
})
.then(response => {
// Store pdf in newly created folder
var fileMetaData = {
'name': 'filename.pdf',
'parents': [response.data.id],
};
var media = {
mimeType: 'application/pdf',
body: fs.createReadStream('./temp/output.pdf'),
};
drive.files.create({
resource: fileMetaData,
media: media,
fields: 'id'
}, function(err, file) {
if(err){
console.error('Error:', err);
}else{
// I have considered piping 'file' back in the response here but can't figure out how
console.log('File uploaded');
}
});
})
.catch(error => {
console.error('Error:', error);
});
// Finally, I attempt to send the pdf to client/browser
res.setHeader('Content-Type', 'application/pdf');
res.send(fs.readFileSync('./temp/output.pdf'));
})
Edit: After some more searching, I've found a similar question which explains that the fs module is for reading/writing to local filestore.
EROFS error when executing a File Write function in Firebase
I eventually came to a solution after some further reading. I'm not sure who this will be useful for, but...
Turns out the Firebase filesystem only has 1 directory which allows you to write to (the rest are read-only). This directory is named tmp and I accessed it using the tmp node module [installed with: npm i tmp], since trying to manually reference the path with pdf.save('./tmp/output.pdf') didn't work.
So the only changes to my code were to add in the lines:
var tmp = require('tmp');
var tmpPath = tmp.tmpNameSync();
and then replacing all the instances of './temp/output.pdf' with tmpPath
I have a website running with node.js, with the backend running on Firebase Functions. I want to store a bunch of JSON to Firebase Storage. The below snippet works just fine when I'm running on localhost, but when I upload it to Firebase functions, it says Error: EROFS: read-only file system, open 'export-stock-trades.json. Anyone know how to get around this?
fs.writeFile(fileNameToReadWrite, JSON.stringify(jsonObjToUploadAsFile), function(err){
bucket.upload(fileNameToReadWrite, {
destination: destinationPath,
});
res.send({success: true});
});
I can't tell for sure, since much of the context of your function is missing, but it looks like you function is attempting to write a file to local disk first (fs.writeFile), then upload it (bucket.upload).
On Cloud Functions, code you write only has write access to /tmp,
which is os.tmpdir() in node. Read more about that in the
documentation:
The only writeable part of the filesystem is the /tmp directory, which
you can use to store temporary files in a function instance. This is a
local disk mount point known as a "tmpfs" volume in which data written
to the volume is stored in memory. Note that it will consume memory
resources provisioned for the function.
This is probably what's causing your code to fail.
Incidentally, if the data you want to upload is in memory, you don't have to write it to a file first as you're doing now. You can instead use file.save() for that.
Another way I feel this could work is to convert the JSON file into a buffer and then perform an action like this (the code snippet below). I wrote an article on how you can do this using Google Cloud Storage but it works fine with Firebase storage. The only thing you need to change is the "service-account-key.json" file.
The link to the article can be found here: Link to article on medium
const util = require('util')
const gc = require('./config/')
const bucket = gc.bucket('all-mighti') // should be your bucket name
/**
*
* #param { File } object file object that will be uploaded
* #description - This function does the following
* - It uploads a file to the image bucket on Google Cloud
* - It accepts an object as an argument with the
* "originalname" and "buffer" as keys
*/
export const uploadImage = (file) => new Promise((resolve, reject) => {
const { originalname, buffer } = file
const blob = bucket.file(originalname.replace(/ /g, "_"))
const blobStream = blob.createWriteStream({
resumable: false
})
blobStream.on('finish', () => {
const publicUrl = format(
`https://storage.googleapis.com/${bucket.name}/${blob.name}`
)
resolve(publicUrl)
})
.on('error', () => {
reject(`Unable to upload image, something went wrong`)
})
.end(buffer)
})
I am downloading a file through puppeteer into my directory. I need to upload this file to an s3 bucket so I need to pick up the file name. But the problem is, this file name has a time stamp that changes every time so I can't keep a hard coded name. So is there a way around this to get a constant name every time (even if the old file is replaced), or how to rename the file being downloaded?
I thought of using node's fs.rename() function but that would again require the current file name.
I want a constant file name to hard code and then upload into the s3 bucket.
await page._client.send('Page.setDownloadBehavior', {behavior: 'allow', downloadPath: './xml'}); // This sets the directory
await page.keyboard.press('Tab');
await page.keyboard.press('Enter'); // This downloads an XML file.
You have two options:
Monitor the requests/responses to log the name of the file and rename it via Node.js
Use the Chrome DevTools Protocol to modify the response header
Option 1: Monitor the requests / response
This is the most straight-forward way to do it. Monitor all responses and in case you notice the response that is being downloaded, use the name to rename it locally via fs.rename.
Code Sample
const path = require('path');
// ...
page.on('response', response => {
const url = response.request().url();
const contentType = response.headers()['content-type'];
if (/* URL and/or contentType matches pattern */) {
const fileName = path.basename(response.request().url());
// handle and rename file name (after making sure it's downloaded)
}
});
The code listens to all responses and wait for a specific pattern (e.g. contentType === 'application/pdf'). Then it takes the file name from the request. Depending on your use case, you might want to check the Content-Disposition header in addition. After that, you have to wait until the file is downloaded (e.g. file is present and file size does not change) and then you can rename it.
Option 2: Use the Chrome DevTools Protocol to modify the response header
I'm 99% sure, that this is possible. You need to intercept the response which is currently not supported by puppeteer itself. But as the Chrome DevTools Protocol is supporting this functionality, you can use it using the low-level protocol.
The idea is to intercept the response and change the Content-Disposition header to your desired file name.
Here is the idea:
Use chrome-remote-interface or a CDP Session to activate Network.requestIntercepted
Listen for Network.requestIntercepted events
Send Network.getResponseBodyForInterception to receive the body of the response
Modify the body and add (or change) the Content-Disposition header to include your filename
Call Network.continueInterceptedRequest with your modified response
Your file should then be save with your modified file name. Check out this comment on github for a code sample. As I already explained it is a rather sophisticated approach as long as puppeteer does not support modifying responses.
You can save the file using GUID as the filename and rename it when the download is completed.
const puppeteer = require('puppeteer');
const fs = require('fs');
const path = require('path');
const downloadFolder = path.resolve('./DOWNLOAD-FOLDER-HERE');
// Act like a dictionary storing the filename for each file with guid
let guids = {};
const browser = await puppeteer.launch({
headless: false
});
let client = await browser.target().createCDPSession();
await client.send('Browser.setDownloadBehavior', {
behavior: 'allowAndName', //allow downloading file and save the file using guid as the filename
downloadPath: downloadFolder, // specify the download folder
eventsEnabled: true //set true to emit download events (e.g. Browser.downloadWillBegin and Browser.downloadProgress)
});
client.on('Browser.downloadWillBegin', async (event) => {
//some logic here to determine the filename
//the event provides event.suggestedFilename and event.url
guids[event.guid] = 'FILENAME.pdf';
});
client.on('Browser.downloadProgress', async (event) => {
// when the file has been downloaded, locate the file by guid and rename it
if(event.state === 'completed') {
fs.renameSync(path.resolve(downloadFolder, event.guid), path.resolve(downloadFolder, guids[event.guid]));
}
});