I am using sails.js and am trying to stream files from the Amazon s3 server directly to the client.
To connect to S3, I use the s3 Module : https://www.npmjs.org/package/s3
This module provides capabilities like client.downloadFile(params) and client.downloadBuffer(s3Params).
My current code looks like the following:
var view = client.downloadBuffer(params);
view.on('error', function(err) {
cb({success: 0, message: 'Could not open file.'}, null);
});
view.on('end', function(buffer) {
cb(null, buffer);
});
I catch this buffer in a controller using:
User.showImage( params , function (err, buffer){
// this is where I can get the buffer
});
Is it possible to stream this data as an image file (using buffer.pipe(res) doesn't work of course). But is there something similar to completely avoid saving file to server disk first?
The other option client.downloadFile(params) requires a local path (i.e. a server path in our case)
The GitHub issue contains the "official" answer to this question: https://github.com/andrewrk/node-s3-client/issues/53
Related
Im trying to look over the ways AWS has to offer in order to upload files to s3. When I looked into their docs it confused the hell of out me. Looking up to the various resources I came to know a bit more resources like s3.upload vs s3.putObject and others realised there are physical limitations in API gateway and using lambda function to upload a file.
Particularly in case of uploading large file like 1-100 GB AWS suggests multiple methods to upload file to s3. Amongst them are createMultipartUpload, ManagedUpload, getSignedURL and tons of other.
So my Question is:
What is the best and the easiest way to upload large files to s3 where I also can cancel the upload process. The multipart upload seems to tedious.
There's no Best Way to upload file to S3
It depends on what you want especially what are the sizes of the object that you want to upload.
putObject - Ideal for objects which are under 20MB
Presigned Url - Allows you to bypass API Gateway and Put object under 5GB to s3 bucket
Multipart Upload - Allows you to upload files in chunks which means you can continue your upload even the connection went off temporarily. The maximum file size you can upload via this method is 5TB.
Use Streams to upload to S3, this way the Node.JS server doesn't take too much of the resources.
const AWS = require('aws-sdk');
const S3 = new AWS.S3();
const stream = require('stream');
function upload(S3) {
let pass = new stream.PassThrough();
let params = {
Bucket: BUCKET,
Key: KEY,
Body: pass
};
S3.upload(params, function (error, data) {
console.error(error);
console.info(data);
});
return pass;
}
const readStream = fs.createReadStream('/path/to/your/file');
readStream.pipe(upload(S3));
This is via streaming local file, the stream can be from request as well.
If want to listen to the progress can use ManagedUpload
const manager = S3.upload(params);
manager.on('httpUploadProgress', (progress) => {
console.log('progress', progress)
// { loaded: 6472, total: 345486, part: 3, key: 'large-file.dat' }
});
Im following rackspace example on file upload to cloud storage on docs. It is works but the upload are taking too loong. Like really longer! No matter what region do I use 17,kb file take more than 3sec is this actual behaviour of rackspace cloud, they really slow?
Im using rackspace with nodejs, whith the help of pacakage named pkgcloud.
// taken from pkgcloud docs
var readStream = fs.createReadStream('a-file.txt');
var writeStream = client.upload({
container: 'a-container',
remote: 'remote-file-name.txt'
});
writeStream.on('error', function(err) {
// handle your error case
});
writeStream.on('success', function(file) {
// success, file will be a File model
});
readStream.pipe(writeStream);
The purpose here is, I do image processing on the backend then I send back CDN URL to the user, but a user cannot wait too long 2MB took forever to upload -- timeout and held my server until crash since the stream aren't finished yet
i'm developing a software that provide direct download to my S3 bucket in Node JS.
I have develop the following code:
const stream = s3.getObject(getParams).createReadStream();
stream.on("error", (err) => {
res.send(err);
});
res.setHeader("Content-disposition", `attachment; filename=${path.basename(req.params[ 0 ])}`);
stream.pipe(res);
and it works!
But if the file doesn't exist, the download will start and this is wrong (it contains the error message).
Where is the best place to put the Header? in a way to set it only if the file exist?
I have try several events like on("data") or on("end") but it doesn't works as expected.
After that, i need to close the stream after this operation?
I am using Meteor project to upload images to the google cloud from iOS device and download the same images to iOS device.
I don't get any issues while uploading the images, it gets stored in google storage bucket. The issue I am facing is while downloading the images, I am using below code which downloads the images on server's path.
bucket.file(srcFilename).download(options);
I want to download and store the images on iOS device.
When I tried to read the file using createReadStream, my app get stuck without any progress (Not getting any callback).
bucket.file(srcFilename).createReadStream()
.on('error', function(err) {
console.log("error");
})
.on('response', function(response) {
// Server connected and responded with the specified status and
console.log("response");
})
.on('end', function() {
// The file is fully downloaded.
console.log("The file is fully downloaded.");
})
I hope that I am not missing anything while downloading the images to iOS device. I looked but unable to find any other option to do the same.
Any help in this regard is really appreciated as I am stuck at this very point.
I used below code to get the file from google cloud and download the chunks which I converted to binary format. Then I used this binary format to display image and store in my local storage from client side.
var chunkNew = new Buffer('');
bucket.file(srcFilename).createReadStream().on('data', function (chunk) {
chunkNew = Buffer.concat([chunkNew,chunk]);
})
.on('end', function() {
// The file is fully downloaded.
callback(null, chunkNew.toString('base64'));
})
More information can be found in this link http://codewinds.com/blog/2013-08-04-nodejs-readable-streams.html which uses data chunk to show the image as array buffer.
My issue is to get image uploading to amazon working.
I was looking for a solution that doesnt save the file on the server and then upload it to Amazon.
Googling I found pkgcloud and on the README.md it says:
Special attention has been paid so that methods are streams and
pipe-capable.
Can someone explain what that means and if it is what I am looking for?
Yupp, that means you've found the right kind of s3 library.
What it means is that this library exposes "streams". Here is the API that defines a stream: http://nodejs.org/api/stream.html
Using node's stream interface, you can pipe any readable stream (in this case the POST's body) to any writable stream (in this case the S3 upload).
Here is an example of how to pipe a file upload directly to another kind of library that supports streams: How to handle POSTed files in Express.js without doing a disk write
EDIT: Here is an example
var pkgcloud = require('pkgcloud'),
fs = require('fs');
var s3client = pkgcloud.storage.createClient({ /* ... */ });
app.post('/upload', function(req, res) {
var s3upload = s3client.upload({
container: 'a-container',
remote: 'remote-file-name.txt'
})
// pipe the image data directly to S3
req.pipe(s3upload);
});
EDIT: To finish answering the questions that came up in the chat:
req.end() will automatically call s3upload.end() thanks to stream magic. If the OP wants to do anything else on req's end, he can do so easily: req.on('end', res.send("done!"))