So my express server will be handling requests of all kinds, most of them are very lightweight. But theres a request a user can make to archive their files, which requires me to upload 100s, maybe even 1000s of files to AWS S3. The files are each 1-10MB.
I am doing this through streams. So I would first create a read stream from the source of the file, then use that to stream the file to S3. And repeat for every single file.
It would look something like this:
const uploadFile = (stream, path) => {
let params = { Bucket, Key: path, Body: stream };
S3.upload(params, (err) => {
if (err) {
console.log("Error uploading file: " + path);
console.log(err);
} else {
console.log("Uploaded: " + path);
}
});
};
How would this affect the server's ability to handle other requests? The other requests all involve the server fetching some data from the DB or setting some data to the DB.
Related
Hey everyone so I am trying to make this type of request in nodejs. I assume you can do it with multer but there is one major catch I don't want to download the file or upload it from a form I want to pull it directly from s3, get the object and send it as a file along with the other data to my route. Is it possible to do that?
Yes it's completely possible. Assuming you know your way around the aws-sdk, you can create a method for retrieving the file and use this method to get the data in your route and do whatever you please with them.
Example: (Helper Method)
getDataFromS3(filename, bucket, callback) {
var params = {
Bucket: bucket,
Key: filename
};
s3.getObject(params, function(err, data) {
if (err) {
callback(true, err.stack); // an error occurred
}
else {
callback(false, data); //success in retrieving data.
}
});
}
Your Route:
app.post('/something', (req, res) => {
var s3Object = getDataFromS3('filename', 'bucket', (err, file) => {
if(err) {
return res.json({ message: 'File retrieval failed' });
}
var routeProperties = {};
routeProperties.file = file;
routeProperties.someOtherdata = req.body.someOtherData;
return res.json({routeProperties});
});
});
Of course, the code might not be totally correct. But this is an approach that you can use to get what you want. Hope this helps.
There are two ways that I see here, you can either:
pipe this request to user, it means that you still download it and pass it through but you don't save it anywhere, just stream it through your backend.
There is a very similar question asked here: Streaming file from S3 with Express including information on length and filetype
I'm just gonna copy & paste code snippet just for the reference how it could be done
function sendResponseStream(req, res){
const s3 = new AWS.S3();
s3.getObject({Bucket: myBucket, Key: myFile})
.createReadStream()
.pipe(res);
}
if the file gets too big for you to easily handle, create presigned URL in S3 and send it through. User then can download the file himself straight from S3 for a limited amount of time, more details here: https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html
This question already has answers here:
Nodejs POST request multipart/form-data
(5 answers)
Closed 5 years ago.
File uploading using multer is not happening
My code:
File read from html and pass to external url
router.post('/fileUpload',function (req,res) {
request({
uri: http//example.com/upload, // url of other server
method: "POST",
form: {
"name":req.body.name,
"image":? //image from html - no idea how to pass the req.files here
}
}, function (error, response, body) {
------------
------------
}
});
other server url : /example.com/upload
This is the code to upload image using multer
app.post('/upload',function(req,res){
var storage = multer.diskStorage({
destination: function (req, file, callback) {
callback(null, 'uploads');
},
filename: function (req, file, callback) {
callback(null, file.fieldname + '-' + Date.now());
}
});
var upload = multer({ storage : storage }).array('productImage');
upload(req,res,function(err) {
if(err) {
return res.json({'success':0,'result':{},'errorMessage':'Unknown Error'});
}
return res.json({'success':1,'result':{},'errorMessage':''});
});
});
Create readStream from file uploaded and pipe it to the another server.check this link https://www.npmjs.com/package/request, you will easily get this done.
The simple answer would be to create a read stream from the uploaded file and pipe it to the second server, like so:
fs.createReadStream(req.files[0].path).pipe(request.post('http//example.com/upload'))
However, there are many ways to make this work. The most efficient of which is to use a binary stream from the initial upload all the way to the second server. You want to avoid using your first server as a storage for all of the uploads.
Here's how I would do it instead:
Use jQuery file upload on client side upload
(Or any other library/approach to upload the raw binary stream)
$('#fileupload').fileupload({
url: 'https://server1.com/upload'
})
Use Formidable (or other library like multer) to handle upload server-side
(This will allow you to read the binary stream in parts, and handle each part as it comes)
app.post('/upload',function(req,res){
var form = new formidable.IncomingForm();
form.parse(req);
form.onPart = function(part) {
fs.createReadStream(part).pipe( request.post('http//example.com/upload'))
}
}
When each part of the binary upload is received, you can to stream the binary directly to the second server via pipe() to avoid having to store it on the first server whatsoever.
The key to making this work is to look at the file upload and transfer as a stream of binary bits. When the user uploads a file (req.body) you want to create a read stream from the file, and pipe the binary over the request.post().
I have a server in Node.js and say I have a POST request that uploads a multipart file to my server and then I upload it to AWS S3.
The issue is, with multer, I have to save the file to disk first.
If I deploy my server onto EC2 then how will file uploading work as it won't have a destination to temporarily store the file?
Thanks!
You can use streams with busboy. I don't have experience with the AWS Node SDK, but here's the general idea:
req.busboy.on('file', function (fieldname, file, filename) {
const params = { Bucket: 'bucket', Key: 'key', Body: file };
s3.upload(params, (err, data) => {
console.log(err, data);
});
});
Im struggling to find material on this
I have a rest API, written in node.js, that uses mongoDB.
I want users to be able to upload images (profile pictures) and have them saved on the server (in mongoDB).
A few questions, Ive seen it is recommended to use GridFS, is this the best solution?
How do i send these files? Ive seen res.sendFile, but again is this the best solution?
If anyone has any material they can link me I would be appreciative
thanks
You won't be able to get the file object on the server directly. To get file object on the server, use connect-multiparty middleware. This will allow you to access the file on the server.
var multipart = require('connect-multiparty');
var multipartmiddleware = multipart();
var mv = require('mv');
var path = require('path');
app.post("/URL",multipartmiddleware,function(req,res){
var uploadedImage = req.files.file;
for (var i = 0; i < uploadedImage.length; i++) {
var tempPath = uploadedImage[i].path;
var targetPath = path.join(__dirname ,"../../../img/Ads/" + i + uploadedImage[i].name);
mv(tempPath, targetPath, function (err) {
if (err) { throw err; }
});
}
})
Use file system
Generally in any database you store the image location in the data as a string that tells the application where the image is stored on the file system.
Unless your database needs to be portable as a single unit, the storing of images inside of the database as binary objects generally adds unnecessary size and complexity to your database.
-Michael Stearne
In MongoDB, use GridFS for storing files larger than 16 MB.
- Mongo Documentation
Therefore unless your images will be over 16 MB, you should either store the file on a CDN (preferable) or the server's own file system and save its URL to user's document on the database.
Local file system implementation
This method uses Busboy to parse the photo upload.
in relevant html file:
<input type="file" title="Choose a file to upload" accept="image/*" autofocus="1">
Handler function for your photo upload route in server file (you will need to fill in the variables that apply to you and require the necessary modules):
function photoUploadHandlerFunction (req, res) {
var busboy = new Busboy({ headers: req.headers })
busboy.on('file', function (fieldname, file, filename, encoding, mimetype) {
const saveToDir = path.join(__dirname, uploadsPath, user.id)
const saveToFile = path.join(saveToDir, filename)
const pathToFile = path.join(uploadsPath, user.id, filename)
const writeStream = fs.createWriteStream(saveToFile)
createDirIfNotExist(saveToDir)
.then(pipeUploadToDisk(file, writeStream))
.then(findUserAndUpdateProfilePic(user, pathToFile))
.catch((err) => {
res.writeHead(500)
res.end(`Server broke its promise ${err}`)
})
})
busboy.on('finish', function () {
res.writeHead(200, { 'Connection': 'close' })
res.end("That's all folks!")
})
return req.pipe(busboy)
}
Where the promise functions createDirIfNotExist and pipeUploadToDisk could look like this:
function createDirIfNotExist (directory, callback) {
return new Promise(function (resolve, reject) {
fs.stat(directory, function (err, stats) {
// Check if error defined and the error code is "not exists"
if (err) {
if (err.code === 'ENOENT') {
fs.mkdir(directory, (err) => {
if (err) reject(err)
resolve('made folder')
})
} else {
// just in case there was a different error:
reject(err)
}
} else {
resolve('folder already existed')
}
})
})
}
function pipeUploadToDisk (file, writeStream) {
return new Promise((resolve, reject) => {
const fileWriteStream = file.pipe(writeStream)
fileWriteStream.on('finish', function () {
resolve('file written to file system')
})
fileWriteStream.on('error', function () {
reject('write to file system failed')
})
})
}
To answer your question 'How do I send these files?', I would need to know where to (MongoDB, to the client...). If you mean to the client, you could serve the static folder where they are saved.
If you still want to learn about implementing GridFs tutorialspoint have a good tutorial
More material
Good tutorial on handling form uploads
Tutorial using the node-formidable module
If you're using the mongoose odm you can use the mongoose-crate module and send the file wherever for storage.
Also, this is a good case for shared object storage like AWS S3 or Azure blob storage. If you are running a distributed setup in something like AWS, you usually don't want to store photos on the local server.
Store the url or key name in the database that points to the S3 object. This also integrates with CloudFront CDN pretty easily.
As suggested before. MultiPart for the actual upload.
I'm trying to take an image and upload it to an Amazon S3 bucket using Node.js. In the end, I want to be able to push the image up to S3, and then be able to access that S3 URL and see the image in a browser. I'm using a Curl query to do an HTTP POST request with the image as the body.
curl -kvX POST --data-binary "#test.jpg" 'http://localhost:3031/upload/image'
Then on the Node.js side, I do this:
exports.pushImage = function(req, res) {
var image = new Buffer(req.body);
var s3bucket = new AWS.S3();
s3bucket.createBucket(function() {
var params = {Bucket: 'My/bucket', Key: 'test.jpg', Body: image};
// Put the object into the bucket.
s3bucket.putObject(params, function(err) {
if (err) {
res.writeHead(403, {'Content-Type':'text/plain'});
res.write("Error uploading data");
res.end()
} else {
res.writeHead(200, {'Content-Type':'text/plain'});
res.write("Success");
res.end()
}
});
});
};
My file is 0 bytes, as shown on Amazon S3. How do I make it so that I can use Node.js to push the binary file up to S3? What am I doing wrong with binary data and buffers?
UPDATE:
I found out what I needed to do. The curl query is the first thing that should be changed. This is the working one:
curl -kvX POST -F foobar=#my_image_name.jpg 'http://localhost:3031/upload/image'
Then, I added a line to convert to a Stream. This is the working code:
exports.pushImage = function(req, res) {
var image = new Buffer(req.body);
var s3bucket = new AWS.S3();
s3bucket.createBucket(function() {
var bodyStream = fs.createReadStream(req.files.foobar.path);
var params = {Bucket: 'My/bucket', Key: 'test.jpg', Body: bodyStream};
// Put the object into the bucket.
s3bucket.putObject(params, function(err) {
if (err) {
res.writeHead(403, {'Content-Type':'text/plain'});
res.write("Error uploading data");
res.end()
} else {
res.writeHead(200, {'Content-Type':'text/plain'});
res.write("Success");
res.end()
}
});
});
};
So, in order to upload a file to an API endpoint (using Node.js and Express) and have the API push that file to Amazon S3, first you need to perform a POST request with the "files" field populated. The file ends up on the API side, where it resides probably in some tmp directory. Amazon's S3 putObject method requires a Stream, so you need to create a read stream by giving the 'fs' module the path where the uploaded file exists.
I don't know if this is the proper way to upload data, but it works. Does anyone know if there is a way to POST binary data inside the request body and have the API send that to S3? I don't quite know what the difference is between a multi-part upload vs a standard POST to body.
I believe you need to pass the content-length in the header as documented on the S3 docs: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html
After spending quite a bit of time working on pushing assets to S3, I ended up using the AwsSum library with excellent results in production:
https://github.com/awssum/awssum-amazon-s3/
(See the documentation on setting your AWS credentials)
Example:
var fs = require('fs');
var bucket_name = 'your-bucket name'; // AwsSum also has the API for this if you need to create the buckets
var img_path = 'path_to_file';
var filename = 'your_new_filename';
// using stat to get the size to set contentLength
fs.stat(img_path, function(err, file_info) {
var bodyStream = fs.createReadStream( img_path );
var params = {
BucketName : bucket_name,
ObjectName : filename,
ContentLength : file_info.size,
Body : bodyStream
};
s3.putObject(params, function(err, data) {
if(err) //handle
var aws_url = 'https://s3.amazonaws.com/' + DEFAULT_BUCKET + '/' + filename;
});
});
UPDATE
So, if you are using something like Express or Connect which are built on Formidable, then you don't have access to the file stream as Formidable writes files to disk. So depending on how you upload it on the client side the image will either be in req.body or req.files. In my case, I use Express and on the client side, I post other data as well so the image has it's own parameter and is accessed as req.files.img_data. However you access it, that param is what you pass in as img_path in the above example.
If you need to / want to Stream the file that is trickier, though certainly possible and if you aren't manipulating the image you may want to look at taking a CORS approach and uploading directly to S3 as discussed here: Stream that user uploads directly to Amazon s3