I have a server in Node.js and say I have a POST request that uploads a multipart file to my server and then I upload it to AWS S3.
The issue is, with multer, I have to save the file to disk first.
If I deploy my server onto EC2 then how will file uploading work as it won't have a destination to temporarily store the file?
Thanks!
You can use streams with busboy. I don't have experience with the AWS Node SDK, but here's the general idea:
req.busboy.on('file', function (fieldname, file, filename) {
const params = { Bucket: 'bucket', Key: 'key', Body: file };
s3.upload(params, (err, data) => {
console.log(err, data);
});
});
Related
So my express server will be handling requests of all kinds, most of them are very lightweight. But theres a request a user can make to archive their files, which requires me to upload 100s, maybe even 1000s of files to AWS S3. The files are each 1-10MB.
I am doing this through streams. So I would first create a read stream from the source of the file, then use that to stream the file to S3. And repeat for every single file.
It would look something like this:
const uploadFile = (stream, path) => {
let params = { Bucket, Key: path, Body: stream };
S3.upload(params, (err) => {
if (err) {
console.log("Error uploading file: " + path);
console.log(err);
} else {
console.log("Uploaded: " + path);
}
});
};
How would this affect the server's ability to handle other requests? The other requests all involve the server fetching some data from the DB or setting some data to the DB.
I'm using a service called opentok which store video on their cloud and give me a callback url when file are ready so that i can download it and store on my cloud provider.
We use gcloud where we work and i need to download the file, then store it on my gcloud bucket with a firebase cloud functions.
Here is my code :
const archiveFile = await axios.get(
'https://sample-videos.com/video701/mp4/720/big_buck_bunny_720p_2mb.mp4'
);
console.log('file downloaded from opentokCloud !');
fs.writeFile('archive.mp4', archiveFile, err => {
if (err) throw err;
// success case, the file was saved
console.log('File Saved in container');
});
await firebaseBucket.upload('archive.mp4', {
gzip: true,
// destination: `archivedStreams/${archiveInfo.id}/archive.mp4`,
destination: 'test/lapin.mp4',
metadata: {
cacheControl: 'no-cache',
},
});
I tried to put directly the file downloaded in the upload() but it does not work i have to provide a String (path of my file)
How can i have the path of my downloaded file in the cloud function ? Is it still in the RAM of my container or in a cache folder ?
As you can see i tried to write with FS but i have no write access in the container of the cloud function
Thanks in advance to the community
If someone is looking for this in the futur here is how i solved it :
With Firebase cloud functions you can write temporary files in /tmp (see here for more information https://cloud.google.com/functions/docs/concepts/exec#file_system)
I solved the problem by using node-fetch package andwriteStream function of Node.JS :
const fetch = require('node-fetch');
const fs = require('fs');
await fetch(archiveInfo.url).then(res => {
console.log('start writing data');
const dest = fs.createWriteStream('/tmp/archive.mp4');
res.body.pipe(dest);
//Listen when the writing of the file is done
dest.on('finish', async () => {
console.log('start uploading in gcloud !');
await firebaseBucket.upload('/tmp/archive.mp4', {
gzip: true,
destination: `{pathToFileInBucket}/archive.mp4`,
metadata: {
cacheControl: 'no-cache',
},
});
console.log('uploading finished');
});
});
firebaseBucket is my gcloud bucket already configured elsewhere, define your own bucket using #google-cloud/storage
As my function where triggered by link, don't forget to put a response and catch errors to avoid timed out cloud functions (running for nothing, billed for nothing :D)
i want to
1-choose an image from my filesystem and upload it to server/local
2- get its url back using node js service . i managed to do step 1 and now i want to get the image url instead of getting the success message in res.end
here is my code
app.post("/api/Upload", function(req, res) {
upload(req, res, function(err) {
if (err) {
return res.end("Something went wrong!");
}
return res.end("File uploaded sucessfully!.");
});
});
i'm using multer to upload the image.
You can do something like this, using AWS S3 and it returns the url of the image uploaded
const AWS = require('aws-sdk')
AWS.config.update({
accessKeyId: <AWS_ACCESS_KEY>,
secretAccessKey: <AWS_SECRET>
})
const uploadImage = file => {
const replaceFile = file.data_uri.replace(/^data:image\/\w+;base64,/, '')
const buf = new Buffer(replaceFile, 'base64')
const s3 = new AWS.S3()
s3.upload({
Bucket: <YOUR_BUCKET>,
Key: <NAME_TO_SAVE>,
Body: buf,
ACL: 'public-read'
}, (err, data) => {
if (err) throw err;
return data.Location; // this is the URL
})
}
also you can check this express generator, which has the route to upload images to AWS S3 https://www.npmjs.com/package/speedbe
I am assuming that you are saving the image on the server file system and not a Storage solution like AWS S3 or Google Cloud Storage, where you get the url after upload.
Since, you are storing it on the filesystem, you can rename the file with a unique identifier like uuid or something else.
Then you can make a GET route and request that ID in query or path parameter and then read the file having that ID as the name and send it back.
I am using aws-sdk to upload files on Amazon S3. It is working fine and uploading files, but my problem is; it changed file name after uploaded to the server. For example, if I upload sample.jpg, and it renamed to something like b4c743c8a2332525.jpg. Here is my code.
AWS.config.update({
accessKeyId: key,
secretAccessKey: secret
});
var fileStream = fs.createReadStream(path);
fileStream.on('error', function (err) {
if (err) { throw err; }
});
fileStream.on('open', function () {
var s3 = new AWS.S3();
s3.putObject({
Bucket: bucket,
Key: directory + file,
Body: fileStream
}, function (err) {
if (err)
res.send(err);
fs.unlinkSync(path);
});
});
Is it normal to change file name after uploaded files to S3 server, or is there any options to upload the same file name? Thank you.
Neither S3 nor the AWS SDK pick arbitrary file names for things you upload. The names are set by your own code.
Check the value of directory + file when you set it as the S3 object key. You may be uploading 'sample.jpg' from your browser (so the file is called sample.jpg locally on your disk), but the temporary file name that node.js uses to identify the file on it's disk may be using a hash like b4c743c8a2332525.jpg.
I'm trying to take an image and upload it to an Amazon S3 bucket using Node.js. In the end, I want to be able to push the image up to S3, and then be able to access that S3 URL and see the image in a browser. I'm using a Curl query to do an HTTP POST request with the image as the body.
curl -kvX POST --data-binary "#test.jpg" 'http://localhost:3031/upload/image'
Then on the Node.js side, I do this:
exports.pushImage = function(req, res) {
var image = new Buffer(req.body);
var s3bucket = new AWS.S3();
s3bucket.createBucket(function() {
var params = {Bucket: 'My/bucket', Key: 'test.jpg', Body: image};
// Put the object into the bucket.
s3bucket.putObject(params, function(err) {
if (err) {
res.writeHead(403, {'Content-Type':'text/plain'});
res.write("Error uploading data");
res.end()
} else {
res.writeHead(200, {'Content-Type':'text/plain'});
res.write("Success");
res.end()
}
});
});
};
My file is 0 bytes, as shown on Amazon S3. How do I make it so that I can use Node.js to push the binary file up to S3? What am I doing wrong with binary data and buffers?
UPDATE:
I found out what I needed to do. The curl query is the first thing that should be changed. This is the working one:
curl -kvX POST -F foobar=#my_image_name.jpg 'http://localhost:3031/upload/image'
Then, I added a line to convert to a Stream. This is the working code:
exports.pushImage = function(req, res) {
var image = new Buffer(req.body);
var s3bucket = new AWS.S3();
s3bucket.createBucket(function() {
var bodyStream = fs.createReadStream(req.files.foobar.path);
var params = {Bucket: 'My/bucket', Key: 'test.jpg', Body: bodyStream};
// Put the object into the bucket.
s3bucket.putObject(params, function(err) {
if (err) {
res.writeHead(403, {'Content-Type':'text/plain'});
res.write("Error uploading data");
res.end()
} else {
res.writeHead(200, {'Content-Type':'text/plain'});
res.write("Success");
res.end()
}
});
});
};
So, in order to upload a file to an API endpoint (using Node.js and Express) and have the API push that file to Amazon S3, first you need to perform a POST request with the "files" field populated. The file ends up on the API side, where it resides probably in some tmp directory. Amazon's S3 putObject method requires a Stream, so you need to create a read stream by giving the 'fs' module the path where the uploaded file exists.
I don't know if this is the proper way to upload data, but it works. Does anyone know if there is a way to POST binary data inside the request body and have the API send that to S3? I don't quite know what the difference is between a multi-part upload vs a standard POST to body.
I believe you need to pass the content-length in the header as documented on the S3 docs: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html
After spending quite a bit of time working on pushing assets to S3, I ended up using the AwsSum library with excellent results in production:
https://github.com/awssum/awssum-amazon-s3/
(See the documentation on setting your AWS credentials)
Example:
var fs = require('fs');
var bucket_name = 'your-bucket name'; // AwsSum also has the API for this if you need to create the buckets
var img_path = 'path_to_file';
var filename = 'your_new_filename';
// using stat to get the size to set contentLength
fs.stat(img_path, function(err, file_info) {
var bodyStream = fs.createReadStream( img_path );
var params = {
BucketName : bucket_name,
ObjectName : filename,
ContentLength : file_info.size,
Body : bodyStream
};
s3.putObject(params, function(err, data) {
if(err) //handle
var aws_url = 'https://s3.amazonaws.com/' + DEFAULT_BUCKET + '/' + filename;
});
});
UPDATE
So, if you are using something like Express or Connect which are built on Formidable, then you don't have access to the file stream as Formidable writes files to disk. So depending on how you upload it on the client side the image will either be in req.body or req.files. In my case, I use Express and on the client side, I post other data as well so the image has it's own parameter and is accessed as req.files.img_data. However you access it, that param is what you pass in as img_path in the above example.
If you need to / want to Stream the file that is trickier, though certainly possible and if you aren't manipulating the image you may want to look at taking a CORS approach and uploading directly to S3 as discussed here: Stream that user uploads directly to Amazon s3