I am making an app which will access a phones camera from the browser. Allow the user to take a picture and then save the picture to a Amazon S3 bucket. I am having trouble making a successful API post. The error message that comes back is just a copy of the very long Body that I am sending.
Here is my server side code
const fs = require('fs');
const express = require('express');
const pool = require('../modules/pool');
const router = express.Router();
const { rejectUnauthenticated } = require('../modules/authentication-middleware');
const aws = require('aws-sdk');
const s3 = new aws.S3();
aws.config.update({
accessKeyId: 'Access_KeyID',
secretAccessKey: 'Access_Key',
region: 'us-east-2',
});
//POST request to Amazon API
router.post('/', (req,res)=>{
let pic = req.body.picture[0]
console.log(pic)
let params = {
Bucket: 'beerphoto',
Key: req.body.filename,
Body : fs.createReadStream(pic),
ContentType: 'image/png',
Expires: 60,
}
s3.upload(params, function(err,data){
if(err) {
console.log('err')
}
if (data) {
console.log("Uploaded in:", data.Location);
}
})
})
module.exports = router;
I am wondering if the issue is with the picture data. Here is what is being passed into the Body,
"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAASQA"
The actual string is much, much, longer (around 26,000 characters). Any help would be appreciated.
Using fs.createReadStream() is not your option. This method creates a stream for data from local file system. However, you already have your picture in RAM, you even have a reference for it with req.body.picture[0]. I think there's no need to leverage streams at all, just use your base64 string, or maybe convert to Buffer instance, according to docs.
Also, you mentioned that your pictures in base64 are at around 26000 characters, which is at least 52 kbs in UTF-8. Imagine a high load for your server with 1000 request with large pictures, your process might run out of memory, because pictures are just too big to fit in it. To avoid that you can let your clients (not server) to upload to S3 directly and securely (without exposing secret keys) with signed urls.
Related
I have a react web app that allows users to record mp3 files in the browser. These mp3 files are saved in an AWS S3 bucket and can be retrieved and loaded back into the react app during the user's next session.
Saving the file works just fine, but when I try to retrieve the file with getObject() and try to create an mp3 blob on the client-side, I get a small, unusable blob:
Here's the journey the recorded mp3 file goes on:
1) Saving to S3
In my Express/Node server, I receive the uploaded mp3 file and save to the S3 bucket:
//SAVE THE COMPLETED AUDIO TO S3
router.post("/", [auth, upload.array('audio', 12)], async (req, res) => {
try {
//get file
const audioFile = req.files[0];
//create object key
const userId = req.user;
const projectId = req.cookies.currentProject;
const { sectionId } = req.body;
const key = `${userId}/${projectId}/${sectionId}.mp3`;
const fileStream = fs.createReadStream(audioFile.path)
const uploadParams = {
Bucket: bucketName,
Body: fileStream,
Key: key,
ContentType: "audio/mp3"
}
const result = await s3.upload(uploadParams).promise();
res.send(result.key);
} catch (error) {
console.error(error);
res.status(500).send();
}
});
As far as I know, there are no problems at this stage. The file ends up in my S3 bucket with "type: mp3" and "Content-Type: audio/mp3".
2) Loading file from S3 Bucket
When the react app is loaded up, an HTTP GET Request is made in my Express/Node server to retrieve the mp3 file from the S3 Bucket
//LOAD A FILE FROM S3
router.get("/:sectionId", auth, async(req, res) => {
try {
//create key from user/project/section IDs
const sectionId = req.params.sectionId;
const userId = req.user;
const projectId = req.cookies.currentProject;
const key = `${userId}/${projectId}/${sectionId}.mp3`;
const downloadParams = {
Key: key,
Bucket: bucketName
}
s3.getObject(downloadParams, function (error, data) {
if (error) {
console.error(error);
res.status(500).send();
}
res.send(data);
});
} catch (error) {
console.error(error);
res.status(500).send();
}
});
The "data" returned here is as such:
3) Making a Blob URL on the client
Finally, in the React client, I try to create an 'audio/mp3' blob from the returned array buffer
const loadAudio = async () => {
const res = await api.loadAudio(activeSection.sectionId);
const blob = new Blob([res.data.Body], {type: 'audio/mp3' });
const url = URL.createObjectURL(blob);
globalDispatch({ type: "setFullAudioURL", payload: url });
}
The created blob is severely undersized and appears to be completely unusable. Downloading the file results in a 'Failed - No file' error.
I've been stuck on this for a couple of days now with no luck. I would seriously appreciate any advice you can give!
Thanks
EDIT 1
Just some additional info here: in the upload parameters, I set the Content-Type as audio/mp3 explicitly. This is because when not set, the Content-Type defaults to 'application/octet-stream'. Either way, I encounter the same issue with the same result.
EDIT 2
At the request of a commenter, here is the res.data available on the client-side after the call is complete:
Based on the output of res.data on the client, there are a couple of things that you'd need to do:
Replace uses of res.data.Body with res.data.Body.data (as the actual data array is in the data attribute of res.data.Body)
Pass a Uint8Array to the Blob constructor, as the existing array is of a larger type, which will create an invalid blob
Putting that together, you would end up replacing:
const blob = new Blob([res.data.Body], {type: 'audio/mp3' });
with:
const blob = new Blob([new Uint8Array(res.data.Body.data)], {type: 'audio/mp3' });
Having said all that, the underlying issue is that the NodeJS server is sending the content over as a JSON encoded serialisation of the response from S3, which is likely overkill for what you are doing. Instead, you can send the Buffer across directly, which would involve, on the server side, replacing:
res.send(data);
with:
res.set('Content-Type', 'audio/mp3');
res.send(data.Body);
and on the client side (likely in the loadAudio method) processing the response as a blob instead of JSON. If using the Fetch API then it could be as simple as:
const blob = await fetch(<URL>).then(x => x.blob());
Your server side code seems alright to me. I'm not super clear about the client-side approach. Do you load this into the blob into the HTML5 Audio player.
I have a few approaches, assuming you're trying to load this into an audio tag in the UI.
<audio controls src="data:audio/mpeg;base64,blahblahblah or html src" />
Assuming that the file you had uploaded to S3 is valid here are two approaches:
Return the data as a base64 string instead of as a buffer directly from S3. You can do this in your server side by returning as
const base64MP3 = data.Body.toString('base64');
You can then pass this in to the MP3 player in the src property and it will play the audio. Prefix it with data:audio/mpeg;base64
Instead of returning the entire MP3 file, have your sectionID method return a presigned S3 URL. Essentially, this is a direct link to the object in S3 that is authorized for say 5 minutes.
Then you should be able to use this URL directly to stream the audio
and set it as the src. Keep in mind that it will expire.
const url = s3.getSignedUrl('getObject', {
Bucket: myBucket,
Key: myKey,
Expires: signedUrlExpireSeconds
});
You stated: "The created blob is severely undersized and appears to be completely unusable"
This appears to me that you have an encoding issue. Once you read the MP3 from the Amazon S3 bucket, you need to encode it properly so it functions in a web page.
I did a similar multimedia use case that involved MP4 and a Java app. That is, i wanted a MP4 obtained from a bucket to play in the web page - as shown in this example web app.
Once I read the byte stream from the S3 bucket, I had to encode it so it would play in a HTML Video tag. Here is a good reference to properly encode a MP3 file.
I am using Node.js API to upload/download profile images. All the profile images are stored in the AWS s3 storage.
How to download the file from s3 to Node.js route and return the file to the client in response. I can't use file hosting because all the routes are protected with token and user may or may not have the access of certain profile images based on some logic.
Although s3 SDK works great but I have used simple stream from AWS to read and send the chuncks of data to the response.
Use getObject() to download the files form s3 bucket.
See the following example :-
var stream = require('stream');
const s3 = require('../config/s3.config.js');
exports.doDownload = (req, res) => {
const s3Client = s3.s3Client;
const params = s3.downloadParams;
params.Key = req.params.filename;
s3Client.getObject(params)
.createReadStream()
.on('error', function(err){
res.status(500).json({error:"Error -> " + err});
}).pipe(res);
}
Hi I would like to store images in amazon s3. I am making a react application with node js and express at the back end. I have a code which is saving the images locally, in images folder as desired. I am using jimp library to convert the images into black and white. What i want is to store these black and white images directly to aws instead of saving to local hdd. I need to do this because in the end the app has to be deployed to heroku, and heroku is not able to read images from local hdd.
Here is the code through which i was able to store images in a particular directory as required.
const input = req.body.input;
google.list({
keyword: input,
num: 15,
detail: true,
})
.then(function (res) {
res.map((data,index)=>{
const url = data.url;
const extension = url.split('.')[url.split('.').length-1]
const foldername=input
Jimp.read(url, function (err, image) {
image.resize(250, 250)
.greyscale()
.write(path.join(__dirname,"../../public/images/"+foldername+"/"+foldername+index+"."+extension));
});
});
})
}).catch(function(err) {
res.send('There was some error')
})
I need to store images in the same path ie., awsbucketname/foldername/foldername.jpg. I tried converting the image to buffer but still i don't understand how to proceed with it. Some one please help me :(
(Disclaimer: I have no practical experience with Jimp!)
It seems like you are on the right track with writing the image to a buffer instead of a local file. Once you have initialized the AWS SDK and instantiated the S3 interface, it should be easy to pass the buffer to the upload function. Something along the lines of:
const s3 = new AWS.S3({ params: { Bucket: 'yourBucketName' } });
// ...
Jimp.read(url, (err, image) => {
const bucketPath = `/${foldername}/${index}.${extension}`;
image.resize(250, 250)
.greyscale()
.getBuffer(Jimp.AUTO).then(buffer => {
s3.upload({ Key: bucketPath, Body: buffer })
.then(() => console.log('yay!'));
});
}
);
This is just a sketch of course, missing error handling etc.
I am a newbie in nodejs and firebase, but i need to upload files. I saw this tutorial but could not grab much. I am totally confused. In this tutorial which is the function to pass the choosed file?
the code is:
const keyFilename = "./my-private-api-key-file.json"; //replace this with api key file
const projectId = "my-project-id-should-go-here" //replace with your project id
const bucketName = `${projectId}.appspot.com`;
const mime = require('mime');
const gcs = require('#google-cloud/storage')({
projectId,
keyFilename
});
const bucket = gcs.bucket(bucketName);
const filePath = `./package.json`;
const uploadTo = `subfolder/package.json`;
const fileMime = mime.lookup(filePath);
bucket.upload(filePath, {
destination: uploadTo,
public: true,
metadata: {
contentType: fileMime,
cacheControl: "public, max-age=300"
}
}, function (err, file) {
if (err) {
console.log(err);
return;
}
console.log(createPublicFileURL(uploadTo));
});
function createPublicFileURL(storageName) {
return `http://storage.googleapis.com/${bucketName}/${encodeURIComponent(storageName)}`;
}
I want to upload file when the users selects a file. Can anyone provide me something to start ? thanks.
The purpose of the tutorial you're following is for handling files AFTER they have been uploaded to your server. This is useful for services (like Heroku or OpenShift, as the tutorial mentions) with ephemeral file systems that won't hold uploaded content permanently.
You should try following this tutorial instead, which explains how to build a front-end user-interface with AJAX. Once you've done that, then you might need to follow the tutorial linked in your question to get those files into permanent storage.
Hope this helped.
use multiparty library it helped me a lot in my current nodejs project
I'm trying to take an image and upload it to an Amazon S3 bucket using Node.js. In the end, I want to be able to push the image up to S3, and then be able to access that S3 URL and see the image in a browser. I'm using a Curl query to do an HTTP POST request with the image as the body.
curl -kvX POST --data-binary "#test.jpg" 'http://localhost:3031/upload/image'
Then on the Node.js side, I do this:
exports.pushImage = function(req, res) {
var image = new Buffer(req.body);
var s3bucket = new AWS.S3();
s3bucket.createBucket(function() {
var params = {Bucket: 'My/bucket', Key: 'test.jpg', Body: image};
// Put the object into the bucket.
s3bucket.putObject(params, function(err) {
if (err) {
res.writeHead(403, {'Content-Type':'text/plain'});
res.write("Error uploading data");
res.end()
} else {
res.writeHead(200, {'Content-Type':'text/plain'});
res.write("Success");
res.end()
}
});
});
};
My file is 0 bytes, as shown on Amazon S3. How do I make it so that I can use Node.js to push the binary file up to S3? What am I doing wrong with binary data and buffers?
UPDATE:
I found out what I needed to do. The curl query is the first thing that should be changed. This is the working one:
curl -kvX POST -F foobar=#my_image_name.jpg 'http://localhost:3031/upload/image'
Then, I added a line to convert to a Stream. This is the working code:
exports.pushImage = function(req, res) {
var image = new Buffer(req.body);
var s3bucket = new AWS.S3();
s3bucket.createBucket(function() {
var bodyStream = fs.createReadStream(req.files.foobar.path);
var params = {Bucket: 'My/bucket', Key: 'test.jpg', Body: bodyStream};
// Put the object into the bucket.
s3bucket.putObject(params, function(err) {
if (err) {
res.writeHead(403, {'Content-Type':'text/plain'});
res.write("Error uploading data");
res.end()
} else {
res.writeHead(200, {'Content-Type':'text/plain'});
res.write("Success");
res.end()
}
});
});
};
So, in order to upload a file to an API endpoint (using Node.js and Express) and have the API push that file to Amazon S3, first you need to perform a POST request with the "files" field populated. The file ends up on the API side, where it resides probably in some tmp directory. Amazon's S3 putObject method requires a Stream, so you need to create a read stream by giving the 'fs' module the path where the uploaded file exists.
I don't know if this is the proper way to upload data, but it works. Does anyone know if there is a way to POST binary data inside the request body and have the API send that to S3? I don't quite know what the difference is between a multi-part upload vs a standard POST to body.
I believe you need to pass the content-length in the header as documented on the S3 docs: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html
After spending quite a bit of time working on pushing assets to S3, I ended up using the AwsSum library with excellent results in production:
https://github.com/awssum/awssum-amazon-s3/
(See the documentation on setting your AWS credentials)
Example:
var fs = require('fs');
var bucket_name = 'your-bucket name'; // AwsSum also has the API for this if you need to create the buckets
var img_path = 'path_to_file';
var filename = 'your_new_filename';
// using stat to get the size to set contentLength
fs.stat(img_path, function(err, file_info) {
var bodyStream = fs.createReadStream( img_path );
var params = {
BucketName : bucket_name,
ObjectName : filename,
ContentLength : file_info.size,
Body : bodyStream
};
s3.putObject(params, function(err, data) {
if(err) //handle
var aws_url = 'https://s3.amazonaws.com/' + DEFAULT_BUCKET + '/' + filename;
});
});
UPDATE
So, if you are using something like Express or Connect which are built on Formidable, then you don't have access to the file stream as Formidable writes files to disk. So depending on how you upload it on the client side the image will either be in req.body or req.files. In my case, I use Express and on the client side, I post other data as well so the image has it's own parameter and is accessed as req.files.img_data. However you access it, that param is what you pass in as img_path in the above example.
If you need to / want to Stream the file that is trickier, though certainly possible and if you aren't manipulating the image you may want to look at taking a CORS approach and uploading directly to S3 as discussed here: Stream that user uploads directly to Amazon s3