I finally have file uploads working through Node and the AWS SDK...there's just one thing that's missing now that I haven't been able to crack yet, which is that I need to get the URL to the newly uploaded file on S3 and save it to my database.
const s3 = new AWS.S3();
const fileContent = Buffer.from(req.files.listPDF.data, 'binary');
const params = {
Bucket: 'my_bucket',
Key: filename,
Body: fileContent
};
s3.upload(params, function(err, data){
if (err) {
throw err;
}
});
I'm guessing it's promise-related, but I haven't had success with "await" yet. The data parameter in the function has a "Location" attribute, which I need. Originally, I was trying to set a previously-declared var to it. However, it wasn't doing anything since the upload was not yet completed. If anyone's grappled with this and cracked the code, I'd really appreciate your thoughts!
Related
Hey everyone so I am trying to make this type of request in nodejs. I assume you can do it with multer but there is one major catch I don't want to download the file or upload it from a form I want to pull it directly from s3, get the object and send it as a file along with the other data to my route. Is it possible to do that?
Yes it's completely possible. Assuming you know your way around the aws-sdk, you can create a method for retrieving the file and use this method to get the data in your route and do whatever you please with them.
Example: (Helper Method)
getDataFromS3(filename, bucket, callback) {
var params = {
Bucket: bucket,
Key: filename
};
s3.getObject(params, function(err, data) {
if (err) {
callback(true, err.stack); // an error occurred
}
else {
callback(false, data); //success in retrieving data.
}
});
}
Your Route:
app.post('/something', (req, res) => {
var s3Object = getDataFromS3('filename', 'bucket', (err, file) => {
if(err) {
return res.json({ message: 'File retrieval failed' });
}
var routeProperties = {};
routeProperties.file = file;
routeProperties.someOtherdata = req.body.someOtherData;
return res.json({routeProperties});
});
});
Of course, the code might not be totally correct. But this is an approach that you can use to get what you want. Hope this helps.
There are two ways that I see here, you can either:
pipe this request to user, it means that you still download it and pass it through but you don't save it anywhere, just stream it through your backend.
There is a very similar question asked here: Streaming file from S3 with Express including information on length and filetype
I'm just gonna copy & paste code snippet just for the reference how it could be done
function sendResponseStream(req, res){
const s3 = new AWS.S3();
s3.getObject({Bucket: myBucket, Key: myFile})
.createReadStream()
.pipe(res);
}
if the file gets too big for you to easily handle, create presigned URL in S3 and send it through. User then can download the file himself straight from S3 for a limited amount of time, more details here: https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html
i want to
1-choose an image from my filesystem and upload it to server/local
2- get its url back using node js service . i managed to do step 1 and now i want to get the image url instead of getting the success message in res.end
here is my code
app.post("/api/Upload", function(req, res) {
upload(req, res, function(err) {
if (err) {
return res.end("Something went wrong!");
}
return res.end("File uploaded sucessfully!.");
});
});
i'm using multer to upload the image.
You can do something like this, using AWS S3 and it returns the url of the image uploaded
const AWS = require('aws-sdk')
AWS.config.update({
accessKeyId: <AWS_ACCESS_KEY>,
secretAccessKey: <AWS_SECRET>
})
const uploadImage = file => {
const replaceFile = file.data_uri.replace(/^data:image\/\w+;base64,/, '')
const buf = new Buffer(replaceFile, 'base64')
const s3 = new AWS.S3()
s3.upload({
Bucket: <YOUR_BUCKET>,
Key: <NAME_TO_SAVE>,
Body: buf,
ACL: 'public-read'
}, (err, data) => {
if (err) throw err;
return data.Location; // this is the URL
})
}
also you can check this express generator, which has the route to upload images to AWS S3 https://www.npmjs.com/package/speedbe
I am assuming that you are saving the image on the server file system and not a Storage solution like AWS S3 or Google Cloud Storage, where you get the url after upload.
Since, you are storing it on the filesystem, you can rename the file with a unique identifier like uuid or something else.
Then you can make a GET route and request that ID in query or path parameter and then read the file having that ID as the name and send it back.
So in my routes I have a post which lets me send a multipart post, which then gets saved to a database and sends the pictures attached in the post request to amazon s3.
Here's my code for the route
exports.post = function(req, res){
var pictureNames = [];
var s3Bucket = new AWS.S3({params: {Bucket: 'anonybox'}});
for(key in req.files){
tp = req.files[key].path;
fn = req.files[key].name;
ftype = req.files[key].type;
pictureUrls.push(fn);
fs.readFile(tp, function(err, fileBuffer){
var params = {
Key: fn,
Body: fileBuffer,
ACL: 'public-read',
ContentType: ftype
};
s3Bucket.putObject(params, function(err, data){
if(err){
console.log("error" + err);
}else{
console.log("worked, data: "+JSON.stringify(data));
}
});
});
}
messageObject = {
message: req.body.message,
staytime: req.body.staytime,
picturenames: pictureNames
};
var postMessage = new MessageModel(messageObject);
postMessage.save(function(err, doc){
if(err || !doc){
throw 'Error';
}else{
console.log("created");
console.log(doc);
// res.json(doc);
}
});
When I try to send two images in the post request In the console the putObject function returns:
worked, data: {"ETag":"\"24c1d19724ca10f40bc633aa29315931\""}
worked, data: {"ETag":"\"4201e2a997779c5595dc35925e954191\""}
But only one of the files shows up in my bucket on s3, the last file in the req.files object.
The problem was not with the aws-sdk It was that the for loop kept running before the first file was read in fs.readfile, the solution is detailed here
Asynchronously reading and caching multiple files in nodejs
I had a similar issue with two file uploads, I'm just mentioning it for posterity in case someone has a similiar issue in the future: my image uploads were not embedded in a for-loop, but just sequential calls to s3.putObject(). The first image would show up just fine on page refresh, but the second image upload was appearing to "fail silently" without working. The answer, it turns out, for some reason the first file upload would update the cache, but the second image upload would not update the cache, thus appearing to not work (even though the actual file transfer to s3 was actually working). If you are having an issue, try clearing your cache and see if the fresh upload is actually working but your browser isn't showing it properly.
I'm using AWS SDK for Node.js to create a folder or key on s3. I searched on google, but I got nothing. Does anybody know how can I create a folder under my bucket with AWS SDK for Node.js?
and how can you check if this folder exists in your bucket already?
if you use console.aws.amazon.com, you can create a folder in your bucket easily. it seems I didn't figure it out how to create it with AWS SDK for Node.js?
S3 is not your typical file system. It's an object store. It has buckets and objects. Buckets are used to store objects, and objects comprise data (basically a file) and metadata (information about the file). When compared to a traditional file system, it's more natural to think of an S3 bucket as a drive rather than as a folder.
You don't need to pre-create a folder structure in an S3 bucket. You can simply put an object with the key cars/ford/focus.png even if cars/ford/ does not exist.
It's valuable to understand what happens at the API level in this case:
the putObject call will create an object at cars/ford/focus.png but it will not create anything representing the intermediate folder structure of cars/ or cars/ford/.
the actual folder structure does not exist, but is implied through delimiter=/ when you call listObjects, returning folders in CommonPrefixes and files in Contents.
you will not be able to test for the ford sub-folder using headObject because cars/ford/ does not actually exist (it is not an object). Instead you have 2 options to see if it (logically) exists:
call listObjects with prefix=cars/ford/ and find it in Contents
call listObjects with prefix=cars/, delimiter=/ and find it in CommonPrefixes
It is possible to create an S3 object that represents a folder, if you really want to. The AWS S3 console does this, for example. To create myfolder in a bucket named mybucket, you can issue a putObject call with bucket=mybucket, key=myfolder/, and size 0. Note the trailing forward slash.
Here's an example of creating a folder-like object using the awscli:
aws s3api put-object --bucket mybucket --key cars/ --content-length 0
In this case:
the folder is actually a zero-sized object whose key ends in /. Note that if you leave off the trailing / then you will get a zero-sized object that appears to be a file rather than a folder.
you are now able to test for the presence of cars/ in mybucket by issuing a headObject call with bucket=mybucket and key=cars/.
Finally, note that your folder delimiter can be anything you like, for example +, because it is simply part of the key and is not actually a folder separator (there are no folders). You can vary your folder delimiter from listObjects call to call if you like.
The code from #user2837831 doesn't seem to work anymore, probably with the new version of javascript sdk. So I am adding here the version of code that I am using to create a folder inside a bucket using node.js. This works with the 2.1.31 sdk. What is important is the '/' at the end of the Key value in params - using that it thinks you are trying to create a folder and not a file.
var AWS = require('aws-sdk');
AWS.config.region = 'us-east-1';
var s3Client = new AWS.S3();
var params = { Bucket: 'your_bucket_goes_here', Key: 'folderInBucket/', ACL: 'public-read', Body:'body does not matter' };
s3Client.upload(params, function (err, data) {
if (err) {
console.log("Error creating the folder: ", err);
} else {
console.log("Successfully created a folder on S3");
}
});
This is really straightforward you can do it by using the following, just remember the trailing slash.
var AWS = require("aws-sdk");
var s3 = new AWS.S3();
var params = {
Bucket: "mybucket",
Key: "mykey/"
};
s3.putObject(params).promise();
I find that we do not need an explicit directory creation call anymore.
Just the following works for me and automatically creates a directory hierarchy as I need.
var userFolder = 'your_bucket_name' + '/' + variable-with-dir-1-name + '/' + variable-with-dir-2-name;
// IMPORTANT : No trailing '/' at the end of the last directory name
AWS.config.region = 'us-east-1';
AWS.config.update({
accessKeyId: 'YOUR_KEY_HERE',
secretAccessKey: 'your_secret_access_key_here'
});
var bucket = new AWS.S3({
params: {
Bucket: userFolder
}
});
var contentToPost = {
Key: <<your_filename_here>>,
Body: <<your_file_here>>,
ContentEncoding: 'base64',
ContentType: <<your_file_content_type>>,
ServerSideEncryption: 'AES256'
};
bucket.putObject(contentToPost, function (error, data) {
if (error) {
console.log("Error in posting Content [" + error + "]");
return false;
} /* end if error */
else {
console.log("Successfully posted Content");
} /* end else error */
})
.on('httpUploadProgress',function (progress) {
// Log Progress Information
console.log(Math.round(progress.loaded / progress.total * 100) + '% done');
});
In console, the link generated first would be the bucket created path and second would be the folder structure.
var AWS = require("aws-sdk");
var path = require('path')
// Set the region
AWS.config.update({
region: "us-east-2",
accessKeyId: "your aws acces id ",
secretAccessKey: "your secret access key"
});
s3 = new AWS.S3();
var bucketParams = {
Bucket: "imageurrllll",
ACL: "public-read"
};
s3.createBucket(bucketParams, function(err, data) {
if (err) {
console.log("Error", err);
} else {
console.log("Success", data.Location);
var folder_name = 'root_folder'
//this is for local folder data path
var filePath = "./public/stylesheets/user.png"
//var child_folder='child'
var date = Date.now()
var imgData = `${folder_name}_${date}/` +
path.basename(filePath);
var params = {
Bucket: 'imageurrllll',
Body: '', //here you can give image data url from your local directory
Key: imgData,
ACL: 'public-read'
};
//in this section we are creating the folder structre
s3.upload(params, async function(err, aws_uploaded_url) {
//handle error
if (err) {
console.log("Error", err);
}
//success
else {
console.log("Data Uploaded in:", aws_uploaded_url.Location)
}
})
}
});
I'm trying to upload a PDF to an S3 bucket using the Knox library, but I keep getting 505 errors and the PDFs won't save. My code:
// all of this works well
var knox = require('knox');
var client = knox.createClient(require('../path/to/config.js').knox);
client.putFile('tmp/file', '/prefix/key',
function(err, res) {
if (err) {
console.log("Error PUTing file in S3:", err);
}
console.log("S3 RESPONSE:", res.statusCode); // returns 505
}
);
Anyone have any insight into what I'm doing wrong? I've also tried setting my own headers using client.put(..), but I got the same 505 response.
2 Possible reasons.
1) If this is your complete code, then you forgot to enter the key,secret and bucket.
var client = knox.createClient({
key: '<api-key-here>'
, secret: '<secret-here>'
, bucket: 'learnboost'
});
2) There is a space in the file name that you are trying to upload.
This isn't an answer per se, and I'm still unsure about the 505 response above, but the AWS SDK that Amazon puts out works great if anyone is having similar issues with Knox. The above just becomes:
var aws = require('aws-sdk');
aws.config.loadFromPath('./path/to/config.json');
var s3 = new aws.S3();
var params = { Bucket: 'your-bucket',
Key: 'your-key',
Body: fs.readFileSync('/path/to/file.pdf') };
s3.putObject(params, function(err, data) {
if (err) {
console.log("Error PUTing file:", err);
}
console.log("S3 RESPONSE:", data);
});