I am following this tutorial wherein the programmer used this code:
await s3
.upload({ Bucket: bucket, Key: target_filename, Body: file_stream })
.promise();
Now, I understand that the method above would use the initialized variables file_stream, bucket, and target_filename (which he didn't bother typing out in his tutorial).
But the tutorial is hard to follow since (for what I know) the Key parameter inside the upload is the actual directory of the file to be re-uploaded back to S3.
This is confusing because at the file_stream variable, another Key parameter exists inside the method getObject().
So, is the filename inside the getObject() method should be the same as target_filename of the upload() method? and can you initialize the variables mentioned just to make it clearer for this question? Thank you.
No, the filename inside the getObject() method may not be the same as the target_filename in upload(). Let's look at a concrete example. Suppose you have a photo.zip file stored on S3 and its key is a/b/photo.zip, and you want to unzip it and reupload it to c/d/photo.jpg assuming that the photo.zip only contains one file. Then, the filename should be a/b/photo.zip, and the target_filename should be c/d/photo.jpg. As you can see, they are clearly different.
Related
I want to know how to invoke a lambda function using boto3 library with large payload. As of now I am able to invoke it with payload less than 6 mb.
Also I want to know what is the maximum limit for the payload.
Once the above issue is fixed...I have another doubt...
How should I pass this payload in the invoke function..
Earlier I was doing it as below :
lambda_payload = open('fileName.txt','r').read()
lambda_client.invoke( FunctionName='##FName', InvocationType='Request Response', Payload=lambda_payload)
# arn copied is in the below format :
# arn:aws:s3:::dev-abc/fileName.txt
Now what should be my new payload..
The invocation payload of a lambda can only be 6MB when invoked synchronously or 256KB when invoked asynchronously. An easy workaround for this is to upload your payload to S3 and pass the S3 object location as payload to your lambda. Your lambda can then read or stream the contents of the S3 object.
You could add the S3 URI, S3 object ARN or simply the name of the bucket and the name of the object as string values to the invocation payload. You can then use boto3 inside your lambda function to read out the contents of that file.
If you need a larger payload in order to execute an upload, have a look at pre-signing S3 URLs. This would allow you to return a URL that can be used to upload directly to an S3 location.
I need to move a file in AWS s3 bucket to another location, example:
From: http://aws.xxxxx/xxxx/locationA/file.png
To: http://aws.xxxxx/xxxx/locationB/file.png
I've looked over the documentation: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html, but haven't found any mention of either moving nor updating the file (I'm thinking I could update the file Key path...).
So far, it seems I need to copy the file then remove the old one? Is there a more straightforward way of doing it?
My current code which copies then removes old file:
function moveFileInAws(fromLocation, toLocation, callback) {
awsSdk.copyObject({
Bucket: BUCKET_NAME,
ACL: 'public-read',
CopySource: fromLocation,
Key: toLocation
}, (err, data) => {
if (err) {
console.log(err)
return callback("Couldn't copy files in directory")
}
// callback()
awsSdk.deleteObject({ Key: fromLocation }, (err, data) => {
if (err) {
console.log("Couldn't delete files in directory")
console.log(err)
return callback("Couldn't delete files in directory")
}
callback()
})
})
}
Based on the answer here: AWS S3 - Move an object to a different folder in which user #Michael-sqlbot comments:
That's because S3, itself, doesn't have an atomic move or rename operation... so there isn't really an alternative
And the docs here: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-s3-objects.html#copy-object (for the Java SDK, but seems to be useful here) which notes:
You can use copyObject with deleteObject to move or rename an object, by first copying the object to a new name (you can use the same bucket as both the source and destination) and then deleting the object from its old location.
It sounds like S3's infrastructure simply doesn't support moving or renaming in a single operation. You must copy, then delete.
Amazon S3 doesn’t provide an API to move or rename an object from one bucket to another in a single step.
As in your example, you can use copyObject with deleteObject to move or rename an object, by first copying the object to a new name (you can use the same bucket as both the source and destination) and then deleting the object from its old location.
For more information see Performing Operations on Amazon S3 Objects
I'm not at all familiar w/ the JavaScript SDK that you're using, but using aws cli, there is:
aws s3 mv s3://bucket/folder/file s3://bucket/folder2/file
that seems it would do what you want. Not sure about JavaScript SDK.
I am trying to convert S3 video file to audio file through Lambda function. Whenever video files are uploaded into an S3 bucket I have to generate an audio file and save it back to S3 bucket by triggering the AWS Lambda function. I can convert the video file to audio in local. ( Convert video to an audio file using FFMPEG). But I am wondering, how to do this conversion part in Lambda function every time the video file is uploaded into an S3 bucket. I have no idea how to do this AWS Lambda function. Please share your suggestions.
Sample code:
var ffmpeg = require('fluent-ffmpeg');
/**
* input - string, path of input file
* output - string, path of output file
* callback - function, node-style callback fn (error, result)
*/
function convert(input, output, callback) {
ffmpeg(input)
.output(output)
.on('end', function() {
console.log('conversion ended');
callback(null);
}).on('error', function(err){
console.log('error: ', e.code, e.msg);
callback(err);
}).run();
}
convert('./df.mp4', './output.mp3', function(err){
if(!err) {
console.log('conversion complete');
//...
}
});
Thanks,
You just need to set up an event on s3 bucket - put object - to trigger lambda function (you will get access to the description of the object uploaded to that S3 bucket through the first parameter of the lambda function).
If you can convert the video file to audio on your local machine, using some external libraries, then you need to create a zip file containing your lambda function (in the root of the zip file) as well as the dependencies.
This is pretty simple in case of Node. Create a new folder, run npm init, install needed modules, create index.js file where you put your Node code. Zip all the contents of this folder (not the folder itself). When you create new lambda function, choose to upload this zip file.
If you are wondering about how to programatically communicate with AWS resources and manipulate them, then check aws-sdk which you can import as a module and use it for that purpose.
So basically what you will need to inside of your lambda function is to parse event argument (the first parameter) to obtain bucket and key of the uploaded object. Then you will call s3.getObject method to get the data. Process the data with your custom logic. Call s3.putObject to store the newly transformed data to new S3 location.
Lambda has access to its own local file system, if your code needs to store some data there. You just need to specify absolute path to the file, such as /tmp/output.mp3. To retrieve it, you can use fs module. Then, you can continue with s3.putObject.
I'm trying to save a remote image file into a database, but I'm having some issues with it since I've never done it before.
I need to download the image and pass it along (with node-request) with a few other properties to another node api that saves it into a mysql database (using sequelize). I've managed to get some data to save, but when I download it manually and try to open it, it's not really usable and no image shows up.
I've tried a few things: getting the image with node-request, converting it to a base64 string (read about that somewhere) and passing it along in a json payload, but that didn't work. Tried sending it as a multipart, but that didn't work either. Haven't worked with streams/buffers/multipart all that much before and never in node. I've tried looking into node-request pipes, but I couldn't really figure out how possibly apply them to this context.
Here's what I currently have (it's a part es6 class so there's no 'function' keywords; also, request is promisified):
function getImageData(imageUrl) {
return request({
url: imageUrl,
encoding: null,
json: false
});
}
function createEntry(entry) {
return getImageData(entry.image)
.then((imageData) => {
entry.image_src = imageData.toString('base64');
var requestObject = {
url: 'http://localhost:3000/api/entry',
method: 'post',
json: false,
formData: entry
};
return request(requestObject);
});
}
I'm almost 100% certain the problem is in this part because the api just takes what it gets and gives it to sequelize to put into the table, but I could be wrong. Image field is set as longblob.
I'm sure it's something simple once I figure it out, but so far I'm stumped.
This is not a direct answer to your question but it is rarely needed to actually store an image in the database. What is usually done is storing an image on storage like S3, a CDN like CloudFront or even just in a file system of a static file server, and then storing only the file name or some ID of the image in the actual database.
If there is any chance that you are going to serve those images to some clients then serving them from the database instead of a CDN or file system will be very inefficient. If you're not going to serve those images then there is still very little reason to actually put them in the database. It's not like you're going to query the database for specific contents of the image or sort the results on the particular serialization of an image format that you use.
The simplest thing you can do is save the images with a unique filename (either a random string, UUID or a key from your database) and keep the ID or filename in the database with other data that you need. If you need to serve it efficiently then consider using S3 or some CDN for that.
I'm facing issue of deleting folder which contains photos inside on Amazon S3
1. Create folder
var params = {Bucket: S3_BUCKET, Key: "test/", ACL:"public-read"};
s3.putObject(params, function(err, data) {
});
2. Upload photo
var body = fs.createReadStream(filePath);
var params = {Bucket: S3_BUCKET, Key: "test/flower.jpgg", Body: body, ContentType:"image/jpeg", ACL:"public-read"};
s3.upload(params, function(err, data) {
});
3. Delete folder
var params = {Bucket: S3_BUCKET, Key: "test/"};
s3.deleteObject(params, function(err, data) {
});
If folder has no photo, delete function works well. But it contains photos, delete will not work.
Please help. Thank for all supports.
The problem here is a conceptual one, and starts at step 1.
This does not create a folder. It creates a placeholder object that the console will display as a folder.
An object named with a trailing "/" displays as a folder in the Amazon S3 console.
http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html
It's not necessary to do this -- creating objects with this key prefix will still cause the console to display a folder, even without creating this object. From the same page:
Amazon S3 has a flat structure with no hierarchy like you would see in a typical file system. However, for the sake of organizational simplicity, the Amazon S3 console supports the folder concept as a means of grouping objects. Amazon S3 does this by using key name prefixes for objects.
Since, at step 1, you are not actually creating a folder, it makes sense that removing the placeholder object also does not delete the folder.
Folders do not actually exist in S3 -- they're just used for display purposes in the console -- so objects cannot properly be said to be "in" folders. The only way to remove all the objects "in" a folder is to explicitly remove the objects individually. Similarly, the only way to rename a folder is to rename the objects in it... and the only way to rename an object is to make a copy of an the object with a new key and then delete the old object.