I'm searching how check if an object exist in my aws s3 bucket in nodejs without list all my object (~1500) and check the prefix of the object but I cannot find how.
The format is like that:
<prefix I want to search>.<random string>/
Ex:
tutturuuu.dhbsfd7z63hd7833u/
Because you don't know the entire object Key, you will need to perform a list and filter by prefix. The AWS nodejs sdk provides such a method. Here is an example:
s3.listObjectsV2({
Bucket: 'youBucket',
MaxKeys: 1,
Prefix: 'tutturuuu.'
}, function(err, data) {
if (err) throw err;
const objectExists = data.Contents.length > 0
console.log(objectExists);
});
Note that it is important to use MaxKeys in order to reduce network usage. If more than one object has the prefix, then you will need to return everything and decide which one you need.
This API call will return metadata only. After you have the full key you can use getObject to retrieve the object contents.
Related
In my S3 bucket, I have a folder structure of rootFolder1/year/month/day/myfile.csv. Depending on the time, the rootFolder folder could have multiple subfolder, organised by year, month and date, with each date folder containing multiple csv files. There could also be a rootFolder2 in my bucket, with the same folder structure.
I know for downloading object from S3, I can use the getObject method where I can pass in the params object with the bucket Key like so:
var params = {
Key: "book/2020/01/01/all_books_000.csv",
Bucket: 'my-bucket-test'
}
s3.getObject(params, (err, data) => {
if (err) throw err;
// do something with data
})
But in my case I don't want to hardcode the Key in. Is there any elegant way I can call getObject where I can dynamically set the Key, so it downloads the files under each date folder?
edit: for some further clarity, these are the keys I get back when I call s3.listObjects
'author/',
'author/2020/',
'author/2020/01/',
'author/2020/01/01/',
'author/2020/01/01/all_authors_000.csv',
'book/',
'book/2020/',
'book/2020/01/',
'book/2020/01/01/',
'book/2020/01/01/all_books_000.csv',
'book/2020/01/01/all_books_001.csv'
I have stored video files in S3 bucket and now i want to show the files to clients through an API. Here is my code for it
app.get('/vid', async(req, res) => {
AWS.config.update({
accessKeyId: config.awsAccessKey,
secretAccessKey: config.awsSecretKey,
region: "ap-south-1"
});
let s3 = new AWS.S3();
var p = req.query.p
res.attachment(p);
var options = {
Bucket: BUCKET_NAME,
Key: p,
};
console.log(p, "name")
try {
await s3.getObject(options).
createReadStream().pipe(res);
} catch (e) {
console.log(e)
}
})
This is the output I am getting when ther is this file available in S3 bucket -
vid_kdc5stoqnrIjEkL9M.mp4 name
NoSuchKey: The specified key does not exist.
This is likely caused by invalid parameters being passed into the function.
To check for invalid parameters you should double check the strings that are being passed in. For the object check the following:
Check the value of p, ensure it is the exact same name of the full object key.
Validate that the correct BUCKET_NAME is being used
No trailing characters (such as /)
Perform any necessary decoding before passing parameters in.
If in doubt use logging to output the exact value, also to test the function try testing with hard coded values to validate you can actually retrieve the objects.
For more information take a look at the How can I troubleshoot the 404 "NoSuchKey" error from Amazon S3? page.
Having certain characters in the bucketname leads to this error.
In your case, there is an underscore. Try renaming the file.
Also refer to this
S3 Bucket Naming Requirements Docs
Example from Docs:
The following example bucket names are not valid:
aws_example_bucket (contains underscores)
AwsExampleBucket (contains uppercase letters)
aws-example-bucket- (ends with a hyphen)
I am using aws transcribe to get the text of the video using node js. I can specify the particular destination bucket in params but not the particular folder. Can anyone help me with this ? This is my code
var params = {
LanguageCode: "en-US",
Media: { /* required */
MediaFileUri: "s3://bucket-name/public/events/545/videoplayback1.mp4"
},
TranscriptionJobName: 'STRING_VALUE', /* required */
MediaFormat: "mp4", //mp3 | mp4 | wav | flac,
OutputBucketName: 'test-rekognition',
// }
};
transcribeservice.startTranscriptionJob(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
I have specified the destination bucket name in OutputBucketName field. But how to specify a particular folder ?
I would recommend creating a designated S3 Bucket for Transcribe output and adding a trigger with a lambda function to respond to that trigger on 'Object create (All)'. Essentially, as soon as there is a new object added to your S3 bucket, a lambda function is invoked to move/process that output by placing it in a specific 'folder' of your choice.
This doesn't solve the API issue but I hope it serves as a good workaround - you could look at this article ( https://linuxacademy.com/hands-on-lab/0e291fc6-52a4-4ed3-ad65-8cf2fd84e0df/ ) as a guide.
have a good one.
Very late to the party, but I just had the same issue.
It appeared that Amazon added a new parameter called OutputKey that allows you to save your data in a specific folder in your bucket :
You can use output keys to specify the Amazon S3 prefix and file name
of the transcription output. For example, specifying the Amazon S3
prefix, "folder1/folder2/", as an output key would lead to the output
being stored as "folder1/folder2/your-transcription-job-name.json". If
you specify "my-other-job-name.json" as the output key, the object key
is changed to "my-other-job-name.json". You can use an output key to
change both the prefix and the file name, for example
"folder/my-other-job-name.json".
Just make sure to put 'folder/' as an OutputKey (with '/' symbol at the end), otherwise it will be interpreted as a name for your file and not the folder where to store it.
Hope it'll be useful to someone.
I'm using signed url to upload a file from my react application to a s3 bucket. I specify the path as part of my Key and the folders are getting created properly:
let params = {
Bucket: vars.aws.bucket,
Key: `${req.body.path}/${req.body.fileName}`,
Expires: 5000,
ACL: 'public-read-write',
ContentType: req.body.fileType,
};
s3.getSignedUrl('putObject', params, (err, data)=>{...
However, when I use s3.listObject, the folders that are created this way are not getting returned. Here is my node api code:
const getFiles = (req, res) => {
let params = {
s3Params:{
Bucket: vars.aws.bucket,
Delimiter: '',
Prefix: req.body.path
}
}
s3.listObjects(params.s3Params, function (err, data) {
if (err) {
res.status(401).json(err);
} else {
res.status(200).json(data);
}
});
}
The folders that are getting created through the portal are showing in the returned object properly. Is there any attribute I need to set as part of generating the signed URL to make the folder recognized as an object?
I specify the path as part of my Key and the folders are getting created properly
Actually, they aren't.
Rule 1: The console displays a folder icon for the folder foo because one or more objects exists in the bucket with the prefix foo/.
The console appears to allow you to create "folders," but that isn't what's happening when you do that. If you create a folder named foo in the console, what actually happens is that an ordinary object, zero bytes in length, with the name foo/ is created. Because this now means there is at least one object that exists in the bucket with the prefix foo/, a folder is displayed in the console (see Rule 1).
But that folder is not really a folder. It's just a feature of the console interacting with another feature of the console. You can actually delete the foo/ object using the API/SDK and nothing happens, because the console till shows that folder as long as there remains at least one object in the bucket with the prefix foo/. (Deleting the folder in the console sends delete requests for all objects with that prefix. Deleting the dummy object via the API does not.)
In short, the behavior you are observing is normal.
If you set the delimiter to /, then the listObjects response will include CommonPrefixes -- and this is where you should be looking if you want to see "folders." Objects ending with / are just the dummy objects the console creates. CommonPrefixes does not depend on these.
So far I've been able to successfully use node.js, express, and knox to add/update/delete/retrieve objects in Amazon S3. Trying to move things to the next level I'm trying to figure out how to use knox (if it's possible) to do two things:
1) Set the object to use server-side encryption when adding/updating the object.
2) Get a particular version of an object or get a list of versions of the object.
I know this is an old question, but it is possible to upload a file with knox using server-side encryption by specifying a header:
client.putFile('test.txt', '/test.txt', {"x-amz-server-side-encryption": "AES256"}, function(err, res) {
//Do something here
});
Andy (who wrote AwsSum) here.
Using AwsSum, when you put an object, just set the 'ServerSideEncryption' to the value you want (currently S3 only supports 'AES256'). Easy! :)
e.g.
var body = ...; // a buffer, a string, a stream
var options = {
BucketName : 'chilts',
ObjectName : 'my-object.ext',
ContentLength : Buffer.byteLength(body),
Body : body,
ServerSideEncryption : 'AES256'
};
s3.PutObject(options, function(err, data) {
console.log("\nputting an object to pie-18 - expecting success");
console.log(err, 'Error');
console.log(data, 'Data');
});