I have a lambda function which gets an image from one bucket, resizes it and puts it into another bucket. The lambda function is set to trigger when a file is created the source bucket. Fairly standard, tutorial level stuff.
When I use the aws web UI to put an image in the source bucket, everything works as expected.
However, when I use xhr from my web app to put an image into the same bucket, I get the following error (thrown from my s3.getObject call):
AccessDenied: Access Denied
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/services/s3.js:585:35
Having extensively searched around, most folk are saying 403 errors usually boil down to role/policy permissions for the lambda function. But when I trawl the logs, the only difference I see between my xhr upload and an aws web UI upload is the eventName and userIdentity.
For a web UI upload it's Put and principalId:
eventName: 'ObjectCreated:Put',
userIdentity: { principalId: 'AWS:AIDAJ2VMZPNX5NJD2VBLM' }
But on a xhr call it's Post and Anonymous:
eventName: 'ObjectCreated:Post',
userIdentity: { principalId: 'Anonymous' }
My Lambda role has two policies attached:
AWSLambdaExecute
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:*"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::*"
}
]
}
AWSLambdaBasicExecutionRole
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "*"
}
]
}
My S3 buckets have the following policies:
Source bucket:
{
"Version": "2012-10-17",
"Id": "Lambda access bucket policy",
"Statement": [
{
"Sid": "All on objects in bucket lambda",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::source-bucket-name/*"
},
{
"Sid": "All on bucket by lambda",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::source-bucket-name"
}
]
}
Destination bucket:
{
"Version": "2012-10-17",
"Id": "Lambda access bucket policy",
"Statement": [
{
"Sid": "All on objects in bucket lambda",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::destination-bucket-name/*"
},
{
"Sid": "All on bucket by lambda",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::destination-bucket-name"
}
]
}
Do I need to somehow pass (or assign) a Principal Id to my xhr call to get it to work? Or do I need to add permissions/policies/roles to my function to let it fire the function without a Principal Id attached to the trigger?
EDIT:
Here's the JS code that sends a POST'ed file to the source bucket:
function uploadFileAttachment(attachment, form) {
var formButtons = document.querySelectorAll("form.form--trix button.btn");
formButtons.forEach((button) => {
button.setAttribute("disabled", "disabled");
});
uploadFile(attachment.file, setProgress, setAttributes)
function setProgress(progress) {
attachment.setUploadProgress(progress)
}
function setAttributes(attributes) {
attachment.setAttributes(attributes)
formButtons.forEach((button) => {
button.removeAttribute("disabled");
});
}
}
function uploadFile(file, progressCallback, successCallback) {
var key = createStorageKey(file)
var formData = createFormData(key, file)
var xhr = new XMLHttpRequest()
xhr.open("POST", global.s3url, true)
xhr.upload.addEventListener("progress", function(event) {
var progress = event.loaded / event.total * 100
progressCallback(progress)
})
xhr.addEventListener("load", function(event) {
if (xhr.status == 204) {
var attributes = {
url: global.s3url + key,
href: global.s3url + key + "?content-disposition=attachment"
}
successCallback(attributes)
}
})
xhr.send(formData);
}
function createStorageKey(file) {
var date = new Date()
var day = date.toISOString().slice(0,10)
var name = date.getTime() + "-" + file.name
return [ "trix", day, name ].join("/")
}
function createFormData(key, file) {
var data = new FormData()
data.append("key", key)
data.append("Content-Type", file.type)
data.append("file", file)
return data
}
Related
I am trying to upload an object to an AWS bucket using NodeJs (aws-sdk), but I am get access denied error.
The IAM user of which I am using accessKeyId and secretAccessKey also have been given access to the s3 bucket to which I am trying to upload.
Backend Code
const s3 = new AWS.S3({
accessKeyId: this.configService.get<string>('awsAccessKeyId'),
secretAccessKey: this.configService.get<string>('awsSecretAccessKey'),
params: {
Bucket: this.configService.get<string>('awsPublicBucketName'),
},
region: 'ap-south-1',
});
const uploadResult = await s3
.upload({
Bucket: this.configService.get<string>('awsPublicBucketName'),
Body: dataBuffer,
Key: `${folder}/${uuid()}-${filename}`,
})
.promise();
Bucket Policy
{
"Version": "2012-10-17",
"Id": "PolicyXXXXXXXXX",
"Statement": [
{
"Sid": "StmtXXXXXXXXXXXXXX",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::some-random-bucket"
},
{
"Sid": "StmtXXXXXXXXXXX",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::XXXXXXXXXX:user/some-random-user"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::some-random-bucket"
}
]
}
You have an explicit deny statement, denying anyone from doing anything S3-related on some-random-bucket.
This will override any allow statements in the policy, according to the official IAM policy evaluation logic.
You can do any of the following:
Remove the deny statement from the policy
Modify the deny statement & use NotPrincipal to exclude some-random-user from the deny statement
Modify the deny statement & use the aws:PrincipalArn condition key with the ArnNotEquals condition operator to exclude some-random-user from the deny statement i.e.
{
"Version": "2012-10-17",
"Id": "PolicyXXXXXXXXX",
"Statement": [
{
"Sid": "StmtXXXXXXXXXXXXXX",
"Effect": "Deny",
"Action": "s3:*",
"Principal": "*",
"Resource": "arn:aws:s3:::some-random-bucket",
"Condition": {
"ArnNotEquals": {
"aws:PrincipalArn": "arn:aws:iam::XXXXXXXXXX:user/some-random-user"
}
}
},
{
"Sid": "StmtXXXXXXXXXXX",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::XXXXXXXXXX:user/some-random-user"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::some-random-bucket"
}
]
}
I'm using the AWS NodeJS SDK to upload and download files to s3 buckets, recently I updated the bucket policy so no one beside my domain and the ec2 elastic beanstalk role can access these images.
Everything seems to be working fine, except actually downloading the files
AccessDenied: Access Denied at Request.extractError (/node_modules/aws-sdk/lib/services/s3.js:714:35)
S3 Bucket policy:
{
"Version": "2012-10-17",
"Id": "http referer policy",
"Statement": [
{
"Sid": "Allow get requests originating from www.*.domain.com and *.domain.com.",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::data/*",
"Condition": {
"StringLike": {
"aws:Referer": [
"https://www.*.domain.com/*",
"https://*.domain.com/*"
]
}
}
},
{
"Sid": "Deny get requests originating not from www.*.domain.com and *.domain.com.",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::data/*",
"Condition": {
"StringNotLike": {
"aws:Referer": [
"https://www.*.domain.com/*",
"https://*.domain.com/*"
]
}
}
},
{
"Sid": "Allow get/put requests from api.",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::[redacted]:role/aws-elasticbeanstalk-ec2-role"
},
"Action": [
"s3:GetObject",
"s3:GetObjectAcl",
"s3:GetObjectVersion",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::data",
"arn:aws:s3:::data/*"
]
}
]
}
I am able to list contents of the bucket, so thats not the issue in this case because uploading is working just fine
This is my code that upload files:
const params = {
Bucket: "data",
Key: String(fileName),
Body: file.buffer,
ContentType: file.mimetype,
ACL: 'public-read',
};
await s3.upload(params).promise();
For downloading:
await s3.getObject({ Bucket: this.bucketS3, Key: fileId }).promise();
Uploading/Downloading was working fine before setting up policies, but I would rather limit who can view/download these files to only the api and domains
I have a website working with nodeJs. In here users can upload their pictures. I'm using Wasabi to store that pictures.
My problem is everytime a user send a picture to server it will save it with private condition. Because of that after uploading users can't see the picture. I need to make it public on the bucket so users can see it.
My all buckets are free to read but everytime I upload a file it will be private. How can I make every uploaded picture public?
these are my upload params
const paramsForUpload = {
Bucket: bucketName,
Key: filePath,
Body: file.data,
};
const options = {
partSize: 10 * 1024 * 1024, // 10 MB
queueSize: 10,
};
s3.upload(paramsForUpload,options, (err) =>{
....
})
My policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowPublicRead",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::bucketname"
},
{
"Sid": "AddPerm",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::bucketname"
}
]
}
Thanks for helping
You can make public upload with adding
ACL: 'public-read',
to your upload params like
const paramsForUpload = {
Bucket: bucketName,
Key: filePath,
Body: file.data,
ACL: 'public-read',
};
The bucket policy should refer to the contents of the bucket, with a trailing /*:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowPublicRead",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::bucketname/*"
}
]
}
So I am struggling getting images from an s3 bucket to show up client side in a meteor app. I am using aws-sdk node package. I am getting the objects in my console on the server side and if I change my bucket policy to public it works as well. The fact that the objects show up in the console on startup but are not showing on the client side make me think it is a meteor problem I am just having trouble finding good documentation on the subject.
server.js
Meteor.startup(() => {
AWS.config.update({
accessKeyId: Meteor.settings.AWS.S3.accessId,
secretAccessKey: Meteor.settings.AWS.S3.secret
});
s3 = new AWS.S3({
region: "<my-region>"
});
var params = {
Bucket: "<my-bucket-name>"
};
s3.listObjects(
params,
Meteor.bindEnvironment(function(err, data) {
if (err) throw err;
console.log(data);
})
);
})
Here is my bucket policy( when I set the resource to * it works but I want the resource set to a specific IAM account):
{
"Version": "2008-10-17",
"Id": "Policy-some-number",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<current-iam-account-in>"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::<bucket-name>/*"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<current-iam-account-in>"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::<bucket-name>/*"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<current-iam-account-in>"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::<bucket-name>"
}
]
}
I am using lambda function to thumbnail images in s3 bucket. And I found a sample here: Image conversion using Amazon Lambda and S3 in Node.js. However, after refactoring the code, I got access denied when invocating s3.getObject(). I checked if my IAM policy granted permissions incorrectly, but I had full access to Lambda, S3 and CloudFront. Here's how exception thrown:
async.waterfall([
function download(next) {
console.log(srcBucket+ " "+srcKey);
s3.getObject({
Bucket: srcBucket,
Key: srcKey
}, next);
console.log(srcBucket+ " "+srcKey+ "After");
}
], function(err, result) {
if (err) {
console.error(err);
}
// result now equals 'done'
console.log("End of step " + key);
callback();
});
Also, how my matchng regex setting is the same as the sample:
var srcBucket = event.Records[0].s3.bucket.name;
var typeMatch = srcKey.match(/\.([^.]*)$/);
var fileName = path.basename(srcKey);
if (!typeMatch) {
console.error('unable to infer image type for key ' + srcKey);
return;
}
var imageType = typeMatch[1].toLowerCase();
if (imageType != "jpg" && imageType != "gif" && imageType != "png" &&
imageType != "eps") {
console.log('skipping non-image ' + srcKey);
return;
}
My Lambda policy is:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
When a Lambda gets an Access Denied error when trying to use S3, it's almost always a problem with the Lambda's Role Policy. In general, you need something like this to grant access to an S3 bucket from Lambda:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::mybucket",
"arn:aws:s3:::mybucket/*"
],
"Effect": "Allow"
}
]
}
For those who reached this question and checked all the steps below:
Resource definition is correct for action "s3:*"
"Resource": [
"arn:aws:s3:::mybucket",
"arn:aws:s3:::mybucket/*"
]
Lambda role has S3 Permissions (S3FullAccess probably)
File is in the bucket (throws 403 for not found) and bucket and object key information is correct
S3 object level permission for read is not overwritten.
There is an extra step which solved my problem. You may want to check IAM Role definition to remove if any permission boundary blocks to reach S3. You can also look at Permission Boundary Documentation for more information.
I got the inspiration from this as this answer: Reference to error
Therefore, I changed my test event to:
"s3": {
"configurationId": "testConfigRule",
"object": {
"eTag": "0123456789abcdef0123456789abcdef",
"sequencer": "0A1B2C3D4E5F678901",
"key": "images/HappyFace.jpg",
"size": 1024
}
by given prefix to images/ in the key setting, which I should have done that in my regex setting.
This error message is really confusing, please do check your policy first, and then try to give a full path as your key.
I've given necessary permissions from S3 side for the Lambda Execution Role ARN
{
"Version": "2012-10-17",
"Id": "Policy1663698473036",
"Statement": [
{
"Sid": "Stmt1663698470845",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::434324234:role/file-conversion-lambdaRole"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::file-conversion-bucket/*"
},
{
"Sid": "Stmt1663698470842",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::434324234:role/file-conversion-lambdaRole"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::file-conversion-bucket/processed/*"
}
]
}