I need to access a file and print its content from a subfolder in an s3 bucket. My file (file_abc) is in a sub folder (subfolder_abc) in a folder (folder_abc) in s3 bucket.
I am using the following code to do so -
s3_client = boto3.client('s3')
response = s3_client.get_object(Bucket='Bucket_abc',
Key='folder_abc/subfolder_abc' + "/" + 'file_abc')
result = str(response["Body"].read())
print (result)
I am getting the following error -
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
How to access data of files in subfolders?
Can you show us the permissions for the bucket?
The way you are attempting to read the file looks correct, I assume you have an issue with the permissions for reading files in that bucket.
If you can show us the permissions for the bucket and the role you function is executing as, we can be of more help.
Here is a policy example that would allow all access:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::MyExampleBucket/*"
],
}
]
}
Related
I got a website hosted in cloudfront using an origin id to acess angualr app data in my non-public s3 bucket.
things work fine except that users cannot access paths like www.exmaple.com/path, users do get access denied. They always have to start at root.
I need to enable that access but am struggeling. I found that post, but frank, I dont understand it.
AWS CloudFront redirect to path
I found a workaround by making bucket public and use static website hosting, problem is only that users can bybass cloudfront and go to bucket right away.
anybody got a hint on how to enable paths?
I found some post explaining it.
U need to use the static website hosting enpoint an make it public
In cloudfront you set a header named Referer and give some random secret as value.
You put a bucket policy like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion"
],
"Resource": "arn:aws:s3:::yours3bucket/",
"Condition": {
"StringLike": {
"aws:Referer": "yoursecretvalue"
}
}
},
{
"Effect": "Deny",
"Principal": "",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion"
],
"Resource": "arn:aws:s3:::yours3bucket/",
"Condition": {
"StringNotLike": {
"aws:Referer": "yoursecretvalue"
}
}
}
]
}
I have created lambda in region A and a S3 bucket in region B , trying to access bucket from lambda boto-3 client but getting an error(access denied).Please suggest some solution for this in python CDK. Will I need to create any specific policy for it.
Your lambda function requires permissions to read S3.
The easiest way to enable that is to add AWS managed policy:
arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
to your lambda execution role.
Specifying region is not required, as S3 buckets have global scope.
You have to explicitly pass the region name of the bucket if it is not in the same region as the lambda (because AWS have region specific endpoints for S3 which needs to be explicitly queried when working with s3 api).
Initialize your boto3 S3 client as:
import boto3
client = boto3.client('s3', region_name='region_name where bucket is')
see this for full reference of boto3 client:
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.client
---------Edited------------
you also need the following policy attached to (or inline in) the role of your lambda:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ExampleStmt",
"Action": [
"s3:GetObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::YOUR-BUCKET-NAME/*"
]
}
]
}
If you need to list and delete the objects too, then you need to have the following policy instead, attached to (or inline in) the role of the lambda:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ExampleStmt1",
"Action": [
"s3:GetObject",
"s3:DeleteObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::YOUR-BUCKET-NAME/*"
]
},
{
"Sid": "ExampleStmt2",
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::YOUR-BUCKET-NAME"
]
}
]
}
I'm getting a malformed syntax error when running boto3 to create_policy command surprisingly I don't get the error in AWS console. I tried to debug this using the AWS Console's "Policy Editor" and click the "Validate" button and it creates the policy No error. Does anyone know what I'm doing wrong?
iam_client.create_policy(PolicyName='xxxxx-policy',
PolicyDocument=json.dumps(dir_name + 'xxxxx-policy.json'))
This policy contains the following error: botocore.errorfactory.MalformedPolicyDocumentException: An error occurred (MalformedPolicyDocument) when calling the CreatePolicy operation: Syntax errors in policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:Describe*",
"iam:ListRoles",
"sts:AssumeRole"
],
"Resource": "*"
}
]
}
json.dumps will turn a Python dictionary into a JSON string. The input shouldn't be a file name. In fact, you don't need json package to do this.
import boto3
with open('xxx-policy.json', 'r') as fp:
iam_client = boto3.client('iam')
iam_client.create_policy(
PolicyName='xxx-policy',
PolicyDocument=fp.read()
)
You are reading your document from file:
with open(dir_name + 'xxxxx-policy.json', 'r') as f:
policy_document = f.read()
iam_client.create_policy(
PolicyName='xxxxx-policy',
PolicyDocument=policy_document)
I have an EMR cluster that involves steps to write and delete objects on S3 bucket. I have been trying to create a bucket policy in the S3 bucket that denies deleting access to all principals except for the EMR role and the instance profile. Below is my policy.
{
"Version": "2008-10-17",
"Id": "ExamplePolicyId123458",
"Statement": [
{
"Sid": "ExampleStmtSid12345678",
"Effect": "Deny",
"Principal": "*",
"Action": [
"s3:DeleteBucket",
"s3:DeleteObject*"
],
"Resource": [
"arn:aws:s3:::bucket-name",
"arn:aws:s3:::bucket-name/*"
],
"Condition": {
"StringNotLike": {
"aws:userId": [
"AROAI3FK4OGNWXLHB7IXM:*", #EMR Role Id
"AROAISVF3UYNPH33RYIZ6:*", # Instance Profile Role ID
"AIPAIDBGE7J475ON6BAEU" # Instance Profile ID
]
}
}
}
]
}
As I found somewhere, it is not possible to use wildcard entries to specify every Role session in the "NotPrincipal" section so I have used the condition of aws:userId to match.
Whenever I run the EMR step without the bucket policy, the step completes successfully. But when I add the policy to bucket and re run, the step fails with following error.
diagnostics: User class threw exception:
org.apache.hadoop.fs.s3a.AWSS3IOException: delete on s3://vr-dump/metadata/test:
com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more objects could not be deleted
(Service: null; Status Code: 200; Error Code: null; Request ID: 9FC4797479021CEE; S3 Extended Request ID: QWit1wER1s70BJb90H/0zLu4yW5oI5M4Je5aK8STjCYkkhZNVWDAyUlS4uHW5uXYIdWo27nHTak=), S3 Extended Request ID: QWit1wER1s70BJb90H/0zLu4yW5oI5M4Je5aK8STjCYkkhZNVWDAyUlS4uHW5uXYIdWo27nHTak=: One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: 9FC4797479021CEE; S3 Extended Request ID: QWit1wER1s70BJb90H/0zLu4yW5oI5M4Je5aK8STjCYkkhZNVWDAyUlS4uHW5uXYIdWo27nHTak=)
What is the problem here? Is this related to EMR Spark Configuration or the bucket policy?
Assuming these role ids are correct (they start in AROA so they have a valid format) I believe you also need the aws account number on the policy. For example:
{
"Version": "2008-10-17",
"Id": "ExamplePolicyId123458",
"Statement": [
{
"Sid": "ExampleStmtSid12345678",
"Effect": "Deny",
"Principal": "*",
"Action": [
"s3:DeleteBucket",
"s3:DeleteObject*"
],
"Resource": [
"arn:aws:s3:::vr-dump",
"arn:aws:s3:::vr-dump/*"
],
"Condition": {
"StringNotLike": {
"aws:userId": [
"AROAI3FK4OGNWXLHB7IXM:*", #EMR Role Id
"AROAISVF3UYNPH33RYIZ6:*", # Instance Profile Role ID
"AIPAIDBGE7J475ON6BAEU", # Instance Profile ID
"1234567890" # Your AWS Account Number
]
}
}
}
]
}
How to provide public read access to a all files in folder by s3fs in node.js?
I am trying to rename the folder name by using following code:
fsImpl.mkdirp(main_folder_path).then(function () {
fsImpl.copyDir(copy_folder_path+'/', main_folder_path+'/').then(function () {
console.log("copy directory")
}, function (reason) {
res.json('error');
});
});
Successfully all files are get copying to new folder,But the copies files don't have public access..I have searched some sources and finally i get following answer
fsImpl.writeFile(fileName, stream,{ACL: 'public-read'}).then(function (err,data) {
// code
})
But I need to give access to the folder . How do i do that??
Thanks in Advance.
Did your S3 bucket have public read access?? if not go to your S3 account and there you find Permissions Tab after click on that you will have three new Tabs. from those tabs Click on Bucket Policy. and Add the following JSON code there.
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "AllowPublicRead",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::YourBucket name/*"
}
]}
It will resolve your problem.