I am trying to list the metadata about all the S3 buckets available in my AWS account using the boto3 client.
I tried below api:
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.list_buckets but it just returns the bucket name and created date.
I am looking for more details like:
Bucket Region
Bucket status like Active
Bucket Id (if there is any)
It would be more helpful if I can get a single API which can return all these details as describe_instances of EC2 returns more metadata.
Any help is highly appreciated
Hope, this post of some help!
S3 Region? - I don't think, S3 is region-specific anymore. Also, the bucket name is already a unique value.
As you can see in the same document you have below function for collecting a different kinds of metadata.
get_bucket_accelerate_configuration()
get_bucket_acl()
get_bucket_analytics_configuration()
get_bucket_cors()
get_bucket_encryption()
get_bucket_inventory_configuration()
get_bucket_lifecycle()
get_bucket_lifecycle_configuration()
get_bucket_location()
get_bucket_logging()
get_bucket_metrics_configuration()
get_bucket_notification()
get_bucket_notification_configuration()
get_bucket_policy()
get_bucket_policy_status()
get_bucket_replication()
get_bucket_request_payment()
get_bucket_tagging()
get_bucket_versioning()
get_bucket_website()
These are created just for the reason of segregating information that specifically required for user.
In my opinion, you are looking for get_bucket_inventory_configuration - Returns an inventory configuration (identified by the inventory ID) from the bucket.
This will also return ARN which is a unique id for all AWS resources.
A sample ARN looks like
'Bucket' :"arn:aws:s3:::10012346561237-rawdata-bucket"
Related
I want to upload image , audio files of small size from form to the S3 using postman for test. I successfully uploaded file in AWS S3 bucket from my application running on my local machine. Following is the part of the code I used for file uploading .
import boto3
s3_client = boto3.client('s3',aws_access_key_id =AWS_ACCESS_KEY_ID,aws_secret_access_key = AWS_SECRET_ACCESS_KEY,)
async def save_file_static_folder(file, endpoint, user_id):
_, ext = os.path.splitext(file.filename)
raw_file_name = f'{uuid.uuid4().hex}{ext}'
# Save image file in folder
if ext.lower() in image_file_extension:
relative_file_folder =user_id+'/'+endpoint
contents = await file.read()
try:
response = s3_client.put_object(Bucket = S3_BUCKET_NAME,Key = (relative_file_folder+'/'+raw_file_name),Body = contents)
except:
return FileEnum.ERROR_ON_INSERT
I called this function from another endpoint and form data (e.g. name, date of birth and other details) are successfully saved in Mongodb database and files are uploaded in S3 bucket.
This app is using fastapi and files are uploaded in S3 bucket while deploying this app in local machine.
Same app is delpoyed in AWS lambda and S3 bucket as storage. For handling whole app , following is added in endpoint file.
handler = Mangum(app)
After deploying app in AWS creating lambda function from root user account of AWS, files didnot get uploaded in S3 bucket.
If I didnot provide files during form then the AWS API endpoint successfully works. Form data gets stored in MongoDB database (Mongodb atlas) and app works fine hosted using Lambda.
App deployed using Lambda function works successfully except file uploads in form. FOr local machine, file uploads in S3 get success.
EDIT
While tracing in Cloudwatch I got following error
exception An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The AWS Access Key Id you provided does not exist in our records.
I checked AWS Access Key Id and secret key many times and they are correct and root user credentials are kept.
It looks like you have configured your Lambda function with an execution IAM role, but you are overriding the AWS credentials supplied to the boto3 SDK here:
s3_client = boto3.client('s3',aws_access_key_id =AWS_ACCESS_KEY_ID,aws_secret_access_key = AWS_SECRET_ACCESS_KEY,)
You don't need to provide credentials explicitly because the boto3 SDK (and all language SDKs) will automatically retrieve credentials dynamically for you. So, ensure that your Lambda function is configured with the correct IAM role, and then change your code as follows:
s3_client = boto3.client('s3')
As an aside, you indicated that you may be using AWS root credentials. It's generally a best security practice in AWS to not use root credentials. Instead, create IAM roles and IAM users.
We strongly recommend that you do not use the root user for your everyday tasks, even the administrative ones. Instead, adhere to the best practice of using the root user only to create your first IAM user. Then securely lock away the root user credentials and use them to perform only a few account and service management tasks.
I am working on one issue where I need Lambda to write the logs in S3 bucket but the tricky part here is, Lambda will read the logs and write in another s3 bucket which is in another AWS account. Can we achieve this?
I wrote some code but it isn't working.
from urllib.request import urlopen
import boto3
import os
import time
BUCKET_NAME = '***'
CSV_URL = f'***'
def lambda_handler(event, context):
response = urlopen(CSV_URL)
s3 = boto3.client('s3')
s3.upload_fileobj(response, BUCKET_NAME, time.strftime('%Y/%m/%d'))
response.close()
It sounds like you are asking how to allow the Lambda function to create an object in an Amazon S3 bucket that belongs to a different AWS Account.
Bucket Policy on target bucket
The simplest method is to ask the owner of the target bucket (that is, somebody with Admin permissions in that other AWS Account) to add a Bucket Policy that permits PutObject access to the IAM Role being used by the AWS Lambda function. You will need to supply them with the ARN of the IAM Role being used by the Lambda function.
Also, make sure that the IAM Role has been given permission to write to the target bucket. Please note that two sets of permissions are required: The IAM Role needs to be allowed to write to the bucket in the other account, AND the bucket needs to permit access by the IAM Role. This double-set of permissions is required because access both accounts need to permit this access.
It is possible that you might need to grant some additional permissions, such as PutObjectACL.
Assuming an IAM Role from the target account
An alternative method (instead of using the Bucket Policy) is:
Create an IAM Role in the target account and give it permission to access the bucket
Grant trust permissions so that the IAM Role used by the Lambda function is allowed to 'Assume' the IAM Role in the target account
Within the Lambda function, use the AssumeRole() API call to obtain credentials from the target account
Use those credentials when connecting to S3, which will allow you to access the bucket in the other account
Frankly, creating the Bucket Policy is a lot easier.
import requests
url = 'https://s3.amazonaws.com/<some-bucket-name>'
data = { 'key': 'test/test.jpeg' }
files = { 'file': open('test.jpeg', 'rb') }
r = requests.post(url, data=data, files=files)
I want to upload an image to the S3 bucket as above.The S3 bucket is enabled with AES256 encryption. How will I be able to specify the encryption in post requests?
Warning
It seems like you have configured your bucket in a way that allows unauthenticated PUT requests into it - this is dangerous and may become expensive, because essentially anybody that knows your bucket name can put data into it and you'll have to pay the bill. I recommend you change that.
If you want it to stay that way, you can use headers to configure the encryption type for each object as described in the PutObject API-Reference.
The most relevant (excluding SSE-C encryption) are these two:
x-amz-server-side-encryption
The server-side encryption algorithm used when storing this object in Amazon S3 (for example, AES256, aws:kms).
Valid Values: AES256 | aws:kms
x-amz-server-side-encryption-aws-kms-key-id
If x-amz-server-side-encryption is present and has the value of aws:kms,
this header specifies the ID of the AWS Key Management Service (AWS
KMS) symmetrical customer managed customer master key (CMK) that was
used for the object.
If the value of x-amz-server-side-encryption is aws:kms, this header
specifies the ID of the symmetric customer managed AWS KMS CMK that
will be used for the object. If you specify
x-amz-server-side-encryption:aws:kms, but do not provide
x-amz-server-side-encryption-aws-kms-key-id, Amazon S3 uses the AWS
managed CMK in AWS to protect the data.
You can add these in your requests.post call.
The API-Docs of the requests library specify how to do that, so it should look roughly like this:
requests.post(
url,
data=data,
files=files,
headers={"x-amz-server-side-encryption": "AES:256"}
)
I want to copy a file from one s3 bucket to another in same region. Both buckets have different access key and secret key. How do I provide these secret and access key using the following python code snippet:
import boto3
s3 = boto3.resource('s3')
copy_source = {
'Bucket': 'mybucket',
'Key': 'mykey'
}
bucket = s3.Bucket('otherbucket')
bucket.copy(copy_source, 'otherkey')
You don't. Copying objects, whether from one bucket to another or within the same bucket, requires you to use one set of credentials that has the necessary permissions in both buckets.
When you perform a copy object, the request is actually sent by your client to the destination bucket, which sends the request for the content to the source bucket using a path that is internal to S3, but using the same credentials you used for the first request. The object is transferred without you needing to download it and then upload it again.
If you don't have a single set of credentials that can access both buckets, you have to resort to downloading and re-uploading.
I'm trying to access AWS Glacier (from the command line on Ubuntu 14.04) using something like:
aws glacier list-vaults -
rather than
aws glacier list-vaults --account-id 123456789
The documentation suggests that this should be possible:
You can specify either the AWS Account ID or optionally a '-', in
which case Amazon Glacier uses the AWS Account ID associated with the
credentials used to sign the request.
Unless "credentials used to sign the request" means that I have to explicitly include credentials in the command, rather than rely on my .aws/credentials file, I would expect this to work. Instead, I get:
aws: error: argument --account-id is required
Does anyone have any idea how to solve this?
The - is supposed to be passed as the value of --account-id, so like
aws glacier list-vaults --account-id -
--account-id is in fact a required option.
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/glacier/list-vaults.html
Says that "--account-id" is a required parameter for the glacier section of the full aws api. A little wierd, but documented. So yay.