I'm trying to check if a bucket exists on s3 and have been following this link: https://stackoverflow.com/a/49817544/19505278
s3 = boto3.resource('s3')
bucket = s3.Bucket('my-bucket-name')
if bucket.creation_date:
print("The bucket exists")
else:
print("The bucket does not exist")
However, I'm unable to get this to work due to a potential missing permission.
I was able to try this on a different s3 bucket and can verify this works. However, the s3 bucket I'm working with does not and is likely due to missing permissions. Unfortunately, I do not have access to the working bucket's permissions.
Is there a permission that I need to enable to retrieve bucket metadata?
Here is how you would typically test for the existence of an S3 bucket:
import boto3
from botocore.exceptions import ClientError
Bucket = "my-bucket"
s3 = boto3.client("s3")
try:
response = s3.head_bucket(Bucket=Bucket)
print("The bucket exists")
except ClientError as e:
if e.response["Error"]["Code"] == "404":
print("No such bucket")
elif e.response["Error"]["Code"] == "403":
print("Access denied")
else:
print("Unexpected error:", e)
If you think that there is a permission issue, you might want to check the documentation on permissions on s3. If you simply want to make sure you can check existence of all buckets, s3:ListAllMyBuckets would work nicely.
For the code, you usually want to make it light-weight by using head_bucket for buckets, head_object for objects etc. #jarmod above provided sample code.
As for question on client vs resource, client is close to metal i.e. actual back-end api powering the service. Resource is higher level. It tries to create meaningful objects that you would create from client response. They both use botocore underneath. There are sometimes slight differences when requesting something as resource would already have the knowledge of underlying object.
For example, if you first create a Bucket Resource object, you can simply use a method that's meaningful for that bucket without specifying Bucket Name again.
resource = boto3.resource('s3')
bucket = resource.Bucket('some_bucket_name')
# you can do stuff with this bucket, e.g. create it without supplying any params
bucket.create()
# if you are using client, story is different. You dont have access to objects, so you need to supply everything
client = boto3.client('s3')
client.create_bucket(BucketName='some_bucket_name')
# here you would need to supply
client.create_bucket()
Related
I am trying to delete the particular prefix life cycle by using python boto3
I have tried the below code, but below code is deleting the entire bucket life cycle configuration.
import boto3
client = boto3.client('s3')
response = client.delete_bucket_lifecycle(Bucket='my_bucket_name')
I want to delete the particular prefix life cycle.
The delete_bucket_policy() API call will delete a Bucket Policy, which is used to grant access to an Amazon S3 bucket.
It seems that you actually wish to delete a Lifecycle policy, which can be done with the delete_bucket_lifecycle() API call.
I'm working on an aws lambda function on python that reads videos uploaded to an s3 bucket and extracts a few frames from it, i already have the script for extracting the frames with opencv but i don't know what parameter i should pass to cv2.VideoCapture since the file is only accessible through the s3 bucket.
I've tried passing the video as an s3 object with s3.get_object() as well as with s3.download_fileobj, none of this seemed to work tho.
I've also tried passing just the key of the video file in s3 but it didn't work either (I didn't expect this to work, but i was hopeless).
Code i have now:
import boto3
import cv2
import io
def lambda_handler(event, context):
s3 = boto3.client("s3")
bucket_name = "my_bucket"
video_key = "videos/video.mp4"
vidcap = cv2.VideoCapture(s3.get_object(Bucket=bucket_name,Key=video_path))
success,image = vidcap.read()
I've also tried with:
vidcap = cv2.VideoCapture(s3.download_fileobj(Bucket=bucket_name, Key=video_key, Fileobj=io.BytesIO())
But with no luck either
I'm getting success = False and image=None. I expect the output of success to be True and the image to be a numpy array to be able to read it.
A presigned url for S3 object can be used.
url = s3_client.generate_presigned_url( ClientMethod='get_object', Params={ 'Bucket': bucket, 'Key': key } )
vidcap = cv2.VideoCapture(url)
OpenCV is expecting to access a file on the local disk.
You would need to download the file from Amazon S3, then reference that file.
Please note that AWS Lambda only provides 500MB of disk space, and only in the /tmp/ directory.
You can try to create a AWS CloudFront distribution for s3 bucket. Here is the tutorial link: Use CloudFront to serve HTTPS requests S3
I'm trying to create a aws bucket from python3 using boto3. create_bucket() is the method I use. Still I get an error botocore.errorfactory.BucketAlreadyExists
MY CODE:
import boto3
ACCESS_KEY = 'theaccesskey'
SECRET_KEY = 'thesecretkey'
S3 = boto3.client('s3',
aws_access_key_id = ACCESS_KEY,
aws_secret_access_key = SECRET_KEY)
response = S3.create_bucket(Bucket='mynewbucket',
CreateBucketConfiguration={'LocationConstraint':'ap-south-1'})
ERROR:
botocore.errorfactory.BucketAlreadyExists: An error occurred (BucketAlreadyExists)
when calling the CreateBucket operation: The requested bucket name is not available.
The bucket namespace is shared by all users of the system.
Please select a different name and try again.
However, the Bucket does not exist and it still failed to create the bucket.
EDIT
I found the reason from the link and I also posted that in answers in-order to help someone.
I got it after reading few articles on-line. The bucket name should be globally unique once it satifies that condition it works as I expect.
I share this to help someone wonders just like me
Reference
I am trying to copy from one bucket to another bucket and each bucket has their own access key and secret.
I can connect to the first bucket and down load a file just fine. It might be important to note that I do not have full access to the bucket I am copying from, meaning I can not read all keys in the bucket, just a subset I have access to. I have complete control on the second bucket I am copying to.
client2 is where I am copying to and client is where I am copying from.
copy_source = {
'Bucket': bucketName,
'Key': key
}
client2.copy(CopySource = copy_source,Bucket=bucketName2,Key=key,SourceClient=client)
Here is the error I get:
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the UploadPartCopy operation: Access Denied
I am a newbie and any help would be greatly appreciated!!
The reason you're likely getting the Access Denied on this is because the SourceClient is only used for getting the size of the object to determine if it can be copied directly, or if a multi-part upload is required.
When it comes to the actual copy itself, the underlying the underlying copy_object method on the client, which does not accept a SourceClient, and calls out to the S3 APIs PUT Object - Copy method.
As such, if you want to be able to perform an S3 copy from one bucket to another, you can either give the user associated with the access key used by client2 permission to read from the Source bucket, or you can perform an S3 Get using client1 then an S3 Put with client2.
Looking for some help writing some code that will pull down all bucket names and identify which ones are externally visible (open to the internet for read or write). I've read over the documentation for boto3 s3 and cannot find any commands that will allow me to make this query... should I be looking under IAM?
So far, I am only able to print the bucket names... I would like to report name + its internet presence. The goal is to identify which s3 buckets are visible from the internet so we can periodically review the data/objects within them.
print("############# S3 Bucket Dump ############")
s3 = boto3.resource('s3')
f = open('s3buckets.txt', 'w')
for bucket in s3.buckets.all():
print(bucket.name, file=f)