If you have an app where users have data in S3 buckets but can select who they share it with, what's the best technique for protecting this data? For example, how would Instagram protect their image data if they were using S3 (or some other centralized storage provider) so you could only see pictures you were authorized to see?
Obscurity from large url strings seems like one approach, but I was curious if there was a better technique?
By default, all objects in Amazon S3 are private. You can then add permissions so that people can access your objects. This can be done via:
Access Control List that applies to individual objects
A Bucket Policy that applies rules to the whole bucket
IAM to apply permissions to specific Users and Groups
A Pre-Signed URL that grants temporary access to an individual object
If you wish to "select who to share it with", there are two choices:
If the person is defined as a User in IAM, then assign permissions against that User
If the person is not defined in IAM (eg an Instagram user), then use a pre-signed URL
A Pre-Signed URL grants access to S3 objects as a way of "overriding" access controls. A normally private object can be accessed via a URL by appending an expiry time and signature. This is a great way to serve private content from Amazon S3.
Basically, if the application determines that the user is entitled to access an object in Amazon S3, it can generate a link that provides temporary access to the object. Anyone with that link can access the object, but it will no longer work once the time period has expired.
The pre-signed URL can be generated via the AWS SDK (available for most popular programming languages). It can also be generated via the aws s3 presign command in the AWS Command-Line Interface (CLI).
Pre-signed URLs can even be used within web pages. For example, the HTML might refer to a picture using an <img> tag, where the src is a pre-signed URL. That way, a private picture can be displayed on the page, but search engines would not be able to scrape the picture.
Related
I have a few questions regarding firebase storage?
I am generating download URLs for firebase storage objects using and admin account (has custom claims) and storing the URL on Firestore.
Users can read the Firestore document to get the URL instead of having to call getDownloadUrl on the client side code.
Q1) I noticed there is a token at the end of the storage URLs. Is this specific to my admin account and is it safe that none admin users can now read this token?
Q2) Furthermore if a non admin user called getDownloadUrl on the same storage path would they receive the same URL as the admin account or a different one?
Q3) If I switch to using getDownloadUrl on the client side would this increase my cost when using firebase storage?
Q4) If i am caching the content by URL and the URL changes it will redownload and not use cache.. Are these download links unique or can getDownloadURL return different URLs on subsequent calls?
Thanks a lot
Edit ---
Sorry I have an additional question
Q5)To move files on firebase storage I currently download them to my local pc and reupload them to another location -- seems very inefficient.
I have seem people using file.move() (as can be seen here.)
Would this be possible to call in a firebase function (as they talk storage rules being an issue in the comments, although its from 2016) and if so how would this be cheaper than my manual download and upload?
Sorry for many questions :)
Q1) I noticed there is a token at the end of the storage URLs. Is this specific to my admin account and is it safe that none admin users can now read this token?
This token is a a random ID generated for this specific file. It won't change, unless you change it intentionally (you can "revoke" the token from the Firebase Console, which will replace it with a new token). Everyone who possesses the URL can view the file whether they are authenticated or not. However, the URL is "hard to guess", so unless you share it with anyone, it will stay secret, practically speaking.
Q2) Furthermore if a non admin user called getDownloadUrl on the same storage path would they receive the same URL as the admin account or a different one?
The returned URL will always be the same, unless you invalidate it in the Firebase Console. If you don't want clients to call getDownloadURL on the files, add a Storage Security Rule that denies reads:
match /path/to/{file} {
allow read: if false;
// Or, if only authed users should be able to call getDownloadURL:
allow read: if request.auth != null;
}
Q3) If I switch to using getDownloadUrl on the client side would this increase my cost when using firebase storage?
A call to getDownloadUrl() does utilize some Google Cloud resources that you will have to pay for, whether you do it server-side or client-side. It's a "Class B" operation (check Google Cloud pricing), and a bit of data transfer.
Q4) If i am caching the content by URL and the URL changes it will redownload and not use cache.. Are these download links unique or can getDownloadURL return different URLs on subsequent calls?
The same URL is return each time, unless you manually invalidate the token. (By the way, the caching policy that sets the Cache-Control header is set on the object as metadata when you upload it.)
Q5) To move files on firebase storage I currently download them to my local pc and reupload them to another location -- seems very inefficient. [..] Would this be possible to call in a firebase function
Yes, you can move files in a Firebase Cloud Function. The Firebase Admin SDKs bypasses security rules.
1) I noticed there is a token at the end of the storage URLs. Is this specific to my admin account and is it safe that none admin users can now read this token?
Depends on what you have at the moment since you can integrate Custom Authentication with Firebase which will allow you to create custom tokens that can be used to sign into the Firebase Authentication service on a client application and assume the identity described by the token’s claim. This can be used when accessing other Firebase services, such as Cloud Storage, etc.
In general your server should create a custom token with a unique identifier.
2) Furthermore if a non admin user called getDownloadUrl on the same storage path would they receive the same URL as the admin account or a different one?
Depends on how you are setting the permissions for the getDownloadUrl. If you have a customized one they can receive a different one but usually it returns a new instance that points to the current reference.
3) If I switch to using getDownloadUrl on the client side would this increase my cost when using firebase storage?
I am not sure about this, I have checked the documentation and there is nothing that would indicate a quota or pricing on this specific method so I would go ahead and assume that it would not do it but I might be wrong on this one.
4) If i am caching the content by URL and the URL changes it will redownload and not use cache.. Are these download links unique or can getDownloadURL return different URLs on subsequent calls?
As specified before, it returns a new instance that points to the current reference so these download links are unique.
5) To move files on firebase storage I currently download them to my local pc and reupload them to another location -- seems very inefficient.
For this question and the last part of your initial post I would suggest you to create a support ticket and ask more details to the Firebase Support Team where you can get more information regarding this since it is more suited for them than to StackOverflow. (https://firebase.google.com/support)
As it turned out there is no API for invalidating of presigned urls, but it is possible to drop access from IAM policy. If I have a service with many users (in Cognito Userpool) - what is the correct design for some kind of url invalidation? Do I need to have as many IAM accounts as a users I have? Now I think that I can use {directory-names-of-long-random-crypto-tokens-for-s3-that-I-will-rename}/{when-filename-will-remained-the-same.txt}, tokens from secrets.urlsafe_token(99), but with such approach I will get duplicated files - different users can have access to the same files.
Another solution - I can use lambda for every call for a file, but I think that simpler, cheaper and faster would be to have duplicated files in S3 (maybe a few percent of wasted storage).
I checked CloudFront - looks like no way to use different urls/params mapping to the same S3 file.
i have an Azure Blob Storage with blobs that are pdf that are categorized by client number. So for each client, they have multiple pdf reports. I only want the client to be able to access the blobs for their client number. (There are hundreds of clients.)
I've researched, but only see shared access signatures, but this doesn't look like what i need.
There is no user-level blob permissions, other than Shared Access Signatures (and Policies).
It's going to be up to you to manage access to specific user content (and how you manage that is really up to you and your app, and how you manage a user's content metadata).
When providing a link to a user's content: if you assume all content is always private, then simply create an on-demand SAS link when requested. There's no way for the user to modify a SAS link to guess sequential numbers or neighboring blobs, since the SAS is for a specific URL.
As Andrés suggested, you could also use your app to stream blob content, and never worry about SAS. However, you will now be consuming resources of your web app (network, CPU, memory), and this will have an impact on your app's scale requirements. You will no longer be able to offload this to the storage service.
Sounds like you already have the users authenticate, and you know which pdfs belong to them. My suggestion is to add to your current application a simple proxy (for instance if you have an MVC application, you could add a new controller and action method that will retrieve the pdfs on behalf of the user).
This way you don't need to use shared access signature and can keep the blob container private. Your controller/action method would simply use the storage SDK to retrieve the blob. An added bonus is that you could check to make sure that they are requesting their own PDF file and reject the request if they guess the ID of someone else's file.
I have an image in my AWS S3 bucket. Is it safe to include this image in my website by placing the AWS URL in an <img> tag? The URL includes parameters such as "Amz-Signature", "Amz-Credential", and "amz-security-token. Could these be used maliciously to get to access other files in my S3 bucket?
Here is an example URL:
https://s3.amazonaws.com/MyBucketName/FileName.jpg?X-Amz-Date=20160126T141139Z&X-Amz-Expires=300&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Signature=Lots_of_letters_and_Numbers2&X-Amz-Credential=MYAMAZON_CREDENTIALS/20160126/us-east-1/s3/aws4_request&X-Amz-SignedHeaders=Host&x-amz-security-token=REALLY_LONG_SECURITYTOKEN
Alternatively, I can generate an expiry URL from my C# code using the AWS SDK. Something like:
var expiryUrlRequest = new GetPreSignedUrlRequest
{
BucketName = WebConfigurationManager.AppSettings["AWSBucketName"],
Key = fileName,
Expires = DateTime.Now.AddHours(3)
};
This yields a URL that has "AWSAccessKeyId" as a parameter.
Are either of these URL's safe to use in my webpage? What risks would be involved in using them on my site?
Thank you very much for your time. Please let me know if you need additional information or if I am being unclear.
EDIT: To provide some further insight into my application, users are uploading a file to an S3 bucket. I'm using SignalR to confirm that the image is in the bucket by displaying the image from S3 on my webpage for the user to see.
Do not make the bucket public. If you do, then potentially user1 could see user2's uploaded files.
You can allow users to retrieve single files for a specific period of time using pre-signed URLs.
Mark the S3 bucket as private.
Use GetPreSignedUrlRequest to generate a pre-signed URL for the file you want the user to download.
Use that URL in your <img> tag.
Using this technique is safe:
The user can only download the file during the timeframe that you permit, until the expiration date (which you set as part of the GetPreSignedUrlRequest call)
The credentials you see in the URL are may be the same as those that were used to create the URL. But they are safe to show the user.
The user cannot download any other files from the bucket.
The URL uses a hashing technique to ensure the URL cannot be modified, nor can it be abused to get other files.
If displaying the access key ID is a concern, you can either (a) create an IAM user specifically for the purpose of downloading the files from S3, or (b) use an IAM role on your EC2 instance to generate the pre-signed URL.
References:
http://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html
First of all, there are two ways of restricting access to the content of a bucket:
Private - users need the AWS credentials to access that file (in the same way as shown in your answer)
Public - everybody can access the content of the bucket (https://myBucket.s3.amazonaws.com/myFolder/myFile.jpg)
If you want other users to access the image (for example by providing a URL for your website), you should mark the bucket as public.
DO NOT post the url with your AWS credentials anywhere!!!
Is this safe? Maintaining security using a pre-signed url with AWS S3 Bucket object?
my link
Another words - part 1...
say I'm storing a bunch of separate individual's files in a bucket. I want to provide a link to a file for a user. Obviously, each file is uniquely but consecutively named, I don't want people to be able to change the link from 40.pdf to 30.pdf and get a different file. This URL seems to do that.
part 2, and more importantly....
Is this safe or is a it dangerous method of displaying a URL in terms of the security of my bucket? Clearly, i will be giving away my "access key" here, but of course, not my "secret".
Already answered 3 years ago... sorry.
How secure are Amazon AWS Access keys?
AWS Security Credentials are used when making API calls to AWS. They consist of two components:
Access Key (eg AKIAISEMTXNOG4ABPC6Q): This is similar to a username. It is okay for people to see it.
Secret Key: This is a long string of random characters that is a shared secret between you and AWS. When making API calls, the SDK uses the shared secret to 'sign' your API calls. This is a one-way hash, so people cannot reverse-engineer your secret key. The secret key should be kept private.
A Signed URL is a method of granting time-limited access to an S3 object. The URL contains the Access Key and a Signature, which is a one-way hash calculated from the object, expiry time and the Secret Key.
A Signed URL is safe because:
It is valid for only a limited time period that you specify
It is valid only for the Amazon S3 object that you specify
It cannot be used to retrieve a different object nor can the time period be modified (because it would invalidate the signature)
However, anyone can use the URL during the valid time period. So, if somebody Tweets the URL, many people could potentially access the object until the expiry time. This potential security threat should be weighed against the benefit of serving traffic directly from Amazon S3 rather than having to run your own web servers.