I'm quite new to Amazon S3 and looking for a way to secure large amount of data on s3 that will be accessed by users via our website.
The issue I have is that I don't want the data on Amazon s3 to be public at all so it need sto be private but still accessible via our site.
Presigning each url request would be a pain due to the amount of new files being accessed and added all the time. I know there is option http_referer mod, should be
quick and easy, and maybe in combination with very obscure URLs
(long random codes in the path) that might be enough?
Is there any other way to make s3 urls secure?
Cheers
S3 does provide the ability to sign an image so that the URL is only valid for a limited period of time. There are more details of that here
There are a number of ways of handling access as described here
Related
I'm building a social app, where users can choose what profile-picture they wish to display to others. I've currently implemented it as the following:
User sends the profile-picture file to my server
My server validates that the file size is not too large
My server uploads the file to a AWS S3 bucket, with a UUID prefix
The S3 bucket allows anyone in the world to download specific files, but not perform write/list operations
I store the profile-picture URL in my database, and send the link to other users as appropriate
The client for other users, upon encountering the profile-picture URL, will download the file and display it to the user
I'm now wondering if it's possible for a malicious user to upload a malware-file as his profile-picture, and take advantage of the fact that this malware will be stored in my S3 bucket, and downloaded onto many other users' phones.
How much of a concern is the above? Is there anything simple I can do to prevent such concerns? If not, what is the recommended way to store/distribute user-uploaded profile-pictures to others?
Edit: My server is in Java. Any Java specific library/tool advice will be appreciated as well
I have to build a project where users will login and upload some images and videos, and it should be such that the files uploaded should only be visible by the uploader or the site admin.
I main Node.js, so i tried using express middle wares to restrict the media files by user, but it came to my notice that this isn't the best way to handle this as express isn't good at rendering static content.
Here are some options i can think of after some google sessions
Amazon S3 bucket where each user gets their own folder/permissions and files no into this (but are the files truly private when we have a url)
Generate a temporary URL of the files using pre-signed URLs from S3 bucket ( the file will be public for 20 min, i don't want this)
Restrict access on Nginx ( again i don't know if Nginx can access the database and authenticate the request it got)
Use GridFS with mongoDB? (i will probably not use this, but wanna see if this can be a solution)
is there any other way to do this?
Each user is given a unique ID where content (files & videos etc) is referenced in part by this ID such that clients are given access only to their ID content
User logs in, nodejs pairs that user with their unique ID as retrieved from mongodb. No need to offload this to nginx. Where you stow the content is independent of this logic. Could be a local filesystem, mongo, S3, etc ...
Avoid putting username or ID into any URL, its redundant, server maintains this knowledge internally. No need to muddy up URL. For example Amazon AWS has this URL when I interact with my private content
https://console.aws.amazon.com/route53/home?#resource-record-sets:ZAHKXPA7IKZ8W
See it just shows the content ID which is unique to my username's resources. Goal is to minimize clutter in URL.
Alternatively, in the world of sharing resources, like on flicker, here the URL does give username and resource ID https://www.flickr.com/photos/okinawa-soba/31419099190/in/photostream/ In your case even if someone picks up the URL of another's content the server must reject since its not authorized to muck with another's content
The most elegant way would be to politely ask them to refrain for accessing other people content.
I'm not sure it would be the most effective way, though.
Seriously, though, I would suggest using express.static middleware together with Passport or something to implement authentication and permissions. You're not "rendering" any static content here. You're just streaming files straight from the disk.
I am trying to utilize S3 to let my clients download my software package. What I envision is creating a simple web app with authorization For example (download.mysoftware.com) Once the user is authenticated, they will be presented with a S3 url used to download the software. I will create user accounts based on my customers.
My concern is, what happens if the user copies the S3 URL link and then gives it to someone who isn't authenticated to download the software?
Is it possible to create an S3 policy that would prevent this and work for my usecase? I looked at allowing only specific IPs, however, I won't have a way to find out IP of my customers and wouldn't want to ask them first and then add it to the policy each time.
One way allowing specific IPs would work is if I allow downloads only from the IP that is linked to (download.mysoftware.com) but then the downloads will really be happened from my web application as opposed to from S3. Which seems like double effort.
When a user makes a request to download your application, generate a pre-signed URL for them with a short expiration. It only needs to be valid for as long as it takes them to start downloading your file, so even a few minutes of validity is likely to be plenty.
While it's technically possible for a user to share one of these URLs, they would have to convey the URL to someone else and get them to download it very quickly, which is probably sufficient to deter them from trying to share the URL. (A perfect defense is more difficult, and is probably unnecessary anyways; there's no way to prevent a user from personally transferring a file they downloaded to someone else.)
Suppose we have a web application for ebook shopping, users can only download books they have paid. And all those ebooks data(like epub data) are stored in S3, users will got a S3 download url when our app validated the user already bought one book.
I know S3 will generate a temporary and time-limited URLs to the user. But the problem is even the url is temporary and time-limited, what will happen if by any chance another user get the temporary url within the limited time?
Ok so one way seems to be letting the limited-time very short, like 10 seconds, but the potential risks still there.
I know this question is rigorous, how would you deal with it? Or forget it?
If the concern is that the buyer would distribute the URL, they could equally well distribute the file they've downloaded.
If the concern is that someone would get hold of the URL without the buyer's knowledge, then perhaps you could protect the download with a token that only the buyer would know (e.g. a password of some sort). This would make the URL unusable to anyone other than the buyer. The downside is that this will inconvenience the very people who've paid for your product.
I'm trying to create a private podcast feeds. Each user of my service gets an account and depending on what they pay, they will receive different content. Some content is sensitive so security is reasonably important; if any of the enclosed audio files made the rounds around the internet, it could be catastrophic for our business.
I am currently prototyping the service and MP3s are stored on S3 and they are not secured. It's time to secure them. From my research, I understand that I can, in fact, secure files on S3 with an access expiry period. But, because I have many users, signing the request with my "global" key and have an expiry probably isn't a good idea because
If I need to revoke access, I'll need to do it for everyone
Since I don't know when, exactly, their podcatcher will request the file, I don't know when to set the expiry. Sometimes, the podcatcher downloads the feed XML, but only later fetches the MP3 file so the URLs could expire before the client has a chance to fetch them (I'm thinking about iTunes, but there could be others).
The way I see it, I have two options and I'm not sure if either are workable:
Edit Another potential way, I suppose, would be to role my own security, and simply redirect to an MP3 on S3 with a short expiry if the user is good to go. This seems most sensible.
I can create user accounts on Amazon for ALL of my users and link them to an Amazon token in my database. Everyone's MP3 urls are signed with their secret token and expiry a long time from now. I don't like the idea of storing their tokens on my database, and I'm not sure if the Amazon ACL was designed for this scenario. It also means the files can still be shared if you know the URL.
I can proxy every request through my server. This means I only have one Amazon account, and I can role my own security system. But proxying every MP3 download through my server sounds slow, wasteful and expensive.
Any ideas on the best way to do this.
P.S. I'm not married to S3. Other solutions could be considered. And I'm on Heroku using Ruby, in case you care.
I am going with the redirect solution suggested elsewhere. This seems to give me the most flexibility and very low overhead. What you definitely should not do is is using links directly to S3 in your feed, because they likely expire before they are downloaded by the client.