I'm creating a way for clients to upload large files to S3 for us to process.
I built a mechanism that allows clients to send the list of files they have and in return they get an HTTP request they need to send to S3, along with the file attached, for one of the files they offered. This removes the strain of uploads from our server and we can pick up any file that has been uploaded via notification from the S3 bucket.
My problem is with replay attacks. If a certain party asks to send a file and receives the request back, they can replay the same request over and over again, costing us in requests. I don't care about overwriting the file since the Contents-MD5 header forces the file to be the same file (conflicts notwithstanding). I also don't care about being notified about the file completing upload again.
I thought about generating a policy that only allows uploads with a specific token, which changes every X minutes. Should someone want to replay an attack, they would fail and have to re-request an S3 request from us (which would fail, since the upload already completed beforehand). I'm not sure how much of a best practice it would be to rotate such a token and worried it would also cause lots of legitimate requests that are taking too long to start to fail.
Is there any other mechanism I'm not aware of that should be used in this case?
worried it would also cause lots of legitimate requests that are taking too long to start to fail.
You can dismiss that particular worry, by signing the URLs with a short expiration time. Authentication and authorization, including signature validation, happens at the beginning of the request. S3 won't cut an upload or download short because the signature expires in the middle of a long request.
Changing bucket policies programmatically, repeatedly, on the fly is definitely not a best practice.
Note that it is does not appear to be clearly documented, but when S3 denies a request, the (negligible, but still non-zero) per-request charge apparently still applies, so having S3 refuse a redundant overwrite of the same object with identical content is unlikely to be a solution worth achieving.
Related
I am creating a Chromium/Electron based Mac app. The app is essentially a browser for my customers to use a web service that I have no control over. My requirement is that users of my app (who may have root access on their Mac) should not be able to view the URLs the app is visiting, and should be unable to gain access to the cookies the app is storing. Normally it is not hard to MITM yourself, or attach a debugger to an app and dump memory to see the URLs and cookies.
How can I prevent these types of leaks to the user? If it's impossible, it may be acceptable to make it very hard so that a very high level of sophistication is needed.
Your users have full control of their devices, it is not possible to securely prevent them from proxying or exploring what your client-side app does. Obfuscation would seem like an option, but in the end, the http request that leaves your app will traverse the whole OS through different layers, and your user can easily observe that, if not else then in network packets (but usually much easier).
The only way it is possible to prevent the user from knowing what's happening is if you have your own backend. The frontend app (Electron) would make a request to your backend, which in turn could make any request with any parameters without the user being aware.
Note though that your backend could still be used as a proxy or oracle just like if the user was connecting to the real service. This might or might not be a problem in your case, depending on what you actually want to achieve and why.
The app is essentially a browser for my customers to use a web service that I have no control over. My requirement is that users of my app (who may have root access on their Mac) should not be able to view the URLs the app is visiting, and should be unable to gain access to the cookies the app is storing
Basically, you cannot (you could with the appropriate infrastructure. But you lack that infrastructure).
Network communications can be secured, to a point, using HTTPS (if you can't even use that, then you're completely out of luck - users wouldn't even need root access to the Mac to sniff traffic). You need to verify the server certificate to be sure you're connecting to the correct server.
One thing you might do - effectual just against wannabes, I'm afraid - is first run a test API call on some random server and verify that the connection either fully succeeds, with the proper server identification and matching IP, if the server exists, or that it properly fails if the server never existed. Anything else would be a telltale that someone has taken over the network layer, and at that point you could connect to a different server, making different calls, and lament that the server isn't answering properly.
Strings in memory can be (air quote) protected (end air quote) by having them available only for the shortest time, and otherwise stored in a different form - you can have for example an URL and a random byte sequence with the same length, then store the sequence and the XOR of the URL and the sequence. You can then reconstruct the URL every time you need it, remembering to clear it off any app caches it might find its way into. Also, just for the lols, you can keep a baker's dozen of different URLs sprinkled in the clear throughout the code. A memory dump at that point will turn out nothing useful.
Files, of course, can be encrypted with any one of several schemes - the files residing on the same machine that has to know how to decode them makes all such schemes ultimately vulnerable, but there again, you can try and obfuscate things. I once stored some information in a ZIP file - but it was just the header of an encrypted ZIP file, with the appropriate directory entry block glued at the end. The data were actually just gzipped in the clear, there was no password whatsoever. The guys that tried to decode the file thought it was a plain encrypted Zip file with the extension changed, wasted a significant amount of time with several Zip cracking tools, and ended up owing me a beer.
More than that, there is not much that can realistically be done.
A big advantage would be in outsourcing the API calls and "cookie" maintenance to an external service that you control, e.g. on Amazon AWS or Azure or similar. Then you could employ all kinds of protection schemes (for example: all outbound API calls could be stored in an opaque object, timestamped, nonced, and encrypted with your server's public key, and the responses sent encrypted with your client's unique key). Since this is relatively simple and cost-effective, it would also be my recommendation.
I am trying to utilize S3 to let my clients download my software package. What I envision is creating a simple web app with authorization For example (download.mysoftware.com) Once the user is authenticated, they will be presented with a S3 url used to download the software. I will create user accounts based on my customers.
My concern is, what happens if the user copies the S3 URL link and then gives it to someone who isn't authenticated to download the software?
Is it possible to create an S3 policy that would prevent this and work for my usecase? I looked at allowing only specific IPs, however, I won't have a way to find out IP of my customers and wouldn't want to ask them first and then add it to the policy each time.
One way allowing specific IPs would work is if I allow downloads only from the IP that is linked to (download.mysoftware.com) but then the downloads will really be happened from my web application as opposed to from S3. Which seems like double effort.
When a user makes a request to download your application, generate a pre-signed URL for them with a short expiration. It only needs to be valid for as long as it takes them to start downloading your file, so even a few minutes of validity is likely to be plenty.
While it's technically possible for a user to share one of these URLs, they would have to convey the URL to someone else and get them to download it very quickly, which is probably sufficient to deter them from trying to share the URL. (A perfect defense is more difficult, and is probably unnecessary anyways; there's no way to prevent a user from personally transferring a file they downloaded to someone else.)
I'm trying to create a private podcast feeds. Each user of my service gets an account and depending on what they pay, they will receive different content. Some content is sensitive so security is reasonably important; if any of the enclosed audio files made the rounds around the internet, it could be catastrophic for our business.
I am currently prototyping the service and MP3s are stored on S3 and they are not secured. It's time to secure them. From my research, I understand that I can, in fact, secure files on S3 with an access expiry period. But, because I have many users, signing the request with my "global" key and have an expiry probably isn't a good idea because
If I need to revoke access, I'll need to do it for everyone
Since I don't know when, exactly, their podcatcher will request the file, I don't know when to set the expiry. Sometimes, the podcatcher downloads the feed XML, but only later fetches the MP3 file so the URLs could expire before the client has a chance to fetch them (I'm thinking about iTunes, but there could be others).
The way I see it, I have two options and I'm not sure if either are workable:
Edit Another potential way, I suppose, would be to role my own security, and simply redirect to an MP3 on S3 with a short expiry if the user is good to go. This seems most sensible.
I can create user accounts on Amazon for ALL of my users and link them to an Amazon token in my database. Everyone's MP3 urls are signed with their secret token and expiry a long time from now. I don't like the idea of storing their tokens on my database, and I'm not sure if the Amazon ACL was designed for this scenario. It also means the files can still be shared if you know the URL.
I can proxy every request through my server. This means I only have one Amazon account, and I can role my own security system. But proxying every MP3 download through my server sounds slow, wasteful and expensive.
Any ideas on the best way to do this.
P.S. I'm not married to S3. Other solutions could be considered. And I'm on Heroku using Ruby, in case you care.
I am going with the redirect solution suggested elsewhere. This seems to give me the most flexibility and very low overhead. What you definitely should not do is is using links directly to S3 in your feed, because they likely expire before they are downloaded by the client.
I have a question concerning cookie storage design. I am developing a web application which should cache it's server-fetched data to a local storage. No user credentials will be stored.
What is in the cookie:
list of data and it's properties
proof for up-to-dateness
Proof for up-to-dateness will not be a hash, but most likely a timestamp of the last write to the server which is checked against the DB. This is to ensure the user gets valid info if he has used the website on another browser/computer/device and is out of sync.
The cookie should be able to handle more than one user in it and most probably some sort of encryption so other people can't see plain-text data. Military-grade security isn't needed here as the information here is not so important. But everything hacked in less than 30-60 minutes should be considered unsafe.
Questions:
How to encrypt my data
How to enable the cookie for multiple users
How to prevent the cookie from being stolen
What would be a good and simple way to present the option to disable cookie caching and explain to my users the risks of using caching on public computers
Is the whole idea any good at all
What are some potential issues I haven't accounted for
I'll answer #5 (which renders the rest of the questions moot).
Cookies aren't designed for that sort of thing. They are make a round trip with every HTTP requests - including on same domain CSS, images, JS, etc.
I suggest you look at HTML 5 local storage or just sending the data down to the browser every time with a minimal key cookie.
For #3 - there isn't a way to fool proof way prevent someone from stealing a cookie or forging a duplicate.
Introduction
I want to create a Java web application for storing and backing up user files, similar to Dropbox. One of the interesting Dropbox feature is that it can detect whether a certain file already exists on server. For example, if one user upload a file onto server, another user who tries to upload the same file will not need to upload the same file content. Server will only need mark that he has the same file. This helps to save the bandwidth/space and increases the speed in many ways.
The most basic solution to this problem is to use a file hash string, e.g. sha1, md5, etc., to identify the file. The client software check whether a certain hash exists on server or not. If it exists, then it can skip the uploading process and mark that user has the same file.
Problem
The web application is implemented based on REST architecture so that user can easily write their own client software to upload their files. For security reasons, the SSL is enabled for all transactions. But my most security concern is about users faking that they have a file without actually owning it if I use sha1 or any other standard hash alogorithms. This cannot be prevented by SSL or encryption. If a user manage to get the hash string, e.g. md5 and sha1 of many files can be found by googling, he can mark that he has the file using REST service on the web application.
So one of the possible solution is that the server requests a set of certain random bytes from the file as well as the hash of the whole file. Here is example steps:
Client checks whether a certain hash exists on server or not. Then, server returns the required positions of random bytes if the file already exists.
Client sends random bytes as per request if the server has the file. Client software will not be able to response to it without having the actual file.
In this way, it can save the bandwidth as well as ensure that user owns the file they want to upload.
Question
I am no expert in Security over the web so I have no idea whether this is a good idea or not. I have read some articles about implementing their own fancy process might lead to the reduction in security strength because the security cannot be tested and the extra information may provide a cracking method.
Does anyone has any comment on the process?
Will it reduce the sucurity?
Does anyone have an idea to solve this problem differently?
I understand that there might not be an exactly answer to this question but I would like to hear if anyone has encounter the same problem and has any good solution to it.
Rather than asking the client to upload some random bytes of the file's contents, it may be better to ask the client to upload the hash of a random region the file. That way you can use a wider range of sizes that you ask the client to verify.
Better yet, though, may be to send the client a random number and require the client to compute an HMAC of the entire file's contents using that number as the key. This is more computationally-expensive since the server must compute the HMAC too, but it verifies that the client has the entire file, not just a small portion of it.
One unavoidable side effect of this hash feature, even with a verification scheme, is that it reveals that a copy of the file already exists somewhere on the server. That by itself may be sensitive information.
For the most stringent privacy protection, you should forego this feature and make each user upload their own copy of the file. You can use hash comparison on the server to avoid storing multiple copies of the file, transparently to the clients.