As you can connect your S3 bucket to a simple domain like images.yourdomain.com, what is the advantage in using the S3Reader rather than simply using the RemoteReader?
I know that RemoteReader will open and resample all images for security reasons, but thats no problem.
Could I even switch between cloud providers by simply setting the nameserver to another destination?
You can use RemoteReader to connect to any public HTTP-accessible datastore, including Amazon S3.
However, S3Reader offers many benefits
Less error-prone whitelisting. Validate by bucket name instead of managing the permutations for each S3 region and address method.
Optional file invalidation (RemoteReader cannot invalidate if a resource changes)
Optional SSL encryption
Optional IAM authentication so you can hide the source files, while serving resized versions.
Related
What's the workaround for having multiple AWS keys in Hadoop environment? My hadoop jobs will require access to two different S3 buckets (two different keys). Tried with "credential" provider but looks like it's pretty limited. It stores all keys in lower case, as a result I cannot use "s3a" for one job and "s3n" for other job. For example: for s3a, it looks for:
fs.s3a.access.key
fs.s3a.secret.key
And for s3n:
fs.s3n.awsAccessKeyId
fs.s3n.awsSecretAccessKey
But if I create provider with "fs.s3n.awsAccessKeyId", it stores as "fs.s3n.awsaccesskeyid", as a result, during runtime it fails to load the expected key.
As a workaround, I tried to generate two different credential providers and pass as:
--Dhadoop.security.credential.provider.path=key1,key2
But it didn't work togher as both of the keys have fs.s3a.access.key & fs.s3a.secrety.key pair.
I don't want to pass access and secret key using -D option as it's visible. Is there any better way to handle this scenario?
If you upgrade to Hadoop 2.8 you can use the per-bucket configurations to address this problem. Everything in fs.s3a.bucket.$BUCKETNAME is patched into the config for the FS instance for that bucket, overriding any other configs
fs.s3a.bucket.engineering.access.key=AAID..
fs.s3a.bucket.logs.access.key=AB14...
We use this a lot for talking to buckets in different regions, encryption, other things. Works well, so far. Though I would say that.
Special exception: if you encrypt credential secrets in JCECKS files. The docs cover this.
I am trying to utilize S3 to let my clients download my software package. What I envision is creating a simple web app with authorization For example (download.mysoftware.com) Once the user is authenticated, they will be presented with a S3 url used to download the software. I will create user accounts based on my customers.
My concern is, what happens if the user copies the S3 URL link and then gives it to someone who isn't authenticated to download the software?
Is it possible to create an S3 policy that would prevent this and work for my usecase? I looked at allowing only specific IPs, however, I won't have a way to find out IP of my customers and wouldn't want to ask them first and then add it to the policy each time.
One way allowing specific IPs would work is if I allow downloads only from the IP that is linked to (download.mysoftware.com) but then the downloads will really be happened from my web application as opposed to from S3. Which seems like double effort.
When a user makes a request to download your application, generate a pre-signed URL for them with a short expiration. It only needs to be valid for as long as it takes them to start downloading your file, so even a few minutes of validity is likely to be plenty.
While it's technically possible for a user to share one of these URLs, they would have to convey the URL to someone else and get them to download it very quickly, which is probably sufficient to deter them from trying to share the URL. (A perfect defense is more difficult, and is probably unnecessary anyways; there's no way to prevent a user from personally transferring a file they downloaded to someone else.)
I want to host copyrighted data on a Amazon S3 bucket (to have a larger bandwidth available than what my servers can handle) and provide access to these copyrighted data for a large numbers of authorized clients.
My problem is:
i create signed expiring HTTPS URL for these resources on the server side
these URL are sent to clients via a HTTPS connection
when the client uses these URL to download the contents, the URL can be seen in clear for any man-in-the-middle
In details, the URL are created via a Ruby On Rails server using the fog gem.
The mobile clients I'm talking about are iOS devices.
The proxy I've used for my test is mitmproxy.
The URL I generated looked like this:
https://mybucket.s3.amazonaws.com/myFileKey?AWSAccessKeyId=AAA&Signature=BBB&Expires=CCC
I'm not a network or security expert but I had found resources stating nothing was going clear over HTTPS connections (for instance, cf. Are HTTPS headers encrypted?). Is it a misconfiguration of my test that led to this clear URL? Any tip on what could have gone wrong here? Is there a real chance I can prevent S3 URL to go clear over the network?
So firstly, when sending a request over SSL all parameters are encrypted. If you were to look at the traffic going through a normal proxy you wouldn't be able to read them.
However, many proxies allow interception of SSL data by creating dummy certificates. This is exactly what mitmproxy does. You may well have enabled this and not realised it (although you would have had to install a client-side certificate to do this).
The bottom line is that your AWS URLs could be easily intercepted by somebody looking to reverse engineer your app, either through a proxy or by tapping into the binary itself. However, this isn't a 'bad thing' per se: Amazon themselves know this happens, and that's why they're not sending the secret key directly in the URL itself, but using a signature.
I don't think this is a huge problem for you: after all, you're creating URLs that expire, so even if someone can get hold of them through a proxy they'll only be able to access the URL for as long as it is valid. To access your resources post-expiry would require direct access to your secret key. Now, it actually turns out this isn't impossible (since you've probably hard-coded it into your binary), but it's difficult enough that most users won't be bothering with it.
I'd encourage you to be realistic with your security and copyright prevention: when you've got client-side native code it's not a matter of if it gets broken but when.
I'm quite new to Amazon S3 and looking for a way to secure large amount of data on s3 that will be accessed by users via our website.
The issue I have is that I don't want the data on Amazon s3 to be public at all so it need sto be private but still accessible via our site.
Presigning each url request would be a pain due to the amount of new files being accessed and added all the time. I know there is option http_referer mod, should be
quick and easy, and maybe in combination with very obscure URLs
(long random codes in the path) that might be enough?
Is there any other way to make s3 urls secure?
Cheers
S3 does provide the ability to sign an image so that the URL is only valid for a limited period of time. There are more details of that here
There are a number of ways of handling access as described here
What are the options for secure password/credential storage on a host and propagation of changes across a fleet of hosts?
An example would be you have fleet of size N and you want to store credentials, such as AWS access keys, on those hosts. The simple approach is to store it in the source code or a config file, but this is bad because it's usually plain text. Also, if you make a change to the credentials you then want them to propagate to all of the hosts in the fleet.
Are there any solutions that would allow for something along these lines?
Credentials credentials = new Credentials();
system.doAction( credentials.getCredential("CredentialKeyId") );
This is assuming a non-Windows fleet.
Are you asking for something similar to NIS? To Kerberos, maybe?