I would like to use the Content-MD5 header to supply Azure with a client-side generated MD5 hash of the files being uploaded. How would I accomplish this task in FineUploader?
Target audience: Firefox, Chrome, Safari, Opera and IE10+.
Judging by this answer I gather that it should be possible but at least half of our clients will run on very old (slow) computers. Is it feasible to do this client side MD5 hashing approach then?
Somewhat related side question: I was also thinking that it might be wise to set a maximum file size to reduce strain on the MD5 hashing algorithm. Maybe just below Azure's "chunking" limit? From a different view: Can I disable chunking?
Fine Uploader provides direct access to the underlying File or Blob object via the getFile(id) API method. Once you have access to this, you can make use of FileReader to read the file/blob bytes and generate a hash. To make hashing even easier, consider using a JS library like SparkMD5.
The best place to calculate this hash is probably in an onSubmit callback handler. Since the process of reading a file is asynchronous, you will have to return a promise in your handler, which is supported by Fine Uploader. Once the hash has been calculated, resolve the promise and set the hash as a parameter for the file using Fine Uploader's setParams API method.
Can I disable chunking?
Yes, you can, but you will lose all of the benefits associated with chunking, along with the resume feature and efficient retries should the file upload fail midway though.
Related
I'm using node:crypto API, namely createCipheriv() and createDecipheriv() with aes-256-gcm cipher to encode/decode a stream of data. However, it looks like I need to call decipher.setAuthTag() in order to decode the stream correctly, otherwise it throws an authentication error in the end (however the data is decoded correctly).
Is there a way to avoid using the authentication checks with this cipher? I'm using streams of data and it's very inconvenient to store auth tag with the data (I'm using multiple storage options, one of which is a plain filesystem). The data consistency can be checked by other means.
Or maybe you could recommend a universal auth tag storage option that I could use with streams (which doesn't require random access and rewind)?
You need to use the auth tag if you are using GCM. GCM is CTR with authentication built in, you could look into using a CTR cipher Node Forge has this option.
Alternatively, you could append your tag to your ciphertext and store everything together? Not sure what the details of your storage system are that make that inconvenient.
Our Google cloud Function has access to an encrypted API key which it can unencrypt by using an external service. Once the API key is unencrypted, is it then safe to cache the API key as a global variable so that in cases where a Google Cloud Function is reused, the unencrypted variable can be used instead of contacting the unencryption service?
EDIT:
Our thinking is that the function will use an unencrypted version of the API key when running (i.e. store it in its memory for use) and that it's cache, I believe, is in memory and per function, which to the best of my knowledge would make it no less safe to cache the unencrypted API key per function than get it and unencrypt it on every function invocation?
'Safe' was a bad word choice - there is no such thing as safe, everything is, to an extent, a balancing act.
Statistically speaking, the longer you hold sensitive information in memory, the easier it is for a bad actor to get a hold of it. But you can never really eliminate the chance of this happening. The issue is really how this bad actor gets into Cloud Functions. The moment this becomes a possibility, you've got a problem. This can happen by trusting third party code into your deployment, or someone getting a hold of your project's admin credentials or a lapse in security at Google.
But if you assume that there is no possibility of a bad actor entering the system, it doesn't really matter how long you hold on to something in memory, since you trust every bit of code that could access it (and of course Google for providing that memory).
The memory isn't held strictly "per function". It would be held "per function per instance". Depending on the load, you could have many server instances all decrypting and holding sensitive information. But the code running on the instance would only be triggered from the one function and never others.
Caching API keys in memory in this manner does make changing API keys a bit more complex if you had to quickly change them i.e. due to a leak - a way round could be to also store a timestamp in a global variable and invalidate the key after x amount of time has passed or to restart all of the functions so the memory cache is cleared and fresh versions of the API keys are fetched, which would happen if you push a new version of the function to Google Cloud Functions
I'm using the Javascript SDK to make a web page that is entirely static HTML and Javascript (i.e., it's not dynamically produced web markup via some web app). This web page occasionally uses Javascript to POST data to a server--data which should be tied to a particular Facebook user. I use FB.getLoginStatus to determine who the user is. This gives me authRepsonse JSON data which looks like this:
authResponse:
accessToken:"AAAC91..."
expiresIn: 3786
signedRequest:"Ws93YNGWQeOi..."
userID: "670..."
I can send the signedRequest to the server and decode it and validate it there (using my app's secret key), and then I know that the user is, in this case, "670...", so I can presumably safetly perform whatever operation is supposed to happen on the server. Here's the data I extract from the signed request:
{"algorithm"=>"HMAC-SHA256",
"code"=>
"2.AQAKT...|5hVFYWcu5a...",
"issued_at"=>1323403518,
"user_id"=>"670..."}
My question is, what prevents an adversary (who somehow got ahold of the encoded authResponse above) from just "replaying" the signedRequest data to my server at a much later time?
The "issued_at" param at first looked promising, but I don't have anything to compare that issued_at time to to see if I should accept this signedRequest or not. The "expiresIn" is another time related parameter, but it's not signed, so I can't trust it. Maybe "code" provides me with extra info, but I don't see how to decode that.
I expect I'm just thinking about this wrong, or using the API in a way I'm not supposed to. Any insights? Thanks.
First of all using an Message Authentication Code (MAC) is a fundamentally insecure approach to the problem of authentication. You should be storing this information as a server side state, so that this is never a threat. By using a cryptographic hash function as an HMAC you introduce the possibility of someone brute forcing your secret key. Cryptography should only be used when there is no other solution, instead you are using it to introduce a weakness. This is a gross misuse of cryptography.
That being said, you have an issued_at timestamp. Just take the current timestamp and subtract. make sure that value is greater than your session timeout.
Introduction
I want to create a Java web application for storing and backing up user files, similar to Dropbox. One of the interesting Dropbox feature is that it can detect whether a certain file already exists on server. For example, if one user upload a file onto server, another user who tries to upload the same file will not need to upload the same file content. Server will only need mark that he has the same file. This helps to save the bandwidth/space and increases the speed in many ways.
The most basic solution to this problem is to use a file hash string, e.g. sha1, md5, etc., to identify the file. The client software check whether a certain hash exists on server or not. If it exists, then it can skip the uploading process and mark that user has the same file.
Problem
The web application is implemented based on REST architecture so that user can easily write their own client software to upload their files. For security reasons, the SSL is enabled for all transactions. But my most security concern is about users faking that they have a file without actually owning it if I use sha1 or any other standard hash alogorithms. This cannot be prevented by SSL or encryption. If a user manage to get the hash string, e.g. md5 and sha1 of many files can be found by googling, he can mark that he has the file using REST service on the web application.
So one of the possible solution is that the server requests a set of certain random bytes from the file as well as the hash of the whole file. Here is example steps:
Client checks whether a certain hash exists on server or not. Then, server returns the required positions of random bytes if the file already exists.
Client sends random bytes as per request if the server has the file. Client software will not be able to response to it without having the actual file.
In this way, it can save the bandwidth as well as ensure that user owns the file they want to upload.
Question
I am no expert in Security over the web so I have no idea whether this is a good idea or not. I have read some articles about implementing their own fancy process might lead to the reduction in security strength because the security cannot be tested and the extra information may provide a cracking method.
Does anyone has any comment on the process?
Will it reduce the sucurity?
Does anyone have an idea to solve this problem differently?
I understand that there might not be an exactly answer to this question but I would like to hear if anyone has encounter the same problem and has any good solution to it.
Rather than asking the client to upload some random bytes of the file's contents, it may be better to ask the client to upload the hash of a random region the file. That way you can use a wider range of sizes that you ask the client to verify.
Better yet, though, may be to send the client a random number and require the client to compute an HMAC of the entire file's contents using that number as the key. This is more computationally-expensive since the server must compute the HMAC too, but it verifies that the client has the entire file, not just a small portion of it.
One unavoidable side effect of this hash feature, even with a verification scheme, is that it reveals that a copy of the file already exists somewhere on the server. That by itself may be sensitive information.
For the most stringent privacy protection, you should forego this feature and make each user upload their own copy of the file. You can use hash comparison on the server to avoid storing multiple copies of the file, transparently to the clients.
I want to create a website where a user enters content (say a couple of sentences) which eventually gets stored in a backend database (maybe MySQL). But before the content leaves the client side, I want it to get encrypted using something on client like maybe javascript.
The data will travel over the web encrypted, but more importantly, will also be permanently stored in the backend database encrypted.
Is JavaScript appropriate to use for this? Would 256 bit encryption take too long?
Also, how do you query an encrypted database later on if you want to pull down the content that a user may have submitted over the past 2 months?
I'm looking for tips, suggestions and any pointers you guys may have in how to go about learning about and accomplishing this.
Thanks!
You shouldn't implement the encryption for the communication between the client and the server yourself, use SSL (https) for that. As for encrypting data in the database, you can always use MySQL's built-in methods, such as AES_ENCRYPT and AES_DECRYPT, see reference manual for details.
Look at http://www.farfarfar.com/scripts/encrypt/ (encrypt/decrypt).
Tried text/text/XXTEA with success.
However, that's about as far you can go with JS encryption.
As long as you're not using SSL/https, the main disadvantage is:
A fuzz to protect the private key, (it can be done though, like a form field, not submitted, but requires user to enter value on each session.)
About searching/extracting data on encypted data, I belive nothing is going to work.
Well, to point out something: if it's client side encrypted, it'll likely also be easily decrypted, seeing as with languages like javascript, they're being handed the source code of your encryption scheme. Plenty of encryption schemes out there, AES, Blowfish, etc, but if the data is traveling on an encrypted connection, I suppose the encryption of the data only adds very little security, maybe packet sniffers locally installed or something to that effect.
I would also suggestion you look into using compression as well, I myself have used LZMA, Huffman, and even base64 encoding with javascript to at least obscure the content from casual observers. Point being, no matter how good the encryption, you're handing over the process to the client, and they can just view your source and be able to easily reverse the encryption, given that they know the encryption scheme and aren't to lazy to do a bit of Googling or searching on Wikipedia. I personally prefer compression because it also reduces the size of the data being sent, and unless one is trained in analysis of encryption and compression, it is isn't easy to figure out the compression being used from sniffing packets.
EDIT: But if you want high levels of security, I would suggest against using just compression, and instead, using an asymmetric encryption method like RSA or Diffie-Hellman encryption on data traveling between the server and client, as the private key will never be shared.
For client side, javascript or better, a Java application, would work (at least you can't right click -> view source with a Java app in two seconds). 256 bit encryption wouldn't take too long if you're just wanting to save a few sentences like you said. A modern processor with blast through that in milliseconds. Then, when it arrives at the server, encrypt it using AES or another powerful algorithm (note that the US government uses AES 256 for their Top Secret documents) before sending it to the database. (Also want to add that you can also store a hash (MD5 is a function readily available in PHP, you can opt for something like Whirlpool as well, but you'll have to find a library for that) of the data on a separate database, and check against it when you access it to ensure that your database hasn't been compromised)
For querying, I suggest PHP. Read the data, decrypt it, then send to the client (in still encrypted form), or if you want, and have the client decrypt it also, if you don't trust the networks the data is being sent through.
Here's a good source on Javascript and AES: http://point-at-infinity.org/jsaes/
My two saved links using RSA in javascript: http://www.ohdave.com/rsa/ and http://www.hanewin.net/encrypt/rsa/rsa.htm
As for the data being sent over the network encrypted, this is the kind of thing that HTTPS was made for. Nothing is transmitted in the clear. It's encrypted securely enough for most banks to trust it. Don't reinvent the wheel here; it'll never be as round as what a bunch of people who make wheels for a living have come up with.
Have your form submit to an https:// url (rather than http://), and the script/app at that URL talk to the database, encrypting and decrypting the data as needed. Javascript won't easily talk directly to MySQL anyway, and if you do all the encryption client-side, anyone who can get the page can still decrypt it (since by necessity the client would need to be given the decryption key as well).
Check out this blogpost: http://www.ravellosystems.com/blog/all-you-need-to-know-to-configure-ssl-offloading/
It takes you through all the steps you need to do to configure your webs server to serve content under an encrypted channel (a.k.a SSL termination).