How can I apply AES encryption to hadoop? Is it possible? If not, help me encrypt the data file in hadoop
The latest versions of hadoop supports encryption. We can create encrypted zones and the data that we transfer to these zones will be encrypted automatically and the data retrieved from this zone will be decrypted automatically. This is also known as REST data encryption. The detailed steps are given in the apache website. This doesn't need any change in the code that access this data.
This can be also equated to server side encryption.
If you want custom encryption to be applied to the files in hdfs, it will be little complex, because you have to apply the encryption/decryption logic in all the programs that uses these data. If the data is encrypted using custom encryption logic, the RecordReader and RecordWriter classes needs to be modified to work with the data.
Related
My team is using Kafka; we need to add encryption for security compliance reasons so that data is encrypted before it is published to Kafka and is decrypted when an authorized consumer consumes it from Kafka. I see Kafka offers TLS security options, but that doesn't seem to address our needs. TLS secures communication, but internally, data is still stored unencrypted. With some searching I see KIP-317: End to end encryption (https://cwiki.apache.org/confluence/display/KAFKA/KIP-317%3A+Add+end-to-end+data+encryption+functionality+to+Apache+Kafka
), which seems to address our use case, but that KIP seems like it stalled and never got finished.
One simple option is to add a simple custom encryption layer on top of the Kafka API. Programs publishing events to Kafka use an encryption library and encrypt the data before publishing events. Programs consuming events use an encryption library to decrypt messages consumed from Kafka. This would work and is simple.
Is there a better solution or a more standard solution?
As you suggested the easiest and most straight forward way to solve this is by encrypting the message before sending it and after you receive it at the application level and instead of sending an object you are sending a blob.
For a more elegant and optimized approach I would go with a custom serde though. One of the advantages of kafka is that the data is actually processed and manipulated in binary form so you are already using a serde to convert to and from binary.
Now by writing a custom serde you should be able to get no overhead other than the obvious one required to encrypt/decrypt the bytes. Furthermore, going this way allows you to make the encryption completely transparent to the application. You could easily have an unencrypted dev environment while using the encrypted serde in production just by changing 2 lines in application.properties (or equivalent), no recompile required. Furthermore you can have a single person working on the serde while the rest of the team works on the software. When the serde is done, you just drop it in and you have encryption.
You could also try and check repositories like this and this. You might be able to use them as they are, fork them or at least get some inspiration.
Disclaimer: Never tested any of the 3 links I referenced in this answer but the principle behind them is sound.
As part of developing a cross-platform application, I need to store username password locally on disk. On Linux, I'm planning to make use of libsecret, which makes use of the Secret Service API and takes care of how to encrypt and where to store the data. On Windows, I'm planning to use DPAPI's CryptProtectData to encrypt the password and store the cipher text in a local text/config file.
Are there any better/safer ways to do this (especially related to storing the encrypted contents on Windows)? What are the usual best practices for storing secrets locally on Windows and Linux?
Could anyone point me to documentation of encryption options available for couchbase? With SQL Server we have the option to encrypt data at table or db level. Is it possible to encrypt couchbase data in a similar way?
I believe that Gazzang is what you want to look at in regards to data encryption for Couchbase:
Gazzang for Couchbase offers a powerful, policy-driven solution that enables you to encrypt your data stored in Couchbase Server. With Gazzang, data files in Couchbase Server are encrypted on disk. This ensures that your data is not compromised if your database is stolen, copied, lost, or otherwise improperly accessed.
Source and more information on Gazzang: http://www.couchbase.com/couchbase-server/connectors/gazzang
You could also use file system encryption - windows provides the option.
I know that in iOS, if the .sqlite is updated / read by SQL statement, then SQLCipher can be used to do encryption and decryption.
Is there any suggestion if Core Data is used instead ? The requirements are:
Core Data is used instead
Backup via itune for the application data is still be encrypted
Thanks
If you want the SQLite to be encrypted when the device is locked only, then the on-disk encryption feature introduced in iOS 4 is your answer.
Otherwise, you may need to encrypt stuff programmatically. Cf. Core Data SQLite encryption?.
But AFAIK, there's no built-in support of any SQLite encryption feature in Core Data.
I am using firebird server 2.50. As far as I know there is no way to encrypt a database in Firebird. So how to secure the user data?
Manually encrypting all data before saving would cause trouble since i will not be able to use something like "starting with".
I use CentOs for Database servers. These servers are communicating with an Application Server which runs on Windows Server 2008.
Encryption is one kind of several protection measures which can be done against potential adversaries. And there are other methods too. You need common security analysis before you go with decision whether to encrypt or not, and if not than what. You have to look who are adversaries, where they could hit, etc-etc-etc. Blind use of encryption may be waste of resource/time/money/etc. Do security analysis first.
DB encryption is possible in version 3:
With Firebird 3 comes the ability to encrypt data stored in database. Not all of the database file is encrypted:
just data, index and blob pages.
To make it possible to encrypt a database you need to obtain or write a database crypt plug-in.
Refer to Firebird-3.0.0_Alpha1-ReleaseNotes for details