I would like to use Columnar Encryption or parquet encryption with the envelope technique. I have read this here.
An example of this technique for an open-source KMS is explained here
My question is now is it possible to use Azure managed HSM instead of an open-source KMS? Do you know an example which uses MEK (master encryption keys), that is on Azure managed HSM to wrap/unwrap the DEKs (Data Encryption Keys) for parquet encryption?
Related
My requirement is to encrypt newly added files in data lake storage container. For this I am creating an event grid that will trigger when a new file is added to a specified container. On the end side, I want to create a function that will encrypt the file that was added.
How to write code to create that function? or is there any alternate way?
Azure Storage automatically encrypts all data in a storage account at the service level using 256-bit AES encryption, one of the strongest block ciphers available, and is FIPS 140-2 compliant. Customers who require higher levels of assurance that their data is secure can also enable 256-bit AES encryption at the Azure Storage infrastructure level for double encryption. Double encryption of Azure Storage data protects against a scenario where one of the encryption algorithms or keys may be compromised. In this scenario, the additional layer of encryption continues to protect your data.
Infrastructure encryption can be enabled for the entire storage account, or for an encryption scope within an account. When infrastructure encryption is enabled for a storage account or an encryption scope, data is encrypted twice — once at the service level and once at the infrastructure level — with two different encryption algorithms and two different keys.
Service-level encryption supports the use of either Microsoft-managed keys or customer-managed keys with Azure Key Vault or Key Vault Managed Hardware Security Model (HSM) (preview). Infrastructure-level encryption relies on Microsoft-managed keys and always uses a separate key.
https://learn.microsoft.com/en-us/azure/storage/common/infrastructure-encryption-enable?tabs=portal
Azure SQL Server provides data encrypted at rest. Can someone help to find what is the encryption standard and version that it uses to achieve this
Transparent data encryption (TDE) encrypts SQL Server, Azure SQL Database, and Azure Synapse Analytics data files. This encryption is known as encrypting data at rest.
TDE does real-time I/O encryption and decryption of data and log files. The encryption uses a database encryption key (DEK). The database boot record stores the key for availability during recovery. The DEK is a symmetric key. It's secured by a certificate that the server's master database stores or by an asymmetric key that an EKM module protects.
TDE protects data at rest, which is the data and log files. It lets you follow many laws, regulations, and guidelines established in various industries. This ability lets software developers encrypt data by using AES and 3DES encryption algorithms without changing existing applications.
In Azure, the default setting for TDE is that the DEK is protected by
a built-in server certificate. The built-in server certificate is
unique for each server and the encryption algorithm used is AES 256.
You can find other applicable TDE approach here.
There are three scenarios for server-side encryption:
Server-side encryption using Service-Managed keys
Server-side encryption using customer-managed keys in Azure Key Vault
Server-side encryption using customer-managed keys on customer-controlled hardware
The Azure SQL Server supports all the scenarios, based upon your feasibility and requirement. Check the detailed description here.
My source data is in Azure blobs. I am loading this in Snowflake table via ADF pipeline. I know that snowflake uses E2E encryption for its data and also that we can use master key when giving COPY COMMAND in snowflake.
I have understood that key vault can be used in Azure for source and sink.
However, it is not much clear to me. How exactly does encryption work for files moving from azure to snowflake table. Is client-side encryption (master key) possible ?
Are there any other ways to do this ?
When the data is stored in cloud storage service, the user may optionally encrypt the data files using client-side encryption.
Client-side encryption provides a secure system for managing data in cloud storage. Client-side encryption means that a user encrypts stored data before loading it into Snowflake. The cloud storage service only stores the encrypted version of the data and never includes data in the clear.
To load client-side encrypted data from a customer-provided stage, you create a named stage object with an additional MASTER_KEY parameter using CREATE STAGE, and then load data from the stage into your Snowflake tables. The MASTER_KEY parameter requires a 256-bit Advanced Encryption Standard (AES) key encoded in Base64.
The master key is the Base64-encoded
string of the customer’s secret master key. As with all other
credentials, this master key is transmitted over Transport Layer
Security (HTTPS) to Snowflake and is stored encrypted in metadata
storage. Only the customer and the query-processing components of
Snowflake are exposed to the master key and are therefore able to
decrypt data stored in the stage.
Please visit Data Encryption in Snowflake to get in-depth details.
You can also check https://www.snowflake.com/blog/data-encryption-with-customer-managed-keys-for-azure/
I have sensitive data in my application, just because encryption is not available with the Azure Table Storage, I can't store the sensitive data into it.
As of 31 August 2017 the Azure storage service encrypts all data at rest using 256 bit AES for blobs, files, tables and queues as per this post Announcing Default Encryption for Azure Blobs, Files, Table and Queue Storage. This has also been applied to any storage accounts created before this date.
This is automatic and cannot be disabled but you can choose to use your own encryption keys instead of the Microsoft managed ones. It applies to both standard and premium performance tiers and Azure Resource Manager and classic deployment models.
Docs here Azure Storage Service Encryption for Data at Rest
If you have secure data then it's better to encrypt, you can encrypt with Azure Key Vault one more link
Basically, in your entity, you will mark properties with attribute
[EncryptProperty]
public string EncryptedProperty1 { get; set; }
The benefit would be that you will always communicate with encrypted data, even if somebody, somehow get access to storage he will not be able to extract data.
I guess that depends on how you're managing your data. If only parts of your data need to be encrypted and the rest can remain clear then you could potentially encrypt those sensitive fields in your app before storing them in Table Storage. You'd have to implement the encryption yourself, of course, but you should be able to find plenty of libraries to help with that.
If you had to encrypt everything before storing I don't think it would work well. Table Storage is key:value pairs and all your values would be encrypted, rendering Table Storage useless for any kind of lookup. You'd have to read in all your data, sort and index it in your app, then get on with your regular work.
I deploy Asp.Net Core web app and I need to build a storage for private keys of my clients (it is a lot of values). What should I use: Azure Key Vault or Data Protection APIs?
The second seems more easy to programming, however there is information from docs:
The ASP.NET Core data protection APIs are not primarily intended for
indefinite persistence of confidential payloads...
But I need to store keys long-term.
If you're using the keys to protect data for long term storage, I would advise you to use Azure Key vault.
Azure Key vault is a high availability service designed for storage of secrets and keys. Keys and secrets are automatically copied to Key vault instances in multiple regions and easily backed up securely using PowerShell cmdlets. You can store them in an HSM if you are dealing with highly sensitive data.
The Data Protection APIs are more designed to protect local or ephemeral data.
You should use Azure Key Vault to store your keys. The Data Protection API is always used in your application, for example it is used to encrypt and secure your session cookie.
But don't forget that you also do need to configure and store the data protection keys in a secure place. If you don't do it properly, then uses might be kicked out of your site when you redeploy.
See this document for more details:
Key storage providers in ASP.NET Core
If you do want to store the Data Protection Key ring in AKZ, then check out my implementation here:
Storing the ASP.NET Core Data Protection Key Ring in Azure Key Vault