I'm using ADF dataflow to process data from CosmosDB. Is there a way to auto update the connection key in linked service when the cosmosDB keys are rotated ?
based on my understanding unfortunately it is not possible directly. What you can do is have a custom logic via azure automation or any tool to rotate the keys and push the keys to the key vault when rotated. And you can map the linked service in adf ti fetch from key vault rather than manually updating it
I would recommend you consider authenticating using the Data Factory Managed Service Identity. This approach avoids having to manage any passwords or keys and is the preferred approach from a security best practice.
Related
We are trying to ingest some data from DataLake to Azure Cosmos DB and Spark OLTP Connector seems to be the easiest to use.
But due to the company's policy, we are not supposed to use the master keys and we usually use managed identity for the applications. I see the Cosmos DB Java client builder has the 'TokenCredential' option with sample code as:
CosmosAsyncClient client = CosmosClientBuilder
.credential(new DefaultAzureCredentialBuilder().build())
.buildAsyncClient();
Is there anyway to setup the connector to use the same authentication mechanism with managed identity?
I see the Cosmos DB Java client builder has the 'TokenCredential' option with sample code
In CosmosAsyncClient you also have to mention the maker key. there is no such way to use managed identities.
we are not supposed to use the master keys and we usually use managed identity for the applications.
As you want to transfer data from Datalake to CosmosDB with Managed Identities you can use Copy Data Tool in Azur data factory. Create Linked service for cosmos db and in authentication type select Managed identity either system or user.
You can refer this So Thread by #KarthikBhyresh-MT for more understanding on Copy data tool.
Currently, the Spark Connector does not support MSI. I see you correctly created the Issue on the repo that holds the source code: https://github.com/Azure/azure-sdk-for-java/issues/29958
That will surely be used for tracking purposes or at least linking to the workitem that tracks the progress on that area. The feature will be available in the future but there is currently no ETA.
Can data stored in Azure using BYOK (storage accounts, databases etc) be technically (rather than contractually) assured to be not to be access even from Microsoft? For example we assume the HSM key has been securely transferred to HSM backed Key Vault. How can application write to and read from storage using BYOK without Microsoft being able to peek in configuration, or in memory process, or while data is saved to storage?
In public preview now you can use Managed HSM (MHSM). You can provision an MHSM similar to a Key Vault (KV), but to activate and use it you need to set up 3 or more keys to download a security domain from the HSM. Microsoft has no access to decrypt the key - only a quorum of the 3 or more public keys you uploaded. While a bit specific to our testing environment, we have a script that shows how we create certificates and download the security domain using those public keys in order to test MHSM.
You can use the existing Key Vault SDKs and tools like the Azure CLI to access MHSM just like you would KV. For the Azure CLI you need to pass --hsm-name instead of --vault-name, but otherwise works the same for keys.
We are soon releasing another beta of the Azure SDKs for .NET, Java, JavaScript, and Python that support other algorithms supported by MHSM (AES-CBC, AES-CBC-PAD, and AES-GCM). Check out our blog for announcements.
Like most Azure Functions in the beginning we have a connection string to the associated storageaccount that includes the Accountkey like this
DefaultEndpointsProtocol=https;AccountName=ourstorageAccount;EndpointSuffix=core.windows.net;AccountKey=WQfbn+VBhaY1fi/l0eRBzvAvngiCiOwPmx/==
We obviously want to remove that AccountKey. I had hoped we could use ManagedIdentity and the 'Contributor' Role but what I am reading is telling me you cannot use Managed Identity to access Tables in a Storage Account only Blobs.
I know that we could move the whole connection string to KeyVault but that just becomes ann Azure Management Issue if we want to rotate the keys.
Has anyone succesfully controlled access to Azure Table Storage with Managed Identities?
If not what is the next best approach that preferably allows for simple rotation of keys?
Has anyone successfully controlled access to Azure Table Storage with Managed Identities?
Definitely it is unable to access azure table storage with MSI(managed identity, essentially it is a service principal in azure ad), when using MSI to access some azure resources, it essentially uses the azure ad client credential flow to get the token, then uses the token to access the resource.
However, azure ad auth just supported by azure blob and queue storage, table storage doesn't support it currently, see - Authorize access to blobs and queues using Azure Active Directory.
If not what is the next best approach that preferably allows for simple rotation of keys?
You could use azure function to do that, follow this doc - Automate the rotation of a secret for resources with two sets of authentication credentials, I think it completely meets your requirement, this tutorial rotates Azure Storage account keys stored in Azure Key Vault as secrets using a function triggered by Azure Event Grid notification.
I have been facing issues with understanding what would be the best way to store encrypted data in Azure Tables. The main objective here is to avoid someone with access to database to be able to read that data on the storage explorer.
One approach I have looked at is encrypting it on our server logic before saving it to the db, but the solution is causing a performance hit on the application.
Is there a way to achieve this directly on the Azure Tables? If it isn't, what else would be the best way to achieve this?
Data in Azure Storage is encrypted at rest by default. The scenario you're describing probably fits best with this option:
[...] create a storage account that relies on a key that is scoped to the account. When the account is first created, Microsoft uses the account key to encrypt the data in the account, and Microsoft manages the key. You can subsequently configure customer-managed keys for the account to take advantage of those benefits, including the ability to provide your own keys, update the key version, rotate the keys, and revoke access controls.
Source: Create an account that supports customer-managed keys for tables and queues
I am following this article to create tables in Azure Mobile service as Easy tables in portal without using any Code or SQL server management studio.
But I don't find any option there to set the primary key or foreign key for a column.
Please suggest me some ways of doing it.
About Primary Key azure will create a Primary key automatically for you.
If you need a foreign key you should just connect to the SQL Azure that it creates for you and modify the tables in side the SQL Azure using TSQL.
In general if your DB structure is too complicated with a lot of relationships you should think of coding your own service as these easy tables as their name suggest are for "Easy" stuff to give you a boost start...