I've created and published (to Azure) a working and fairly simple Azure Function App (HTTP Trigger) which inserts data retrieved from an API call to an Azure SQL database after authenticating via User identity (if testing locally) or Managed Identity (when running on VM).
Everything is functional, however, I now need to encrypt one of the SQL table columns which I have done using SSMS. The next step as I understand it is authenticating a provider to access the CMK via Azure Key Vault (I'm following this Microsoft guide).
I'm wondering in the following code how to InitializeAzureKeyVaultProvider without using applicationId and clientKey from Azure App registration resource, but with a user or managed identity role. Or, if there's any other way to get/use applicationId, and clientKey without creating/using an Azure App registration resource.
Is there a newer / easier way to access Azure Key Vault for Always Encrypted Columns sql queries?
static void InitializeAzureKeyVaultProvider() {
_clientCredential = new ClientCredential(applicationId, clientKey);
SqlColumnEncryptionAzureKeyVaultProvider azureKeyVaultProvider = new SqlColumnEncryptionAzureKeyVaultProvider(GetToken);
Dictionary<string, SqlColumnEncryptionKeyStoreProvider> providers = new Dictionary<string, SqlColumnEncryptionKeyStoreProvider>();
providers.Add(SqlColumnEncryptionAzureKeyVaultProvider.ProviderName, azureKeyVaultProvider);
SqlConnection.RegisterColumnEncryptionKeyStoreProviders(providers);
}
}
Here is the other way I've been attempting, however, upon installing and using Microsoft.Data.SqlClient.AlwaysEncrypted.AzureKeyVaultProvider, I get the error following this code sample:
private static bool EncryptionProviderInitialized = false;
private static void InitializeAzureKeyVaultProvider()
{
if (!EncryptionProviderInitialized)
{
SqlColumnEncryptionAzureKeyVaultProvider akvProvider = null;
#if DEBUG
if (Debugger.IsAttached)
{
Console.WriteLine("Debugger attached - configuring KeyVaultProvider via VisualStudioCredential");
akvProvider = new SqlColumnEncryptionAzureKeyVaultProvider(new VisualStudioCredential());
}
if (akvProvider == null)
#endif
akvProvider = new SqlColumnEncryptionAzureKeyVaultProvider(new ManagedIdentityCredential());
SqlConnection.RegisterColumnEncryptionKeyStoreProviders(customProviders: new Dictionary<string, SqlColumnEncryptionKeyStoreProvider>(capacity: 1, comparer: StringComparer.OrdinalIgnoreCase)
{
{ SqlColumnEncryptionAzureKeyVaultProvider.ProviderName, akvProvider}
});
EncryptionProviderInitialized = true;
}
}
ERROR:
[2021-10-08T01:36:24.209Z] Executed 'EncryptedSQLMigration' (Failed, Id=1323dcbb-e671-4ed4-8a7c-6259447326c5, Duration=537ms)
[2021-10-08T01:36:24.209Z] System.Private.CoreLib: Exception while executing function: EncryptedSQLMigration. FreshFirstApproach: Method not found: 'Microsoft.Extensions.Primitives.StringValues Microsoft.AspNetCore.Http.IQueryCollection.get_Item(System.String)'.
Initially, I got that same error, however, with Microsoft.Extensions.Logging.Abstractions - upon removing the ILogger from my main function just for the sake of moving on as I wasn't able to solve that issue either, I now get this Microsoft.Extensions exception.
Any help with my goal of using Always Encrypted Columns with Azure Function App and Azure Key Vault is very much appreciated!
Thank you very much.
It is not possible to use applicationId and clientKey without creating or using an Azure App registration resource. There is an alternative way where you can pass clientId and clientSecret as shown below, but here also you will need application registration.
static void InitializeAzureKeyVaultProvider()
{
string clientId = ConfigurationManager.AppSettings["AuthClientId"];
string clientSecret = ConfigurationManager.AppSettings["AuthClientSecret"];
_clientCredential = new ClientCredential(clientId, clientSecret);
....
....
}
As for user or managed identity, if you check this document you won't find the Azure Key Vault in the list of services that support managed resources.
So managed service identity should be created for Azure Function instead of Azure Key Vault. Check this Retrieve Azure Key Vault Secrets using Azure Functions and Managed Service Identity document for more information.
Related
I would like to retrieve Azure Key Vault secrets from a client application(outside of Azure) using certificate authentication for Azure Key Vault through Azure AD. My initial development is done through a dotnet console application, and eventually, I would like to use similar logic within a WCF web service hosted on an IIS server outside Azure (an on-premise server). I have everything set up on cloud side: Azure Key Vault set up, client application registration, and a certificate within Azure Key Vault for client Application authentication. I also install the same certificate in my local machine current user certificate store and the certificate on the windows service server hosting my WCF web service. So that I can retrieve the certificate from the certificate by using its thumbprint value:
var certCollection = certStore.Certificates.Find(X509FindType.FindByThumbprint, certificateThumbprint, onlyAllowValidCerts);
I tried a couple of approaches to access Azure Key Vault secrets, but nothing has met the expectations I set up initially. I would like to know what is the best/latest approach to achieve what I want without compromising response time and using deprecated APIs
Use AzureKeyVaultConfigurationProvider to retrieve secrets
Issues:
1.1)I have to use a few deprecated packages for this purpose.
1.2)Also, the performance (response time) to retrieve the secrets from Azure Key Vault and populate the AzureKeyVaultConfiguration is not ideal...
Code Example:
// create IConfigurationRoot to read Azure key vault
IConfigurationRoot config = new ConfigurationBuilder()
.AddAzureKeyVault(
keyVaultUrl,
CLIENT_ID,
KeyVaultUtility.AssertionCert2,
new DefaultKeyVaultSecretManager())
.Build();
Use AzureKeyVault package to create a KeyVault Utility. Then use GetSecretAsync method:
Issues: 1)Azure Key Vault package is deprecated
2)When I use this logic with WCF service, and test locally, I run into following error:
Method not found: 'Void Microsoft.Azure.KeyVault.KeyVaultClient..ctor(AuthenticationCallback, System.Net.Http.DelegatingHandler[])'.
Code Example:
var client = KeyVaultUtility.GetClient();
var secret = Task.Run(async () => await client.GetSecretAsync(keyVaultUrl, "Jon--Test")).Result.Value;
KeyVaultUtility logic:
public static KeyVaultClient GetClient()
{
if (AssertionCert == null)
{
throw new Exception("Call Initialise before calling GetClient.");
}
return new KeyVaultClient(new KeyVaultClient.AuthenticationCallback((a, r, s) => GetAccessToken(a, r, s, AssertionCert)));
}
private static async Task<string> GetAccessToken(string authority, string resource, string scope, ClientAssertionCertificate cert)
{
var context = new AuthenticationContext(authority, TokenCache.DefaultShared);
var result = await context.AcquireTokenAsync(resource, cert).ConfigureAwait(false);
return result.AccessToken;
}
The task is to download google sheet in excel format and store it in Azure blob storage on timely basics using the Azure time trigger function.
Access Method to users google drive - OAuth Client ID.
I have created an Azure function locally and it works fine as expected and performs the task but when I deploy azure function I get this error.
Code for DriveService where the error occurs according to stack trace when deployed
public string[] Scopes = { DriveService.Scope.Drive, DriveService.Scope.DriveReadonly };
public DriveService GetService()
{
UserCredential _credential;
//Error Occurs at line below
Google.Apis.Auth.OAuth2.Flows.GoogleAuthorizationCodeFlow googleAuthFlow = new GoogleAuthorizationCodeFlow(new GoogleAuthorizationCodeFlow.Initializer()
{
ClientSecrets = new ClientSecrets
{
ClientId = _config[Constant.ClientId],
ClientSecret = _config[Constant.ClientSecret],
}
});
string FilePath = Path.GetDirectoryName(_driveCredentialsPath);
_credential = GoogleWebAuthorizationBroker.AuthorizeAsync(
googleAuthFlow.ClientSecrets,
Scopes,
"user",
CancellationToken.None,
new FileDataStore(FilePath, true)).Result;
DriveService service = new DriveService(new BaseClientService.Initializer()
{
HttpClientInitializer = _credential,
ApplicationName = Constant.ApplicationName,
});
return service;
}
I think there are two situations where it can go wrong but I am not sure about it.
When I am running the application locally a consent screen appears and gives permission to access the drive.
When this same function is running on azure who and how it will grant permission to access the drive.
I have provided my Azure App URL on Google OAuth Consent Screen as mentioned below to overcome this situation.
When I am running locally after giving permission to access drive it creates a TOKENRESPONSE-USER file a which consists of the access token, expiry date refresh token, and scope.
Is this possible that when the function is deployed it is unable to create a TOKENRESPONSE-USER file on azure function?
Please let me know why I am getting this error or do I need to change something in my process.
You can configure your function app to use Google login for authentication purposes when running on Azure. To achieve this you have to generate client id and client secret using the Google sign-in for server-side apps, using this connection you can store the tokens obtained in the token store. Please refer to this document to configure your function app to use Google Login, refer to this document regarding the token store and how to retrieve and refresh the token obtained.
I'm experimenting with various Azure features and currently want to retrieve a secret from KeyVault.
Straight to the case:
I'm using this nuget package to interact with my azure resources.
I've developed a simple .NET Core console app and run it locally.
I have a KeyVault resource with one secret defined which is active and not expired.
I've registered an App in AAD so my locally shipped .NET Core console app has an identity within AAD.
Than I've created a "client secret" within this registered app in AAD to use it to authenticate myself as an app.
After that I've added access policy in my KeyVault resource to allow GET operation for secrets for this registered app:
Then I've developed a small piece of code which should retrieve the desired secret:
public class AzureAuthentication
{
public async Task<string> GetAdminPasswordFromKeyVault()
{
const string clientId = "--my-client-id--";
const string tenantId = "--my-tenant-id--";
const string clientSecret = "--my-client-secret--";
var credentials = new ClientSecretCredential(tenantId, clientId, clientSecret);
var client = new SecretClient(new Uri("https://mykeyvaultresource.vault.azure.net"), credentials);
var secret = await client.GetSecretAsync("admincreds");
return secret.Value.Value;
}
}
However when I'm trying to do this I'm getting an AccessDenied error:
Am I missing something painfully obvious here? Or there is some latency (>30 min for this moment) for which changes from Access policies screen in KeyVault resource are applied?
I test your code and Get permission, it works fine.
From your screenshot, it looks you didn't add the correct service principal related to the AD App to the Access policies.
If you add the service principal related to the AD App, it will appear as APPLICATION, not COMPOUND IDENTITY.
So when you add it, you could search for the client Id(i.e. application Id) or the name of your App Registration directly, make sure you add the correct one.
Make sure your AD App(service principal) has the correct permission in your keyvault -> Access policies
Is it possible to give access to a key vault to a user assigned identity?
In Managed Identities from the azure portal I created a new Identity "KeyVaultIdentity", which I assigned it to a web application (in Identity, user assigned identities tab). In access policies from key vault I added the new created "KeyVaultIdentity" identity and offered permissions to access the secrets.
I am using the following code to access the key vault:
try
{
/* The below 4 lines of code shows you how to use AppAuthentication library to fetch secrets from your Key Vault*/
AzureServiceTokenProvider azureServiceTokenProvider = new AzureServiceTokenProvider();
KeyVaultClient keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
var secret = await keyVaultClient.GetSecretAsync("https://play9VaultDemo.vault.azure.net/secrets/AppSecret")
.ConfigureAwait(false);
Message = secret.Value;
/* The below do while logic is to handle throttling errors thrown by Azure Key Vault. It shows how to do exponential backoff which is the recommended client side throttling*/
do
{
long waitTime = Math.Min(getWaitTime(retries), 2000000);
secret = await keyVaultClient.GetSecretAsync("https://play9VaultDemo.vault.azure.net/secrets/AppSecret")
.ConfigureAwait(false);
retry = false;
}
while (retry && (retries++ < 10));
}
/// <exception cref="KeyVaultErrorException">
/// Thrown when the operation returned an invalid status code
/// </exception>
catch (KeyVaultErrorException keyVaultException)
{
Message = keyVaultException.Message;
if ((int)keyVaultException.Response.StatusCode == 429)
retry = true;
}
But it says that access is forbidden when I try to access the secret. However if in Key Vault I give access to the System Assigned Identity of the Web application, I can access the secret,
Do you have any idea how can I make this work with the user assigned identity?
I had the same problem, and I had to do two things to step getting "forbidden" every time I tried to access KeyVault with a user-assigned identity:
Upgrade the version of Microsoft.Azure.Services.AppAuthentication I was using to 1.2.0-preview2. Earlier versions don't have support for user-assigned identities.
Pass a connection string into the AzureServiceTokenProvider constructor to tell the service which identity to use. This is the bit that all of the links above omit. So I had:
var connectionString = "RunAs=App;AppId=";
var azureServiceTokenProvider = new AzureServiceTokenProvider(connectionString);
instead of:
var azureServiceTokenProvider = new AzureServiceTokenProvider();
To find the value for your clientId, open up your managed identity in the Azure Portal. You should see a field marked "Client ID". That's the one you want.
Do you have any idea how can I make this work with the user assigned identity?
You could follow the steps from How to use managed identities for App Service and Azure Functions.
Here is the steps:
1.In webapp Identity, click User Assigned(preview) and add your user-assigned managed identity.
2.Adding the user-assigned type and a cotells Azure to create and manage the identity for your application using an Azure Resource Manager template.
"identity": {
"type": "UserAssigned",
"userAssignedIdentities": {
"[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', variables('identityName'))]": {}
}
}
3.If you request a token to Key Vault, you need to make sure you have added an access policy that includes your application's identity. Otherwise, your calls to Key Vault will be rejected, even if they include the token.
4.Using the Microsoft.Azure.Services.AppAuthentication library for .NET to get secret.
var azureServiceTokenProvider = new AzureServiceTokenProvider();
var keyVaultClient = new KeyVaultClient(
new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
var secret = keyVaultClient.GetSecretAsync("https://yourkeyvaultname.vault.azure.net/secrets/secretname").Result.Value;
5.The output is as below:
If you run into this error Connection string is not valid. Must contain 'TenantId' then make sure you explicitly reference Microsoft.Azure.Services.AppAuthentication >= v1.2.0
I was using Microsoft.Extensions.Configuration.AzureKeyVault v3.1.3 which ships with a lower version of AppAuthentication so user assigned identities didn't work.
More info here:
https://github.com/MicrosoftDocs/azure-docs/issues/28729
following Azure reference site for analysis service processing costume .NET activity
https://github.com/Azure/Azure-DataFactory/tree/master/Samples/AzureAnalysisServicesProcessSample
in this example we have Pipeline to Process Cube
` "TabularDatabaseName": "<DATABASE_NAME>",
"AzureADAuthority": "https://login.windows.net/<TENANT_ID>",
"AzureADResource": "https://<LOCATION>.asazure.windows.net",
"AzureADClientId": "<CLIENT_ID>",
"AzureADClientSecret": "<CLIENT_SECRET>"`
first we need to know how we get AzureADResource and AzureADAuthority information.
also this pipeline working fine when we pass hard-coded password instead of {0)
so we didn't understand where is problem. is ServicePrincipalAuth we provided here is correct or it is mandatory to provide password
Regards,
Manish
Please follow this article from MSDN which mentions how to create an application in the AD and create AD service principal.
https://learn.microsoft.com/en-us/azure/data-factory/data-factory-create-data-factories-programmatically
Step 7 would return the AD service principal.
Also, to be able to run a pipeline you would require a client when creating a connection with ADF, which would require client Id, client Key, subscription Id and tenant Id.
This is how I created my client to ADF.
private void CreateADFClient()
{
AuthenticationContext authenticationContext = new AuthenticationContext($"https://login.windows.net/{tenant_id}");
ClientCredential credential = new ClientCredential(clientId: this.client_id, clientSecret: this.client_key);
Task<AuthenticationResult> result = authenticationContext.AcquireTokenAsync(resource: "https://management.core.windows.net/", clientCredential: credential);
if (result == null)
{
throw new InvalidOperationException("Failed to obtain the JWT token");
}
string token = result.Result.AccessToken;
TokenCloudCredentials credentials = new TokenCloudCredentials(subscription_id, token);
this.inner_client = new DataFactoryManagementClient(credentials);
}
The application name that you provided while performing the steps from MSDN, will return the name of the application, take note of the application name & find it on the Azure portal by Azure Active Directory --> App Registrations --> . Corresponding to it would be your application id which is the client Id here.
To create the client key, click on the application name and go to 'Keys' section and add a key description and validation and save. Upon saving the value would be shown, this is your client key. Please take note of it, this would be visible the next time you come to this page, but of course, you can create another.
The tenant id can be downloaded by clicking on Help --> show diagnostics. A file will be downloaded where you can search your tenant Id from.
Hope this solves your questions around it!