I am trying to create an Azure Function App which is an EventHub Trigger. The thing is that to connect to Azure Eventhub, I don't have the full connection string. Instead I have the Eventhub SAS token stored as secret in a Key Vault.
I would like to know if in the App Settings section there is a way to build the connection string by passing that token that I get from the key vault.
I have a variable KEYVAULT_SAS_SECRET whose value I want to use in another variable within the App Setting.
Would it be possible to reference the KEYVAULT_SAS_SECRET variable to construct the connection string that is stored in a second variable?
Something like this:
Endpoint=sb://some-namespace.servicebus.windows.net/;SharedAccessKeyName=policy;SharedAccessKey=[KEYVAULT_SAS_SECRET];EntityPath=eventhub-topic.
Thank you very much in advance
Unfortunately it is not possible to reference Azure Function App Settings variable in another App Settings variables.
In your particular case you have to construct EventHub connection string from two App Settings variables directly in your code
Related
I have followed the following for How to reference Key Vault connection strings from an Azure App Service.
https://davecallan.com/how-to-reference-key-vault-connection-strings-from-an-azure-app-
service/
The connection string has a name of "sqlconnection" in the App Service Configuration. I then deleted this entry from the Configuration Json file and restarted the Web app.
I then try to connect but it gives a database error with no details.
Any ideas on this ?
I tried to reproduce the same in my environment like below:
I created a key vault with Azure role-based access control (RBAC) like below:
Make sure to add Key Vault Administrator role in Access control like below:
Generate a key vault secret under setting. While generating secret of type as Manual name and set the value to the current SQL Database connection string stored in the Application web.config
Try to use Secret value connection string of your sql database like below:
Then you can able to see Secret Identifier like below:
In your web app -> under setting click configuration -> click new connection string
Type the name of the connection string and set the value #Microsoft.KeyVault(SecretUri=VALUE), where the VALUE
In Identity under setting -> make sure to add system assigned status as on like below:
Click on the "Role Assignments" button and then Add keyvault secrets user role like below:
As you are facing database error with no details you can check the Log4net errors under Portals_default\Logs.
Reference:
Connection strings and app settings not working properly on Azure App Service · Issue #4227 · dnnsoftware/Dnn.Platform · GitHub
In Azure you can setup an App Config and a KeyVault. The point of the KeyVault being to store more sensitive data than your App Config and be able to regulate access to the config and vault separately.
So what is the benefit of using a keyvault reference in the app config?
You are basically allowing anyone with access to the app config to access certain values in your keyvault and are bypassing the additional layer of security the vault normally provides.
The additional layer being required auth to the vault to access those same values if they aren't referenced in the config.
I really don't understand what benefit keyvault references give you.
This blog article by Jan de Vries explains them in more detail: https://jan-v.nl/post/2021/using-key-vault-with-azure-app-configuration/.
The relevant part for your question:
As it happens, the code for accessing App Configuration doesn’t give your application permission to retrieve secrets from Key Vault.
The application retrieves them from Key Vault, not from App Configuration.
App Config only holds the reference, not the actual value.
Official docs also mention this:
Your application uses the App Configuration client provider to retrieve Key Vault references, just as it does for any other keys stored in App Configuration. In this case, the values stored in App Configuration are URIs that reference the values in the Key Vault. They are not Key Vault values or credentials. Because the client provider recognizes the keys as Key Vault references, it uses Key Vault to retrieve their values.
Your application is responsible for authenticating properly to both App Configuration and Key Vault. The two services don't communicate directly.
I suppose there are different approaches to using the KeyVault, but the way I tend to use it is as follows.
My application will have a set of secrets, which I store locally using the Secrets Manager, you would add the secret for your application:
dotnet user-secrets set "Movies:ServiceApiKey" "12345"
Your application can then read this setting using _moviesApiKey = Configuration["Movies:ServiceApiKey"]; as you'll see in the link above. Obviously, there's no way you can see this value in the code, but your application can read it from the Secrets Manager.
If you do forget the values, you can use the following command to retrieve them:
dotnet user-secrets list
KeyVault will work as your Secrets Manager within Azure. So, your application will need to have permission to access the KeyVault, and in my case I store the Vault name in the appsettings.json, and during the bootstrapping, I include the KeyVault configuration if running in Production mode i.e. on the Azure Server and not locally.
public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.ConfigureLogging(logging =>
{
logging.ClearProviders();
logging.AddConsole();
logging.AddAzureWebAppDiagnostics();
})
.ConfigureAppConfiguration((context, config) =>
{
if (context.HostingEnvironment.IsProduction())
{
IConfigurationRoot builtConfig = config.Build();
ConfigurationBuilder keyVaultConfigBuilder = new ConfigurationBuilder();
keyVaultConfigBuilder.AddAzureKeyVault(builtConfig["VaultName"]);
IConfigurationRoot keyVaultConfig = keyVaultConfigBuilder.Build();
config.AddConfiguration(keyVaultConfig);
}
})
.UseStartup<Startup>();
Note, the check for context.HostingEnvironment.IsProduction(). Within the appsettings, I have:
"VaultName": "https://yourkvname.vault.azure.net/"
So, the only reference I have to the KeyVault from the application is the name, and that should be secure as only the application will have access to the keys/secrets.
One thing to note, you need to make sure that the names match both for your local secrets and the ones in the KeyVault. In my case, I am running on a Windows platform, so I needed to make a small change to the names using double dashes (--) in place of the colon (:), so...
Movies:ServiceApiKey
Becomes
Movies--ServiceApiKey
When working in Azure, storing secrets in Key Vault is a good idea. And to make it better, there’s the Key Vault Reference notation. This feature makes sure no one can read the secret(s) unless someone grants permission.
Speaking of secrets, they should never be directly stored in application settings of a Function App (same goes for App Services by the way). Why not ? Because secrets would be available to anyone who has access to the Function App in the Azure Portal. The right way is to use an Azure Key Vault which is the Azure component for securely storing and accessing secrets 🔒. Once your secrets are in the key vault, you have to grant the Key Vault access to the identity of your Function App and you can then reference the secrets you need directly in your application settings. These are called Key Vault references because an application setting does not contain directly the value of a secret but a reference to the secret which is stored in Key Vault. When running, your function will automatically have access to the secret and its value as an environment variable, as if it was a normal application setting.
Key Vault references work for both App Services and Function Apps and are particularly useful for existing applications that have their secrets stored in settings because securing the secrets with Azure Key Vault references does not require any code change.
Reference: https://www.techwatching.dev/posts/azure-functions-custom-configuration
https://www.sharepointeurope.com/using-key-vault-references-with-azure-app-configuration/
I have a requirement where I need to point our DEV Azure Data Factory to a Production Azure SQL database and also have the ability to switch the data source back to the Dev database should we need to.
I've been looking at creating parameters against the linked services but unsure of the best approach.
Should I create parameters as follows and choose the relevant parameters depending on the environment I want to pull data from?
DevFullyQualifiedDomainName
ProdFullyQualifiedDomainName
DevDatabaseName
ProdDatabaseName
DevUserName
ProdUserName
Thanks
Any sort of trigger can also have parameters attached to it. Check out the following example, assuming you have a custom event trigger and SQL server as a source:
Create a string parameter for the database name field while establishing a SQL server connected service as a dataset.
Create New parameter in dataset, assign the dataset parameter to that same Linked service parameter, which will be used to store the trigger data.
A custom event trigger has the ability to parse and deliver a custom data payload to your pipeline. You define the pipeline parameters and then populate the values on the Parameters page. To parse the data payload and provide values to the pipeline parameters, use the format #triggerBody().event.data. keyName_.
As per Microsoft Official Documents, which could be referred:
Reference trigger metadata in pipelines
System variables in custom event trigger
When you utilize a pipeline activity in a source, it will request you for a dataset parameter. In this case, utilize dynamic content and choose the parameter containing the trigger data.
I would suggest using Azure Key Vault for that.
Create an Azure Key Vault for each environment (dev, prod, etc.)
Create secrets inside both key vaults with the same name but different values.
For example, for the database server name, create the same secret "database-server" in both dev and prod key vaults but with the correct value representing the connection string of the dev and prod server respectively, in the following format:
integrated security=False;encrypt=True;connection timeout=30;data source=<serverName>.database.windows.net;initial catalog=<databaseName>;user id=<userName>;password=<loginPassword>
In your Azure Data Factory, create a Key Vault linked service pointing to your key vault.
In your Azure Data Factory, create a new Azure SQL Database linked service selecting the Key Vault created in step 1 and the secret created in step 2.
Now you can easily switch between dev and prod by simply adjusting your Key Vault linked service to point to the desired environment.
Have fun ;)
Reference:
https://learn.microsoft.com/en-us/azure/data-factory/store-credentials-in-key-vault
I was able to use the Key Vault inside a function app as described here but when I tried to use the Key Vault to hold the connection string of a function with a queue trigger I have issues with the storage account connection string. The function seems to find the parameter I provide but either doesn't get the secret back or doesn't like the information when it throws the error No valid combination of account information found.
My function is defined as:
[FunctionName("ReadQueueForMessage")]
public static async Task Run([QueueTrigger("%AzureQueueTrigger%", Connection = "AzureWebJobsStorage")] string myQueueItem,
Binder binderinputblob,
ILogger log)
This works fine if I just define the connection string in my local.settings.json. What I'm trying to do is instead of just putting the connection string in the json file, I want to point the function to the Key Vault with this syntax:
"AzureWebJobsStorage": "#Microsoft.KeyVault(SecretUri=https://myappkeyvault.vault.azure.net/secrets/myapp-AzureWebJobsStorage-Queue/the-guid-of-secret)",
I did go to the Key Vault and update the access policies to include the function app so it can read/list the secrets. The documentation here shows making an update to the configuration after it was deployed to Azure. I'm trying to test first in Visual Studio. Maybe that is the issue? Or is it not possible at all to use the secret in this manner?
I'm trying to test first in Visual Studio.
For now using Azure Key Vault references with Azure Functions does not support to work on local, as confirmed by Azure Functions team. If you still want to test on local, you could implemented an incomplete local workaround like this issue.
I test on portal and it works well. You could refer to the following steps as below：
1.In VS Function.cs, then publish to azure：
public static void Run([QueueTrigger("queue", Connection = "AzureWebJobsStorage")]string myQueueItem, TraceWriter log)
{
log.Info($"C# Queue trigger function processed: {myQueueItem}");
string connectionString = System.Environment.GetEnvironmentVariable("AzureWebJobsStorage");
log.Info($"The connection string is {connectionString}");
}
2.Set AzureWebJobsStorage on Appsettings setting on portal.
3.Then it will work fine.
With the Nuget packages Azure.Extensions.AspNetCore.Configuration.Secrets and Azure.Identity you can now use the KeyVault as a configuration provider, the same way as in ASP.NET Core. You need to add a class derived from FunctionsStartup to add the KeyVault as a configuration provider, see Add FunctionsStartup class with the KeyVault as a configuration provider.
If you add the AzureWebJobsStorage connection string as a secret to the KeyVault, you can remove it from the Configuration section of your Function App in Azure. Make sure to turn on System assigned in the Identity section and in the KeyVault add an Access Policy with the Secret permissions Get and List for your Function App.
When you run your Function App local for debugging, Azure.Identity automatically uses your Microsoft Account for access to the KeyVault, if it has at least Get and List access to the secrets.
Unfortunately, when you test local, the Function App does NOT read AzureWebJobsStorage from the configuration/KeyVault, but requires it to be stored in local.settings.json. To prevent storing keys on your local computer, you can set AzureWebJobsStorage to "UseDevelopmentStorage=true" in local.settings.json.
For detailed instructions see: Create Azure Function App with Dependency Injection and the Key Vault as Configuration Provider.
Example project: https://github.com/Forestbrook/FunctionWithKeyVaultAndDI
I have a parameterized connection string in Azure Data Factory Linked Services as below:
Integrated
Security=False;Encrypt=True;Connection Timeout=30;Data
Source=xyz;User ID=admin;Password=password;Initial
Catalog=#{linkedService().LSDBName};
The value for database is passed from a pipeline variable at runtime.
I want to save this connection string to Azure Key Vault, but the issue is that after the value is read from the key vault, the linked service parameter "LSDBName" is not dynamically replaced by the actual value and it tries to connect to "#{linkedService().LSDBName}" as the database name.
Is there any way to secure a dynamically parameterized connection string in key vault? Or a workaround to achieve this?
Thanks!
If you want to store the entire connection string in key vault then you have to pass the connection string in "Server=myServerAddress;Database=myDataBase;User Id=myUsername;Password=myPassword;" format. Create separate connection string for each database and store it in key vault with different secrets and then create parameterized linked service in ADF, giving these secrets as parameter.
My idea is using Set Variable Activity+Azure Function Activity.
First step is using Set Variable Activity to get the LinkedService Connection String.
Second Step is passing the variable as parameter into Azure Function Activity. Then use AKV sdk to store the connection string value in the azure function inside.
Incidentally, I think your connection string has been parameterized already, security issues have been avoided. You don't have to store it into AKV again because mostly we read private information from AKV, rather than write store information into AKV in ADF. Just my own opinion.