I was wondering if it's possible to initialize the queue trigger or even the blob trigger off a connection string that is read from azure vault.
Right now, we have to set these data connection via environment settings via blade properties. However, I wanted to just use the service principal to retrieve the token for the azure key vault to get all these connection strings.
I'm trying to figure how to get this working in java.
Thanks,
Derek
This feature is tracked and in progress here:
Feature request: retrieve Azure Functions' secrets from Key Vault
Add binding to Key Vault
EDIT 28/11/2018: It is currently in preview
Simplifying security for serverless and web apps with Azure Functions and App Service
Former answer 07/10/2018
This solution won't work for Triggers using the consumption plan.
In the mean time I did some research about your problem and it is possible to read config from key vault if you use Azure Function v2.
I've created an Azure Functions v2 (.NET Standard) from Visual Studio.
It uses:
NETStandard.Library v2.0.3
Microsoft.NET.Sdk.Functions v1.0.22
Microsoft.Azure.WebJobs v3.0.0
Microsoft.Azure.WebJobs.Extensions.Storage v3.0.0
Because Azure Functions v2 uses ASP.NET core, I was able to reference this link to configure my functions app to use Azure Key Vault:
Azure Key Vault configuration provider in ASP.NET Core
I've added this nuget package:
Microsoft.Extensions.Configuration.AzureKeyVault
I've configured my app to use this nuget package:
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using System.Linq;
[assembly: WebJobsStartup(typeof(FunctionApp1.WebJobsExtensionStartup), "A Web Jobs Extension Sample")]
namespace FunctionApp1
{
public class WebJobsExtensionStartup : IWebJobsStartup
{
public void Configure(IWebJobsBuilder builder)
{
// Get the existing configuration
var serviceProvider = builder.Services.BuildServiceProvider();
var existingConfig = serviceProvider.GetRequiredService<IConfiguration>();
// Create a new config based on the existing one and add kv
var configuration = new ConfigurationBuilder()
.AddConfiguration(existingConfig)
.AddAzureKeyVault($"https://{existingConfig["keyVaultName"]}.vault.azure.net/")
.Build();
// replace the existing configuration
builder.Services
.Replace(ServiceDescriptor.Singleton(typeof(IConfiguration), configuration));
}
}
}
My Azure functions uses MSI:
I've granted Read/List secrets permissions to the function app on my key vault:
I have a small queue triggered function:
public static class Function2
{
[FunctionName("Function2")]
public static void Run([QueueTrigger("%queueName%", Connection = "queueConnectionString")]string myQueueItem, ILogger log)
{
log.LogInformation($"C# Queue trigger function processed: {myQueueItem}");
}
}
The queueName is defined in the local.settings.json file (App settings blade once deployed):
{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "UseDevelopmentStorage=true",
"FUNCTIONS_WORKER_RUNTIME": "dotnet",
"keyVaultName": "thomastestkv",
"queueName": "myqueue"
}
}
The queueConnectionString is configured in my keyvault:
Sourcing Application Settings from Key Vault
The Key Vault references feature makes it so that your app can work as if it were using App Settings as they have been, meaning no code changes are required. You can get all of the details from our Key Vault reference documentation, but I’ll outline the basics here.
This feature requires a system-assigned managed identity for your app. Later in this post I’ll be talking about user-assigned identities, but we’re keeping these previews separate for now.
You’ll then need to configure an access policy on your Key Vault which gives your application the GET permission for secrets. Learn how to configure an access policy.
Lastly, set the value of any application setting to a reference of the following format:
#Microsoft.KeyVault(SecretUri=secret_uri_with_version)
Where secret_uri_with_version is the full URI for a secret in Key Vault. For example, this would be something like: https://myvault.vault.azure.net/secrets/mysecret/ec96f02080254f109c51a1f14cdb1931
That’s it! No changes to your code required!
For this initial preview, you need to explicitly set a secret version, as we don’t yet have built-in rotation handling. This is something we look forward to making available as soon as we can.
User-assigned managed identities (public preview)
Our existing support for managed identities is called system-assigned. The idea is that the identity is created by the platform for a specific application and is tied to the lifecycle of the application. If you delete the application, the identity is removed from Azure Active Directory immediately.
Today we’re previewing user-assigned identities, which are created as their own Azure resource and then assigned to a given application. A user-assigned identity can also be assigned to multiple applications, and an application can have multiple user-assigned identities.
more details check this
Update
This is now GA
This was just released as preview a couple days ago.
This feature requires a system-assigned managed identity for your app. Later in this post I’ll be talking about user-assigned identities, but we’re keeping these previews separate for now.
You’ll then need to configure an access policy on your Key Vault which gives your application the GET permission for secrets. Learn how to configure an access policy.
Lastly, set the value of any application setting to a reference of the following format:
#Microsoft.KeyVault(SecretUri=secret_uri_with_version)
Where secret_uri_with_version is the full URI for a secret in Key Vault. For example, this would be something like: https://myvault.vault.azure.net/secrets/mysecret/ec96f02080254f109c51a1f14cdb1931
Using Keyvault integration within the function runtime
I just implemented it in Java following below two references.
https://learn.microsoft.com/en-us/azure/app-service/app-service-key-vault-references
https://medium.com/statuscode/getting-key-vault-secrets-in-azure-functions-37620fd20a0b
in java use System.getenv("SECRET_KEY") to read the values from your app settings.
Happy to help if you need further assistance.
I have already given my answer in above, this answer is for #Matt Sanders's comment,
i just want to explain how MSI working in the Azure Environment,
"I have 2 user assigned identities, 1 has permissions to KeyVault, the other does not. How can you specify the correct user assigned an identity to use for retrieving the secret? I'm guessing this is not possible and User Assigned Identities are not supported even though they are listed in your answer. – Matt Sanders"
when you want to use Azure Manage Identity Service, your application must register in the Azure AD, for an example, lets say multiple users accessing your web application and, within your web application, you 'r trying to access vVault's secrets, In that case, vault doesnt care about the users that consume your application, it cares about the application,
please reffer below image,
I as showing the picture, only Azure Function added as an Identity to the vault, other applications are not,
so whoever using Azure function can access vault's secrets, according to this example only A and B can access secrets,
Related
In Azure you can setup an App Config and a KeyVault. The point of the KeyVault being to store more sensitive data than your App Config and be able to regulate access to the config and vault separately.
So what is the benefit of using a keyvault reference in the app config?
You are basically allowing anyone with access to the app config to access certain values in your keyvault and are bypassing the additional layer of security the vault normally provides.
The additional layer being required auth to the vault to access those same values if they aren't referenced in the config.
I really don't understand what benefit keyvault references give you.
This blog article by Jan de Vries explains them in more detail: https://jan-v.nl/post/2021/using-key-vault-with-azure-app-configuration/.
The relevant part for your question:
As it happens, the code for accessing App Configuration doesn’t give your application permission to retrieve secrets from Key Vault.
The application retrieves them from Key Vault, not from App Configuration.
App Config only holds the reference, not the actual value.
Official docs also mention this:
Your application uses the App Configuration client provider to retrieve Key Vault references, just as it does for any other keys stored in App Configuration. In this case, the values stored in App Configuration are URIs that reference the values in the Key Vault. They are not Key Vault values or credentials. Because the client provider recognizes the keys as Key Vault references, it uses Key Vault to retrieve their values.
Your application is responsible for authenticating properly to both App Configuration and Key Vault. The two services don't communicate directly.
I suppose there are different approaches to using the KeyVault, but the way I tend to use it is as follows.
My application will have a set of secrets, which I store locally using the Secrets Manager, you would add the secret for your application:
dotnet user-secrets set "Movies:ServiceApiKey" "12345"
Your application can then read this setting using _moviesApiKey = Configuration["Movies:ServiceApiKey"]; as you'll see in the link above. Obviously, there's no way you can see this value in the code, but your application can read it from the Secrets Manager.
If you do forget the values, you can use the following command to retrieve them:
dotnet user-secrets list
KeyVault will work as your Secrets Manager within Azure. So, your application will need to have permission to access the KeyVault, and in my case I store the Vault name in the appsettings.json, and during the bootstrapping, I include the KeyVault configuration if running in Production mode i.e. on the Azure Server and not locally.
public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.ConfigureLogging(logging =>
{
logging.ClearProviders();
logging.AddConsole();
logging.AddAzureWebAppDiagnostics();
})
.ConfigureAppConfiguration((context, config) =>
{
if (context.HostingEnvironment.IsProduction())
{
IConfigurationRoot builtConfig = config.Build();
ConfigurationBuilder keyVaultConfigBuilder = new ConfigurationBuilder();
keyVaultConfigBuilder.AddAzureKeyVault(builtConfig["VaultName"]);
IConfigurationRoot keyVaultConfig = keyVaultConfigBuilder.Build();
config.AddConfiguration(keyVaultConfig);
}
})
.UseStartup<Startup>();
Note, the check for context.HostingEnvironment.IsProduction(). Within the appsettings, I have:
"VaultName": "https://yourkvname.vault.azure.net/"
So, the only reference I have to the KeyVault from the application is the name, and that should be secure as only the application will have access to the keys/secrets.
One thing to note, you need to make sure that the names match both for your local secrets and the ones in the KeyVault. In my case, I am running on a Windows platform, so I needed to make a small change to the names using double dashes (--) in place of the colon (:), so...
Movies:ServiceApiKey
Becomes
Movies--ServiceApiKey
When working in Azure, storing secrets in Key Vault is a good idea. And to make it better, there’s the Key Vault Reference notation. This feature makes sure no one can read the secret(s) unless someone grants permission.
Speaking of secrets, they should never be directly stored in application settings of a Function App (same goes for App Services by the way). Why not ? Because secrets would be available to anyone who has access to the Function App in the Azure Portal. The right way is to use an Azure Key Vault which is the Azure component for securely storing and accessing secrets 🔒. Once your secrets are in the key vault, you have to grant the Key Vault access to the identity of your Function App and you can then reference the secrets you need directly in your application settings. These are called Key Vault references because an application setting does not contain directly the value of a secret but a reference to the secret which is stored in Key Vault. When running, your function will automatically have access to the secret and its value as an environment variable, as if it was a normal application setting.
Key Vault references work for both App Services and Function Apps and are particularly useful for existing applications that have their secrets stored in settings because securing the secrets with Azure Key Vault references does not require any code change.
Reference: https://www.techwatching.dev/posts/azure-functions-custom-configuration
https://www.sharepointeurope.com/using-key-vault-references-with-azure-app-configuration/
I was able to use the Key Vault inside a function app as described here but when I tried to use the Key Vault to hold the connection string of a function with a queue trigger I have issues with the storage account connection string. The function seems to find the parameter I provide but either doesn't get the secret back or doesn't like the information when it throws the error No valid combination of account information found.
My function is defined as:
[FunctionName("ReadQueueForMessage")]
public static async Task Run([QueueTrigger("%AzureQueueTrigger%", Connection = "AzureWebJobsStorage")] string myQueueItem,
Binder binderinputblob,
ILogger log)
This works fine if I just define the connection string in my local.settings.json. What I'm trying to do is instead of just putting the connection string in the json file, I want to point the function to the Key Vault with this syntax:
"AzureWebJobsStorage": "#Microsoft.KeyVault(SecretUri=https://myappkeyvault.vault.azure.net/secrets/myapp-AzureWebJobsStorage-Queue/the-guid-of-secret)",
I did go to the Key Vault and update the access policies to include the function app so it can read/list the secrets. The documentation here shows making an update to the configuration after it was deployed to Azure. I'm trying to test first in Visual Studio. Maybe that is the issue? Or is it not possible at all to use the secret in this manner?
I'm trying to test first in Visual Studio.
For now using Azure Key Vault references with Azure Functions does not support to work on local, as confirmed by Azure Functions team. If you still want to test on local, you could implemented an incomplete local workaround like this issue.
I test on portal and it works well. You could refer to the following steps as below：
1.In VS Function.cs, then publish to azure：
public static void Run([QueueTrigger("queue", Connection = "AzureWebJobsStorage")]string myQueueItem, TraceWriter log)
{
log.Info($"C# Queue trigger function processed: {myQueueItem}");
string connectionString = System.Environment.GetEnvironmentVariable("AzureWebJobsStorage");
log.Info($"The connection string is {connectionString}");
}
2.Set AzureWebJobsStorage on Appsettings setting on portal.
3.Then it will work fine.
With the Nuget packages Azure.Extensions.AspNetCore.Configuration.Secrets and Azure.Identity you can now use the KeyVault as a configuration provider, the same way as in ASP.NET Core. You need to add a class derived from FunctionsStartup to add the KeyVault as a configuration provider, see Add FunctionsStartup class with the KeyVault as a configuration provider.
If you add the AzureWebJobsStorage connection string as a secret to the KeyVault, you can remove it from the Configuration section of your Function App in Azure. Make sure to turn on System assigned in the Identity section and in the KeyVault add an Access Policy with the Secret permissions Get and List for your Function App.
When you run your Function App local for debugging, Azure.Identity automatically uses your Microsoft Account for access to the KeyVault, if it has at least Get and List access to the secrets.
Unfortunately, when you test local, the Function App does NOT read AzureWebJobsStorage from the configuration/KeyVault, but requires it to be stored in local.settings.json. To prevent storing keys on your local computer, you can set AzureWebJobsStorage to "UseDevelopmentStorage=true" in local.settings.json.
For detailed instructions see: Create Azure Function App with Dependency Injection and the Key Vault as Configuration Provider.
Example project: https://github.com/Forestbrook/FunctionWithKeyVaultAndDI
I have an environment variable that references a secret in Azure KeyVault:
{
"name": "SECRET",
"value": "#Microsoft.KeyVault(SecretUri=https://keyvault_name.vault.azure.net/secrets/secret_name/)",
"slotSetting": false
}
This is loaded when on Startup.cs in my web api solution:
public void ConfigureServices(IServiceCollection services)
{
services.AddOptions<Secret>().Configure(o => o.ClearText = Configuration["SECRET"]);
services.AddControllers();
}
How is the secret resolved? Is it resolved on every option call? Eg everytime it gets injected into my constructor, or is it resolved when the environment variable is resoved and loaded into the Configuration object?
Reason why I ask is that every call to the vault costs a tiny bit, but it all adds up.
UPDATE
I have added this as an issue on github: https://github.com/MicrosoftDocs/azure-docs/issues/44064#event-2855607916
Based on my testing, Azure web app will load data from Azure key vault in two scenarios :
You submitted a Application settings modify request to Azure.
You restart your Azure Web app.
Your secret will be cached in Azure app service. You can try this scenario : After you add an environment variable that references a secret in Azure KeyVault , create a new version with different value in Azure key vault , you will found that the value in Azure web app will not change : still the old version.
It will not change until you restart your web app or modify your Application settings on portal again.
I also checked metrics of my Azure key vault, I called my Azure web app to retrieve the secret I configed in app settings for about 10 times , but the access metrics of my Azure key vault at that time point is 0. But when I restart my web app , the access metrics of Azure key vault will increase .
Hope it helps .
When I develop for Azure I usually start copying in some keyvault client code so only keyvault urls will be in my settings file, no secrets can ever end up my git repositories.
After starting to make Azure functions I realized that it was not possible to do this for the trigger connection string for e.g. service bus or blob storage.
The recommended approach seems to connect the app to keyvault directly in Azure when deployed, and just manage secrets locally in Secret Manager, like suggested in
this article
I am not developing alone, so while I am not adverse to using a tool like Secret Manager, I need to still have my offline secrets connected to the Azure keyvault! If others change anything.
Question: How do I manage secrets offline in a way that is synchronized with Azure keyvault?
it was not possible to do this for the trigger connection string for e.g. service bus or blob storage.
In short, it's possible.
Here are steps you could follow and refer to the detailed article.
1.Add a System Assigned Managed Identity to the Azure Function.
2.Go to the Access Control section of your Key Vault and click on Add a role assignment blade.
3.Go to your Key Vault and click on Access Policies and then click on Add service principal with secret GET permission.
4.When you use ServiceBusTrigger, you set ServiceBusConnectionString in Function ->Configuration ->Application settings.
public static void Run([ServiceBusTrigger(_topicName, _subscriptionName, Connection = "ServiceBusConnectionString")] string mySbMsg, ILogger log)
{ ....
}
5.Now you change the value of ServiceBusConnectionString to the Azure Key Vault reference with #Microsoft.KeyVault(SecretUri=Secret URI with version). Then you could run your function successfully with Key Vault.
I'd like to use managed identities instead of username and password configured in the storage account connection. I only see the option to configure the queue trigger with a connection string, but can't add a managed identity to avoid secrets to be configured.
Is that possible at all?
I'm fairly certain it is not possible as of now, you can only use managed identities when the function runs to access resources, not for the trigger. I cannot dig up a proof for that right now, saw it on some GH issue.
This is now possible using the Microsoft Azure Function Extension Libraries, e.g. "Microsoft.Azure.WebJobs.Extensions.Storage.Blobs".
Exampleconnection configuration for managed identities:
"QueueSettings:StorageAccount": "",
"QueueSettings:StorageAccount__queueServiceUri": "https://mytestfa.queue.core.windows.net/",
"QueueSettings:StorageAccount__credential": "managedidentity"
And reference the connection in the function trigger like this:
[Function("ProcessUserData")]
public async Task ProcessUserData([QueueTrigger("%QueueSettings:UserDataQueue%", Connection = "QueueSettings:StorageAccount")] string queueItem, FunctionContext context)
{
var logger = context.GetLogger<QueueListener>();
...
}
Original announcement from the Microsoft DevBlog here:
https://devblogs.microsoft.com/azure-sdk/introducing-the-new-azure-function-extension-libraries-beta/
also reference here:
Azure Functions - use queue trigger with managed identity