Azure WebJobs QueueTrigger ignores environment when reading configuration - azure

I have an Azure WebJobs with QueueTrigger:
public void ProcessTestQueueByTrigger(
[QueueTrigger("test-queue", Connection = "MyCustomStorageConnection")] string queueMessage,
int dequeueCount,
TextWriter log)
{
var message = $"ProcessTestQueueByTrigger executed: {queueMessage}, dequeue: {dequeueCount}";
log.WriteLine(message);
}
}
Where MyCustomStorageConnection is not the same as default jobs connection (that is why I define it in the QueueTrigger attribute). I also have two local files which defines settings for dev and prod environment: appsettings.json and appsettings.Production.json.
When I start the WebJobs, I'm reading the valid configuration and settings to the config object
var configuration = new JobHostConfiguration
{
DashboardConnectionString = config.GetConnectionString("AzureWebJobsDashboard"),
StorageConnectionString = config.GetConnectionString("AzureWebJobsStorage"),
};
and the valid connection stored in config.GetConnectionString("MyCustomStorageConnection") but there is no place to set it with the host. And whenever host is started, the QueueTrigger is reading the value from the appsettings.json and totally ignores the appsettings.Production.json.
How can I force QueueTrigger to use the proper config or just define the value for the QueueTrigger connection string?

You can achieve this easily by setting environment variable.
The following are the steps, on windows 10 with visual studio 2017.
1.Create a .NET core webjob(you can follow this doc if not familiar), and the completed code as below:
Program.cs:
using System;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
namespace MySampleWebjob
{
class Program
{
static void Main(string[] args)
{
//since we have 2 .json files, we can control which .json file is to be used by setting a Environment variable. And then read the value here.
var environment = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");
Console.WriteLine($"the environment is: "+ environment);
var builder = new HostBuilder();
builder.ConfigureWebJobs(b =>
{
b.AddAzureStorageCoreServices();
b.AddAzureStorage();
})
.ConfigureAppConfiguration((hostContext, configApp) => {
configApp.AddJsonFile("appsettings.json", optional: true,reloadOnChange:true);
configApp.AddJsonFile($"appsettings.{environment}.json", optional: true, reloadOnChange: true);
configApp.AddEnvironmentVariables();
})
.ConfigureLogging((context, b) =>
{
b.AddConsole();
})
;
var host = builder.Build();
using (host)
{
host.Run();
}
}
}
}
Functions.cs:
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
namespace MySampleWebjob
{
public class Functions
{
public static void ProcessQueueMessage([QueueTrigger("myqueue-items")] string message, ILogger logger)
{
logger.LogInformation(message);
}
}
}
Then add 2 .json files in the project: appsettings.Production.json and appsettings.json. The structure of the 2 .json files are the same, but have different storage connectiong strings.
appsettings.json:
{
"AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=xxx;AccountKey=xx;EndpointSuffix=core.windows.net"
}
appsettings.Production.json:
{
"AzureWebJobsStorage": "Another azure storage connection string"
}
Note that in visual studio, right click each of the json file -> properties -> set "Copy to Output Directory" to "Copy if newer".
At last, set environment variable via cmd or UI. The cmd command is setx ASPNETCORE_ENVIRONMENT "Production", then restart the visual studio if you want to use the environment variable.
So if you want to use the storage connection string from appsettings.json, don't need to set the environment variable. If you want to use the connection string from appsettings.Production.json, need to set the environment variable.
Note that, if you set an environment variable or remove an environment variable, remember to restart the visual studio to take effect.

Related

CosmosDBTrigger not getting invoked when CosmosDB binding is also present

I'm trying to make a CosmosDBTriggered function in my precompiled C# CI/CD deployed project.
Here's the function implementation, which gets deployed with no complaints. I've tried static and instance methods.
There are no errors but also no invocations as reported by the monitoring/Insights tools even though the watched Collection has items and changes while it's deployed.
The function says it's enabled and has a Cosmosdb trigger:
I've tried adding these dependencies individually, but no changes:
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.CosmosDB" Version="3.0.10" />
<PackageReference Include="Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator" Version="1.2.1" />
This function DOES NOT appear in the Triggers of any CosmosDB Collection as I might expect, but I think that's possibly for a different kind of Trigger.
What configuration step am I missing??
UPDATE
When I comment out this [CosmosDB] DocumentClient binding (and anything that relies on it), the function is invoked. So I guess it's a problem with those bindings being used together?
Are you sure you set the CosmosDbConnection in azure function app on azure?
For example, this is my function app:
Function1.cs
using System;
using System.Collections.Generic;
using Microsoft.Azure.Documents;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Extensions.Logging;
namespace FunctionApp109
{
public static class Function1
{
[FunctionName("Function1")]
public static void Run([CosmosDBTrigger(
databaseName: "testbowman",
collectionName: "testbowman",
ConnectionStringSetting = "str",
CreateLeaseCollectionIfNotExists = true,
LeaseCollectionName = "lease")]IReadOnlyList<Document> input, ILogger log)
{
if (input != null && input.Count > 0)
{
log.LogInformation("Documents modified " + input.Count);
log.LogInformation("First document Id " + input[0].Id);
}
}
}
}
local.settings.json
{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=0730bowmanwindow;AccountKey=xxxxxx;EndpointSuffix=core.windows.net",
"FUNCTIONS_WORKER_RUNTIME": "dotnet",
"str": "AccountEndpoint=https://testbowman.documents.azure.com:443/;AccountKey=xxxxxx;"
}
}
But when deploy function app to azure, the local.settings.json will not be used, you need to set the connection string here:
The function app on azure will not tell you this thing, it just doesn't work.
Based on your updated post, the issue is that the Functions runtime is not initializing your Function because of some configuration issue in your bindings.
Normally, the actual error should be in your Application Insights logs.
The Cosmos DB output binding you are using is missing the collection and database properties. Checking the official samples: https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-cosmosdb-v2-input?tabs=csharp#http-trigger-get-multiple-docs-using-documentclient-c and assuming CosmosDBConnection is a variable that points to the setting name of a setting that contains the connection string:
[CosmosDB(
databaseName: "<your-db-name>",
collectionName: "<your-collection-name>",
ConnectionStringSetting = CosmosDBConnection)] DocumentClient dbClient,

How do I pass in the storage account connection string for a CosmosDBTrigger?

I'm trying to figure out the proper way to pass in a storage account connection string to a CosmosDBTrigger. I have a function that runs when there is a change on a CosmosDB container. This function copies image blobs from one container to another. If you look at the code below, I have commented out the line where I am trying to fine the storage account that I want to connect to. This function runs when that is commented out. It does not run when I have that un-commented. Why?
public static class Function1
{
[FunctionName("ImageCopier")]
public static async Task Run([CosmosDBTrigger(
databaseName: "MyDatabase",
collectionName: "Orders",
ConnectionStringSetting = "databaseConnection",
CreateLeaseCollectionIfNotExists = true,
LeaseDatabaseName = "TriggerLeases",
LeaseCollectionName = "TriggerLeases",
LeaseCollectionPrefix = "ImageCopier")]IReadOnlyList<Document> input,
//[StorageAccount("MyStorageAccount")]string storageConnectionString,
ILogger log)
{
I have MyStorageAccount defined in my local.settings.json file and I also have it in my Azure Function Configuration settings. I copied the connection string directly from the storage account keys panel.
When you set up a CosmosDB trigger, the information that is supplied in that trigger is specific to the trigger. If you need a setting or configuration not related to the trigger in your code, you can use the Environment.GetEnvironmentVariable method.
In your local environment, you can set these variables by editing the local.settings.json file, specifically the Values array. For example:
{
"IsEncrypted": false,
"Values": {
"JobUri": "https://yourapiendpointurl.com",
"BlobStorageConnectionString" : "the connection string",
"AzureWebJobsStorage": "UseDevelopmentStorage=true",
"FUNCTIONS_WORKER_RUNTIME": "dotnet"
}
}
In your method, you may grab that value like so:
public static class Function1
{
[FunctionName("ImageCopier")]
public static async Task Run([CosmosDBTrigger(
databaseName: "MyDatabase",
...
ILogger log)
{
var connectionString =
Environment.GetEnvironmentVariable("BlobStorageConnectionString");
}
}
The local.settings.json file will not be used when it's running in Azure.
I am not sure that when you publish the function if your local.settings.json file will migrate the settings to your Azure Function app's configuration, so I would check to make sure that your settings are in there after publishing.
Side note: Be carful when committing code to repos .. you don't want "secrets" in your repositories in case someone gets in to your repo and discovers it.
While you can access raw configuration values using GetEnvironmentVariable, a more robust/idiomatic approach with .NET in particular is to leverage the built-in dependency injection of configuration.
Using this, you can accept an IConfiguration or strongly-typed IOptions through the function's constructor and use the values in your code. For example:
public class Function1
{
private readonly IConfiguration configuration;
public Function1(IConfiguration configuration)
{
this.configuration = configuration;
}
[FunctionName("ImageCopier")]
public async Task Run([CosmosDBTrigger(/* trigger params */)] IReadOnlyList<Document> input)
{
var connectionString = configuration["MyStorageAccount"];
// Use connection string
}
}
You can take this further to inject services like an "ImageBlobService" into your function that have already been configured in a common Startup Configure method just like ASP.NET Core. That way the individual functions don't need to know anything about configuration and just ask for the relevant service to use.

Azure Storage Queue Trigger - Use remote queue name

I am using Azure App Configuration Store to store configuration. I am using the following code in startup.cs to load my config from Azure.
var builder = new ConfigurationBuilder();
builder.AddAzureAppConfiguration(options =>
{
options.Connect(this.Values.AppConfigConnectionString);
options.Select(keyFilter: KeyFilter.Any, labelFilter: this.Values.Env);
});
var config = builder.Build();
Now this config variable contains my queue names. I need this dynamic so to create and handle it in 4 different environments. Dev / Stage / QA / Prod.
public async Task Run(
[QueueTrigger("%QueueName%", Connection = "StorageConnection")]VoiceHubEvent item)
This isn't working as my local.settings.json file doesn't contain QueueName entry.
Is it possible to make use of config variable in Run() to resolve queuename? By reloading queue trigger function or something?
Thanks,
Kiran.
Is it possible to make use of config variable in Run() to resolve queuename? By reloading queue trigger function or something?
Yes, you can.
Create an extensions method for the IWebJobsBuilder interface to set up a connection to AzureAppConfiguration.
public static IWebJobsBuilder AddAzureConfiguration(this IWebJobsBuilder webJobsBuilder)
{
//-- Get current configuration
var configBuilder = new ConfigurationBuilder();
var descriptor = webJobsBuilder.Services.FirstOrDefault(d => d.ServiceType == typeof(IConfiguration));
if (descriptor?.ImplementationInstance is IConfigurationRoot configuration)
configBuilder.AddConfiguration(configuration);
var config = configBuilder.Build();
//-- Add Azure Configuration
configBuilder.AddAzureAppConfiguration(options =>
{
var azureConnectionString = config[TRS.Shared.Constants.CONFIGURATION.KEY_AZURECONFIGURATION_CONNECTIONSTRING];
if (string.IsNullOrWhiteSpace(azureConnectionString)
|| !azureConnectionString.StartsWith("Endpoint=https://"))
throw new InvalidOperationException($"Missing/wrong configuration value for key '{TRS.Shared.Constants.CONFIGURATION.KEY_AZURECONFIGURATION_CONNECTIONSTRING}'.");
options.Connect(azureConnectionString);
});
//build the config again so it has the key vault provider
config = configBuilder.Build();
return webJobsBuilder;
}
Where the azureConnectionString is read from you appsetting.json and should contain the url to the Azure App Configuration.
In startup.cs:
public void Configure(IWebJobsBuilder builder)
{
builder.AddAzureConfiguration();
ConfigureServices(builder.Services)
.BuildServiceProvider(true);
}
For more details, you could refer to this SO thread.

QueueTrigger is not picking messages- Azure WebJobs SDK 3.0

I'm trying to develop WebJob using SDK 3.0.x, and testing it locally. I've followed the sample in github without any success.
When running it locally everything is going ok, it also see the ProcessQueueMessage function but it doesn't pick the messages from the queue.
Program.cs
static void Main(string[] args)
{
var builder = new HostBuilder();
//builder.UseEnvironment(EnvironmentName.Development);
builder.ConfigureWebJobs(b =>
{
b.AddAzureStorageCoreServices();
b.AddAzureStorage();
});
builder.ConfigureAppConfiguration((context, config) =>
{
config.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);
});
builder.ConfigureLogging((context, b) =>
{
b.AddConsole();
// If the key exists in settings, use it to enable Application Insights.
string instrumentationKey = context.Configuration["APPINSIGHTS_INSTRUMENTATIONKEY"];
if (!string.IsNullOrEmpty(instrumentationKey))
{
b.AddApplicationInsights(o => o.InstrumentationKey = instrumentationKey);
}
});
builder.ConfigureServices((context, services) =>
{
//services.AddSingleton<IJobActivator, MyJobActivator>();
services.AddScoped<Functions, Functions>();
services.AddSingleton<IHostService, HostService>();
})
.UseConsoleLifetime();
var host = builder.Build();
using (host)
{
host.Run();
}
}
Functions.cs
public class Functions
{
private readonly IHostService _hostService;
public Functions(IHostService hostService)
{
_hostService = hostService;
}
// This function will get triggered/executed when a new message is written
// on an Azure Queue called queue.
public void ProcessQueueMessage([QueueTrigger("newrequests")] string dd,
//DateTimeOffset expirationTime,
//DateTimeOffset insertionTime,
//DateTimeOffset nextVisibleTime,
//string queueTrigger,
//string id,
//string popReceipt,
//int dequeueCount,
ILogger logger)
{
var newRequestItem = new RequestQueueItem();
logger.LogTrace($"New queue item received...");
//logger.LogInformation($" QueueRef = {id} - DequeueCount = {dequeueCount} - Message Content [Id = {newRequestItem.Id}, RequestDate = {newRequestItem.RequestDate}, Mobile = {newRequestItem.Mobile}, ProviderCode = {newRequestItem.ProviderCode}, ItemIDClass = {newRequestItem.MappingIDClass}]");
// TODO: Read the DatabaseConnectionString from App.config
logger.LogTrace($" Getting DB ConnectionString...");
var connectionString = ConfigurationManager.ConnectionStrings["DatabaseConnection"].ConnectionString;
// TODO: Initiation of provider service instance
logger.LogTrace($" Init IalbayanmtnclientserviceClient service instance...");
var bayanService = new AlbayanMtnWCFService.IalbayanmtnclientserviceClient();
// TODO: sending request to provider service endpoint and wait for response
logger.LogTrace($" Sending request to Service Endpoint...");
var response= bayanService.requestpaymenttransactionAsync("agentcode", "agentpassword", "accountno", int.Parse(newRequestItem.TransactionType), newRequestItem.MappingIDClass, newRequestItem.Mobile, (int)newRequestItem.Id).Result;
logger.LogTrace($"Done processing queue item");
}
}
Here is the screen shot for the output
Appreciate your help
Screen shot for queue messages 'newrequests'
enter image description here
From your snapshot, your webjob runs well on local. It didn't pick message because you don't add message in the newrequests queue.
The function only be triggered after you add the message. Or I will get the same result just like yours.
About the tutorial , your could refer to the official doc:Get started with the Azure WebJobs SDK. And make sure you set the right storage account. The below is my appsettings.json. Make sure the "Copy to output directory" property of the appSettings.json file is set to either Copy if newer or Copy always. Or it will run into exception:Storage account 'Storage' is not configured.
{
"ConnectionStrings": {
"AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=mystorage;AccountKey=key;..."
}
}
Hope this could help you, if you still have other questions, please let me know.

Web app running on Azure is not picking up and using an environment variable connection string (ASP,NET Core RC1)

Github link for reproduction.
I have an ASP.NET Core (RC1) application that works fine locally. The issue I'm having is that my connection string is not being picked up by my Azure app. I've asked similar questions to this, but I've narrowed down the issue on my end in this app. Note, it requires an app on Azure to reproduce it.
Here's the issue I'm seeing.
First, my configuration is setup as such:
public Startup()
{
var builder = new ConfigurationBuilder()
.AddJsonFile("config.json")
.AddEnvironmentVariables();
mConfiguration = builder.Build();
}
And EF7 is setup here:
public void ConfigureServices(IServiceCollection services)
{
services.AddEntityFramework()
.AddSqlServer()
.AddDbContext<FooDbContext>(options =>
{
// I'm assuming it's failing here.
// I'm not sure how to debug it running on Azure.
// All the developer exception page shows is:
// 500 Internal Server Error "An error occurred while starting the application."
options.UseSqlServer(mConfiguration["Data:ConnectionStringTest:ConnectionString"]);
});
services.AddScoped<IFooDataService, FooSqlDataService>();
}
My config.json has:
{
"Data": {
"ConnectionStringTest": {
"ConnectionString": "Data Source=(localdb)\\mssqllocaldb;Initial Catalog=ConnectionStringTest"
}
}
}
And this should be overridden by the connection string I've setup in Azure:
When going to the Kudu SCM and looking at the environment variables on the Azure web app instance, I see the following:
SQLAZURECONNSTR_Data:ConnectionStringTest:ConnectionString = my_connection_string_here
I am assuming this is the class that is being used under the hood when my environment variable is used at runtime: EnvironmentVariablesConfigurationProvider
Ok here's what I found, and this feels awkward.
It seems that you need to use Data:{my_connection_string_key}:ConnectionString everywhere EXCEPT in Azure. This environment variable converter will construct the proper connection string using this format automatically if the connection string is prefixed with SQLAZURECONNSTR_.
This means when you setup your connection string in Azure, you need to omit EVERYTHING except the key to your connection string. Do not insert Data: or :ConnectionString... simply use {connection_string_key} (refer to the above format) instead. If you include the entire format in your Azure key/value pair, the EnvironmentVariablesConfigurationProvider will add another Data: and :ConnectionString around it, resulting in something like Data:Data:{my_connection_string_key}:ConnectionString:ConnectionString.
In ConfigureServices(...), use the format that ASP expects:
... options.UseSqlServer(mConfiguration["Data:ConnectionStringTest:ConnectionString"]);
You can therefore locally use this for your local JSON, for testing in development/fallback:
{
"Data": {
"ConnectionStringTest": {
"ConnectionString": "Data Source=(localdb)\\mssqllocaldb;Initial Catalog=ConnectionStringTest"
}
}
}
Just make sure your Azure connection string has the middle part of that format (ConnectionStringTest using this example).
This will make your environment variable in Azure look like this in raw format:
SQLAZURECONNSTR_ConnectionStringTest = {insert connection string here}
And the EnvironmentVariablesConfigurationProvider will strip off the Azure prefix string, and wrap your key in the hardcoded format: Data:{0}:ConnectionString
Experiment Results
To augment your excellent answer, I did a local experiment to confirm that the SQLAZURECONNSTR_connection_string_key environmental variable becomes this configuration:
mConfiguration["Data:connection_string_key:ConnectionString"]
Local Experiment
A local environmental variable emulates an Azure SQL Database connection string named connection_string_key.
PS> $env:SQLAZURECONNSTR_connection_string_key = "an azure conn string"
The following code dumps all the environmental variables and configuration sections to the page.
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
app.Run(async (context) =>
{
await context.Response.WriteAsync("# Environmental Variables \r\n");
await DumpAllEnvVariables(context, Environment.GetEnvironmentVariables());
await context.Response.WriteAsync("# Configuration Sections \r\n");
await DumpAllConfigItems(context, mConfiguration.GetChildren());
});
}
private async Task DumpAllEnvVariables(HttpContext context, IDictionary envVariables)
{
foreach (var envVar in envVariables.Cast<DictionaryEntry>())
{
await context.Response.WriteAsync($"{envVar.Key}"); // : {envVar.Value}
await context.Response.WriteAsync($"\r\n");
}
}
private async Task DumpAllConfigItems(HttpContext context,
IEnumerable<IConfigurationSection> sections, string prefix = "")
{
foreach (var section in sections)
{
await context.Response.WriteAsync($"{prefix}{section.Key}"); // : {envVar.Value}
await context.Response.WriteAsync($"\r\n");
if(section.GetChildren().Any())
{
await DumpAllConfigItems(context, section.GetChildren(), prefix + " ");
}
}
}

Resources