We used this blog post https://blog.mexia.com.au/dependency-injections-on-azure-functions-v2 to set up dependencies in our Function App project.
DI part works fine, however, some classes require configuration settings.
In DiModule we have the following code:
var configuration = new ConfigurationBuilder()
.SetBasePath(Environment.CurrentDirectory)
.AddJsonFile("serilog.json")
.Build();
Environment.CurrentDirectory works on local machine, but doesn't work in Azure. I know that working implementation is
.SetBasePath(context.FunctionAppDirectory)
where context is of type ExecutionContext.
Is there a way to get ExecutionContext except from function parameter?
Or is there better way to load config in Function App (without loading it from every function call)?
We end up using 2 kinds of settings with Azure Functions:
1. App settings
2. Custom settings stored in the database with a cache layer
Serilog settings are stored in app settings.
Related
I'm revisiting a project which hasn't been updated for a while.
In production/online environment, it uses environment variables defined at:
openshift online console > applications > deployments > my node app > environment
In development/offline environment, it uses environment variables defined at:
./src/js/my_modules/local_settings (this file is ignored by .gitignore)
The code looks something like:
// check which environment we are in
if (process.env.MONGODB_USER) {
var online_status = "online";
}
else {
var online_status = "offline";
}
// if online, use environment variables defined in red hat openshift
if (online_status === 'online') {
var site_title = process.env.SITE_TITLE;
var site_description = process.env.SITE_DESCRIPTION;
//etc
}
// if offline, get settings from a local file
else if (online_status === 'offline') {
var local_settings = require('./src/js/my_modules/local_settings');
var site_title = local_settings.SITE_TITLE;
var site_description = local_settings.SITE_DESCRIPTION;
// etc
}
I would like to install the dotenv package in my local project repo via:
npm install dotenv
So that I can:
Have my local settings in a .env file in the root of my project (ignored in .gitignore)
Be able to use process.env.SOME_VARIABLE rather than local_settings.SOME_VARIABLE
Get rid of some if/else blocks as both scenarios would point to process.env.SOME_VARIABLE
I'm a bit confused as to how this would effect the online environment.
Seeing as both production/online and development/offline environments would use:
var some_variable = process.env.SOME_VARIABLE_HERE
would the application automatically know to:
Look at the local .env file when in development?
Look at the Red Hat environment variables when in production?
And would adding the required instantiation at the beginning of the server-side file:
require('dotenv').config()
somehow make Red Hat OpenShift freak out (as it seems to already have its own 'things' in place to resolve references to process.env.SOME_VARIABLE_HERE to the relevant values defined in the OpenShift console)?
To have a file by any environment (.dev .staging .prod) into the source code repository or manually in the server (it those are in .gitignore) worked for long time, but now it goes against to the devops.
The clean way is to use environment variables but managed remotely and obtained at the start of your application.
How it works?
Basically your apps don't read or need a file (.env .properties, etc) with variables anymore. It loads them from a remote http service.
Not intrusive
In this approach, you don't need specific languages variables (nodejs in your case). You just need to prepare your app to use environment variables. Your application don't care where the variables come from, just needs to be available at operative system level.
To achieve that, you just need to download the variables using a simple shell code or a very basic algorithm (http invocation) in your favorite language.
After that, after the start of your app, variables are ready to use at the most basic level.
var site_title = process.env.SITE_TITLE;
This approach is not intrusive because your app don't need something complex like library or algorithm in some programing language. Just needs the environment variables.
Intrusive
Same as previous alternative but instead to read the variables direct from environment system, you should use or create a class/module in your language. This offer your the variables you need:
var site_title = VariablesManager.getProperty("SITE_TITLE");
VariablesManager at the startup must have consumed the variables from a remote service (http) and the store them to offer them to whoever needs it through getProperty method.
Also this VariablesManager usually has a feature called hot-reload which at intervals, update the variables consuming the remote variables manager. With this, if your application is running in production with real users and some variable needs to be updated, you just need to change it in the variables manager. Automatically your app will load the new values, without restart or touching your app
This approach is intrusive because you need to load advanced libraries in some programing language or create it.
Devops
Your application just needs a few properties or settings related to the consume of remote variables. For example: variables of acme-web-staging:
remote_variables_manager = https://variables.com/api
application_id = acme-web-staging
secure_key = *****
You could hide the secure key and parametrize the application_id using environment variables (created in the platform console)
remote_variables_manager = https://variables.com/api
application_id = ${application_id}
secure_key = ${remote_variables_manager_key}
Or if you want one variable manager by each environment
staging
remote_variables_manager = https://variables-staging.com/api
application_id = acme-web
secure_key = *****
production
remote_variables_manager = https://variables-staging.com/api
application_id = acme-web
secure_key = *****
Variables manager
This concept was introduced many years ago. I used with java. It consist in a web application with features like:
secure login
create applications
create variables of an application
crypt sensitive values
publish http endpoints to download or query the variables by application
Here a list of some ready to use alternatives:
Configurator
Nodejs & mysql solution. I developed this and I use it in various projects.
Doppler
zookeeper
http://www.therore.net/java/2015/05/03/distributed-configuration-with-zookeeper-curator-and-spring-cloud-config.html
Spring Cloud
https://www.baeldung.com/spring-cloud-configuration
This is a java spring framework functionality in which you can create properties file with configurations and configure your applications to read them.
Consul
Consul is a service mesh solution providing a full featured control plane with service discovery, configuration, and segmentation functionality.
doozerd, etcd
In your specific case
Don't use dot-env
Use pure process.env.foo
Deploy a remote variables manager in your openshift infraestructure
Create just one variable in your openshift web console: APP_ENVIRONMENT
In your code at the start, do something like this:
if (process.env.APP_ENVIRONMENT === "PROD")
//get variables from remote service using
//some http client like axios, request, etc
//then inject them to your process.env
process.env.site_url = remoteVariables.site_url
else
//we are in local developer workspace
//so, nothing complex is required
//developer should inject manually
//before the startup: npm run start or dev
//export site_url = "acme.com"
If you can configure an execution of a shell script before the start of your openshift app, you could load and expose the variables at that stage and the previous snippet would not be necessary because the variables will be ready to be retrieved using process.env directly in your app
I've been playing around a bit with Azure App Configuration.
Here's an example configruation:
var environmentVariable = Environment.GetEnvironmentVariable("AppConfigurationConnectionString");
var config =
new ConfigurationBuilder()
.AddAzureAppConfiguration(options =>
{
options.Connect(environmentVariable)
.ConfigureKeyVault(kv =>
{
kv.SetCredential(new DefaultAzureCredential());
});
})
.Build();
services.AddSingleton<IConfiguration>(config);
Following this, I can inject an IConfiguration instance into my services and use _config["settingName"] to access config settings. This all works well and is really quite nice.
One thing that I don't know how to do is to map groups of related settings to a strongly typed object (that is, without having to do it all manually, which I could do, but... hoping there's a better way).
In conventional ASP.NET core configuration, I can group related settings settings as follows (i.e. in appsettings.json)
{
"test": {
"key1": "value1",
"key2": "value2"
}
}
using the IOptions pattern as follows:
services.Configure<Test>(config.GetSection("test"));
which allows me to inject a strongly-typed IOptions<Test> instance into my classes. IMO this is a bit nicer than a big flat indexer, where I use _config["key1"] to get config settings.
Is there an approach for Azure App Configruation to allow me to automatically configure strongly-typed config objects that can be injected into my classes?
TIA
.NET Core flattens objects in appsettings.json when it imports them into IConfiguration. For example, your test object becomes the following two keys in IConfiguration
_config["test:key1"]
_config["test:key2"]
This means that you can accomplish exactly what you want with Azure App Configuration by storing the settings in this flattened manner. The Azure App Configuration UI in the Azure portal has an import utility that will allow you to import an appsettings.json file and it does this importing for you.
Here is an example of the import utility in use:
After you have the flattened object in Azure App Configuration the exact code you have will work.
I have a Core WebJob deployed into an Azure Web App. I'm using WebJobs version 3.0.6.
I've noticed that changes to Connection Strings and App Settings (added via the Azure web UI) are not being picked up immediately by the WebJob code.
This seems to correlate with the same Connection Strings and App Settings not being displayed on the app's KUDU env page straight away (although I acknowledge this may be a red herring and could be some KUDU caching thing which I'm unaware of).
I've deployed a few non-Core WebJobs in the past and have not come across this issue so wonder if it's Core related? Although I can't see how that might affect configs showing up KUDU though.
I was having this issue the other day (where the configs were not getting picked up by the WebJob or shown in KUDU) and was getting nowhere, so left it. When I checked back the following day, the configs were now correctly showing in KUDU and being picked up by the WebJob. So I'd like to know what has happened in the meantime which means the configs are now being picked up as expected.
I've tried re-starting the WebJob and re-starting the app after making config changes but neither seem to have an effect.
It's worth also noting that I'm not loading appSettings.json during the program setup. That being said, the connection string being loaded was consistenly the connection string from that file i.e. my local machine SQL Server/DB. My understanding was always that the anything in the Azure web UI would override any equivalent settings from config files. This post from David Ebbo indicates that by calling AddEnvironmentVariables() during the setup will cause the Azure configs to be observed, but that doesn't seem to be the case here. Has this changed or is it loading the configs from this file by convention because it can't see the stuff from Azure?
Here's my WebJob Program code:
public static void Main(string[] args)
{
var host = new HostBuilder()
.ConfigureHostConfiguration(config =>
{
config.AddEnvironmentVariables();
})
.ConfigureWebJobs(webJobConfiguration =>
{
webJobConfiguration.AddTimers();
webJobConfiguration.AddAzureStorageCoreServices();
}
)
.ConfigureServices((context, services) =>
{
var connectionString = context.Configuration.GetConnectionString("MyConnectionStringKey");
services.AddDbContext<DatabaseContext>(options =>
options
.UseLazyLoadingProxies()
.UseSqlServer(connectionString)
);
// Add other services
})
.Build();
using(host)
{
host.Run();
}
}
So my questions are:
How quickly should configs added/updated via the Azure web UI be displayed in KUDU?
Is the fact they're not showing in KUDU related to my Core WebJob also not seeing the updated configs?
Is appSettings.json getting loaded even though I'm not calling .AddJsonFile("appSettings.json")?
What can I do to force the new configs added via Azure to be available to my WebJob immediately?
The order in which configuration sources are specified is important, as this establishes the precedence with which settings will be applied if they exist in multiple locations. In the example below, if the same setting exists in both appsettings.json and in an environment variable, the setting from the environment variable will be the one that is used. The last configuration source specified “wins” if a setting exists in more than one location. The ASP.NET team recommends specifying environment variables last, so that the environment where your app is running can override anything set in deployed configuration files.
You can refer here for more details on Azure App Services Application Settings and Connection Strings in ASP.NET Core
As described in this article: https://azure.microsoft.com/en-us/blog/windows-azure-web-sites-how-application-strings-and-connection-strings-work/, Azure Web Apps/Web Sites/Web Jobs can take their configuration settings (appSettings, connectionString) from environment variables instead of app.config/web.config.
For example, if an environment variable named "APPSETTING_appSettingKey" exists, it will override the following setting from app.config/web.config:
<appSettings>
<add key="appSettingKey" value="defaultValue" />
</appSettings>
This works fine once the application is deployed in Azure, but I would like to use the same method when testing locally.
I tried to emulate this in a local command line:
> set APPSETTING_appSettingKey=overridedValue
> MyWebJob.exe
The web job accesses this setting using:
ConfigurationManager.AppSettings["appSettingKey"]
When running in Azure, it reads the value "overridedValue" as expected, but locally it reads the value "defaultValue" from the app.config file.
Should I expect this to work, or is this implemented only under an Azure environment?
I could obviously create an abstraction over ConfigurationManager that emulates this, but this wouldn't work when calling code that needs a connection string name instead of a connection string value. Also, I want to use the same method regardless of the environment to simplify management of settings.
There are 3 reasons why I need this:
1) I don't like the idea of deploying to production a web.config file that references connection strings, etc for a developement environment, because there's a risk of an error that would cause the development settings (in web.config) to be used in production (production web app connecting to development database, etc), for example if an environment variable is named incorrectly (after renaming the setting in web.config but forgetting to rename it in environment variables)
2) I'm trying to setup development environments where each developer has his own isolated cloud resources (storage account, databases,...). Currently, everyone has to manually edit his .config files to reference the correct resources, and be careful when checking-in or merging changes to these files.
3) A solution can have multiple projects that need to duplicate the same settings (main web app, web jobs, integration test projects,...). This causes a lot of work to ensure updated settings are replicated across all files.
This would be simplified if there was an environment-independent .config file without any actual configuration, each developer would configure a set of environment variables once and be able to use them for all parts of a solution.
Yes, this special transformation of environment variables into config values is done via a component that is specific to Azure WebApps and won't be in play locally.
Generally people are fine with the local behavior this produces - locally you are reading from config settings as usual, but in Azure you're reading from secure settings that were configured via the App Settings portal blade (so these settings aren't in your source code).
You could write an abstraction over this if you wish, E.g. the WebJobs SDK actually does this internally (code here).
When I am developing locally and want to consistantly use Environment.GetEnvironmentVariable. In my static class Main I have the following code:
if (config.IsDevelopment)
{
config.UseDevelopmentSettings();
Environment.SetEnvironmentVariable("UseDevelopmentSettings", "true");
}
Then in my static class Functions I add a static constructor and in there I call the static method below:
static void AddAppSettingsToEnvironmentVariables()
{
String useDevelopmentSettings = Environment.GetEnvironmentVariable("UseDevelopmentSettings"); ;
if (!(String.IsNullOrEmpty(useDevelopmentSettings)))
{
foreach (String key in ConfigurationManager.AppSettings.AllKeys)
{
Environment.SetEnvironmentVariable(key, ConfigurationManager.AppSettings[key]);
}
}
}
The code is small enough that I can simply comment it out before I test in Azure.
If you want to test the application with the value that will be used in Azure portal AppSettings/Connection String. I would recommend use HostingEnvironment.IsDevelopmentEnvironment. To ensure it will work, please change the <compilation debug="true" targetFramework="4.5.2" /> to <compilation debug="false" targetFramework="4.5.2" />. set the value with the same value in Azure portal if (HostingEnvironment.IsDevelopmentEnvironment == false). I have try with a simple project, hope it helps:
public ActionResult Index()
{
if (HostingEnvironment.IsDevelopmentEnvironment == true)
{
ViewBag.Message = "Is development.";
}
else
{
ViewBag.Message = "Azure environment.";
}
return View();
}
Here is the result:
I've got an azure webjob, that has some appsettings for api keys etc.
I've also got a bunch of PRODUCTION azure app settings (specified in the portal), that should override my webjob config appsettings.. But they don't.
For my website, they work as expected, and all is fine. For the webjobs however, they just get completely ignored, and my app settings from the config are used instead.
Is this a bug in azure? All the docs suggest that this should work.
EDIT
I found this blog all about using CloudConfigurationManager.GetSetting so I've implemented that and it still won't work - still using the settings that are defined in web job's app settings file :(
Thanks
So it looks like I've found a bug!
It's because my AppSettings were referenced from another file, like this (in app.config):
<AppSettings file="appsettings.config"/>
This basically breaks azure's config management.
Use
"D:\home\site\wwwroot\" to create an absolute path to files inside your website.
inside your WebJob to point to any file inside your wwwwroot directory. Using the Absolute path with D:\home\site\wwwroot\appsettings.config should fix you website.
Did you set your settings only in the Web.config file or did you set them in the portal or using the VS "Website Settings" tooling? You need to set the WebJobs-related settings at that level, not just in the Web.config. Once you do that, restart your site, and the settings should be there.
I had a similar problem but for me it was that we set the "is_in_place" settings.job value to true. I guess when it's in place, it doesn't update the config file with the settings. We didn't have a strict requirement that it run in place, so removing that setting fixed the problem for us.
I have a similar problem with several web jobs sharing a common settings file. It would be inconvenient to duplicate the settings in each job.
Someone mentioned using configSource= instead of file= in the app.config to reference the external config settings file. I tried this, and it appears that it now works as expected. The settings are being taken from the Portal App Settings instead of the file now.
Fortunately, the settings for the web jobs all come from the external file. The use of file= allowed me to use additional per-app settings other than just those in the file, but fortunately I don't need to do that.
A workaround to this problem is to place the settings inline in the AppSettings tag of the Web.config like this.
<AppSettings>
<add key="host" value="someHost" />
</AppSettings>
Azure App Service Application Settings are persisted in the Environment Variables. Go to the Azure Portal add your application settings or connection string values. Then, you can check out the environment variables of your app service at https://sitename.scm.azurewebsites.net
If you want your webjobs to share those settings, in your webjob project, create a appsettings.json:
{
"APPSETTING_AzureWebJobsDashboard": "",
"APPSETTING_AzureWebJobsStorage": "",
"SQLAZURECONNSTR_xxx": ""
}
In your Main() method
private static void Main(string[] args)
{
var builder = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.AddEnvironmentVariables();
var configurations = builder.Build();
var services = new ServiceCollection()
.AddDbContext<ApplicationDbContext>(options =>
options.UseSqlServer(configurations["SQLAZURECONNSTR_xxx"]),
ServiceLifetime.Transient)
.BuildServiceProvider();
var host = new JobHost(new JobHostConfiguration
{
DashboardConnectionString = configurations["APPSETTING_AzureWebJobsDashboard"],
StorageConnectionString = configurations["APPSETTING_AzureWebJobsStorage"]
});
host.RunAndBlock();
}
This is simpler than trying to figure out the path to the website config on the app service