I have an app running in Azure and logging to a database. I've got the NLog settings coming from the appsettings.json file and everything seems to be working well. However, now I'd like to override the logging rules by adding an application setting to the app configuration in Azure and it's failing.
This is the NLog entry in the app settings.json:
"NLog": {
"autoReload": true,
"throwConfigExceptions": true,
//"internalLogLevel": "Info",
//"internalLogFile": "x:/internal-nlog.txt",
"extensions": [
{ "assembly": "NLog.Extensions.Logging" },
{ "assembly": "NLog.Web.AspNetCore" },
{ "assembly": "NLog.Database" }
],
"targets": {
"async": true,
"database": {
"type": "Database",
"dbProvider": "System.Data.SqlClient",
"connectionString": "<Connection String>",
"keepConnection": "true",
"commandText": "insert into LoggingMessages(Created, MessageType, Message, CallSite, ExceptionDetail) values(getutcdate(), #level, #message, #callsite, #exception);",
"parameters": [
{
"name": "#level",
"layout": "${level}"
},
{
"name": "#message",
"layout": "${message}"
},
{
"name": "#logger",
"layout": "${logger}"
},
{
"name": "#callsite",
"layout": "${callsite}"
},
{
"name": "#exception",
"layout": "${exception:tostring}"
}
]
},
"logconsole": {
"type": "Console"
}
},
"rules": [
{
"logger": "microsoft.*",
"maxLevel": "Debug",
"final": true
},
{
"logger": "*",
"minLevel": "Trace",
"writeTo": "database"
}
]
}
This line is in the Configuration method of my startup.cs:
var logger = LogManager.Setup()
.LoadConfigurationFromAppSettings()
.GetCurrentClassLogger();
And my CreateHostBuilder method in program.cs looks like this:
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseStartup<Startup>();
})
.UseNLog();
Within the App Configuration screen on the Azure Portal I've then created the following Application Setting entry:
Name
Value
NLog:rules
[{"logger": "microsoft.","maxLevel": "Debug","final": true},{"logger": "","minLevel": "Warning","writeTo": "database"}]
I'm thinking that either the call to LoadConfigurationFromAppSettings() either happens too early or it ignores the updated values, but of course, I could just be completely wrong in what I'm trying to do.
Any help would be great
Maybe add an Application Setting to the Azure App Configuration:
Name
Value
LoggingLevel
Debug
And use ${configsetting:LoggingLevel} in the NLog Logging Rule:
"NLog": {
"autoReload": true,
"throwConfigExceptions": true,
"extensions": [
{ "assembly": "NLog.Extensions.Logging" },
{ "assembly": "NLog.Web.AspNetCore" },
{ "assembly": "NLog.Database" }
],
"targets": {
"async": true,
"database": {
"type": "Database",
"dbProvider": "System.Data.SqlClient",
"connectionString": "<Connection String>",
"keepConnection": "true",
"commandText": "insert into LoggingMessages(Created, MessageType, Message, CallSite, ExceptionDetail) values(getutcdate(), #level, #message, #callsite, #exception);",
"parameters": [
{
"name": "#level",
"layout": "${level}"
},
{
"name": "#message",
"layout": "${message}"
},
{
"name": "#logger",
"layout": "${logger}"
},
{
"name": "#callsite",
"layout": "${callsite}"
},
{
"name": "#exception",
"layout": "${exception:tostring}"
}
]
},
"logconsole": {
"type": "Console"
}
},
"rules": [
{
"logger": "microsoft.*",
"maxLevel": "Debug",
"final": true
},
{
"logger": "*",
"minLevel": "${configsetting:LoggingLevel:whenEmpty=Trace}",
"writeTo": "database"
}
]
}
See also: https://github.com/NLog/NLog/wiki/Filtering-log-messages#semi-dynamic-routing-rules
See also: https://github.com/NLog/NLog/wiki/ConfigSetting-Layout-Renderer
The key-value ("NLog:rules") you created in Azure App Configuration is a JSON object. Make sure you set JSON content-type for it. Otherwise, it will be loaded as a string. For more information, see Use content type to store JSON key-values in App Configuration.
I don't see your code that loads data from Azure App Configuration.
Update your code to use App Configuation by following Quickstart: Create an ASP.NET Core app with Azure App Configuration.
I'm not very familiar with NLog, but you need to make sure NLog can load configuration from the system IConfiguration.
So I couldn't get this to work the way I thought it should - just replacing the current rules - but I got it to work in the way that I need it. All I really wanted to do was change the minimum level for the database rule so that I could adjust this without having to update the code.
What I ended up doing was adding an Application Setting in the Azure App Configuration:
|Name|Value|
|----|-----|
|LoggingLevel|Debug|
then adding some code similar to the following to the Configuration method of the startup.cs after the existing LogManager setup line.
var loggingLevel = Configuration.GetValue<string>("LoggingLevel");
var haveNewLoggingLevel = !string.IsNullOrWhiteSpace(loggingLevel);
// only interested in the 'database' target
var databaseTarget = LogManager.Configuration.FindTargetByName("database") as DatabaseTarget;
if ((databaseTarget != null) && haveNewLoggingLevel)
{
var rules = LogManager.Configuration.LoggingRules;
foreach (var rule in rules)
{
if (rule.Targets.Contains(databaseTarget))
{
rule.SetLoggingLevels(NLog.LogLevel.FromString(loggingLevel), NLog.LogLevel.Fatal);
}
}
LogManager.ReconfigExistingLoggers();
}
With this I was able to achieve what I needed, butt I'm not sure it answers the question I originally asked. I also added similar code to update the connection string to the database from App Settings.
To override a nested value inside the appsettings.json using Azure App Configuration then one must use __.
For example ApplicationInsights__InstrumentationKey will override this value in appsettings.json
"ApplicationInsights": {
"InstrumentationKey": "Will be overrridden"
}
See also: https://learn.microsoft.com/en-us/azure/app-service/configure-common?tabs=portal
Notice to override nested value inside json-array requires extra magic (Ex. a specific NLog-Rule). You need to decorate with json-array-index-name. See also: https://github.com/NLog/NLog.Extensions.Logging/wiki/NLog-configuration-with-appsettings.json#logging-rule-override
Related
I have created azure logic app single tenant project using visual studio code by following this documentation. And then created workflow based on my requirement, this contains the data factory pipeline and send grid actions.
The workflow contains the hardcoded values in the create a pipeline run data factory action.
"Create_a_pipeline_run": {
"inputs": {
"host": {
"connection": {
"referenceName": "azuredatafactory_5"
}
},
"method": "post",
"path": "/subscriptions/#{encodeURIComponent('xxxxxxx-xxxx-xxxx-xxxx-xxxxxx')}/resourcegroups/#{encodeURIComponent('xxxxx')}/providers/Microsoft.DataFactory/factories/#{encodeURIComponent('xxxxxx')}/pipelines/#{encodeURIComponent('xxxxxxx')}/CreateRun",
"queries": {
"x-ms-api-version": "2017-09-01-preview"
}
},
"runAfter": {},
"type": "ApiConnection"
},
And the connections.json file looks file below:
"managedApiConnections": {
"sendgrid": {
"api": {
"id": "/subscriptions/#appsetting('WORKFLOWS_SUBSCRIPTION_ID')/providers/Microsoft.Web/locations/centralus/managedApis/sendgrid"
},
"authentication": {
"type": "ManagedServiceIdentity"
}
},
"azuredatafactory_5": {
"api": {
"id": "/subscriptions/#appsetting('WORKFLOWS_SUBSCRIPTION_ID')/providers/Microsoft.Web/locations/centralus/managedApis/azuredatafactory"
},
"authentication": {
"type": "ManagedServiceIdentity"
}
}
}
The above managed API connections refers the existing API connections from azure. But I want to create the new managed API connections per environment (means parameterize the values in the connections.json file based on the environment).
Can anyone suggest me how to parameterize the values in workflow.json files per environment and parameterize the values in connections.json file per environment.
A logic app standard is just an app service of kind workflowApp.
You can heavily make use of appsettings here.
Logic app parameters.
In your workflow.json, you can use parameters like that:
"Create_a_pipeline_run": {
"inputs": {
"host": {
"connection": {
"referenceName": "azuredatafactory_5"
}
},
"method": "post",
"path": "/subscriptions/#{encodeURIComponent(parameters('subscription_id'))}/resourcegroups/...",
"queries": {
"x-ms-api-version": "2017-09-01-preview"
}
},
"runAfter": {},
"type": "ApiConnection"
}
Then in your parameters.json file, reference app settings like that:
{
"subscription_id": {
"type": "String",
"value": "#appsetting('WORKFLOWS_SUBSCRIPTION_ID')"
}
}
subscription_id must be defined as an app setting in the app service.
Logic app connections.
In the same way you can use app settings and parameters for connection information in your connections.json file:
{
"managedApiConnections": {
"sendgrid": {
"api": {
"id": "/subscriptions/#appsetting('WORKFLOWS_SUBSCRIPTION_ID')/providers/Microsoft.Web/locations/centralus/managedApis/sendgrid"
},
...
"authentication": "#parameters('azure_authentication')"
}
}
}
then in your parameters.json file:
{
"azure_authentication": {
"type": "object",
"value": {
"type": "ManagedServiceIdentity"
}
}
...
}
This way you can easily offload all environment specific parameters to app settings
i am using azure functions written in nodejs.
I have this azure function
module.exports = async function (context, req) {
const title = req.body.title;
console.log('title', title);
if (title) {
req.body.completed = false;
context.bindings.outputDocument = req.body;
context.res = {
body: { success: true, message: `Todo added successfully` }
};
} else {
context.res = {
status: 400,
body: { success: false, message: `Please add title field` }
};
}
};
and the binding
{
"bindings": [
{
"authLevel": "anonymous",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"post"
]
},
{
"type": "http",
"direction": "out",
"name": "res"
},
{
"type": "cosmosDB",
"name": "outputDocument",
"databaseName": "xxx",
"collectionName": "items",
"createIfNotExists": false,
"connectionStringSetting": "xxx",
"partitionKey": "/all",
"direction": "out"
}
],
"disabled": false
}
locally when i run
func host start
i don't see my console log statement.
How can i see on localhost ?
And where i need to go in my azure portal to see the logs there after deployment.
I tried to go in Function App - open my function and there the azure function but i can't see any logs there...
How can i see on localhost ?
And where i need to go in my azure portal to see the logs there after
deployment.
I tried to go in Function App - open my function and there the azure
function but i can't see any logs there...
Of course, you can get the log files on local. I assume you are based on windows, then just edit the host.json like this:
{
"version": "2.0",
"logging": {
"fileLoggingMode": "always",
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[1.*, 2.0.0)"
}
}
And go to
C:\Users\yourusernameonlocal\AppData\Local\Temp\LogFiles\Application\Functions\Function\yourfunctionname
After that you will see the log files.:)
This should probably resolve your issue.
Azure Functions JavaScript developer guide:
Don't use console.log to write trace outputs. Because output from console.log is captured at the function app level, it's not tied to a specific function invocation and isn't displayed in a specific function's logs.
I have an issue with serilog when i want to write to a database based on different environments.
Serilog override my appsettings.production.json and always take the settings in appsettings.json instead even in a production mode !
My code in my local appsettings.json
"Serilog": {
"Using": [ "Serilog.Sinks.MSSqlServer" ],
"MinimumLevel": {
"Default": "Debug",
"Override": {
"Microsoft": "Warning",
"System": "Warning"
}
},
"WriteTo": {
"Console" : {"Name": "Console"},
"Sql": {
"Name": "MSSqlServer",
"Args": {
"connectionString": "Server=MyServerInDev\\SQLSERVER;Database=MyDBDev;user id=ui;password=pw_;ConnectRetryCount=0;MultipleActiveResultSets=true",
"tableName": "Log",
"autoCreateSqlTable": true
}
}
},
appsettings.production.json
"Serilog": {
"Using": [ "Serilog.Sinks.MSSqlServer" ],
"MinimumLevel": {
"Default": "Warning",
"Override": {
"Microsoft": "Warning",
"System": "Warning"
}
},
"WriteTo": {
"Console" : {"Name": "Console"},
"Sql": {
"Name": "MSSqlServer",
"Args": {
"connectionString": "Server=MyServerInProd\\SQLSERVER;Database=MyDBProdv;user id=ui;password=pw_;ConnectRetryCount=0;MultipleActiveResultSets=true",
"tableName": "Log",
"autoCreateSqlTable": true
}
}
},
My program.cs
var configuration = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.AddJsonFile($"appsettings.{Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? "Production"}.json", optional: true)
.AddEnvironmentVariables()
.Build();
Log.Logger = new LoggerConfiguration()
.ReadFrom.Configuration(configuration)
.CreateLogger();
I am missing something ? and why serilog take the appsettings.json settings and not appsettings.production.json even in prod mode
Note that the other settings in the appsettings.production.json work fine. Only the settings in serilog is causing the problem
Found it. The reason is that Serilog adds sink configs to array, and each of two files of appsettings just adds another sinks instead of replacing them. Solution is explicitly indicate array indexes for each sink. Credits for answer: nblumhardt. Tested, works.
"WriteTo": {
"0": {
"Name": "File",
"Args": {
"path": "./logs/myapp-.txt",
"rollingInterval": "Day"
}
}
},
I'm currently implementing Azure Application Insights logging with Serilog which is working fine except for when I use an output template in my Serilog configuration. It seems like the template is ignored when passing the Serilog data to Application insights.
My serilog config in appsetting.json:
"Serilog": {
"Using": [ "Serilog.Sinks.ApplicationInsights" ],
"MinimumLevel": "Debug",
"WriteTo": [
{ "Name": "Console" },
{
"Name": "RollingFile",
"Args": {
"pathFormat": "logs\\log-{Date}.txt",
"outputTemplate": "{Timestamp:yyyy-MM-dd HH:mm:ss.fff zzz} [{Level}] {Message}{NewLine}{Exception}"
}
},
{
"Name": "ApplicationInsights",
"Args": {
"restrictedToMinimumLevel": "Information",
"telemetryConverter": "Serilog.Sinks.ApplicationInsights.Sinks.ApplicationInsights.TelemetryConverters.TraceTelemetryConverter, Serilog.Sinks.ApplicationInsights",
"outputTemplate": "Test Template - {Message}"
}
}
],
"Enrich": [ "FromLogContext", "WithMachineName", "WithThreadId" ],
"Properties": {
"Application": "app"
}
},
The logging statement:
logger.Error("Test Serilog Error For AI - " + DateTime.Now);
The output within application insights:
Is this the correct approach to customising an error message for Application insights?
After checking the source code serilog-sinks-applicationinsights, you will find it did not read the outputTemplate from appsetting.json.
For a workaround, you may implement custom TemplateTraceTelemetryConverter.
TemplateTraceTelemetryConverter
public class TemplateTraceTelemetryConverter : TraceTelemetryConverter
{
public override IEnumerable<ITelemetry> Convert(LogEvent logEvent, IFormatProvider formatProvider)
{
var templateParser = new MessageTemplateParser();
var template = templateParser.Parse($"Test Template - {logEvent.MessageTemplate.Text}");
LogEvent newLogEvent = new LogEvent(logEvent.Timestamp
, logEvent.Level
, logEvent.Exception
, template
, logEvent.Properties.Select(p => new LogEventProperty(p.Key, p.Value)));
return base.Convert(newLogEvent, formatProvider);
}
}
Use TemplateTraceTelemetryConverter
"Serilog": {
"Using": [
"Serilog.Sinks.ApplicationInsights",
],
"MinimumLevel": "Debug",
"WriteTo": [
{
"Name": "Console",
"Args": {
"outputTemplate": "Test Template - {Message}"
}
},
{
"Name": "RollingFile",
"Args": {
"pathFormat": "logs\\log-{Date}.txt",
"outputTemplate": "{Timestamp:yyyy-MM-dd HH:mm:ss.fff zzz} [{Level}] {Message}{NewLine}{Exception}"
}
},
{
"Name": "ApplicationInsights",
"Args": {
"restrictedToMinimumLevel": "Error",
"telemetryConverter": "YourProjectNamespace.TemplateTraceTelemetryConverter, YourProjectNamespace"
//"outputTemplate": "Test Template - {Message}"
}
}
],
"Enrich": [ "FromLogContext", "WithMachineName", "WithThreadId" ],
"Properties": {
"Application": "app"
}
}
I've created a simple Azure Function with the intension of using Serililog for logging to Azure Blob Storage.
When using inline configuration for the Serilog sink, that works perfectly fine, the sink is created and Serilog happily wirtes to Blob storage.
This works:
Log.Logger = new LoggerConfiguration()
.WriteTo.AzureBlobStorage(
"STORAGEACCOUNT CONNECTION STRING", // <-- inline config
LogEventLevel.Information,
null,
"{yyyy}/{MM}/{dd}/MyLogFile.txt")
.CreateLogger();
The problem is that I would like to configure this all via appsettings.json, but when I try this (both locally running in the emulator & in the cloud), it fails to bind the sink settings, and as a result its failing to log to to the Blog storage.
If I debug, I can see that the config values are being loaded as expected into the configuration object, but they are not being applied to the logging configuration.
This doesnt work:
Log.Logger = new LoggerConfiguration()
.ReadFrom.Configuration(configuration)
.CreateLogger();
appsettings.json in this snippet:
{
"Serilog": {
"MinimumLevel": {
"Default": "Information",
"Override": {
"Microsoft": "Information"
}
},
"WriteTo": [{
"Name": "AzureBlobStorage",
"Args": {
"connectionString": " --- REPLACE WITH STORAGEACCOUNT BLOB CONNECTION STRING --- ",
"formatter": "Serilog.Formatting.Compact.RenderedCompactJsonFormatter, Serilog.Formatting.Compact",
"storageFileName": "{yyyy}/{MM}/{dd}/MyLogFile.txt",
"retainedFileCountLimit": 31
}
}
],
"Properties": {
"Application": "int-test-logging",
"Environment": "int"
}
}
}
I'm not sure what I'm doing incorrectly but any help would be appreciated. The following github repo contains code to implement the above (both approaches), and reproduces this behaviour. https://github.com/oneiltomlinson/AzureFunctionsLogging
Just add
"Using": [ "Serilog.Sinks.AzureBlobStorage" ] //under Serilog Config
and it will work!
Working Config
{
"Serilog": {
"Using": [
"Serilog.Sinks.AzureBlobStorage"
],
"MinimumLevel": {
"Default": "Information",
"Override": {
"Microsoft": "Information"
}
},
"WriteTo": [
{
"Name": "AzureBlobStorage",
"Args": {
"connectionString": " --- REPLACE WITH STORAGEACCOUNT BLOB CONNECTION STRING --- ",
"formatter": "Serilog.Formatting.Compact.RenderedCompactJsonFormatter, Serilog.Formatting.Compact",
"storageFileName": "{yyyy}/{MM}/{dd}/MyLogFile.txt",
"retainedFileCountLimit": 31
}
}
],
"Properties": {
"Application": "int-test-logging",
"Environment": "int"
}
}
}
I've the same issue and even with all the required "using" of the used sinks, nothing change. This is my configuration file (just about the Serilog part):
"Serilog": {
"Using": [
"Serilog.Sinks.RollingFile",
"Serilog.Sinks.AzureTableStorage"
],
"MinimumLevel": "Information",
"Override": {
"Microsoft": "Warning",
"Microsoft.EntityFrameworkCore": "Information"
},
"WriteTo": [
{
"Name": "Logger",
"Args": {
"configureLogger": {
"MinimumLevel": "Information",
"WriteTo": [
{
"Name": "Console",
"outputTemplate": "[{Timestamp:yyyy-MM-dd HH:mm:ss.fff}] [{Level:u3}] [TID:{ThreadId}] - {Message:lj}{NewLine}{Exception}"
},
{
"Name": "Debug",
"outputTemplate": "[{Timestamp:yyyy-MM-dd HH:mm:ss.fff}] [{Level:u3}] [TID:{ThreadId}] - {Message:lj}{NewLine}{Exception}"
},
{
"Name": "AzureTableStorage",
"Args": {
"storageTableName": "HeroFunctionsLogs",
"connectionString": "DefaultEndpointsProtocol=https;AccountName=herobugari;AccountKey=NYJcdoCg9mQxqbQaTUdIxOHYQGdcKAwMsuWwzDub29UweHR1c+wd6Sh3IDQNB+eCx3DAe/ccobxu67sJvqQB5g==",
"outputTemplate": "[{Timestamp:yyyy-MM-dd HH:mm:ss.fff}] [{Level:u3}] [TID:{ThreadId}] - {Message:lj}{NewLine}{Exception}",
"restrictedToMinimumLevel": "Information",
"writeInBatches": true,
"batchPostingLimit": 100
}
},
{
"Name": "RollingFile",
"Args": {
"pathFormat": "logs\\Bulgari.Hero.Functions.log",
"retainedFileCountLimit": 10,
"buffered": true,
"flushToDiskInterval": 5,
"outputTemplate": "[{Timestamp:yyyy-MM-dd HH:mm:ss.fff}] [{Level:u3}] [TID:{ThreadId}] - {Message:lj}{NewLine}{Exception}"
}
}
],
"Filter": [
{
"Name": "ByExcludingOnly",
"Args": {
"expression": "SourceContext like '%Microsoft.EntityFrameworkCore%'"
}
}
]
}
}
}
]
}
and this the Startup.cs code that reads and try to use it:
var config = new ConfigurationBuilder()
.SetBasePath(Environment.CurrentDirectory)
.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
.AddEnvironmentVariables().Build();
// Registering Serilog provider
logger = new LoggerConfiguration()
//.WriteTo.Console()
//.WriteTo.File("logs\\herofunctions.log", rollingInterval: RollingInterval.Day)
.ReadFrom.Configuration(config)
.CreateLogger();
builder.Services.AddLogging(lb => lb.AddSerilog(logger));
The logger object has no "sinks" registered as it happens in the question.
I have solved exactly the same issue here by adding entries in local.settings.json.
"Serilog:Using:0": "Serilog.Sinks.Console",
"Serilog:MinimumLevel": "Information",
"Serilog:Override:Microsoft": "Warning",
"Serilog:Override:System": "Warning",
"Serilog:WriteTo:Console:Name": "Console"
then in StartUp.cs:
var provider = builder.Services.BuildServiceProvider();
var config = provider.GetRequiredService<IConfiguration>();
Log.Logger = new LoggerConfiguration().ReadFrom.Configuration(config).CreateLogger();
then the function:
public class TestFunction
{
private readonly ILogger _logger;
public TestFunction()
{
_logger = Log.Logger;
}
[FunctionName("TestFunction")]
public async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)] HttpRequest req)
{
_logger.Information("Hello");
return new OkResult();
}
}
}
I can see the config entries registered and entries are logged!
To write to SQL Server I had to add:
"Using": [
"Serilog.Sinks.MSSqlServer"
],