Serilog: azure webjob logging doesn't seem to work when hosted in azure? - azure

I have azure webjob sdk (v3.0.3) app which has been configured to use serilog for logging.
The log seems to work when I run the app locally in my system. Below is the configuration:
static void Main(string[] args)
{
try
{
var builder = new HostBuilder()
.ConfigureAppConfiguration(SetupConfiguration)
.ConfigureLogging(SetupLogging)
.ConfigureServices(SetupServices)
.ConfigureWebJobs(webJobConfiguration =>
{
webJobConfiguration.AddTimers();
webJobConfiguration.AddAzureStorageCoreServices(); //this is to store logs in azure storage
})
.UseSerilog()
.Build();
builder.Run();
}
}
The code for SetupConfiguration is below:
private static void SetupConfiguration(HostBuilderContext hostingContext, IConfigurationBuilder builder)
{
var env = hostingContext.HostingEnvironment;
_configuration = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: false, reloadOnChange: true)
.AddEnvironmentVariables()
.Build();
}
The code for setting up services:
private static void SetupServices(HostBuilderContext hostingContext, IServiceCollection serviceCollection)
{
serviceCollection.AddSingleton<IConfiguration>(_configuration);
Log.Logger = new LoggerConfiguration()
.ReadFrom.Configuration(_configuration)
.CreateLogger();
_logger = serviceCollection.BuildServiceProvider().GetRequiredService<ILoggerFactory>().CreateLogger("test");
}
The logging is setup as following:
private static void SetupLogging(HostBuilderContext hostingContext, ILoggingBuilder loggingBuilder)
{
loggingBuilder.SetMinimumLevel(LogLevel.Information);
loggingBuilder.AddConsole();
loggingBuilder.AddDebug();
loggingBuilder.AddSerilog(dispose: true);
}
In my TimerTrigger method I'm using the logger:
[Singleton]
public async static Task Trigger([TimerTrigger("%Job%")]TimerInfo myTimer)
{
_logger.LogInformation($"From Trigger {DateTime.UtcNow.ToString()}");
}
In appSettings.json, serilog is configured as follows:
"Serilog": {
"MinimumLevel": "Information",
"WriteTo": [
{
"Name": "RollingFile",
"Args": {
"pathFormat": ".\\Log\\log-{Date}.txt",
"retainedFileCountLimit": 7,
"fileSizeLimitBytes": 5000000,
"outputTemplate": "{Timestamp:yyyy-MM-dd HH:mm:ss} {EventId} [{Level}] [{Properties}] {Message}{NewLine}{Exception}"
}
}
]
}
the folder "Log" and the log files get created when i run the app locally. But when I publish the webjob, the "Log" folder or the log file is not created in the "app_data" folder of webjob. Can anyone help me figureout how to configure serilog to make it work with webjobs?
Following are the nuget packages used:

If you want to use serilog in WebJob , you need to install this package Serilog.Extensions.WebJobs. Then after configuring the serilog, you would be able to use it.
You must inject the ILogger rather than using the global Log.Logger otherwise the log messages will not be written to the Microsoft Azure WebJobs dashboard.
About the detailed description about how to configure and use serilog, you could refer to this doc.
Hope this could help you, if you still have other questions, please let me know.

Related

Azure app insights TelemetryConfiguration not working in Startup.cs file of azure function

my startup file looks like this
[assembly: FunctionsStartup(typeof(Startup))]
{
public override void Configure(IFunctionsHostBuilder builder)
{
var context = builder.GetContext();
var _config = context.Configuration;
var telemetryConfig = new TelemetryConfiguration();
telemetryConfig.ConnectionString = _config.GetConnectionString("ApplicationInsights");
builder.Services.AddSingleton<TelemetryClient>(x => new TelemetryClient(telemetryConfig));
}
}
and the settings file looks like this
{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "UseDevelopmentStorage=true",
"FUNCTIONS_WORKER_RUNTIME": "dotnet",
"APPINSIGHTS_INSTRUMENTATIONKEY": "xxxxxx-fbc9-441b-9869-70bcb4afc93a",
"TimerInterval": "0 */5 * * * *"
},
"ConnectionStrings": {
"ApplicationInsights": "InstrumentationKey=xxxxxx-fbc9-441b-9869-
70bcb4afc93a;IngestionEndpoint=https://xxx-
in.applicationinsights.azure.com/;LiveEndpoint=https://xxxxxx.livediagnostics.monitor.azure.c
om/" }
}
problem here is that telemetryConfig is set properly i can see the InstrumentationKey and ConnString
but new TelemetryClient(telemetryConfig) has InstrumentationKey empty and most of the properties set to null.
even if i set InstrumentationKey directly (which is obsolete) the dependency injected value in the code has empty InstrumentationKey
in code i am using it like this
private readonly TelemetryClient _telemetry;
public FHIRExtract(ILogger<FHIRExtract> logger, TelemetryClient telemetry, IConfiguration configuration)
{
_logger = logger;
_config = configuration;
_telemetry = telemetry;
}
[FunctionName("FHIRExtract")]
public async Task Run([TimerTrigger("%TimerInterval%"
_telemetry has InstrumentationKey empty
but when i set this value in the code
//_telemetry.InstrumentationKey = "xxxxxx-fbc9-441b-9869-70bcb4afc93a";
it works and i am able to send telemetry data
I am trying to use non obsolete functions to configure Telemetry client and want to use it by DI.
what am i doing wrong ?
You shouldn't setup Application Insights this way.
Install the Microsoft.Azure.WebJobs.Logging.ApplicationInsights NuGet package in the azure functions project
In the Configure override add the logging:
builder.Services.AddLogging();
Configure the instrumentatio key by setting APPINSIGHTS_INSTRUMENTATIONKEY in the settings.json file
Inject a TelemetryConfiguration in your function and create an instance of TelemetryClient:
public FHIRExtract(ILogger<FHIRExtract> logger, TelemetryConfiguration
telemetryConfig, IConfiguration configuration)
{
_logger = logger;
_config = configuration;
_telemetry = new TelemetryClient(telemetryConfig);
}
For some reason the custom telemetry logging in azure functions breaks when only setting APPLICATIONINSIGHTS_CONNECTION_STRING in the configuration but I am sure that will be fixed in the near future. In my experience the application insights integration always lags behind a bit.
References:
Offical docs
Sample repository demoing various Application Insights techniques

Azure Function not binding to appsettings when deployed

I have an Azure Function app written in C# with Visual Studio using version 3.0.9 of Functions SDK. In the same directory as the csproj file and the host.json, I have an appsettings.json file with the following content:
{
"test-queue": "test,
"myOptions": {
"batchSize": 5000
}
}
The function works fine when I run it locally but for some reason it doesn't seem to bind to the appsettings file when it is deployed to Azure. I use the following startup class:
public class Startup : FunctionsStartup
{
public override void ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder)
{
builder.ConfigurationBuilder
.SetBasePath(Environment.CurrentDirectory)
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.AddEnvironmentVariables();
var builtConfig = builder.ConfigurationBuilder.Build();
var keyVaultName = builtConfig["AzureKeyVaultName"];
if (!string.IsNullOrEmpty(keyVaultName))
{
var azureServiceTokenProvider = new AzureServiceTokenProvider();
var keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
builder.ConfigurationBuilder.AddAzureKeyVault($"https://{keyVaultName}.vault.azure.net/");
}
builder.ConfigurationBuilder
.AddJsonFile("local.settings.json", true)
.AddEnvironmentVariables()
.Build();
}
public override void Configure(IFunctionsHostBuilder builder)
{
FunctionsHostBuilderContext context = builder.GetContext();
builder.Services.AddOptions<MyOptions>()
.Configure<IConfiguration>((settings, configuration) =>
{
configuration.GetSection("myOptions").Bind(settings);
});
}
}
I have a function class like this:
public class MyFunction
{
private readonly MyOptions options;
public MyFunction(IOptions<MyOptions> options)
{
this.options = options.Value;
}
[FunctionName("Test")]
public async Task Run(
[QueueTrigger("%test-queue%")] MyParameters parameters,
ILogger log)
{
log.LogInformation($"Batch size: {options.BatchSize}");
}
}
The trouble is, the function doesn't seem to be using the appsettings file. I get an InvalidOperationException on startup saying '%test-queue%' does not resolve to a value." I can get rid of the queue binding and hard code the queue name. That makes the funtion run Okay but then the log file says the batch size is 0 instead of 5000.
Again, it works when running locally but not when it is deployed. The optional flag is false when registering the appsettings file in the startup so it must be getting found.
I've got a partial solution by replacing
builder.ConfigurationBuilder
.SetBasePath(Environment.CurrentDirectory)
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.AddEnvironmentVariables();
with
FunctionsHostBuilderContext context = builder.GetContext();
builder.ConfigurationBuilder
.AddJsonFile(Path.Combine(context.ApplicationRootPath, "appsettings.json"), optional: false, reloadOnChange: false)
.AddEnvironmentVariables();
It is still making me hard code the queue names but batchSize variable is getting set at least.

Where in Azure Portal are the logger.Log statements?

I have a .NET Core WebAPI app. The app is deployed on Azure as an App Service.
In the code, I have enabled Application Insights like so
public static IWebHost BuildWebHost(string[] args) =>
WebHost
.CreateDefaultBuilder(args)
.UseApplicationInsights()
.UseStartup<Startup>()
.ConfigureLogging((hostingContext, logging) =>
{
logging.AddConfiguration(hostingContext.Configuration.GetSection("Logging")).SetMinimumLevel(Microsoft.Extensions.Logging.LogLevel.Error);
logging.AddApplicationInsights(" xxxxx-xxxx-xxxx-xxxx--xxxxxxxxxxx").SetMinimumLevel(LogLevel.Trace);
})
.Build();
In the constructor of a controller and inside an method of a controller I have these logging statements.
_logger.LogInformation("ApiController, information");
_logger.LogWarning("ApiController, warning");
_logger.LogCritical("ApiController, critical");
_logger.LogWarning("ApiController, error");
_logger.LogDebug("ApiController, debug");
In the Azure Portal I have Application Insights for my App Service enabled. Here a picture from the portal.
App Insights in Azure Portal
But where do I see the logging statements in the Azure portal?
When I go to Application Insights -> Logs and I query by
search *
I can see the requests made to the API but not the logging statements.
Application Insights Log
Where are the log statements?
First, it is not good practice to configure the log level in code. You can easily configure the log level in the appsettings.json file. So in Program.cs -> public static IWebHost BuildWebHost method, change the code to below:
public static IWebHost BuildWebHost(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.UseApplicationInsights()
.UseStartup<Startup>()
.Build();
Then in appsettings.json(also right click the file -> properties -> set "copy to output directory" to "Copy if newer"):
{
"Logging": {
"IncludeScopes": false,
"ApplicationInsights": {
"LogLevel": {
"Default": "Trace"
}
},
"Console": {
"LogLevel": {
"Default": "Warning"
}
}
},
"ApplicationInsights": {
"InstrumentationKey": "the key"
}
}
In Controller, like ValuesController:
public class ValuesController : Controller
{
private readonly ILogger _logger;
public ValuesController(ILoggerFactory loggerFactory)
{
_logger = loggerFactory.CreateLogger<ValuesController>();
}
// GET api/values
[HttpGet]
public IEnumerable<string> Get()
{
_logger.LogInformation("ApiController, information");
_logger.LogWarning("ApiController, warning");
_logger.LogCritical("ApiController, critical");
_logger.LogWarning("ApiController, error");
_logger.LogDebug("ApiController, debug");
return new string[] { "value1", "value2" };
}
}
Run the project, and wait for a few minutes(application insights would always take 3 to 5 minutes or more to display the data). Then nave to azure portal -> application insights logs, remember that all the logs written by ILogger are stored in "traces" table. Just write the query like "traces" and specify a proper time range, you should see all the logs like below:

ASP.Net Core 2.1 Serilog SQL Server Sink using Azure SQL is working locally but not from Azure app service

I have an ASP.Net Core 2.1 Website that uses Azure SQL Database for the Microsoft Identity component.
I added a Logs table to that database and added Serilog to my website with the SQL Server Sink.
When I run the website locally, while still connected to the Azure SQL database, I can see my log entries in the Logs table just fine. However, when I deploy the website to my Azure App Service, I no longer get any log entries in the Logs table of the database.
Mind you, in the deployed version, I am connecting to and using the Azure SQL database for my MS Identity stuff just fine and I can create new users and edit existing users just fine. So I know the Connection String in my App Service Application Settings is correct.
I have reviewed the Serilog MSSQL Github to compare their configuration recommendations to my own and could not find anything that stood out.
I have this setup working correctly on an ASP.Net Core API that I deploy to another Azure App Service. That service uses a different database but it is on the same SQL Server resource.
I have reviewed the list of SO posts recommended when I started this question with no luck.
I ran the following SQL on the database when I first set up the user account;
EXEC sp_addrolemember N'db_datareader', N'myuser'
EXEC sp_addrolemember N'db_datawriter', N'myuser'
EXEC sp_addrolemember N'db_ddladmin', N'myuser'
And, as I stated, the user account can update and add user data in the AspNetUsers table just fine. So, it doesn't seem like a user account issue.
I have verified that the connection string in my Azure app service DEV deployment slot (the one I am testing), Application Settings, Connection Strings is the exact same as what I have in my local DEV UserSecrets. Plus, again, I can read/write to the AspNet* tables in that same database when deployed to Azure.
Here is my Program.cs class where I set up Serilog;
public class Program
{
public static IConfiguration Configuration { get; } = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.AddJsonFile($"appsettings.{Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? "Production"}.json", optional: true)
.AddUserSecrets<Startup>()
.AddEnvironmentVariables()
.Build();
public static void Main(string[] args)
{
var connectionString = Configuration.GetConnectionString("MyConnectionString");
const string tableName = "Logs";
Log.Logger = new LoggerConfiguration()
.MinimumLevel.Information()
.MinimumLevel.Override("Microsoft", LogEventLevel.Warning)
.Enrich.FromLogContext()
.Enrich.WithMachineName()
.Enrich.WithThreadId()
.WriteTo.MSSqlServer(connectionString, tableName)
.CreateLogger();
// TODO Enable to debug any startup Serilog issues. Make sure to comment out for PROD
//Serilog.Debugging.SelfLog.Enable(msg =>
//{
// Debug.Print(msg);
// Debugger.Break();
//});
try
{
Log.Information("Starting Application");
CreateWebHostBuilder(args).Build().Run();
}
catch (Exception ex)
{
Log.Fatal(ex, "Host terminated unexpectedly");
}
finally
{
Log.CloseAndFlush();
}
}
public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.UseStartup<Startup>()
.UseSerilog();
}
}
The only difference between the API that I have deployed in Azure that is writing logs to Azure SQL and this website is that in the API, which is older, I have
public static IWebHost BuildWebHost(string[] args)
in program.cs whereas the newer website has
public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
So... any ideas would be greatly appreciated.
[UPDATE 1/23/19]
I added the connection string directly to the
var connectionString
in Program.cs rather than getting it from
Configuration.GetConnectionString("MyConnectionString")
and it started logging to the database.
So it seems the issue is with Program.cs being able to read the connection string from Azure App Service deployment slot, Application Settings, Connection Strings section.
This connection string is being correctly read from Startup.cs and has worked since I first created the website.
So, is there some known issue with Azure not being able to read values from deployment slot Application Settings / Connection Strings from Program.cs?
Since there seems to be an issue with Azure, that it doesn't provide the application settings to the web app until CreateWebHostBuilder is invoked, a straightforward workaround (assuming that hardcoding the connection string in the source code isn't a viable option) would be to configure Serilog to use SqlServer in Startup.cs instead of Program.cs.
If it's important to log the initial events that occur during the app's starting up, Serilog can temporarily be configured in Program.cs to write to a file.
Program.cs:
public class Program
{
public static void Main(string[] args)
{
var AzureLogFilePath = #"D:\home\LogFiles\Application\log.txt";
Log.Logger = new LoggerConfiguration()
.MinimumLevel.Information()
.Enrich.FromLogContext()
.WriteTo.File(path: AzureLogFilePath, fileSizeLimitBytes: 1_000_000, rollOnFileSizeLimit: true, shared: true);
.CreateLogger();
try
{
Log.Information("Starting Application");
CreateWebHostBuilder(args).Build().Run();
}
catch (Exception ex)
{
Log.Fatal(ex, "Host terminated unexpectedly");
}
finally
{
Log.CloseAndFlush();
}
}
public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.UseStartup<Startup>()
.UseSerilog();
}
Startup.cs:
public class Startup
{
public Startup(IHostingEnvironment env)
{
var builder = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.AddJsonFile($"appsettings.{env.EnvironmentName ?? "Production"}.json", optional: true)
.AddEnvironmentVariables();
if (env.IsDevelopment()) builder.AddUserSecrets<Startup>(); // according to the docs, AddUserSecrets should be used only in development
Configuration = builder.Build();
}
public IConfiguration Configuration { get; }
public void ConfigureServices(IServiceCollection services)
{
var connectionString = Configuration.GetConnectionString("MyConnectionString");
const string tableName = "Logs";
Log.Logger = new LoggerConfiguration()
.MinimumLevel.Information()
.MinimumLevel.Override("Microsoft", LogEventLevel.Warning)
.Enrich.FromLogContext()
.Enrich.WithMachineName()
.Enrich.WithThreadId()
.WriteTo.MSSqlServer(connectionString, tableName)
.CreateLogger();
//...
}
}

Enable file logging on IIS for serilog

I want to enable file logging on IIS to resolve an issue locally on IIS. Following is the code I am using. It works when I run on visual studio and logs to a file but not when I deployed to IIS. Folder has enough permission to create a file and also I created the folder and file. What am I missing here ?
Program.cs
var path = #"C:\workspace\Logs\Log-{Date}.txt";
Log.Logger = new LoggerConfiguration()
.MinimumLevel.Verbose()
.MinimumLevel.Override("Microsoft", LogEventLevel.Information)
.Enrich.FromLogContext()
.WriteTo.Console(
outputTemplate:
"[{Timestamp:HH:mm:ss} {Level}] {SourceContext}{NewLine}{Message:lj}{NewLine}{Exception}{NewLine}",
theme: AnsiConsoleTheme.Literate)
.WriteTo.File(path, fileSizeLimitBytes: 1_000_000,
rollOnFileSizeLimit: true,
shared: true,
flushToDiskInterval: TimeSpan.FromSeconds(1))
.CreateLogger();
enable useSerialLog()
public static IWebHost BuildWebHost(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.ConfigureServices(services => services.AddAutofac())
.UseStartup<Startup>()
.UseSerilog()
.Build();
Serilog works perfectly on local. For IIS the log file folder need IIS User Permission.
public Startup(IHostingEnvironment env)
{
Log.Logger = new LoggerConfiguration()
.MinimumLevel
.Information()
.WriteTo.RollingFile(#"C:\inetpub\serilog\agilogoservicelogs\{Date}.txt", LogEventLevel.Information)
.WriteTo.Seq("http://localhost:5341")
.CreateLogger();
var builder = new ConfigurationBuilder()
.SetBasePath(env.ContentRootPath)
.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
.AddEnvironmentVariables();
Configuration = builder.Build();
}
Update your controller :
public class ValuesController : Controller
{
private readonly ILogger<ScrumUserController> _logger;
public ValuesController(ILogger<ScrumUserController> logger)
{
_logger = logger;
}
[HttpGet]
public IEnumerable<string> Get()
{
_logger.LogInformation("Agilibo Service start running");
return new string[] { "Agilibo Core", "Service" };
}
}
Now give folder permission on deployed IIS.
If you use .net core ,because default application pool identity:
ApplicationPoolIdentity has no write permission.
You should:
Change application pool user to Local System...
Or-
Give ApplicationPoolIdentity write permission:
ApplicationPoolIdentity is a virtual name , you can use "IIS AppPool\DefaultAppPool" set the permission.
you can use cli do the same work
ICACLS C:\sites\MyWebApp /grant "IIS AppPool\DefaultAppPool":F
Don't forget restart applicationpool and site.
From Microsoft Docs

Resources