ASP.Net Core 2.1 Serilog SQL Server Sink using Azure SQL is working locally but not from Azure app service - azure

I have an ASP.Net Core 2.1 Website that uses Azure SQL Database for the Microsoft Identity component.
I added a Logs table to that database and added Serilog to my website with the SQL Server Sink.
When I run the website locally, while still connected to the Azure SQL database, I can see my log entries in the Logs table just fine. However, when I deploy the website to my Azure App Service, I no longer get any log entries in the Logs table of the database.
Mind you, in the deployed version, I am connecting to and using the Azure SQL database for my MS Identity stuff just fine and I can create new users and edit existing users just fine. So I know the Connection String in my App Service Application Settings is correct.
I have reviewed the Serilog MSSQL Github to compare their configuration recommendations to my own and could not find anything that stood out.
I have this setup working correctly on an ASP.Net Core API that I deploy to another Azure App Service. That service uses a different database but it is on the same SQL Server resource.
I have reviewed the list of SO posts recommended when I started this question with no luck.
I ran the following SQL on the database when I first set up the user account;
EXEC sp_addrolemember N'db_datareader', N'myuser'
EXEC sp_addrolemember N'db_datawriter', N'myuser'
EXEC sp_addrolemember N'db_ddladmin', N'myuser'
And, as I stated, the user account can update and add user data in the AspNetUsers table just fine. So, it doesn't seem like a user account issue.
I have verified that the connection string in my Azure app service DEV deployment slot (the one I am testing), Application Settings, Connection Strings is the exact same as what I have in my local DEV UserSecrets. Plus, again, I can read/write to the AspNet* tables in that same database when deployed to Azure.
Here is my Program.cs class where I set up Serilog;
public class Program
{
public static IConfiguration Configuration { get; } = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.AddJsonFile($"appsettings.{Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? "Production"}.json", optional: true)
.AddUserSecrets<Startup>()
.AddEnvironmentVariables()
.Build();
public static void Main(string[] args)
{
var connectionString = Configuration.GetConnectionString("MyConnectionString");
const string tableName = "Logs";
Log.Logger = new LoggerConfiguration()
.MinimumLevel.Information()
.MinimumLevel.Override("Microsoft", LogEventLevel.Warning)
.Enrich.FromLogContext()
.Enrich.WithMachineName()
.Enrich.WithThreadId()
.WriteTo.MSSqlServer(connectionString, tableName)
.CreateLogger();
// TODO Enable to debug any startup Serilog issues. Make sure to comment out for PROD
//Serilog.Debugging.SelfLog.Enable(msg =>
//{
// Debug.Print(msg);
// Debugger.Break();
//});
try
{
Log.Information("Starting Application");
CreateWebHostBuilder(args).Build().Run();
}
catch (Exception ex)
{
Log.Fatal(ex, "Host terminated unexpectedly");
}
finally
{
Log.CloseAndFlush();
}
}
public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.UseStartup<Startup>()
.UseSerilog();
}
}
The only difference between the API that I have deployed in Azure that is writing logs to Azure SQL and this website is that in the API, which is older, I have
public static IWebHost BuildWebHost(string[] args)
in program.cs whereas the newer website has
public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
So... any ideas would be greatly appreciated.
[UPDATE 1/23/19]
I added the connection string directly to the
var connectionString
in Program.cs rather than getting it from
Configuration.GetConnectionString("MyConnectionString")
and it started logging to the database.
So it seems the issue is with Program.cs being able to read the connection string from Azure App Service deployment slot, Application Settings, Connection Strings section.
This connection string is being correctly read from Startup.cs and has worked since I first created the website.
So, is there some known issue with Azure not being able to read values from deployment slot Application Settings / Connection Strings from Program.cs?

Since there seems to be an issue with Azure, that it doesn't provide the application settings to the web app until CreateWebHostBuilder is invoked, a straightforward workaround (assuming that hardcoding the connection string in the source code isn't a viable option) would be to configure Serilog to use SqlServer in Startup.cs instead of Program.cs.
If it's important to log the initial events that occur during the app's starting up, Serilog can temporarily be configured in Program.cs to write to a file.
Program.cs:
public class Program
{
public static void Main(string[] args)
{
var AzureLogFilePath = #"D:\home\LogFiles\Application\log.txt";
Log.Logger = new LoggerConfiguration()
.MinimumLevel.Information()
.Enrich.FromLogContext()
.WriteTo.File(path: AzureLogFilePath, fileSizeLimitBytes: 1_000_000, rollOnFileSizeLimit: true, shared: true);
.CreateLogger();
try
{
Log.Information("Starting Application");
CreateWebHostBuilder(args).Build().Run();
}
catch (Exception ex)
{
Log.Fatal(ex, "Host terminated unexpectedly");
}
finally
{
Log.CloseAndFlush();
}
}
public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.UseStartup<Startup>()
.UseSerilog();
}
Startup.cs:
public class Startup
{
public Startup(IHostingEnvironment env)
{
var builder = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.AddJsonFile($"appsettings.{env.EnvironmentName ?? "Production"}.json", optional: true)
.AddEnvironmentVariables();
if (env.IsDevelopment()) builder.AddUserSecrets<Startup>(); // according to the docs, AddUserSecrets should be used only in development
Configuration = builder.Build();
}
public IConfiguration Configuration { get; }
public void ConfigureServices(IServiceCollection services)
{
var connectionString = Configuration.GetConnectionString("MyConnectionString");
const string tableName = "Logs";
Log.Logger = new LoggerConfiguration()
.MinimumLevel.Information()
.MinimumLevel.Override("Microsoft", LogEventLevel.Warning)
.Enrich.FromLogContext()
.Enrich.WithMachineName()
.Enrich.WithThreadId()
.WriteTo.MSSqlServer(connectionString, tableName)
.CreateLogger();
//...
}
}

Related

How to retrieve Azure App Configuration Service settings while in Program.cs

I am using Asp.NET Core 5 (v5.0.6) web API deployed as an Azure App Service and I am migrating from using appsettings.json in the API to integrating with Azure App Configuration service. I have the Azure App Configuration Service set up but the issue I am having is how to access values in my Azure App Configuration service for retrieving a connection string for database access while I am still in Program.cs.
Note that in the CreateHostBuilder() method I am examining the context.HostingEnvironment.IsDevelopment() environment variable and if it is "IsDevelopment", implying a local DEV, I am reading an Azure App Configuration Service connection string via User Secrets but if it is not a local DEV, than I rely on the Managed Identity and just pass in the endpoint value from appsettings.json.
The only values I want to get that are not in Azure App Configuration Service is the local DEV Azure App Configuration Service Connection string (from User Secrets) and the Azure App Configuration Service endpoint from Appsettings.json. All other settings should come from Azure App Configuration Service.
The problem I am trying to solve is how to access the values in Azure App Configuration Service, while still in Program.cs, to retrieve the connection string for access to the Azure SQL database I am using for logging.
In the code below, when I link the Azure App Configuration in the CreateHostBuilderMethod and call build, I expected the values in Azure App Configuration Service to then be available via the static Configuration property. However when I try to retrieve the connection string value, it is always null.
How can I correctly retrieve the values for properties in Azure App Configuration Service to use them in Program.cs?
Here is my Program.cs file;
public class Program
{
public static IConfiguration Configuration { get; } = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.AddJsonFile($"appsettings.{Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? "Production"}.json", optional: true)
.AddUserSecrets<Startup>()
.Build();
public static void Main(string[] args)
{
var host = CreateHostBuilder(args)
.Build();
var connectionString = Configuration["CoreApi:Settings:LoggingDb"]; //<-- This returns null
const string tableName = "ApiLogs";
Log.Logger = new LoggerConfiguration()
.MinimumLevel.Information()
.MinimumLevel.Override("Microsoft", LogEventLevel.Warning)
.Filter.ByExcluding(Matching.FromSource("Microsoft.EntityFrameworkCore.Query"))
.WriteTo.MSSqlServer(
connectionString: connectionString,
sinkOptions: new MSSqlServerSinkOptions { TableName = tableName })
.CreateLogger();
// TODO Enable to debug any startup Serilog issues. Make sure to comment out for PROD
// Serilog.Debugging.SelfLog.Enable(msg =>
// {
// Debug.Print(msg);
// Debugger.Break();
// });
//var host = CreateHostBuilder(args)
// .Build();
try
{
Log.Information("Starting ConfirmDelivery API");
host.Run();
}
catch (Exception ex)
{
Log.Fatal(ex, "ConfirmDelivery API Host terminated unexpectedly");
}
finally
{
Log.CloseAndFlush();
}
}
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureAppConfiguration((context, config) =>
{
var settings = config.Build();
if (context.HostingEnvironment.IsDevelopment())
{
var connectionString = settings.GetConnectionString("AzureAppConfiguration");
config.AddAzureAppConfiguration(connectionString);
}
else
{
var endpoint = settings["AppConfigEndpoint"];
var credentials = new ManagedIdentityCredential();
config.AddAzureAppConfiguration(options =>
{
options.Connect(new Uri(endpoint), credentials);
});
}
}).ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseStartup<Startup>();
});
}
Try the code below to get a config value:
using System;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Configuration.AzureAppConfiguration;
namespace myapp
{
class Program
{
static void Main(string[] args)
{
var configItemName = "";
var appConfigConnectionString = "";
var builder = new ConfigurationBuilder();
builder.AddAzureAppConfiguration(appConfigConnectionString);
var config = builder.Build();
Console.WriteLine(config[configItemName]);
}
}
}
Result :

How to use Application settings instead of screts.json when deployed to Azure

I use the secrets.json file to store credentials for my Blazor .Net Core 3.0 Web App and use IConfiguration to get the values like this
public Startup(IConfiguration configuration)
{
Configuration = configuration;
var username = Configuration["Account:Username"];
var password = Configuration["Account:Password"];
new Account(username, password);
}
public IConfiguration Configuration { get; }
Which works fine on my machine in Development, but when I switch to Release in my app Properties > Debug > Environment variables > ASPNETCORE_ENVIRONMENT, the secrets.json doesn't get loaded. Same when I upload the app to Azure, where I have set up the variables as Application settings, it doesn't work. What am I missing here? How can I set up the Web App in Production on Azure, so that I can use the credentials?
edit:
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseStartup<Startup>();
});

Serilog: azure webjob logging doesn't seem to work when hosted in azure?

I have azure webjob sdk (v3.0.3) app which has been configured to use serilog for logging.
The log seems to work when I run the app locally in my system. Below is the configuration:
static void Main(string[] args)
{
try
{
var builder = new HostBuilder()
.ConfigureAppConfiguration(SetupConfiguration)
.ConfigureLogging(SetupLogging)
.ConfigureServices(SetupServices)
.ConfigureWebJobs(webJobConfiguration =>
{
webJobConfiguration.AddTimers();
webJobConfiguration.AddAzureStorageCoreServices(); //this is to store logs in azure storage
})
.UseSerilog()
.Build();
builder.Run();
}
}
The code for SetupConfiguration is below:
private static void SetupConfiguration(HostBuilderContext hostingContext, IConfigurationBuilder builder)
{
var env = hostingContext.HostingEnvironment;
_configuration = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: false, reloadOnChange: true)
.AddEnvironmentVariables()
.Build();
}
The code for setting up services:
private static void SetupServices(HostBuilderContext hostingContext, IServiceCollection serviceCollection)
{
serviceCollection.AddSingleton<IConfiguration>(_configuration);
Log.Logger = new LoggerConfiguration()
.ReadFrom.Configuration(_configuration)
.CreateLogger();
_logger = serviceCollection.BuildServiceProvider().GetRequiredService<ILoggerFactory>().CreateLogger("test");
}
The logging is setup as following:
private static void SetupLogging(HostBuilderContext hostingContext, ILoggingBuilder loggingBuilder)
{
loggingBuilder.SetMinimumLevel(LogLevel.Information);
loggingBuilder.AddConsole();
loggingBuilder.AddDebug();
loggingBuilder.AddSerilog(dispose: true);
}
In my TimerTrigger method I'm using the logger:
[Singleton]
public async static Task Trigger([TimerTrigger("%Job%")]TimerInfo myTimer)
{
_logger.LogInformation($"From Trigger {DateTime.UtcNow.ToString()}");
}
In appSettings.json, serilog is configured as follows:
"Serilog": {
"MinimumLevel": "Information",
"WriteTo": [
{
"Name": "RollingFile",
"Args": {
"pathFormat": ".\\Log\\log-{Date}.txt",
"retainedFileCountLimit": 7,
"fileSizeLimitBytes": 5000000,
"outputTemplate": "{Timestamp:yyyy-MM-dd HH:mm:ss} {EventId} [{Level}] [{Properties}] {Message}{NewLine}{Exception}"
}
}
]
}
the folder "Log" and the log files get created when i run the app locally. But when I publish the webjob, the "Log" folder or the log file is not created in the "app_data" folder of webjob. Can anyone help me figureout how to configure serilog to make it work with webjobs?
Following are the nuget packages used:
If you want to use serilog in WebJob , you need to install this package Serilog.Extensions.WebJobs. Then after configuring the serilog, you would be able to use it.
You must inject the ILogger rather than using the global Log.Logger otherwise the log messages will not be written to the Microsoft Azure WebJobs dashboard.
About the detailed description about how to configure and use serilog, you could refer to this doc.
Hope this could help you, if you still have other questions, please let me know.

how distinguish traces from different instances .net core application in Application Insights

I work on .NET Core 2.2 console application that uses Microsoft.Extensions.Logging and is configured to send logs to Azure Application Insights using Microsoft.ApplicationInsights.Extensibility by:
services.AddSingleton(x =>
new TelemetryClient(
new TelemetryConfiguration
{
InstrumentationKey = "xxxx"
}));
...
var loggerFactory = serviceProvider.GetService<ILoggerFactory>();
loggerFactory.AddApplicationInsights(serviceProvider, logLevel);
It works ok: I can read logs in Application Insights. But the application can be started simultanously in few instances (in different Docker containers). How can I distinguish traces from different instances? I can use source FileName, but I don't know how I should inject it.
I tried to use Scope:
var logger = loggerFactory.CreateLogger<Worker>();
logger.BeginScope(dto.FileName);
logger.LogInformation($"Start logging.");
It's interesting that my configuration is almost identical as in example: https://github.com/MicrosoftDocs/azure-docs/issues/12673
But in my case I can't see the property "FileName" in Application Insights.
For console project, if you want to use the custom ITelemetryInitializer, you should use this format: .TelemetryInitializers.Add(new CustomInitializer());
Official doc is here.
I test it at my side, and it works. The role name can be set.
Sample code is below:
static void Main(string[] args)
{
TelemetryConfiguration configuration = TelemetryConfiguration.CreateDefault();
configuration.InstrumentationKey = "xxxxx";
configuration.TelemetryInitializers.Add(new CustomInitializer());
var client = new TelemetryClient(configuration);
ServiceCollection services = new ServiceCollection();
services.AddSingleton(x => client);
var provider = services.BuildServiceProvider();
var loggerFactory = new LoggerFactory();
loggerFactory.AddApplicationInsights(provider, LogLevel.Information);
var logger = loggerFactory.CreateLogger<Program>();
logger.LogInformation("a test message 111...");
Console.WriteLine("Hello World!");
Console.ReadLine();
}
Check the role name in azure portal:
If you really have no way to distinguish them you can use a custom telemetry initializer like this:
public class CustomInitializer : ITelemetryInitializer
{
public void Initialize(ITelemetry telemetry)
{
telemetry.Context.Cloud.RoleName = Environment.MachineName;
}
}
and/or you can add a custom property:
public class CustomInitializer : ITelemetryInitializer
{
public void Initialize(ITelemetry telemetry)
{
if(telemetry is ISupportProperties)
{
((ISupportProperties)telemetry).Properties["MyIdentifier"] = Environment.MachineName;
}
}
}
In this example I used Environment.MachineName but you can of course use something else if needed. Like this work Id parameter of yours.
the wire it up using:
services.AddSingleton<ITelemetryInitializer, CustomInitializer>();

Enable file logging on IIS for serilog

I want to enable file logging on IIS to resolve an issue locally on IIS. Following is the code I am using. It works when I run on visual studio and logs to a file but not when I deployed to IIS. Folder has enough permission to create a file and also I created the folder and file. What am I missing here ?
Program.cs
var path = #"C:\workspace\Logs\Log-{Date}.txt";
Log.Logger = new LoggerConfiguration()
.MinimumLevel.Verbose()
.MinimumLevel.Override("Microsoft", LogEventLevel.Information)
.Enrich.FromLogContext()
.WriteTo.Console(
outputTemplate:
"[{Timestamp:HH:mm:ss} {Level}] {SourceContext}{NewLine}{Message:lj}{NewLine}{Exception}{NewLine}",
theme: AnsiConsoleTheme.Literate)
.WriteTo.File(path, fileSizeLimitBytes: 1_000_000,
rollOnFileSizeLimit: true,
shared: true,
flushToDiskInterval: TimeSpan.FromSeconds(1))
.CreateLogger();
enable useSerialLog()
public static IWebHost BuildWebHost(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.ConfigureServices(services => services.AddAutofac())
.UseStartup<Startup>()
.UseSerilog()
.Build();
Serilog works perfectly on local. For IIS the log file folder need IIS User Permission.
public Startup(IHostingEnvironment env)
{
Log.Logger = new LoggerConfiguration()
.MinimumLevel
.Information()
.WriteTo.RollingFile(#"C:\inetpub\serilog\agilogoservicelogs\{Date}.txt", LogEventLevel.Information)
.WriteTo.Seq("http://localhost:5341")
.CreateLogger();
var builder = new ConfigurationBuilder()
.SetBasePath(env.ContentRootPath)
.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
.AddEnvironmentVariables();
Configuration = builder.Build();
}
Update your controller :
public class ValuesController : Controller
{
private readonly ILogger<ScrumUserController> _logger;
public ValuesController(ILogger<ScrumUserController> logger)
{
_logger = logger;
}
[HttpGet]
public IEnumerable<string> Get()
{
_logger.LogInformation("Agilibo Service start running");
return new string[] { "Agilibo Core", "Service" };
}
}
Now give folder permission on deployed IIS.
If you use .net core ,because default application pool identity:
ApplicationPoolIdentity has no write permission.
You should:
Change application pool user to Local System...
Or-
Give ApplicationPoolIdentity write permission:
ApplicationPoolIdentity is a virtual name , you can use "IIS AppPool\DefaultAppPool" set the permission.
you can use cli do the same work
ICACLS C:\sites\MyWebApp /grant "IIS AppPool\DefaultAppPool":F
Don't forget restart applicationpool and site.
From Microsoft Docs

Resources