I am evaluating replacing an existing logger in an existing test application. One of my potential candidates is Serilog. The application may start up to 16 threads and each thread is passed in the name of the log file to use. The threads during the test application are started with 3 different log names. I am using Microsoft extensions so that I can replace Serilog with another logger for evaluation.
I need to be able to close, delete and restart the individual log files. The problem I am encountering, is that once I call the CloseAndFlush method, all the logs are closed. Is there some way to individually close a log file in a thread?
My configuration is below. The RunInLogger is an ILogger property that I pass to each thread. If I use Microsoft Extensions, it is a Microsoft ILogger, or if I use the Serilog ILogger is is a serilog ILogger.
var serilogLogger = new LoggerConfiguration()
.MinimumLevel.Debug()
.WriteTo.Map("Name", "Other", (name, wt) => wt.File($"{name}", outputTemplate: outTML))
.CreateLogger();
Log.Logger = serilogLogger;
ILoggerFactory loggerFactory = new LoggerFactory()
.AddSerilog(serilogLogger);
RunInLogger = loggerFactory.CreateLogger("Program");
Each thread is passed the RunInLogger (ILogger) and the name of a file (RunInLogFile) to begin logging, by using the BeginScope method.
using (RunInLogger.BeginScope(new Dictionary<String, object> { { "Name", RunInLogFile } }))
{
RunInLogger.LogInformation(logEntry);
}
I have experimented with using the Serilog ILogger directly, thinking I could add this to my configuration file - .Enrich.FromLogContext() and replace the BeginScope with the Serilog equivalent - LogContext.PushProperty("Name", RunInLogFile). The resulting code I attempted to only close the current log, but still closed all logs.
using (LogContext.PushProperty("Name", RunInLogFile))
{
RunInLogger.Log.Information(logEntry);
Log.CloseAndFlush()
}
Is there a method to call within the LogContext to only close the log associated with my RunInLogFile?
Related
I need to log custom dimensions to Application Insights for which I'm using ILogger.BeginScope(). That works perfectly. That is:
using (logger.BeginScope(new Dictionary<string, object> { "key": "value" }))
{
logger.LogInformation("message");
}
My issue is that I need to call other methods in other classes, and I'm injecting the ILogger into all my classes. So how can I persist the logging scope among all my classes?
I could surely do ILogger.BeginScope() in all my classes, but I would need to pass the custom properties to classes that don't really need that information. Is there a pattern I could use?
If you call BeginScope multiple times in multiple classes within the same execution path, you will get one aggregated scope. There is no need to manually pass the properties. ILogger is injected as a singleton by default (see source code).
See also Since ILogger<T> is a singleton, how different threads can use BeginScope() without affecting others?
A call to BeginScope will put a new item onto that stack, and the adjoining Dispose will pop it off of that stack.
When the logger is invoked via LogInformation or otherwise, the data of the current stack object will be copied to write it to the console or whatever output that logger instance is configured to do.
LoggerExternalScopeProvider.cs#L14
So I've noticed a strange behavior which I would like to share and see if anyone has had the similar problem.
We are using on Prem solution where we pickup a file or a http event request, map it to an outgoing xml xsd/schema and then create the file later on prem.
The problem was that the system where we save the file does not cooperate so good with the logic app, the logic app failes sometime because the system takes the file before the logic app can finish writing the full content.
The system receiving the files only read .xml files, so we though we should first rename the files to tmp, let logic app create the files and then rename them.
This solution sounded quite simple before we started actually applying it to the logic app.
If we take FileSystem function which has Rename File function and use the parameters “Name” from the create file on prem
{
"statusCode": 404,
"message": "Resource not found"
}
We get the message 404 that the resource is not found, now this complicates a lot of things, I’ve checked the privileges on the account that should not be an issue.
What we also have tried is listing all files in the folder, creating a foreach and then adding a rule and the Rename File function. This makes it work but the logic app does not cope well with receiving a lof of files at ones with that solution.
But the Rename Files works when it’s in a foreach loop and we extract the file names in a list from root folder or normal folder.
But why does it not work with just using the Rename Function? Is this perhaps an azure function bug in the Logic app Rename File Function?
So after discussing with Microsoft support on Azure they have actually confirmed that there is a bug with the “Create File” function.
It looks like all the data and information is actually lost during that functions, the support technicians do not know why that is happening but they have had similar cases which people have reported.
I have not stumbled across any of those posts, but I will post how we solved the problem with a work around.
FYI, The support team has taken the case further so that the developers at azure should look into it, because it’s not just “name” tag which is lost from Create a File, ( it’s all valuable options are actually lost ).
So first we initialize a variable and then actually set the variable name with two steps before we create the file:
The name is set with a temp name and a GUID.
Next step is creating the file with the temp-name used in function “Set Variable Temp FileName”
And on the Rename File function we use the Path from where we store the temp file and add \”FILENAME”
And add the “New Name” which we want to use.
This proved to work but is a workaround, support confirmed that you should be able to just use the “RenameFile” after creating the file with a temp name and changing it to the desired name.
But since Create a File does not send or pass any information at all from this list we have to initialize Variables to make it work.
If anyone has stumbled on the same problem where the Backend system reads the files before they are managed to be created by the logic app and you need some workaround this worked good for me.
Hope it helps!
We recently had the same issue; and the workaround of renaming the file also failed.
The cause seems to be that the Azure On Prem Gateway creates a file (or renames a file), then releases its lock, before checking that the file exists. In the gap between releasing the lock and checking that the file exists, the file may be picked up (deleted) thus causing LogicApps to think the step failed (reporting a 404 error), and thus confusion.
Our workaround was to create a Windows service which we hosted on the file servers (so they'd be able to respond to file changes before anything else on the network). This service has a configuration file which accepts a list of paths and file filters, and it uses the FileSystemWatcher to monitor for new files, or renamed files. When it detects a match it takes out a read lock on the file. This ensure it's not blocked by anything writing to the file (i.e. so it doesn't have to wait for the On Prem Gateway's write aciton to complete before obtaining its own lock), but whilst our service holds its lock the file can't be deleted (so the consumer can't remove the file / buying time for the On Prem Gateway to perform it's post-write read and report success). Our service releases its own lock after a defined period (we've gone with 30 seconds, though you could likely get away with much less). At that point, the consumer can successfully consume the file.
Basic code for the file watch & locking logic below:
sing System;
using System.IO;
using System.Diagnostics;
using System.Threading.Tasks;
namespace AzureFileGatewayHelper
{
public class Interceptor: IDisposable
{
object lockable = new object();
bool disposed = false;
readonly FileSystemWatcher watcher;
readonly int lockTimeInMS;
public Interceptor(string path, string filter, int lockTimeInSeconds)
{
lockTimeInMS = lockTimeInSeconds * 1000;
watcher = new FileSystemWatcher();
watcher.Path = path;
watcher.Filter = filter;
watcher.NotifyFilter = NotifyFilters.LastAccess
| NotifyFilters.LastWrite
| NotifyFilters.FileName
| NotifyFilters.DirectoryName;
watcher.Created += OnIncercept;
watcher.Renamed += OnIncercept;
}
public Interceptor(InterceptorConfigElement config) : this(config.Path, config.Filter, config.TimeToLockInSeconds) { Debug.WriteLine($"Loaded config ${config.Key}: Path: '${config.Path}'; Filter: '${config.Filter}'; LockTime: : '${config.TimeToLockInSeconds}'."); }
public void Start()
{
watcher.EnableRaisingEvents = true;
}
public void Stop()
{
if (watcher != null)
watcher.EnableRaisingEvents = false;
}
private async void OnIncercept(object source, FileSystemEventArgs e)
{
using (var fs = new FileStream(e.FullPath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
Debug.WriteLine($"Locked: {e.FullPath} {e.ChangeType}");
await Task.Delay(lockTimeInMS);
}
Debug.WriteLine($"Unlocked {e.FullPath} {e.ChangeType}");
}
public void Dispose()
{
if (disposed) return;
lock (lockable)
{
if (disposed) return;
Stop();
watcher?.Dispose();
disposed = true;
}
}
}
}
I am trying to write a method to create a database and run migrations on it, given the connection string.
I need the multiple connections because I record an audit log in a separate database.
I get the connection strings out of app.config using code like
ConfigurationManager.ConnectionStrings["Master"].ConnectionString;
The code works with the first connection string defined in my app.config but not others, which leads me to think that somehow it is getting the connection string from app.config in some manner I don't know.
My code to create the database if it does not exist is
private static Context MyCreateContext(string ConnectionString)
{
// put the connection string where the factory method can get it
AppDomain.CurrentDomain.SetData("ConnectionString", ConnectionString );
var factory = new ContextFactory();
// I know I need this line - but I cant see how what follows actually uses it
Database.SetInitializer(new MigrateDatabaseToLatestVersion<Context,DataLayer.Migrations.Configuration>());
var context = factory.Create();
context.Database.CreateIfNotExists();
return context
}
The code in the Migrations.Configuration is
Public sealed class Configuration : DbMigrationsConfiguration<DataLayer.Context>
{
public Configuration()
{
AutomaticMigrationsEnabled = false;
}
}
The context factory code is
public class ContextFactory : IDbContextFactory<Context>
{
public Context Create()
{
var s = (string)AppDomain.CurrentDomain.GetData("ConnectionString");
return new Context(s);
}
}
Thus I am setting the connection string before creating the context.
Where can I be going wrong, given that the connection strings are all the same except the database name, and the migration code runs with one connection string, but doesnt run with others?
I wonder if my problem is to do with understanding how How does Database.SetInitializer actually works. I am guessing something about reflection or generics. How do i make the call to SetInitializer tie tie to my actual context?
I have tried the following code but the migrations do not run
private static Context MyCreateContext(string ConnectionString)
{
Database.SetInitializer(new MigrateDatabaseToLatestVersion<Context, DataLayer.Migrations.Configuration>());
var context = new Context(ConnectionString);
context.Database.CreateIfNotExists();
}
This question appears to be related
UPDATE:
I can get the migrations working if I refer to the connection string using
public MyContext() : base("MyContextConnection") - which points to in the config
I was also able to get migrations working on using different instances of the context, if I created a ContextFactory class and passed the connection to it by referencing a global. ( See my answer to the related question link )
Now I am wondering why it has to be so hard.
I'm not sure exactly as to what the problems are you facing, but let me try
The easiest way to provide connection - and be sure it works that way...
1) Use your 'DbContext' class name - and define a connection in the app.config (or web.config). That's easiest, you should have a connection there that matches your context class name,
2) If you put it into the DbContext via constructor - then be consistent and use that one. I'd also suggest to 'read' from config connections - and again name it 'the same' as your context class (use the connection 'name', not the actual string),
3) if none is present - EF/CF makes the 'default' one - based on your provider - and your context's class name - which usually isn't what you want,
You shouldn't customize with initializers for that reason -
initializers should be agnostic and serve other purpose - setup
connection in the .config - or directly on your DbContext
Also check this Entity Framework Code First - How do I tell my app to NOW use the production database once development is complete instead of creating a local db?
Always check 'where your data' goes - before doing anything.
For how the initializer actually works - check this other post of mine, I made a thorough example
How to create initializer to create and migrate mysql database?
Notes: (from the comments)
Connection shouldn't be very dynamic - config is the right place for it to be, unless you have a good reason.
Constructor should work fine too.
CreateDbIfNotExists doesn't work well together with the 'migration' initializer. You can just use the MigrateDatabaseToLatestVersion initializer. Don't 'mix' it
Or - put something like public MyContext() : base("MyContextConnection") - which points to <connectionStrings> in the config
To point to connection - just use its 'name' and put that into constructor.
Or use somehting like ConfigurationManager.ConnectionStrings["CommentsContext"].ConnectionString
Regarding entertaining 'multiple databases' with migrations (local and remote from one app) - not exactly related - but this link - Migration not working as I wish... Asp.net EntityFramework
Update:
(further discussion here - Is adding a class that inherits from something a violation of the solid principles if it changes the behavior of code?)
It is getting interesting here. I did manage to reproduce the problems you're facing actually. Here is a short breakdown on what I think it's happening:
First, this worked 'happily':
Database.SetInitializer(new CreateAndMigrateDatabaseInitializer<MyContext, MyProject.Migrations.Configuration>());
for (var flip = false; true; flip = !flip)
{
using (var db = new MyContext(flip ? "Name=MyContext" : "Name=OtherContext"))
{
// insert some records...
db.SaveChanges();
}
}
(I used custom initializer from my other post, which controls migration/creation 'manually')
That worked fine w/o an Initializer. Once I switched that on, I ran into some curious problems.
I deleted Db-s (two, for each connection). I expected to either not work, or create one db, then another in the next pass (like it did, w/o migrations, just 'Create' initializer).
What happened, to my surprise - is it actually created both databases on the first
pass ??
Then, being a curious person:), I put breakpoints on the MyContext ctor, and debugged through the migrator/initializer. Again empty/no db-s etc.
It created first instance on my call within the flip. Then on the first access to 'model', it invoked the initializer. Migrator took over (having had no db-s). During the migrator.Update(); it actually constructs the MyContext (I'm guessing via generic param in Configuration) - and calls the 'default' empty ctor. That had the 'other connection/name' by default - and creates the other Db all as well.
So, I think this explains what you're experiencing. And why you had to create the 'Factory' to support the Context creation. That seems to be the only way. And setting some 'AppDomain' wide 'connection string' (which you did well actually) which isn't 'overriden' by default ctor call.
Solution that I see is - you just need to run everything through factory - and 'flip' connections in there (no need for static connection, as long as your factory is a singleton.
You can supply a configuration in the MigrateDatabaseToLatestVersion constructor.
If you set the initializer in the DbContext you can also pass a 'true' to use the current connection string.
I want to log in to multiple log files(flume and console). How to set log4j as package level?ie com.mypackage.myclass into flume and other packages into console..
First of all you need to configure log4j to have two named loggers, one that sends to the Console appender, and one that sends to Flume. You can then write use a proxy class for making your logging calls that routes the log4j calls to the different loggers depending on the package the caller is in. You can do this by accessing the stack of the current thread, like so:
public class Logger
{
public static org.apache.log4j.Logger getLogger()
{
// this will get the calling frame, 0=Thread, 1=this, 2=caller
StackTraceElement stackElement = Thread.currentThread().getStackTrace()[2];
if(stackElement.getClassName().startsWith("the.package.that.goes.to.flume"))
{
return org.apache.log4j.Logger.getLogger("Flume");
}
else
{
return org.apache.log4j.Logger.getLogger("Console");
}
}
}
}
The code above is assuming you have named your two loggers 'Flume' and 'Console'.
When ever you make a logging call in your app, use Logger.getLogger() rather than going to log4j directly.
Check this blog post
http://veerasundar.com/blog/2009/07/log4j-tutorial-adding-log4j-logging-to-your-project/
It has a complete PDF for download on how to add log4j to project.
You need to define categories for different packages.Everything is explained in above PDF.
Hope it helps.
Does anyone know if it is possible to do logging in SSIS (SQL Server Integration Services) via log4net? If so, any pointers and pitfalls to be aware of? How's the deployment story?
I know the best solution to my problem is to not use SSIS. The reality is that as much as I hate this POS technology, the company I work with encourages the use of these apps instead of writing code. Meh.
So to answer my own question: it is possible. I'm not sure how our deployment story will be since this will be done in a few weeks from now.
I pretty much took the information from these sources and made it work. This one explains how to make referencing assemblies work with SSIS, click here. TLDR version: place it in the GAC and also copy the dll to the folder of your targetted framework. In my case, C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727. To programmatically configure log4net I ended up using this link as reference.
This is how my logger configuration code looks like for creating a file with the timestamp on it:
using log4net;
using log4net.Config;
using log4net.Layout;
using log4net.Appender;
public class whatever
{
private ILog logger;
public void InitLogger()
{
PatternLayout layout = new PatternLayout("%date [%level] - %message%newline");
FileAppender fileAppenderTrace = new FileAppender();
fileAppenderTrace.Layout = layout;
fileAppenderTrace.AppendToFile = false;
// Insert current date and time to file name
String dateTimeStr = DateTime.Now.ToString("yyyyddMM_hhmm");
fileAppenderTrace.File = string.Format("c:\\{0}{1}", dateTimeStr.Trim() ,".log");
// Configure filter to accept log messages of any level.
log4net.Filter.LevelMatchFilter traceFilter = new log4net.Filter.LevelMatchFilter();
traceFilter.LevelToMatch = log4net.Core.Level.All;
fileAppenderTrace.ClearFilters();
fileAppenderTrace.AddFilter(traceFilter);
fileAppenderTrace.ImmediateFlush = true;
fileAppenderTrace.ActivateOptions();
// Attach appender into hierarchy
log4net.Repository.Hierarchy.Logger root = ((log4net.Repository.Hierarchy.Hierarchy)LogManager.GetRepository()).Root;
root.AddAppender(fileAppenderTrace);
root.Repository.Configured = true;
logger = log4net.LogManager.GetLogger("root");
}
}
Hopefully this might help someone in the future or at least serve as a reference if I ever need to do this again.
Sorry, you didn't dig deep enough. There are 5 different destinations that you can log to, and 7 columns you can choose to include or not include in your logging as well as between 18 to 50 different events that you can capture logging on. You appear to have chosen the default logging, and dismissed it because it didn't work for you out of the box.
Check these two blogs for more information on what can be done with SSIS logging:
http://consultingblogs.emc.com/jamiethomson/archive/2005/06/11/SSIS_3A00_-Custom-Logging-Using-Event-Handlers.aspx
http://www.sqlservercentral.com/blogs/michael_coles/archive/2007/10/09/3012.aspx