Nlog Delete SQLite longer than X days - nlog

xsi:type="File" has maxArchiveFiles="5" archiveEvery="Day" properties. Is that possibble to delete db content older than X days?
xsi:type="Database" has rules for that purpose?
My sample Nlog.config;
<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.nlog-project.org/schemas/NLog.xsd NLog.xsd"
autoReload="true"
throwExceptions="false"
throwConfigExceptions="true"
internalLogLevel="Off" internalLogFile="c:\temp\nlog-internal.log">
<targets>
<!--target name="logfile" xsi:type="File" fileName="file.txt" /!-->
<target xsi:type="Database"
name="db"
dbProvider="System.Data.SQLite.SQLiteConnection, System.Data.SQLite"
connectionString="Data Source=log.db3;"
commandType="Text"
commandText="INSERT into Log(Timestamp, Loglevel, Callsite, Message) values(#Timestamp, #Loglevel, #Callsite, #Message)"
>
<parameter name="#Timestamp" layout="${longdate}"/>
<parameter name="#Loglevel" layout="${level:uppercase=true}"/>
<parameter name="#Callsite" layout="${callsite:filename=true}"/>
<parameter name="#Message" layout="${message}"/>
</target>
<target name="colouredConsole" xsi:type="ColoredConsole" useDefaultRowHighlightingRules="false" layout="${longdate}|${pad:padding=5:inner=${level:uppercase=true}}|${message}">
<highlight-row condition="level == LogLevel.Debug" foregroundColor="DarkGray" />
<highlight-row condition="level == LogLevel.Info" foregroundColor="Gray" />
<highlight-row condition="level == LogLevel.Warn" foregroundColor="Yellow" />
<highlight-row condition="level == LogLevel.Error" foregroundColor="Red" />
<highlight-row condition="level == LogLevel.Fatal" foregroundColor="Red" backgroundColor="White" />
</target>
</targets>
<rules>
<logger name="*" minlevel="Info" maxlevel="Info" writeTo="colouredConsole" />
<logger name="*" minlevel="Error" maxlevel="Error" writeTo="colouredConsole" />
<logger name="*" minlevel="Debug" maxlevel="Debug" writeTo="colouredConsole" />
<logger name="*" minlevel="Debug" maxlevel="Debug" writeTo="db" />
<logger name="*" minlevel="Error" maxlevel="Error" writeTo="db" />
</rules>
</nlog>
How to delete sqlite db content older than 7 days?

Some people suggests EnsureDb(); it works but I want to ask any possibility to set from xsi:type="Database" parameters?
private static void EnsureDb()
{
if (File.Exists("log.db3"))
{
using (SQLiteConnection connection = new SQLiteConnection("Data Source=log.db3"))
using (SQLiteCommand command = new SQLiteCommand(
"DELETE FROM Log WHERE Timestamp <= datetime('now', '-7 day');",
connection))
{
connection.Open();
command.ExecuteNonQuery();
}
return;
}
using (SQLiteConnection connection = new SQLiteConnection("Data Source=log.db3"))
using (SQLiteCommand command = new SQLiteCommand(
"CREATE TABLE Log (Timestamp TEXT, Loglevel TEXT, Callsite TEXT, Message TEXT)",
connection))
{
connection.Open();
command.ExecuteNonQuery();
}
}

Related

NLog Truncate syntax

I'm changing NLog.config file for UiPath. I want to increase truncate value to 20000 and I've tried following syntax but it's not working.
<variable name="truncated_message" value="${replace:replaceWith=...TRUNCATED:regex=true:inner=${message}:searchFor=^[\s\S]{20000}}"/>
<target type="File" name="WorkflowLogFiles" fileName="${WorkflowLoggingDirectory}/${shortdate}_Execution.log" layout="${time} ${level} ${truncated_message}" keepFileOpen="true" openFileCacheTimeout="5" concurrentWrites="true" encoding="utf-8" writeBom="true" />
I've also tried following regex but they do not work for me
(^(?:\S+\s+\n?){0, 20000})
(?\=.\{20000\}).+
Can anyone please tell me what am I doing wrong and how to do set the truncate value to 20000?
This works for me:
<nlog>
<variable name="truncated_message_new" value="${message:truncate=20000}" />
<variable name="truncated_message_old" value="${trim-whitespace:inner=${message:padding=-20000:fixedLength=true}}" />
<targets>
<target type="File" name="WorkflowLogFiles" fileName="C:\temp\nlog/${shortdate}_Execution.log" layout="${time} ${level} ${truncated_message_old}" keepFileOpen="true" openFileCacheTimeout="5" concurrentWrites="true" encoding="utf-8" writeBom="true" />
</targets>
<rules>
<logger name="*" minlevel="Trace" writeTo="WorkflowLogFiles" />
</rules>
</nlog>
It only saves 20000 characters:
var test = string.Empty.PadLeft(40000, '*');
NLog.LogManager.GetLogger("test").Info("Doing hard work! {Action}", test);

dotnet core nlog running on Linux logs only one entry

I am running an aspnet core site on Linux using Kestrel. I have set up nlog to log to csv files. All is well in my Windows development environment, but when I run it on Linux I only get one entry in the log file. I turned on nlog tracing and found the following error in the output:
2016-09-09 07:22:46.2123 Error Error has been raised. Exception: System.IO.IOException: The file '/var/aspnetcore/CSWebSite/releases/20160909061254/logs/archives/own-log.20160908.csv' already exists.
at System.IO.UnixFileSystem.MoveFile(String sourceFullPath, String destFullPath)
at System.IO.File.Move(String sourceFileName, String destFileName)
at NLog.Targets.FileTarget.ArchiveByDate(String fileName, String pattern, LogEventInfo logEvent)
at NLog.Targets.FileTarget.ProcessLogEvent(LogEventInfo logEvent, String fileName, Byte[] bytesToWrite)
at NLog.Targets.FileTarget.FlushCurrentFileWrites(String currentFileName, LogEventInfo firstLogEvent, MemoryStream ms, List`1 pendingContinuations)
2016-09-09 07:22:46.2123 Error Error has been raised. Exception: System.IO.IOException: The file '/var/aspnetcore/CSWebSite/releases/20160909061254/logs/archives/all-log.20160907.csv' already exists.
at System.IO.UnixFileSystem.MoveFile(String sourceFullPath, String destFullPath)
at System.IO.File.Move(String sourceFileName, String destFileName)
at NLog.Targets.FileTarget.ArchiveByDate(String fileName, String pattern, LogEventInfo logEvent)
at NLog.Targets.FileTarget.ProcessLogEvent(LogEventInfo logEvent, String fileName, Byte[] bytesToWrite)
at NLog.Targets.FileTarget.FlushCurrentFileWrites(String currentFileName, LogEventInfo firstLogEvent, MemoryStream ms, List`1 pendingContinuations)
This is my nlog.config:
<?xml version="1.0" encoding="utf-8" ?>
<nlog
xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<extensions>
<!--enable NLog.Web for ASP.NET Core-->
<add assembly="NLog.Web.AspNetCore"/>
</extensions>
<targets>
<!-- https://github.com/nlog/nlog/wiki/File-target -->
<!-- http://stackoverflow.com/questions/34679727/use-nlog-in-asp-net-5-application -->
<target name="allfile" xsi:type="AsyncWrapper" queueLimit="5000" overflowAction="Discard">
<target name="allfile" xsi:type="File"
fileName="${basedir}/logs/all-logfile.csv"
archiveFileName="${basedir}/logs/archives/all-log.{#}.csv"
archiveEvery="Day"
archiveNumbering="Date"
archiveDateFormat="yyyyMMdd"
maxArchiveFiles="28"
concurrentWrites="true"
keepFileOpen="false"
encoding="utf-8">
<layout xsi:type="CSVLayout">
<column name="time" layout="${longdate}" />
<column name="logger" layout="${logger}" />
<column name="authtype" layout="${aspnet-user-authtype}" />
<column name="identity" layout="${aspnet-user-identity}" />
<column name="level" layout="${uppercase:${level}}" />
<column name="message" layout="${message} ${exception:format=ToString,StackTrace}" />
</layout>
</target>
</target>
<target name="ownfile" xsi:type="AsyncWrapper" queueLimit="5000" overflowAction="Discard">
<target name="ownfile" xsi:type="File"
fileName="${basedir}/logs/own-logfile.csv"
archiveFileName="${basedir}/logs/archives/own-log.{#}.csv"
archiveEvery="Day"
archiveNumbering="Date"
archiveDateFormat="yyyyMMdd"
maxArchiveFiles="28"
concurrentWrites="true"
keepFileOpen="false"
encoding="utf-8">
<layout xsi:type="CSVLayout">
<column name="time" layout="${longdate}" />
<column name="logger" layout="${logger}" />
<column name="authtype" layout="${aspnet-user-authtype}" />
<column name="identity" layout="${aspnet-user-identity}" />
<column name="level" layout="${uppercase:${level}}" />
<column name="message" layout="${message} ${exception:format=ToString,StackTrace}" />
</layout>
</target>
</target>
<target xsi:type="Null" name="blackhole" />
</targets>
<rules>
<!--All logs, including from Microsoft-->
<logger name="*" minlevel="Debug" writeTo="allfile" />
<!--Skip Microsoft logs and so log only own logs-->
<logger name="Microsoft.*" minlevel="Trace" writeTo="blackhole" final="true" />
<logger name="*" minlevel="Debug" writeTo="ownFile" />
</rules>
</nlog>
I have concurrent writes true and keepFileOpen false which seems necessary as kestrel forks multiple dotnet processes on Linux.
Is this a bug or is my configuration incorrect?

Log4Net - how to properly use PropertyFilter

Ok, here's the appender:
<appender name="DebugFileAppender" type="log4net.Appender.FileAppender">
<file value="debug.log" />
<lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
<filter type="log4net.Filter.PropertyFilter">
<Key value="ApplicationName" />
<StringToMatch value="Test Application" />
</filter>
<filter type="log4net.Filter.LevelRangeFilter">
<levelMin value="DEBUG" />
<levelMax value="DEBUG" />
</filter>
<filter type="log4net.Filter.DenyAllFilter" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="Date:%date Thread:[%thread] Level:%-5level Logger:%logger - ApplicationName:%property{ApplicationName}; Message:%message%newline" />
</layout>
</appender>
Now, it seems this would check the ThreadContext.Properties["ApplicationName"] string and if it finds 'Test Application' it would log.
Well, it's logging everything, even if ApplicationName="DoNotLog".
Now, I will freely admit this may be due to the way I'm (attempting) to use Log4Net - I need to expose it to COM, so I've wrapped up a singleton Log4Net instance, and call this from COM:
public void LogDebug(LogCodeSource codeSource, LogExecutionSource execSource, object message, Exception exception = null)
{
// Add the thread context properties for appender filtering purposes
log4net.ThreadContext.Properties["ApplicationName"] = codeSource.ApplicationName;
log4net.ThreadContext.Properties["ComponentName"] = codeSource.ComponentName;
log4net.ThreadContext.Properties["MethodName"] = codeSource.MethodName;
log4net.ThreadContext.Properties["ClientMachineName"] = execSource.ClientMachineName;
log4net.ThreadContext.Properties["ServerMachineName"] = execSource.ServerMachineName;
log4net.ThreadContext.Properties["UserName"] = execSource.UserName;
if (exception == null)
{
LoggerContext.GetInstance.MainLogger.Debug(message);
}
else
{
LoggerContext.GetInstance.MainLogger.Debug(message, exception);
}
}
In my tests, if I call:
var logger = new MyLogger();
logger.LogDebug(new LogCodeSource { ApplicationName = "Test Application", ComponentName = "Test component", MethodName = "Test method" }, new LogExecutionSource { ClientMachineName = "Test client", ServerMachineName = "Test server", UserName = "Test user" }, "Test message");
logger.LogDebug(new LogCodeSource { ApplicationName = "DoNotLog", ComponentName = "Test component", MethodName = "Test method" }, new LogExecutionSource { ClientMachineName = "Test client", ServerMachineName = "Test server", UserName = "Test user" }, "Test message");
In root, I've tried:
<root>
<appender-ref ref="DebugFileAppender" />
</root>
As well as:
<logger name="MyLogger">
<level value="DEBUG" />
<appender-ref ref="DebugFileAppender" />
</logger>
Both logs are showing up in my debug.log file.
Any idea why the property filter isn't preventing DoNotLog from going to debug.log?
Here's what I'm seeing in the log:
Date:2015-12-31 11:11:00,928 Thread:[14] Level:DEBUG Logger:MyLogger - ApplicationName:Test Application; Message:Test message
Date:2015-12-31 11:11:00,942 Thread:[14] Level:DEBUG Logger:MyLogger - ApplicationName:DoNotLog; Message:Test message
What you are after is an AND filter since you want to log all message where ApplicationName matches a particular name AND the level is DEBUG. Unfortunately log4net doesn't ship AND filters out of the box however this SO answer details how to implement one.
Property filter are also available when using Microsoft.Extensions.Logging.Abstraction instead if log4Net directly.
Example:
using (_logger.BeginScope(new[] { new KeyValuePair<string, object>("YourPropertyKey", "YourPropertyValue") }))
{
_logger.LogDebug("My special log with context property set");
}
Then in the config an appender has to be setup for the property and key.
Note there is rexexToMatch in oder to stringToMacth if you want to allow multiple values to a key.
<log4net>
<root>
<level value="ALL" />
<appender-ref ref="property_logger" />
</root>
<appender name="property_logger" type="log4net.Appender.RollingFileAppender">
<file value="logs/my_service.log" />
<appendToFile value="true" />
<rollingStyle value="Date" />
<datePattern value="-yyyyMMdd" />
<maxSizeRollBackups value="5" />
<maximumFileSize value="5MB" />
<preserveLogFileNameExtension value="true" />
<staticLogFileName value="false" />
<filter type="log4net.Filter.PropertyFilter">
<Key value="YourPropertyKey" />
<StringToMatch value="YourPropertyValue" />
</filter>
<filter type="log4net.Filter.DenyAllFilter" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date [%2thread] %-5level %.50logger - %message%newline" />
</layout>
</appender>
</log4net>

Why are my exceptions not being logged with ServiceStack NLog?

Given this NLog config file:
<extensions>
<add assembly="Seq.Client.NLog"/>
</extensions>
<variable name="ServiceName" value="LO.Leads.Processor"/>
<targets async="true">
<target name="seq" xsi:type="Seq" serverUrl="http://mywebsite">
<property name="ThreadId" value="${threadid}" as="number" />
<property name="MachineName" value="${machinename}" />
</target>
<target name="file" xsi:type="File"
layout="${longdate}|${logger}|${level}|${threadid}|${message}"
fileName="C:\LogFiles\Leads\Processor\Processor.log"
archiveFileName="C:\LogFiles\Leads\Processor\Processor.{##}.log"
archiveEvery="Day"
archiveNumbering="Rolling"
maxArchiveFiles="45"
concurrentWrites="true"/>
<target name="debugger" xsi:type="Debugger"
layout="${logger}:${message}"/>
<target name="console" xsi:type="Console"
layout="${logger}:${level}:${threadid}:${message}" />
</targets>
<rules>
<logger name="*" minlevel="Info" writeTo="seq" />
<logger name="*" minlevel="Error" writeTo="file" />
<logger name="*" minlevel="Debug" writeTo="debugger" />
<logger name="*" minlevel="Debug" writeTo="console" />
</rules>
and this code block:
public class Program
{
static Program()
{
LogManager.LogFactory = new NLogFactory();
}
public static void Main()
{
HostFactory.Run(x =>
{
x.Service<ProcessorAppHost>(s =>
{
s.ConstructUsing(name => new ProcessorAppHost());
s.WhenStarted(ah =>
{
ah.Init();
ah.Start("http://*:8088/");
"Message processor listening at http://localhost:8088 ".Print();
});
s.WhenStopped(ah => ah.Dispose());
});
x.RunAsLocalSystem();
x.SetDescription("Processes all messages for the Leads application.");
x.SetDisplayName("Leads Message Processor");
x.SetServiceName("LOLeadsProcessor");
});
}
}
//MQHandlers
mqServer.RegisterHandler<LeadInformation>(m =>
{
try
{
var db = container.Resolve<IFrontEndRepository>();
db.SaveMessage(m as Message);
}
catch (Exception exception)
{
_log.Error("This is the only text logged", exception);
}
return ServiceController.ExecuteMessage(m);
});
and this exception
and here is the output from the 'file' logger
2014-11-11 10:06:53.9179|ProcessorAppHost|Error|24|This is the only text logged
Any idea about how to get the overloaded version of Error to work correctly?
Thank you,
Stephen
${message} contains the message of your exception, which is "This is the only text logged".
Take a look at the exception layout renderer.
layout="${longdate}|${logger}|${level}|${threadid}|${message}|${exception}"
Have you told ServiceStack to use NLog anywhere? e.g. before initializing your AppHost:
LogManager.LogFactory = new ServiceStack.Logging.NLogger.NLogFactory();
new AppHost().Init();

NLog is not writing to database table

This is my NLog Config file.
<?xml version="1.0" ?>
<nlog autoReload="true" throwExceptions="true" internalLogLevel="Debug" internalLogToConsole="true"
internalLogFile="c:\\temp\\nlog.txt"
>
<targets>
<!--Useful for debugging-->
<target name="consolelog" type="ColoredConsole"
layout="${date:format=HH\:mm\:ss}|${level}|${stacktrace}|${message}" />
<target name="filelog" type="File" fileName="c:\\temp\\nlog1.txt"
layout="${date}: ${message}" />
<target name="eventlog" type="EventLog" source="My App" log="Application"
layout="${date}: ${message} ${stacktrace}" />
<target name="databaselog" type="Database">
<dbProvider>sqlserver</dbProvider>
<!-- database connection parameters -->
<!-- alternatively you could provide a single 'connectionstring' parameter -->
<connectionString>Data Source=.\SQLEXPRESS;Initial Catalog=SnSolutions;User Id=sa;Password=test#1234#;</connectionString>
<!--<add name="DefaultConnection" connectionString="Data Source=.\SQLEXPRESS;Initial Catalog=MyTest;User Id=mytest;Password=mytest123;" providerName="System.Data.SqlClient" />-->
<commandText>
insert into NLog_Error ([time_stamp],[level],[host],[type],1,[logger],[message],[stacktrace],[allxml]) values(#time_stamp,#level,#host,#type,#source,#logger,#message,#stacktrace,#allxml);
</commandText>
<parameter name="#time_stamp" layout="${date}" />
<parameter name="#level" layout="${level}" />
<parameter name="#host" layout="${machinename}" />
<parameter name="#type" layout="${exception:format=type}" />
<parameter name="#source" layout="${callsite:className=true:fileName=false:includeSourcePath=false:methodName=false}" />
<parameter name="#logger" layout="${logger}" />
<parameter name="#message" layout="${message}" />
<parameter name="#stacktrace" layout="${exception:stacktrace}" />
<parameter name="#allxml" layout="${web_variables}" />
</target>
</targets>
<rules>
<!--
<logger name="*" minlevel="Fatal" writeTo="eventlog" />
-->
<logger name="*" minlevel="Info" writeTo="filelog" />
<logger name="*" minlevel="Info" writeTo="databaselog" />
</rules>
</nlog>
This is my NLogLogger Class
public class NLogLogger
{
private readonly Logger _logger;
public NLogLogger(Logger logger)
{
_logger = logger;
}
public NLogLogger()
{
StackFrame frame = new StackFrame(1, false);
_logger = LogManager.GetLogger(frame.GetMethod().DeclaringType.FullName);
}
public void Trace(string message)
{
_logger.Trace(message);
}
public void Debug(string message)
{
_logger.Debug(message);
}
public void Info(string message)
{
_logger.Info(message);
}
public void Warn(string message)
{
_logger.Warn(message);
}
public void Error(string message)
{
_logger.Error(message);
}
public void Fatal(string message)
{
_logger.Fatal(message);
}
}
which I am trying to use in the following way.
NLogLogger logger = new NLogLogger();
logger.Info("We're on the Index page for Activities");
But not able to see any records in the DB nor any error in the File System.
Please let me know which is the part I am missing.
Thanks in advance.
You have an error in insert command - 1 is not a valid column name.
insert into NLog_Error (...[type],1,[logger]..)
Here is an example regarding logging to DB.

Resources