I'm logging an Azure-based webapp using a Log4Net CSV appender with:
I'm seeing multiple entries with an identical timestamp - clearly not logging the actual instant of a given event on the server:
2018-03-19 21:59:52.000 OrderId: 191096 Starts to validate, multi:
2018-03-19 21:59:52.000 OrderId: 191096 validation request:
2018-03-19 21:59:52.000 OrderId: 191096 passed validation. AuthKey:6128994
2018-03-19 21:59:52.000 OrderId: 191096 Single starts
2018-03-19 21:59:52.000 OrderId: 191096 submits:
2018-03-19 21:59:52.000 SaveOrderChanges: 191096
2018-03-19 21:59:52.000 SaveOrderChanges: 191096
I had thought perhaps it take to do with when the logs are written out to file vs. when the entry is literally generated but unless I'm mis-reading the context, this answer indicates otherwise.
Clearly I have something misconfigured. My CSV is built using code found at: http://element533.blogspot.com/2010/05/writing-to-csv-using-log4net.html
Full appender:
<appender name="CsvFileAppender" type="log4net.Appender.FileAppender">
<file value="D:/home/logfiles/log4netCSV.log" />
<lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
<appendToFile value="true"/>
<threshold value="INFO" />
<layout type=" myWeb.CsvPatternLayout, myWeb">
<header value="DateTime,Thread,Level,Logger,Message,Exception
" />
<conversionPattern value="%date%newfield[%thread]%newfield %-5level%newfield% %property{Ip} _+ %aspnet-request{ASP.NET_SessionId} _+ %logger %newfield%message%newfield%exception%endrow" />
</layout>
</appender>
It is very strange that your dates are showing
2018-03-19 21:59:52.000
The default format is Iso8601 and separator between second and milisec is a comma:
https://github.com/apache/logging-log4net/blob/master/src/DateFormatter/Iso8601DateFormatter.cs#L26
I recommend you to use explicit dates:
%date{yyyy-MM-dd HH:mm:ss.fff}
Also if you want a CSV file you need to separate the values with commas:
<conversionPattern value="%date{yyyy-MM-dd HH:mm:ss.fff},[%thread],%level,..." />
Update:
I just tested this on Azure to see if it had anything to do, and it shows correctly:
http://swagger-net-test.azurewebsites.net/log4net.log
I have two log actions one right after the other and they do show different time stamps
DateTime,Thread,Level,Logger,Message
2018-04-08 13:19:48.658,[20],INFO,Swagger_Test.Controllers.LogController,Test1
2018-04-08 13:19:48.689,[20],ERROR,Swagger_Test.Controllers.LogController,Test2
Related
I am running this command as admin in elevated prompt:
%systemroot%\system32\inetsrv\appcmd list config "website" /section:requestFiltering /text:AllowDoubleEscaping
It returns error message: ERROR (message:Unknown attribute ""AllowDoubleEscaping"". Replace with -? for help.)
So I next ran the following command:
%systemroot%\system32\inetsrv\appcmd set config -section:requestFiltering -?
It returned this output and yes, I can see that the allowDoubleEscaping is missing
ERROR ( message:-allowHighBitCharacters
-unescapeQueryString
-removeServerHeader
-fileExtensions.allowUnlisted
-fileExtensions.applyToWebDAV
-fileExtensions.[fileExtension='string'].fileExtension
-fileExtensions.[fileExtension='string'].allowed
-requestLimits.maxAllowedContentLength
-requestLimits.maxUrl
-requestLimits.maxQueryString
-requestLimits.headerLimits.[header='string'].header
-requestLimits.headerLimits.[header='string'].sizeLimit
-verbs.allowUnlisted
-verbs.applyToWebDAV
-verbs.[verb='string'].verb
-verbs.[verb='string'].allowed
-hiddenSegments.applyToWebDAV
-hiddenSegments.[segment='string'].segment
-alwaysAllowedUrls.[url='string'].url
-alwaysAllowedQueryStrings.[queryString='string'].queryString
-denyUrlSequences.[sequence='string'].sequence
-denyQueryStringSequences.[sequence='string'].sequence
-filteringRules.[name='string'].name
-filteringRules.[name='string'].scanUrl
-filteringRules.[name='string'].scanQueryString
-filteringRules.[name='string'].scanAllRaw
-filteringRules.[name='string'].denyUnescapedPercent
-filteringRules.[name='string'].scanHeaders.[requestHeader='string'].requestHeader
-filteringRules.[name='string'].appliesTo.[fileExtension='string'].fileExtension
-filteringRules.[name='string'].denyStrings.[string='string'].string
So which files is appcmd actually checking for these? I went ahead and checked the C:\Windows\System32\inetsrv\config\schema\IIS_schema.xml file and it does have this attribute defined in there. This seems to be the only place this is defined, so I am confused where else its not finding this attribute to throw the error??
<sectionSchema name="system.webServer/security/requestFiltering">
<attribute name="allowDoubleEscaping" type="bool" defaultValue="false" />
<attribute name="allowHighBitCharacters" type="bool" defaultValue="true" />
<attribute name="unescapeQueryString" type="bool" defaultValue="true" />
In: NLog.Targets.Splunk
https://github.com/AlanBarber/NLog.Targets.Splunk
When use the nlog configuration with:
includeEventProperties="true"
or if I have:
includeEventProperties="false" and use:
<contextproperty name="host" layout="${machinename}" />
<contextproperty name="threadid" layout="${threadid}" />
<contextproperty name="logger" layout="${logger}" />
I get the logs in the following format (properties wrapped in "Properties"):
{"Level":"Info","MessageTemplate":"ApiRequest","RenderedMessage":"ApiRequest","Properties":{"httpMethod":"GET","statusCode":200}, ...}
Is it possible to get rid of the Properties-wrap, and have it more flat?
{ "Level": "Info", "httpMethod": "GET", "statusCode":200, ... }
Many thanks! :-)
I have little problem with JsonLayout
NLog version: 4.7.10
Platform: netcoreapp 3.1
Current Nlog config
<target name="jsonFileMw" xsi:type="File" fileName="logs\mw.log"
archiveAboveSize="10240"
maxArchiveDays="5"
archiveNumbering="DateAndSequence"
archiveEvery="Day"
enableArchiveFileCompression="true">
<layout xsi:type="JsonLayout" includeAllProperties="true">
<attribute name="time" layout="${longdate}" />
<attribute name="level" layout="${level:upperCase=true}"/>
<attribute name="message" layout="${message}" />
</layout>
</target>
my logging code
_logger.LogInformation("request received. {RequestUrl} {RequestBody}", "some url", "some body");
this logging code produces following log line:
{ "time": "2021-08-02 15:07:30.8198", "level": "INFO", "message": "request received. some url some body", "RequestUrl": "some url", "RequestBody": "some body" }
As you can see this adds log properties to message also which means logging same information twice. As a result log file size increases. I just want to keep message simple. Desired output is below:
{ "time": "2021-08-02 15:07:30.8198", "level": "INFO", "message": "request received. {RequestUrl} {RequestBody}", "RequestUrl": "some url", "RequestBody": "some body" }
How can I achieve this?
You can do this:
<attribute name="messagetemplate" layout="${message:raw=true}" />
See also: https://github.com/NLog/NLog/wiki/Message-Layout-Renderer
See also: https://github.com/NLog/NLog/wiki/How-to-use-structured-logging#output-captured-properties
I'm currently using log4net and azure files to store my logs, works ace.
I've been searching and can't find any configuration to make the logger create files no bigger than a given KB size.
This is the configuration I have:
<rollingStyle value="Size" />
<MaxSizeRollBackups value="10" />
<MaximumFileSize value="10KB" />
<AzureStorageConnectionString value="connectiondatahere" />
<ShareName value="filelog" />
<Path value="processor" />
<File value="processor_{yyyy-MM-dd}.txt" />
<layout type="log4net.Layout.PatternLayout">
<ConversionPattern value="%date %-5level %logger %message%newline"/>
</layout>
</appender>
<root>
<level value="ALL" />
<appender-ref ref="AzureFileAppender"/>
</root>
I've tried a few variations of this configuration but no luck.
After reviewed the source code of log4net-appender-azurefilestorage, I found the log file size limit is not support in azure file appender currently. I suggest you rewrite the azure file appender by yourself and add the size limit feature.
Below are the steps to do it.
Step 1, Add a property named MaximumFileSize to AzureFileAppender class.
public int MaximumFileSize { get; set; }
Step 2, add the size limit code when appending log to file.
protected override void Append(LoggingEvent loggingEvent)
{
Initialise(loggingEvent);
var buffer = Encoding.UTF8.GetBytes(RenderLoggingEvent(loggingEvent));
if ((_file.Properties.Length + buffer.Length) > MaximumFileSize)
{
//do something if the file reach the max file size
}
else
{
_file.Resize(_file.Properties.Length + buffer.Length);
using (var fileStream = _file.OpenWrite(null))
{
fileStream.Seek(buffer.Length * -1, SeekOrigin.End);
fileStream.Write(buffer, 0, buffer.Length);
}
}
}
Step 3, After that, you could add the size limit(per byte) to configuration file.
<MaximumFileSize value="10240" />
In my config file, I use
input { log4j {} }
and:
output { stdout { codec => rubydebug } }
I've attached my log4j to logstash using SocketListener. When my app prints something to the log, I see in logstash:
{
"message" => "<the message>",
"#version" => "1",
"#timestamp" => "2015-06-05T20:28:23.312Z",
"type" => "log4j",
"host" => "127.0.0.1:52083",
"path" => "com.ohadr.logs_provider.MyServlet",
"priority" => "INFO",
"logger_name" => "com.ohadr.logs_provider.MyServlet",
"thread" => "http-apr-8080-exec-3",
"class" => "?",
"file" => "?:?",
"method" => "?",
}
the issue is that the "path" field is wrong: AFAI understand, it should be the path of the log file; instead, I get the same value as "logger_name".
I have several apps on my tomcat that I want to collect the logs from. I need "path" to be the path-of-file (including the file-name), so I can distinguish between logs from different apps (each app logs to a different file).
How can it be done?
thanks!
The log4j input is a listener on a TCP socket. There is no file path.
To solve your challenge, you can either configure multiple TCP ports, so every application logs to a different TCP port or you could use GELF. GELF is an UDP-based protocol, but you need additional jars. logstash supports also GELF as native input. You can specify in many GELF appenders static fields, so you can distinguish on application level, which application is currently logging.
You can find here an example:
<appender name="gelf" class="biz.paluch.logging.gelf.log4j.GelfLogAppender">
<param name="Threshold" value="INFO" />
<param name="Host" value="udp:localhost" />
<param name="Port" value="12201" />
<param name="Version" value="1.1" />
<param name="Facility" value="java-test" />
<param name="ExtractStackTrace" value="true" />
<param name="FilterStackTrace" value="true" />
<param name="MdcProfiling" value="true" />
<param name="TimestampPattern" value="yyyy-MM-dd HH:mm:ss,SSSS" />
<param name="MaximumMessageSize" value="8192" />
<!-- This are static fields -->
<param name="AdditionalFields" value="fieldName1=fieldValue1,fieldName2=fieldValue2" />
<!-- This are fields using MDC -->
<param name="MdcFields" value="mdcField1,mdcField2" />
<param name="DynamicMdcFields" value="mdc.*,(mdc|MDC)fields" />
<param name="IncludeFullMdc" value="true" />
</appender>
HTH, Mark