Printing stack trace in readable format using logstash - Newline character not processed by logback logstash JSON appender in Spring boot logger - logstash

I am trying to print the exception stack trace as:-
log.error("Error is {}" , ArrayUtils.toString(e.getStackTrace()));
This prints the entire stack trace in a single line.
I tried replacing every terminal "," character with "\n" but that doesnt work.
I also tried:
log.error("Error is \n {}" , ArrayUtils.toString(e.getStackTrace()));
But instead of printing on new line, it just prints the newline character as-is:
Error is \n java.base. ..... ,\njava.something.....
So i want to know how i can include newline in logger... along with the json appender
logback-spring.xml:
<configuration>
<!-- logs to /tmp/spring.log by default -->
<property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}/}spring.log}"/>
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
<include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
<include resource="logstash-json-appender.xml"/>
<springProfile name="prod | uat | qa | int">
<root level="INFO">
<appender-ref ref="JSON_CONSOLE"/>
</root>
</springProfile>
<springProfile name="default | dev | test | junit">
<root level="INFO">
<appender-ref ref="CONSOLE"/>
</root>
</springProfile>
</configuration>
logstash-json-appender.xml
<included>
<!--
Defines a JSON formatted appender for both the console and log files.
For configuration details, see:
-->
<springProperty scope="context" name="appName" source="spring.application.name"/>
<appender name="JSON_CONSOLE" class="net.logstash.logback.appender.LoggingEventAsyncDisruptorAppender">
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<jsonGeneratorDecorator class="net.logstash.logback.decorate.PrettyPrintingJsonGeneratorDecorator"/>
<providers>
<timestamp>
<timeZone>UTC</timeZone>
</timestamp>
<pattern>
<pattern>
{
"level": "%level",
"service": "${appName:-}",
"traceId": "%X{traceId:-}",
"spanId": "%X{spanId:-}",
"parentSpanId": "%X{parentSpanId:-}",
"pid": "${PID:-}",
"thread": "%thread",
"class": "%logger{40}",
"message": "%message"
}
</pattern>
</pattern>
</providers>
</encoder>
</appender>
</appender>
<appender name="JSON_FILE" class="net.logstash.logback.appender.LoggingEventAsyncDisruptorAppender">
<appender class="ch.qos.logback.core.rolling.RollingFileAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<jsonGeneratorDecorator class="net.logstash.logback.decorate.PrettyPrintingJsonGeneratorDecorator"/>
<timeZone>UTC</timeZone>
</encoder>
<!-- use the default spring boot conventions here, but leverage a different encoder -->
<file>${LOG_FILE}</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOGBACK_ROLLINGPOLICY_FILE_NAME_PATTERN:-${LOG_FILE}.%d{yyyy-MM-dd}.%i.gz}</fileNamePattern>
<cleanHistoryOnStart>${LOGBACK_ROLLINGPOLICY_CLEAN_HISTORY_ON_START:-false}</cleanHistoryOnStart>
<maxFileSize>${LOGBACK_ROLLINGPOLICY_MAX_FILE_SIZE:-10MB}</maxFileSize>
<totalSizeCap>${LOGBACK_ROLLINGPOLICY_TOTAL_SIZE_CAP:-0}</totalSizeCap>
<maxHistory>${LOGBACK_ROLLINGPOLICY_MAX_HISTORY:-7}</maxHistory>
</rollingPolicy>
</appender>
</appender>
</included>

Related

Retrieve MDC values in logback

I am trying to log the MDC values in stdout and in logstash.
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%-5level [%thread] %logger{0}: %msg -someName=%X{mdcFieldKey} %n</pattern>
<charset>utf8</charset>
</encoder>
</appender>
<appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>kibanaurl:port</destination>
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<customFields>
{
"application-name":"${spring.application.name}",
"someName":"${mdcFieldKey}",
"dummyName":"dummyValue"
}
</customFields>
<provider class="net.logstash.logback.composite.loggingevent.ArgumentsJsonProvider"/>
</encoder>
</appender>
Java code:
MDC.put("mdcFieldKey", "value to be displayed in kibana");
The STDOUT appender is displaying the mdc value correctly on kibana but for the logstash appender I see someName_IS_UNDEFINED. The hardcoded dummyname is also displayed correctly.
Why is the mdc value not populated in logstash custom field?
check here https://github.com/logstash/logstash-logback-encoder#mdc-fields
Add the line <includeMdcKeyName>mdcFieldKey</includeMdcKeyName> as below is one option. The field name will be same as mdc key in this case
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%-5level [%thread] %logger{0}: %msg -someName=%X{mdcFieldKey} %n</pattern>
<charset>utf8</charset>
</encoder>
</appender>
<appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>kibanaurl:port</destination>
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<includeMdcKeyName>mdcFieldKey</includeMdcKeyName>
<customFields>
{
"application-name":"${spring.application.name}",
"dummyName":"dummyValue"
}
</customFields>
<provider class="net.logstash.logback.composite.loggingevent.ArgumentsJsonProvider"/>
</encoder>
</appender>

RootLogger to output different levels to different files

I want to have log4j output INFO level to info.log and DEBUG level to debug.log, so that with this piece of code:
logger.info("This is an info message");
logger.debug("This is a debug message");
only the first would go to info.log, while both would go to debug.log.
This doesn't work:
log4j.rootLogger=INFO, R, T
# A1 is set to be a ConsoleAppender.
log4j.appender.A1=org.apache.log4j.ConsoleAppender
# A1 uses PatternLayout.
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%r %-5p %c %x - %m%n
log4j.appender.R=org.apache.log4j.FileAppender
log4j.appender.R.File=info.log
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n
log4j.appender.R.Threshold = INFO
log4j.appender.T=org.apache.log4j.FileAppender
log4j.appender.T.File=debug.log
log4j.appender.T.layout=org.apache.log4j.PatternLayout
log4j.appender.T.layout.ConversionPattern=%p %t %c - %m%n
log4j.appender.T.Threshold = DEBUG
Neither does this:
log4j.rootLogger=INFO, R
log4j.rootLogger=DEBUG, T
log4j.appender.R=org.apache.log4j.FileAppender
log4j.appender.R.File=info.log
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n
log4j.appender.R.Threshold = INFO
log4j.appender.T=org.apache.log4j.FileAppender
log4j.appender.T.File=debug.log
log4j.appender.T.layout=org.apache.log4j.PatternLayout
log4j.appender.T.layout.ConversionPattern=%p %t %c - %m%n
log4j.appender.T.Threshold = DEBUG
You can use a filter, org.apache.log4j.varia.LevelMatchFilter. e.g.:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration PUBLIC "-//log4j/log4j Configuration//EN"
"log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<!-- INFO appender -->
<appender name="INFOFILE" class="org.apache.log4j.FileAppender">
<param name="file" value="info.log" />
<layout class="org.apache.log4j.PatternLayout">
<param name="conversionPattern"
value="%d{yyyyMMdd-HHmmss.SSS} %-5p (%c.java:%L).%M - %m%n" />
</layout>
<filter class="org.apache.log4j.varia.LevelMatchFilter">
<param name="levelToMatch" value="INFO" />
</filter>
</appender>
<!-- DEBUG appender -->
<appender name="DEBUGFILE" class="org.apache.log4j.FileAppender">
<param name="file" value="debug.log" />
<layout class="org.apache.log4j.PatternLayout">
<param name="conversionPattern"
value="%d{yyyyMMdd-HHmmss.SSS} %-5p (%c.java:%L).%M - %m%n" />
</layout>
<filter class="org.apache.log4j.varia.LevelMatchFilter">
<param name="levelToMatch" value="DEBUG" />
</filter>
</appender>
<!-- root -->
<root>
<priority value="ALL" />
<appender-ref ref="INFOFILE" />
<appender-ref ref="DEBUGFILE" />
</root>
</log4j:configuration>
Put all this in one file in the default package: log4j.xml.

Log4j : How do I lookup the datasource in log4jconf.xml

I am using weblogic for application deployment and I have created a datasource with jndi name "MyDataSource". when I try to use it in my log4j configuration, it is not working
<appender name="myDbAppender" class="org.apache.log4j.jdbc.JDBCAppender">
<param name="jndiName" value="MyDataSource"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="INSERT INTO LOGGING (user_id,
correlation_id, first_name, last_name, event_name, role,
status, access_level, message, logger, loglevel)
VALUES ( '%X{USER_ID}', '%X{CORRELATION_ID}', '%X{FIRST_NAME}',
'%X{LAST_NAME}','%X{EVENT_NAME}','%X{ROLE}','%X{STATUS}','%X
{ACCESS_LEVEL}',
'%m' , '%X{LOGGER}','%p' )"/>
</layout>
</appender>
If you want to use this feature, you need to add the jar file of Apache Extras for Apache log4j and use the class org.apache.log4j.DBAppender. e.g.:
<!-- console -->
<appender name="STDOUT" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.SimpleLayout" />
</appender>
<!-- db -->
<appender name="DBOUT" class="org.apache.log4j.DBAppender">
<connectionSource class="org.apache.log4j.receivers.db.JNDIConnectionSource">
<param name="jndiLocation" value="java:/comp/env/jdbc/MySQLDS" />
</connectionSource>
</appender>
<!-- root -->
<root>
<priority value="ALL" />
<appender-ref ref="STDOUT" />
<appender-ref ref="DBOUT" />
</root>

tomcat is logging all the tomcat related Debug logs

I have changed my tomcat logging to use logback.xml instead of util.Logger. I have changed using mentioned steps on https://github.com/grgrzybek/tomcat-slf4j-logback.
But now on starting tomcat it is logging every Debug log of tomcat.
My logback.xml looks like this:
<configuration>
<appender name="CONSOLE" class="org.apache.juli.logging.ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>
%d{HH:mm:ss.SSS} %-5level {%thread} [%logger{20}] : %msg%n
</pattern>
</encoder>
</appender>
<appender name="FILE-CATALINA" class="org.apache.juli.logging.ch.qos.logback.core.rolling.RollingFileAppender">
<file>${catalina.base}/logs/catalina.log</file>
<append>true</append>
<encoder>
<charset>utf-8</charset>
<pattern>
%d{HH:mm:ss.SSS} %-5level {%thread} [%logger{40}] : %msg%n
</pattern>
</encoder>
<rollingPolicy class="org.apache.juli.logging.ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>
${catalina.base}/logs/catalina-%d{yyyyMMdd}-%i.log.zip
</fileNamePattern>
<maxHistory>
60
<!-- days -->
</maxHistory>
<timeBasedFileNamingAndTriggeringPolicy class="org.apache.juli.logging.ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>20MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
</appender>
<appender name="FILE-LOCALHOST" class="org.apache.juli.logging.ch.qos.logback.core.rolling.RollingFileAppender">
<file>${catalina.base}/logs/localhost.log</file>
<append>true</append>
<encoder>
<charset>utf-8</charset>
<pattern>
%d{HH:mm:ss.SSS} %logger{0} {%thread} %level : %msg%n
</pattern>
</encoder>
<rollingPolicy class="org.apache.juli.logging.ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>
${catalina.base}/logs/localhost-%d{yyyyMMdd}-%i.log.zip
</fileNamePattern>
<maxHistory>
60
<!-- days -->
</maxHistory>
<timeBasedFileNamingAndTriggeringPolicy class="org.apache.juli.logging.ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>20MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
</appender>
<appender name="FILE-MANAGER" class="org.apache.juli.logging.ch.qos.logback.core.rolling.RollingFileAppender">
<file>${catalina.base}/logs/manager.log</file>
<append>true</append>
<encoder>
<charset>utf-8</charset>
<pattern>
%d{HH:mm:ss.SSS} %logger{0} {%thread} %level : %msg%n
</pattern>
</encoder>
<rollingPolicy class="org.apache.juli.logging.ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>
${catalina.base}/logs/manager-%d{yyyyMMdd}-%i.log.zip
</fileNamePattern>
<maxHistory>
60
<!-- days -->
</maxHistory>
<timeBasedFileNamingAndTriggeringPolicy class="org.apache.juli.logging.ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>20MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
</appender>
<appender name="FILE-HOST-MANAGER" class="org.apache.juli.logging.ch.qos.logback.core.rolling.RollingFileAppender">
<file>${catalina.base}/logs/host-manager.log</file>
<append>true</append>
<encoder>
<charset>utf-8</charset>
<pattern>
%d{HH:mm:ss.SSS} %logger{0} {%thread} %level : %msg%n
</pattern>
</encoder>
<rollingPolicy class="org.apache.juli.logging.ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>
${catalina.base}/logs/host-manager-%d{yyyyMMdd}-%i.log.zip
</fileNamePattern>
<maxHistory>
60
<!-- days -->
</maxHistory>
<timeBasedFileNamingAndTriggeringPolicy class="org.apache.juli.logging.ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>20MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
</appender>
<logger name="org.apache.catalina" level="INFO" additivity="false">
<appender-ref ref="FILE-CATALINA"/>
</logger>
<logger name="org.apache.catalina.core.ContainerBase.[Catalina].[localhost]" level="INFO" additivity="false">
<appender-ref ref="FILE-LOCALHOST"/>
</logger>
<logger name="org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager]" level="INFO" additivity="false">
<appender-ref ref="FILE-MANAGER"/>
</logger>
<logger name="org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager]" level="INFO" additivity="false">
<appender-ref ref="FILE-HOST-MANAGER"/>
</logger>
<root level="INFO">
<appender-ref ref="CONSOLE"/>
</root>
</configuration>
Logs are like this
07:16:09.218 [main] DEBUG o.a.tomcat.util.digester.Digester - Fire end() for SetNextRule[methodName=addLifecycleListener, paramType=org.apache.catalina.LifecycleListener]
07:16:09.218 [main] DEBUG o.a.tomcat.util.digester.Digester - [SetNextRule]{Server/Listener} Call org.apache.catalina.core.StandardServer.addLifecycleListener(org.apache.catalina.core.AprLifecycleListener#277b8705)
This is line coming when I am starting tomcat :
Using CATALINA_BASE: /mnt/production/apache-tomcat
Using CATALINA_HOME: /mnt/production/apache-tomcat
Using CATALINA_TMPDIR: /mnt/production/apache-tomcat/temp
Using JRE_HOME: /opt/bitnami/java
Using CLASSPATH: /mnt/production/apache-tomcat/bin/bootstrap.jar:/mnt/production/apache-tomcat/bin/tomcat-juli.jar
So classpath looks correct to me.

In Log4J, why %C in ConversionPattern prints '?' (question mark) with AsyncAppender?

I have a trouble when using %C in ConversionPattern with AsyncAppender.
My Lo4J configuration is:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="console" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yyyy/MM/dd HH:mm:ss,SSS} %C{1} - %m%n" />
</layout>
</appender>
<appender name="async_console" class="org.apache.log4j.AsyncAppender">
<param name="BufferSize" value="1000" />
<appender-ref ref="console" />
</appender>
<root>
<level value="debug" />
<!--
<appender-ref ref="console" />
-->
<appender-ref ref="async_console" />
</root>
</log4j:configuration>
And my test code is:
#Test
public void testAsync() {
DOMConfigurator
.configure("src/test/resources/learningtest/log4j/log4j_test_async.xml");
Logger log = Logger.getLogger(getClass());
log.debug("Hello, world!");
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
The result of the test code is:
2012/03/15 11:51:22,570 ? - Hello, world!
Without AsynAppender, it works fine:
2012/03/15 11:51:06,002 Log4jTest - Hello, world!
With %c (category), it works fine, too.
What am I missing?
Please let me know.
Thanks in advance :-)
Reference:
http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html
When using "%C" or "%M", log4J uses Throwable.getStackTrace to get the stackTrace and use this information to get the caller class and method.
The problem is that when using an AsyncAppender, the Throwable is created in another thread and the stackTrace does not contain the caller method.

Resources