Retrieve MDC values in logback - logstash

I am trying to log the MDC values in stdout and in logstash.
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%-5level [%thread] %logger{0}: %msg -someName=%X{mdcFieldKey} %n</pattern>
<charset>utf8</charset>
</encoder>
</appender>
<appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>kibanaurl:port</destination>
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<customFields>
{
"application-name":"${spring.application.name}",
"someName":"${mdcFieldKey}",
"dummyName":"dummyValue"
}
</customFields>
<provider class="net.logstash.logback.composite.loggingevent.ArgumentsJsonProvider"/>
</encoder>
</appender>
Java code:
MDC.put("mdcFieldKey", "value to be displayed in kibana");
The STDOUT appender is displaying the mdc value correctly on kibana but for the logstash appender I see someName_IS_UNDEFINED. The hardcoded dummyname is also displayed correctly.
Why is the mdc value not populated in logstash custom field?

check here https://github.com/logstash/logstash-logback-encoder#mdc-fields
Add the line <includeMdcKeyName>mdcFieldKey</includeMdcKeyName> as below is one option. The field name will be same as mdc key in this case
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%-5level [%thread] %logger{0}: %msg -someName=%X{mdcFieldKey} %n</pattern>
<charset>utf8</charset>
</encoder>
</appender>
<appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>kibanaurl:port</destination>
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<includeMdcKeyName>mdcFieldKey</includeMdcKeyName>
<customFields>
{
"application-name":"${spring.application.name}",
"dummyName":"dummyValue"
}
</customFields>
<provider class="net.logstash.logback.composite.loggingevent.ArgumentsJsonProvider"/>
</encoder>
</appender>

Related

Printing stack trace in readable format using logstash - Newline character not processed by logback logstash JSON appender in Spring boot logger

I am trying to print the exception stack trace as:-
log.error("Error is {}" , ArrayUtils.toString(e.getStackTrace()));
This prints the entire stack trace in a single line.
I tried replacing every terminal "," character with "\n" but that doesnt work.
I also tried:
log.error("Error is \n {}" , ArrayUtils.toString(e.getStackTrace()));
But instead of printing on new line, it just prints the newline character as-is:
Error is \n java.base. ..... ,\njava.something.....
So i want to know how i can include newline in logger... along with the json appender
logback-spring.xml:
<configuration>
<!-- logs to /tmp/spring.log by default -->
<property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}/}spring.log}"/>
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
<include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
<include resource="logstash-json-appender.xml"/>
<springProfile name="prod | uat | qa | int">
<root level="INFO">
<appender-ref ref="JSON_CONSOLE"/>
</root>
</springProfile>
<springProfile name="default | dev | test | junit">
<root level="INFO">
<appender-ref ref="CONSOLE"/>
</root>
</springProfile>
</configuration>
logstash-json-appender.xml
<included>
<!--
Defines a JSON formatted appender for both the console and log files.
For configuration details, see:
-->
<springProperty scope="context" name="appName" source="spring.application.name"/>
<appender name="JSON_CONSOLE" class="net.logstash.logback.appender.LoggingEventAsyncDisruptorAppender">
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<jsonGeneratorDecorator class="net.logstash.logback.decorate.PrettyPrintingJsonGeneratorDecorator"/>
<providers>
<timestamp>
<timeZone>UTC</timeZone>
</timestamp>
<pattern>
<pattern>
{
"level": "%level",
"service": "${appName:-}",
"traceId": "%X{traceId:-}",
"spanId": "%X{spanId:-}",
"parentSpanId": "%X{parentSpanId:-}",
"pid": "${PID:-}",
"thread": "%thread",
"class": "%logger{40}",
"message": "%message"
}
</pattern>
</pattern>
</providers>
</encoder>
</appender>
</appender>
<appender name="JSON_FILE" class="net.logstash.logback.appender.LoggingEventAsyncDisruptorAppender">
<appender class="ch.qos.logback.core.rolling.RollingFileAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<jsonGeneratorDecorator class="net.logstash.logback.decorate.PrettyPrintingJsonGeneratorDecorator"/>
<timeZone>UTC</timeZone>
</encoder>
<!-- use the default spring boot conventions here, but leverage a different encoder -->
<file>${LOG_FILE}</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOGBACK_ROLLINGPOLICY_FILE_NAME_PATTERN:-${LOG_FILE}.%d{yyyy-MM-dd}.%i.gz}</fileNamePattern>
<cleanHistoryOnStart>${LOGBACK_ROLLINGPOLICY_CLEAN_HISTORY_ON_START:-false}</cleanHistoryOnStart>
<maxFileSize>${LOGBACK_ROLLINGPOLICY_MAX_FILE_SIZE:-10MB}</maxFileSize>
<totalSizeCap>${LOGBACK_ROLLINGPOLICY_TOTAL_SIZE_CAP:-0}</totalSizeCap>
<maxHistory>${LOGBACK_ROLLINGPOLICY_MAX_HISTORY:-7}</maxHistory>
</rollingPolicy>
</appender>
</appender>
</included>

UTF-8 is not working Even after charater set is configured in slf4j logback.xml

<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<target>System.out</target>
<encoder>
<charset>UTF-8</charset>
<pattern>%X{akkaTimestamp} %-5level %logger{35} - %msg%n</pattern>
</encoder>
</appender>
<appender name="FILE"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>log/akka.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>log/tests.%i.log.zip</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>250</maxIndex>
</rollingPolicy>
<triggeringPolicy
class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>20MB</maxFileSize>
</triggeringPolicy>
<encoder>
<charset>UTF-8</charset>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n
</pattern>
</encoder>
</appender>
<logger name="akka" level="INFO" />
<root level="INFO">
<appender-ref ref="CONSOLE" />
<appender-ref ref="FILE" />
</root>
Arabic characters are not coming in log file. Characters are coming as
**************************************************message: {"text":"???? ??? ?? ???? ?? ????? ?????? ??."}***********************************
It is working after adding
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<charset>UTF-8</charset>
<outputPatternAsHeader>true</outputPatternAsHeader>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n </pattern>
</encoder>
Full File
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<target>System.out</target>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<charset>UTF-8</charset>
<outputPatternAsHeader>true</outputPatternAsHeader>
<pattern>%d{yyyy-MM-dd HH:mm:ss} %-5level %logger{35} - %msg%n</pattern>
</encoder>
</appender>
<appender name="FILE"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>log/akka.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>log/tests.%i.log.zip</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>250</maxIndex>
</rollingPolicy>
<triggeringPolicy
class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>20MB</maxFileSize>
</triggeringPolicy>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<charset>UTF-8</charset>
<outputPatternAsHeader>true</outputPatternAsHeader>
<pattern>%d{yyyy-MM-dd HH:mm:ss} %-5level %logger{35} - %msg%n
</pattern>
</encoder>
</appender>
<logger name="akka" level="INFO" />
<root level="INFO">
<appender-ref ref="CONSOLE" />
<appender-ref ref="FILE" />
</root>

RootLogger to output different levels to different files

I want to have log4j output INFO level to info.log and DEBUG level to debug.log, so that with this piece of code:
logger.info("This is an info message");
logger.debug("This is a debug message");
only the first would go to info.log, while both would go to debug.log.
This doesn't work:
log4j.rootLogger=INFO, R, T
# A1 is set to be a ConsoleAppender.
log4j.appender.A1=org.apache.log4j.ConsoleAppender
# A1 uses PatternLayout.
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%r %-5p %c %x - %m%n
log4j.appender.R=org.apache.log4j.FileAppender
log4j.appender.R.File=info.log
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n
log4j.appender.R.Threshold = INFO
log4j.appender.T=org.apache.log4j.FileAppender
log4j.appender.T.File=debug.log
log4j.appender.T.layout=org.apache.log4j.PatternLayout
log4j.appender.T.layout.ConversionPattern=%p %t %c - %m%n
log4j.appender.T.Threshold = DEBUG
Neither does this:
log4j.rootLogger=INFO, R
log4j.rootLogger=DEBUG, T
log4j.appender.R=org.apache.log4j.FileAppender
log4j.appender.R.File=info.log
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n
log4j.appender.R.Threshold = INFO
log4j.appender.T=org.apache.log4j.FileAppender
log4j.appender.T.File=debug.log
log4j.appender.T.layout=org.apache.log4j.PatternLayout
log4j.appender.T.layout.ConversionPattern=%p %t %c - %m%n
log4j.appender.T.Threshold = DEBUG
You can use a filter, org.apache.log4j.varia.LevelMatchFilter. e.g.:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration PUBLIC "-//log4j/log4j Configuration//EN"
"log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<!-- INFO appender -->
<appender name="INFOFILE" class="org.apache.log4j.FileAppender">
<param name="file" value="info.log" />
<layout class="org.apache.log4j.PatternLayout">
<param name="conversionPattern"
value="%d{yyyyMMdd-HHmmss.SSS} %-5p (%c.java:%L).%M - %m%n" />
</layout>
<filter class="org.apache.log4j.varia.LevelMatchFilter">
<param name="levelToMatch" value="INFO" />
</filter>
</appender>
<!-- DEBUG appender -->
<appender name="DEBUGFILE" class="org.apache.log4j.FileAppender">
<param name="file" value="debug.log" />
<layout class="org.apache.log4j.PatternLayout">
<param name="conversionPattern"
value="%d{yyyyMMdd-HHmmss.SSS} %-5p (%c.java:%L).%M - %m%n" />
</layout>
<filter class="org.apache.log4j.varia.LevelMatchFilter">
<param name="levelToMatch" value="DEBUG" />
</filter>
</appender>
<!-- root -->
<root>
<priority value="ALL" />
<appender-ref ref="INFOFILE" />
<appender-ref ref="DEBUGFILE" />
</root>
</log4j:configuration>
Put all this in one file in the default package: log4j.xml.

Log4j : How do I lookup the datasource in log4jconf.xml

I am using weblogic for application deployment and I have created a datasource with jndi name "MyDataSource". when I try to use it in my log4j configuration, it is not working
<appender name="myDbAppender" class="org.apache.log4j.jdbc.JDBCAppender">
<param name="jndiName" value="MyDataSource"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="INSERT INTO LOGGING (user_id,
correlation_id, first_name, last_name, event_name, role,
status, access_level, message, logger, loglevel)
VALUES ( '%X{USER_ID}', '%X{CORRELATION_ID}', '%X{FIRST_NAME}',
'%X{LAST_NAME}','%X{EVENT_NAME}','%X{ROLE}','%X{STATUS}','%X
{ACCESS_LEVEL}',
'%m' , '%X{LOGGER}','%p' )"/>
</layout>
</appender>
If you want to use this feature, you need to add the jar file of Apache Extras for Apache log4j and use the class org.apache.log4j.DBAppender. e.g.:
<!-- console -->
<appender name="STDOUT" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.SimpleLayout" />
</appender>
<!-- db -->
<appender name="DBOUT" class="org.apache.log4j.DBAppender">
<connectionSource class="org.apache.log4j.receivers.db.JNDIConnectionSource">
<param name="jndiLocation" value="java:/comp/env/jdbc/MySQLDS" />
</connectionSource>
</appender>
<!-- root -->
<root>
<priority value="ALL" />
<appender-ref ref="STDOUT" />
<appender-ref ref="DBOUT" />
</root>

tomcat is logging all the tomcat related Debug logs

I have changed my tomcat logging to use logback.xml instead of util.Logger. I have changed using mentioned steps on https://github.com/grgrzybek/tomcat-slf4j-logback.
But now on starting tomcat it is logging every Debug log of tomcat.
My logback.xml looks like this:
<configuration>
<appender name="CONSOLE" class="org.apache.juli.logging.ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>
%d{HH:mm:ss.SSS} %-5level {%thread} [%logger{20}] : %msg%n
</pattern>
</encoder>
</appender>
<appender name="FILE-CATALINA" class="org.apache.juli.logging.ch.qos.logback.core.rolling.RollingFileAppender">
<file>${catalina.base}/logs/catalina.log</file>
<append>true</append>
<encoder>
<charset>utf-8</charset>
<pattern>
%d{HH:mm:ss.SSS} %-5level {%thread} [%logger{40}] : %msg%n
</pattern>
</encoder>
<rollingPolicy class="org.apache.juli.logging.ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>
${catalina.base}/logs/catalina-%d{yyyyMMdd}-%i.log.zip
</fileNamePattern>
<maxHistory>
60
<!-- days -->
</maxHistory>
<timeBasedFileNamingAndTriggeringPolicy class="org.apache.juli.logging.ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>20MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
</appender>
<appender name="FILE-LOCALHOST" class="org.apache.juli.logging.ch.qos.logback.core.rolling.RollingFileAppender">
<file>${catalina.base}/logs/localhost.log</file>
<append>true</append>
<encoder>
<charset>utf-8</charset>
<pattern>
%d{HH:mm:ss.SSS} %logger{0} {%thread} %level : %msg%n
</pattern>
</encoder>
<rollingPolicy class="org.apache.juli.logging.ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>
${catalina.base}/logs/localhost-%d{yyyyMMdd}-%i.log.zip
</fileNamePattern>
<maxHistory>
60
<!-- days -->
</maxHistory>
<timeBasedFileNamingAndTriggeringPolicy class="org.apache.juli.logging.ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>20MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
</appender>
<appender name="FILE-MANAGER" class="org.apache.juli.logging.ch.qos.logback.core.rolling.RollingFileAppender">
<file>${catalina.base}/logs/manager.log</file>
<append>true</append>
<encoder>
<charset>utf-8</charset>
<pattern>
%d{HH:mm:ss.SSS} %logger{0} {%thread} %level : %msg%n
</pattern>
</encoder>
<rollingPolicy class="org.apache.juli.logging.ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>
${catalina.base}/logs/manager-%d{yyyyMMdd}-%i.log.zip
</fileNamePattern>
<maxHistory>
60
<!-- days -->
</maxHistory>
<timeBasedFileNamingAndTriggeringPolicy class="org.apache.juli.logging.ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>20MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
</appender>
<appender name="FILE-HOST-MANAGER" class="org.apache.juli.logging.ch.qos.logback.core.rolling.RollingFileAppender">
<file>${catalina.base}/logs/host-manager.log</file>
<append>true</append>
<encoder>
<charset>utf-8</charset>
<pattern>
%d{HH:mm:ss.SSS} %logger{0} {%thread} %level : %msg%n
</pattern>
</encoder>
<rollingPolicy class="org.apache.juli.logging.ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>
${catalina.base}/logs/host-manager-%d{yyyyMMdd}-%i.log.zip
</fileNamePattern>
<maxHistory>
60
<!-- days -->
</maxHistory>
<timeBasedFileNamingAndTriggeringPolicy class="org.apache.juli.logging.ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>20MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
</appender>
<logger name="org.apache.catalina" level="INFO" additivity="false">
<appender-ref ref="FILE-CATALINA"/>
</logger>
<logger name="org.apache.catalina.core.ContainerBase.[Catalina].[localhost]" level="INFO" additivity="false">
<appender-ref ref="FILE-LOCALHOST"/>
</logger>
<logger name="org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager]" level="INFO" additivity="false">
<appender-ref ref="FILE-MANAGER"/>
</logger>
<logger name="org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager]" level="INFO" additivity="false">
<appender-ref ref="FILE-HOST-MANAGER"/>
</logger>
<root level="INFO">
<appender-ref ref="CONSOLE"/>
</root>
</configuration>
Logs are like this
07:16:09.218 [main] DEBUG o.a.tomcat.util.digester.Digester - Fire end() for SetNextRule[methodName=addLifecycleListener, paramType=org.apache.catalina.LifecycleListener]
07:16:09.218 [main] DEBUG o.a.tomcat.util.digester.Digester - [SetNextRule]{Server/Listener} Call org.apache.catalina.core.StandardServer.addLifecycleListener(org.apache.catalina.core.AprLifecycleListener#277b8705)
This is line coming when I am starting tomcat :
Using CATALINA_BASE: /mnt/production/apache-tomcat
Using CATALINA_HOME: /mnt/production/apache-tomcat
Using CATALINA_TMPDIR: /mnt/production/apache-tomcat/temp
Using JRE_HOME: /opt/bitnami/java
Using CLASSPATH: /mnt/production/apache-tomcat/bin/bootstrap.jar:/mnt/production/apache-tomcat/bin/tomcat-juli.jar
So classpath looks correct to me.

Resources