Stop log4j from logging into syslog - log4j

I am trying to configure log4j2 to write logs in several files using Routing & RollingFile appenders, and so far everything works well. However, there is an unwanted side effect - it also logs into Linux syslog. Is there any way to disable logging into syslog? Here is my log4j2.xml:
<?xml version="1.0" encoding="UTF-8" ?>
<Configuration name="Staging" status="warn">
<Properties>
<Property name="pattern" value="%-5level [%d{DEFAULT}] %logger{30} %X %marker %msg%n"/>
<Property name="log-path" value="/var/log/application"/>
<Property name="file-size" value="200 MB"/>
<Property name="file-total" value="10"/>
</Properties>
<Appenders>
<RollingFile name="SESSIONS_APPENDER" fileName="${log-path}/sessions.log" filePattern="${log-path}/sessions-%i.log.gz" append="true">
<PatternLayout pattern="${pattern}"/>
<Policies>
<SizeBasedTriggeringPolicy size="${file-size}"/>
</Policies>
<DefaultRolloverStrategy max="${file-total}"/>
</RollingFile>
<RollingFile name="TRANSACTIONS_APPENDER" fileName="${log-path}/transactions.log" filePattern="${log-path}/transactions-%i.log.gz" append="true">
<PatternLayout pattern="${pattern}"/>
<Policies>
<SizeBasedTriggeringPolicy size="${file-size}"/>
</Policies>
<DefaultRolloverStrategy max="${file-total}"/>
</RollingFile>
<RollingFile name="STDROUT_APPENDER" fileName="${log-path}/stdrout.log" filePattern="${log-path}/stdrout-%i.log.gz" append="true">
<PatternLayout pattern="${pattern}"/>
<Policies>
<SizeBasedTriggeringPolicy size="${file-size}"/>
</Policies>
<DefaultRolloverStrategy max="${file-total}"/>
<!-- Filtering from the stdout log, lines that have their own custom appender -->
<Filters>
<MarkerFilter marker="TRANSACTION" onMatch="DENY" onMismatch="NEUTRAL"/>
<MarkerFilter marker="SESSION" onMatch="DENY" onMismatch="NEUTRAL"/>
</Filters>
</RollingFile>
<Routing>
<name>ROUTING_APPENDER</name>
<Routes>
<pattern>$${marker:}</pattern>
<!-- Default appender if no matching marker found -->
<Route ref="STDROUT_APPENDER"/>
<!-- Appender selection for specific marker -->
<Route key="TRANSACTION" ref="TRANSACTIONS_APPENDER"/>
<Route key="SESSION" ref="SESSIONS_APPENDER"/>
</Routes>
</Routing>
</Appenders>
<Loggers>
<Root level="info">
<AppenderRef ref="STDROUT_APPENDER"/>
</Root>
<Logger name="com.application" level="info" additivity="false">
<AppenderRef ref="ROUTING_APPENDER"/>
</Logger>
<Logger name="org.apache" level="error">
<AppenderRef ref="STDROUT_APPENDER"/>
</Logger>
<Logger name="com.amazonaws" level="error">
<AppenderRef ref="STDROUT_APPENDER"/>
</Logger>
</Loggers>
</Configuration>

Related

How to overwrite the SAME file every hour in log4j2

I have a very simple requirement to write DEBUG logs to a file but shouldn't keep that file for more than an hour due to audit purposes. So, is there a way to overwrite the same file (file_debug) every hour automatically?
I've seen there is "delete" in log4j2 as below but don't want to use this due to security issues. Also, append="false" won't work for me as I need to retain events for an hour - not overwrite every event immediately.
<DefaultRolloverStrategy>
<Delete basePath="${baseDir}" maxDepth="2">
<IfFileName glob="*/app-*.log.gz" />
<IfLastModified age="P60D" />
</Delete>
</DefaultRolloverStrategy>
My current log4j2.xml
<?xml version="1.0" encoding="utf-8"?>
<Configuration>
<Appenders>
<RollingFile name="file" fileName="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}log4j_poc.log" filePattern="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}log4j_poc_%i.log">
<PatternLayout pattern="%-5p %d [%t] [processor: %X{processorPath}; event: %X{correlationId}] %c: %m%n"/>
<SizeBasedTriggeringPolicy size="10 MB"/>
<DefaultRolloverStrategy max="10"/>
</RollingFile>
<RollingFile name="file_debug" fileName="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}log4j_poc_debug.log" filePattern="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}log4j_poc_debug.log">
<Filters>
<ThresholdFilter level="TRACE"/>
<ThresholdFilter level="INFO" onMatch="DENY" onMismatch="NEUTRAL"/>
</Filters>
<PatternLayout pattern="%-5p %d [%t] [processor: %X{processorPath}; event: %X{correlationId}] %c: %m%n"/>
<Policies>
<SizeBasedTriggeringPolicy size="10 MB"/>
<CronTriggeringPolicy schedule="0 0 * * * ?"/>
</Policies>
<DefaultRolloverStrategy max="0"/>
</RollingFile>
</Appenders>
<Loggers>
<!-- Http Logger shows wire traffic on DEBUG. -->
<AsyncLogger name="org.mule.service.http.impl.service.HttpMessageLogger" level="DEBUG" />
<AsyncLogger name="org.mule.service.http" level="WARN"/>
<AsyncLogger name="org.mule.extension.http" level="WARN"/>
<AsyncLogger name="org.mule.extension.db" level="DEBUG"/>
<AsyncLogger name="org.mule.db.commons" level="DEBUG"/>
<!-- Mule logger -->
<AsyncLogger name="org.mule.runtime.core.internal.processor.LoggerMessageProcessor" level="INFO"/>
<AsyncRoot level="INFO">
<AppenderRef ref="file" level="INFO"/>
<AppenderRef ref="file_debug" level="TRACE"/>
</AsyncRoot>
</Loggers>
Please suggest if there is any option to overwrite the same file "file_debug" every hour?

log4j2 AsyncAppender occured a memory leak

log4j2 version : 2.7
My project is a high concurrent system, I use log4j2 asyncAppender to save logs, but every nigh at 00:00:00 when the rollover strategy triggering, the event blocked at rollover process.
the appender create lots of threads and occured a memory leak, here is my config:
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="info">
<Properties>
<Property name="logDir">/data/logs/q-mix</Property>
<Property name="rollingSuffix">log.gz</Property>
<Property name="CONSOLE_LOG_PATTERN">%clr{%d{yyyy-MM-dd HH:mm:ss.SSS}}{faint} %clr{%5p} %clr{%5.5T}{magenta} %clr{---}{faint} %clr{[%20.20t]}{faint} %clr{%-40.40c{1.}}{cyan} %clr{:}{faint} %m%n</Property>
<Property name="FILE_LOG_PATTERN">%d{yyyy-MM-dd HH:mm:ss.SSS} %5p %5.5T --- [%20.20t] %-40.40c{1.} : %m%n</Property>
</Properties>
<appenders>
<console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="${CONSOLE_LOG_PATTERN}" />
</console>
<RollingFile name="RollingFileInfo" fileName="${logDir}/info.log"
filePattern="${logDir}/logs/%d{yyyy-MM-dd}-%i-info.${rollingSuffix}">
<Filters>
<ThresholdFilter level="INFO"/>
</Filters>
<PatternLayout pattern="${FILE_LOG_PATTERN}"/>
<Policies>
<TimeBasedTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="2048 MB"/>
</Policies>
<DefaultRolloverStrategy max="80">
<Delete basePath="${logDir}/logs/" maxDepth="1">
<IfFileName glob="*.${rollingSuffix}">
<IfAny>
<IfAccumulatedFileSize exceeds="50 GB" />
<IfLastModified age="15d" />
</IfAny>
</IfFileName>
</Delete>
</DefaultRolloverStrategy>
</RollingFile>
<RollingFile name="RollingFileWarn" fileName="${logDir}/warn.log"
filePattern="${logDir}/logs/%d{yyyy-MM-dd}-%i-warn.${rollingSuffix}">
<Filters>
<ThresholdFilter level="WARN"/>
</Filters>
<PatternLayout pattern="${FILE_LOG_PATTERN}"/>
<Policies>
<TimeBasedTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="1024 MB"/>
</Policies>
<DefaultRolloverStrategy max="40"/>
</RollingFile>
<RollingFile name="RollingFileError" fileName="${logDir}/error.log"
filePattern="${logDir}/logs/%d{yyyy-MM-dd}-%i-error.${rollingSuffix}">
<ThresholdFilter level="ERROR"/>
<PatternLayout pattern="${FILE_LOG_PATTERN}"/>
<Policies>
<TimeBasedTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="1024 MB"/>
</Policies>
<DefaultRolloverStrategy max="40"/>
</RollingFile>
<RollingFile name="RollingFileAlarm" fileName="${logDir}/alarm.log"
filePattern="${logDir}/logs/%d{yyyy-MM-dd}-%i-alarm.${rollingSuffix}">
<PatternLayout pattern="${FILE_LOG_PATTERN}"/>
<Policies>
<TimeBasedTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="1024 MB"/>
</Policies>
<DefaultRolloverStrategy max="40"/>
</RollingFile>
<Async name="async" bufferSize="1024000">
<!-- <appender-ref ref="Console"/>-->
<appender-ref ref="RollingFileInfo"/>
<!-- <appender-ref ref="RollingFileWarn"/>-->
<appender-ref ref="RollingFileError"/>
</Async>
</appenders>
<loggers>
<logger name="com.upex.exchange.robot.util.AlarmUtils" level="error">
<appender-ref ref="RollingFileAlarm"/>
</logger>
<root level="info">
<appender-ref ref="async"/>
</root>
</loggers>
</configuration>
Here is the stacktrace:
AsyncAppender-async
at sun.misc.Unsafe.park(ZJ)V (Native Method)
at java.util.concurrent.locks.LockSupport.park(Ljava/lang/Object;)V (LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()Z (AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(I)V (AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(I)V (AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.Semaphore.acquire()V (Semaphore.java:312)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.rollover(Lorg/apache/logging/log4j/core/appender/rolling/RolloverStrategy;)Z (RollingFileManager.java:247)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.rollover()V (RollingFileManager.java:192)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.checkRollover(Lorg/apache/logging/log4j/core/LogEvent;)V (RollingFileManager.java:175)
at org.apache.logging.log4j.core.appender.RollingFileAppender.append(Lorg/apache/logging/log4j/core/LogEvent;)V (RollingFileAppender.java:280)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(Lorg/apache/logging/log4j/core/LogEvent;)V (AppenderControl.java:156)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(Lorg/apache/logging/log4j/core/LogEvent;)V (AppenderControl.java:129)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Lorg/apache/logging/log4j/core/LogEvent;)V (AppenderControl.java:120)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(Lorg/apache/logging/log4j/core/LogEvent;)V (AppenderControl.java:84)
at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.callAppenders(Lorg/apache/logging/log4j/core/LogEvent;)Z (AsyncAppender.java:451)
at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.run()V (AsyncAppender.java:404)
This is the Eclipse Memory Analyzer leak report:
One instance of "java.util.concurrent.ArrayBlockingQueue" loaded by "<system class loader>" occupies 897,341,688 (95.23%) bytes.
The instance is referenced by org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread # 0xc0700150 AsyncAppender-async ,
loaded by "org.springframework.boot.loader.LaunchedURLClassLoader # 0xc0400000".
The memory is accumulated in one instance of "java.lang.Object[]" loaded by "<system class loader>".
The stacktrace of this Thread is available. See stacktrace.
Keywords
java.util.concurrent.ArrayBlockingQueue
java.lang.Object[]
org.springframework.boot.loader.LaunchedURLClassLoader # 0xc0400000
How can i change some configs to resolve this problem, I realy need some help. Thanks a lot.
Sorry for my english grammar
According to log4j's sugguest I upgrade the log4j2 to version 2.14
LMAX Disruptor technology. Asynchronous Loggers internally use the Disruptor, a lock-free inter-thread communication library, instead of queues, resulting in higher throughput and lower latency.
the new config xml like this:
<appenders>
<RollingRandomAccessFile name="RollingFileWarn" fileName="${logDir}/warn.log"
filePattern="${logDir}/logs/%d{yyyy-MM-dd}-%i-warn.${rollingSuffix}" immediateFlush="false">
<Filters>
<ThresholdFilter level="WARN"/>
</Filters>
<PatternLayout pattern="${FILE_LOG_PATTERN}"/>
<Policies>
<TimeBasedTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="1024 MB"/>
</Policies>
<DefaultRolloverStrategy max="40"/>
</RollingRandomAccessFile>
</appenders>
<loggers>
<logger name="com.upex.exchange.robot.util.AlarmUtils" level="error">
<appender-ref ref="RollingFileAlarm"/>
</logger>
<AsyncRoot level="info" includeLocation="true">
<appender-ref ref="RollingFileInfo"/>
<appender-ref ref="RollingFileError"/>
</AsyncRoot>
</loggers>
Add a log4j2.component.properties to classpath,content config:
log4j2.AsyncQueueFullPolicy=Discard
AsyncLoggerConfig.SynchronizeEnqueueWhenQueueFull=true
AsyncLoggerConfig.RingBufferSize=131072
This new way may solve the memory leak problem(not confirmed), Because when the TimeBasedTriggeringPolicy started, the cpu load increased five times than usual , it seams still blocked when policy is triggering. However, this configuration limits the queue length and queue full policy. I will continue to observe

how to create log file under project base directory using log4j2

I wanted to create log file in my maven project under project base directory.I am using log4j2 framework kept the log4j2.xml file under src/mian/resources folder. below is the file. please update what is wrong in it.
<?xml version="1.0" encoding="UTF-8"?>
<Configuration xmlns:xi="http://www.w3.org/2001/XInclude"
packages="com.viveo.rtcc.common.client.traces" status="WARN">
<Properties>
<Property name="log-path">appLogs</Property>
</Properties>
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<!-- <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36}
- %msg%n"/> -->
<PatternLayout pattern="[ %d ] %p %t %c : %m%n" />
</Console>
<RollingFile name="rollingFileAppender"
fileName="${log-path}/reLog.log"
filePattern="logs/$${date:yyyy-MM}/app-%d{MM-dd-yyyy}-%i.log.gz">
<PatternLayout>
<Pattern>%d [%t] %p %c - %m%n</Pattern>
</PatternLayout>
<Policies>
<!-- <OnStartupTriggeringPolicy /> <TimeBasedTriggeringPolicy /> -->
<SizeBasedTriggeringPolicy size="10000 KB" />
</Policies>
<DefaultRolloverStrategy max="20" />
</RollingFile>
<MessageAreaAppender name="MessageAreaAppender" />
</Appenders>
<Loggers>
<!-- <Root level="DEBUG"> <AppenderRef ref="RollingFileAppender" /> <AppenderRef
ref="Console" />
<AppenderRef ref="MessageAreaAppender" /> </Root> -->
<Logger name="com.yourloggername" level="debug"
additivity="false">
<!-- <AppenderRef ref="ConsoleAppender" /> -->
<AppenderRef ref="rollingFileAppender" />
</Logger>
<Root level="info">
<!-- <AppenderRef ref="ConsoleAppender" /> -->
<AppenderRef ref="rollingFileAppender" />
</Root>
</Loggers>
</Configuration>
Here is what I see wrong in your configuration:
Maven builds run with the working directory set to the root directory of the project. Any files creates will be respective of that. In your configuration that means it would create the logs in a directory named appLogs. This is a poor practice in Maven builds. You should change it to target/applogs.
Your filePattern contains %d{dMM-dd-yy} but none of your triggering policies are time based. Without a time-based trigger I would expect that time to never changed while the application is running.

Cassandra Appender - How to Log ONLY error messages?

I am using Cassandra appender, console and file appenders in my application. I want to store only ERROR messages in Cassandra table Logs whereas store INFO messages to Console and file.
Below is the log4j2.xml file.
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
<Properties>
<Property name="LOG_PATTERN">%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n"</Property>
<Property name="APP_LOG_ROOT">./log</Property>
<Property name="MINLEVEL">INFO</Property>
<Property name="FILEMAXSIZE">800MB</Property>
<Property name="MAXFILES">5</Property>x
</Properties>
<Appenders>
<Console name="console" target="SYSTEM_OUT" follow="true">
<PatternLayout pattern="${LOG_PATTERN}"/>
</Console>
<RollingFile name="ROLLING" fileName="${APP_LOG_ROOT}/abc.log"
filePattern="${APP_LOG_ROOT}/abc-%d{yyyy-MM-dd}-%i.log">
<PatternLayout pattern="${LOG_PATTERN}"/>
<Policies>
<SizeBasedTriggeringPolicy size="${FILEMAXSIZE}" />
</Policies>
<DefaultRolloverStrategy max="${MAXFILES}"/>
</RollingFile>
<Cassandra name="Cassandra" clusterName="TestCluster" keyspace="Employee" table="Logs" bufferSize="10" username="tomcat" password="password">
<SocketAddress host="IPaddress" port="9042"/>
<ColumnMapping name="userid" pattern="%uuid{TIME}" type="java.util.UUID"/>
<ColumnMapping name="logger" pattern="%logger"/>
<ColumnMapping name="message" pattern="%message"/>
<ColumnMapping name="level" pattern="%level"/>
<ColumnMapping name="timestamp" literal="now()"/>
</Cassandra>
</Appenders>
<Loggers>
<Root level="${MINLEVEL}" additivity="false">
<AppenderRef ref="ROLLING"/>
</Root>
<Logger name="abc.xyz" level="ERROR" additivity="false">
<AppenderRef ref="Cassandra" />
<AppenderRef ref="console" />
</Logger>
</Loggers>
In the last segment when I change the level to ${MINLEVEL} i.e INFO, then it works but ERROR is not working. Please help!
The issue is that additivity is set to false, this is preventing to duplicate the messages to the root and abc.xyz loggers. If you set additivity to false, it will be sent to both loggers.
More information is available in this blog entry.

How to add uuid in log filename .I am using log4j2?

i want to create log filename which includes uuid in name .
<Configuration status="DEBUG">
<Appenders>
<RollingFile name="file" fileName="log/${uuid:TIME}-test.log"
append="true" filePattern="log/${uuid:TIME}-test.log.%i">
<Policies>
<SizeBasedTriggeringPolicy size="100 KB" />
</Policies>
<DefaultRolloverStrategy max="5" />
<PatternLayout>
<Pattern>%d %-5p [%c{3}] (%t) %m%n</Pattern>
</PatternLayout>
</RollingFile>
<Async name="async">
<AppenderRef ref="file" />
</Async>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d %-5p [%c{2}]%m%n" />
</Console>
</Appenders>
<Loggers>
<Root level="warn">
<AppenderRef ref="Console" />
<AppenderRef ref="async" />
</Root>
<Logger name="com.citigroup" level="INFO" additivity="false">
<Appender-ref ref="async" />
</Logger>
</Loggers>
</Configuration>
but it is creating file "uuid:TIME-test.log".I have also try %uuid , %u but none of them is creating file with uuid .
This can be accomplished with a custom lookup. This answer gives example code for how to create a custom lookup in Log4j2 (it's only a few lines of code).

Resources