We're looking to change our size based trigger to time based trigger policy (every 5 minutes). But our configuration looks very different from the current log4j2. Here's our current configuration:
log4j.rootLogger=INFO,file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=${spark.yarn.app.container.log.dir}/spark.log
log4j.appender.file.MaxFileSize=10MB
log4j.appender.file.MaxBackupIndex=100
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d %p %t %c - %m%n
log4j.appender.file.encoding=UTF-8
If I change it to this, I'm not sure if it'll work or if I'm missing anything?
log4j.appender.rolling.type = RollingFile
log4j.appender.rolling.name = RollingFile
log4j.appender.rolling.fileName = spark.log
log4j.appender.rolling.filePattern = ${spark.yarn.app.container.log.dir}/spark-%d{yyyy-MM-dd-HH-mm}-%i.log.gz
log4j.appender.rolling.layout.type = PatternLayout
log4j.appender.rolling.layout.pattern = %d %p %t %c - %m%n
log4j.appender.rolling.policies.type = Policies
log4j.appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
log4j.appender.rolling.policies.time.interval = 5
log4j.appender.rolling.policies.time.modulate = true
Related
Hi I am trying to redirect Spark workers logs to stdout/stderr.
I have added custom log4j.properties file that looks like this:
log4j.rootLogger = INFO, stdout, stderr
# configure stdout
log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Threshold = TRACE
log4j.appender.stdout.filter.filter1=org.apache.log4j.varia.LevelRangeFilter
log4j.appender.stdout.filter.filter1.levelMin = TRACE
log4j.appender.stdout.filter.filter1.levelMax = INFO
log4j.appender.stdout.Target = System.out
log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern = %d{yy/MM/dd HH:mm:ss,SSS} %p %c: %m%n
# configure stderr
log4j.appender.stderr = org.apache.log4j.ConsoleAppender
log4j.appender.stderr.Threshold = WARN
log4j.appender.stderr.Target = System.err
log4j.appender.stderr.layout = org.apache.log4j.PatternLayout
log4j.appender.stderr.layout.ConversionPattern = %d{yy/MM/dd HH:mm:ss,SSS} %p %c: %m%n
# Settings to quiet third party logs that are too verbose
log4j.logger.org.apache.hadoop.util.NativeCodeLoader = ERROR
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.spark=WARN
log4j.logger.org.spark-project.jetty.server=WARN
Still logs are not redirected to stdout/stderr but to stdout and stderr files on the disk.
Anyone knows how can I make it work?
EDIT: reading from this blog post it seems like it is just the way it works
I am using the below configuration to generate the log rolling based on size. The logs are created, however the rolling stops after creating the max files (10). The default file is keep on overwriting and all the rolled over files never gets updated.
appender.sample.type = RollingFile
appender.sample.name = SampleRollingFile
appender.sample.fileName = ${basedir}/logs/${machine}_My_Sample.0.log
appender.sample.filePattern = ${basedir}/logs/${machine}_My_Sample.%i.log
appender.sample.layout.type = PatternLayout
appender.sample.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p [T:%T] %c{1}:%L - %m%n
appender.sample.policies.type = Policies
appender.sample.policies.size.type = SizeBasedTriggeringPolicy
appender.sample.policies.size.size = 100MB
appender.sample.strategy.type = DefaultRolloverStrategy
appender.sample.strategy.max = 10
But when I changed my default initial file name to My_Sample.log (removing the .0 index) it works perfectly fine. My application requirement is initial file with .0 and the rolling the logs from 1 to 10.
I added the below properties to get the desired result. By enabling tracing (status = TRACE) in the log4j2 they use the index 0 to play around with the renaming and purge. To avoid the collision, I started the index from 1 and the continuous rolling is happening !!
appender.sample.strategy.fileIndex = 1
appender.sample.strategy.min = 1
I'd like a log4j2 properties file configuration with a console and a rolling file appender using log4j2 that can be used different application. The log configuration should rotate the log in production environment.
I think there is no such industry standard for logging or log4j2 configuration. Everyone change the configuration as per the need of the application.
Below is one sample log4j2 configuration file having ConsoleAppender and RollingFileAppender -
status = warn
name= properties_configuration
# Give directory path where log files should get stored
property.basePath = ./log/
# ConsoleAppender will print logs on console
appender.console.type = Console
appender.console.name = consoleLogger
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
# Specify the pattern of the logs
appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} %level [%t] [%c] [%M] [%l] - %msg%n
# RollingFileAppender will print logs in file which can be rotated based on time or size
appender.rolling.type = RollingFile
appender.rolling.name = fileLogger
appender.rolling.fileName= ${basePath}app.log
appender.rolling.filePattern= ${basePath}app_%d{yyyyMMdd}.log.gz
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} %level [%t] [%c] [%M] [%l] - %msg%n
appender.rolling.policies.type = Policies
# Rotate log file each day and keep 30 days worth
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.delete.type = Delete
appender.rolling.strategy.delete.basePath = ${basePath}
appender.rolling.strategy.delete.maxDepth = 1
appender.rolling.strategy.delete.ifLastModified.type = IfLastModified
# Delete files older than 30 days
appender.rolling.strategy.delete.ifLastModified.age = 30d
# Mention package name here in place of example. Classes in this package or subpackages will use ConsoleAppender and RollingFileAppender for logging
logger.example.name = example
logger.example.level = debug
logger.example.additivity = false
logger.example.appenderRef.rolling.ref = fileLogger
logger.example.appenderRef.console.ref = consoleLogger
# Configure root logger for logging error logs in classes which are in package other than above specified package
rootLogger.level = error
rootLogger.additivity = false
rootLogger.appenderRef.rolling.ref = fileLogger
rootLogger.appenderRef.console.ref = consoleLogger
Try this out:
# Declare loggers
name=LoggingConfig
appenders=a_console, a_rolling
rootLogger.level=info
rootLogger.appenderRefs=ar_console,ar_rolling
rootLogger.appenderRef.ar_console.ref=StdoutAppender
rootLogger.appenderRef.ar_rolling.ref=DailyRollingAppender
# Console logger
appender.a_console.type=Console
appender.a_console.name=StdoutAppender
appender.a_console.layout.type=PatternLayout
appender.a_console.layout.pattern=%d{ISO8601} [%t] %-5p (%F\:%L) - %m%n
# File logger
appender.a_rolling.type=RollingFile
appender.a_rolling.name=DailyRollingAppender
appender.a_rolling.layout.pattern=%d{ISO8601} [%t] %-5p (%F\:%L) - %m%n
appender.a_rolling.fileName=log4j2-sample.log
appender.a_rolling.filePattern=log4j2-sample-%d{yyyy-MM-dd}.log
appender.a_rolling.layout.type=PatternLayout
appender.a_rolling.policies.type=Policies
appender.a_rolling.policies.time.type=TimeBasedTriggeringPolicy
appender.a_rolling.policies.time.interval=1
This is the simplest config if you just need to use console appender.
name=config
appenders=console
appender.console.type=Console
appender.console.name=STDOUT
appender.console.layout.type=PatternLayout
#appender.console.layout.pattern =%d{HH:mm:ss} [%t] %c{1} [%-5level] - %msg%n
appender.console.layout.pattern=%d{dd-MM-yyyy HH:mm:ss} [%-5p] (%F:%L) - %m%n
rootLogger.level=info
rootLogger.appenderRefs=stdout
rootLogger.appenderRef.stdout.ref=STDOUT
#if you want to make package specific configuration
#logger.org.apache=warn
I'm trying to use Log4j2 in my application and my relevant configuration looks like below code:
*.*.log4j2.appender.file.type = RollingRandomAccessFile;
*.*.log4j2.appender.file.name = file;
*.*.log4j2.appender.file.fileName = "$ROOT/var/output/logs/$APP";
*.*.log4j2.appender.file.filePattern = "$ROOT/var/output/logs/$APP.%d{yyyy-MM-dd-HH}";
*.*.log4j2.appender.file.layout.type = PatternLayout;
*.*.log4j2.appender.file.layout.pattern = "%d{DATE} [%p] %X{requestId} (%t) %c: %m%n";
But, when i run my program i get this error:
955 main ERROR RollingRandomAccessFile contains an invalid element or attribute "fileName"
Can anyone help me on this issue?
Your help is much appreciated.
Best Regards.
I suspect that the *.*.log4j2 prefix may be related. Can you try without this, like in the example in the log4j2 manual?
appender.rolling.type = RollingRandomAccessFile
appender.rolling.name = RollingFile
appender.rolling.fileName = ${filename}
appender.rolling.filePattern = target/rolling2/test1-%d{MM-dd-yy-HH-mm-ss}-%i.log.gz
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = %d %p %C{1.} [%t] %m%n
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 2
appender.rolling.policies.time.modulate = true
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size=100MB
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.max = 5
rootLogger.level = info
rootLogger.appenderRef.rolling.ref = RollingFile
I am having trouble with duplicate log messages when writing to multiple log files using log4j.
At present I am trying to log INFO level data (and upwards) for the specific logger named foobar in my foo.log file and then all WARN level log messages (and upwards) for all loggers in the bar.log file.
As a result of this, duplicate log messages were written to the foo.log file (each line was logged twice) and after some quick research I found that the suggestion to fix this was to add log4j.additivity.foobar=false to my properties file.
The problem with this is that although it stops duplicate lines, the WARN message from the foobar logger are never written to the bar.log file.
My log4j properties file is as follows:
log4j.rootLogger = WARN, FOO, BAR
log4j.logger.foobar = INFO, FOO
log4j.additivity.foobar = false
log4j.appender.FOO = org.apache.log4j.RollingFileAppender
log4j.appender.FOO.layout = org.apache.log4j.PatternLayout
log4j.appender.FOO.layout.ConversionPattern = %d{ISO8601} %-5p %c ~ %m%n
log4j.appender.FOO.File = foo.log
log4j.appender.BAR = org.apache.log4j.RollingFileAppender
log4j.appender.BAR.layout = org.apache.log4j.PatternLayout
log4j.appender.BAR.layout.ConversionPattern = %d{ISO8601} %-5p %c ~ %m%n
log4j.appender.BAR.File = bar.log
Does anyone know how I can write the log messages to both log files (as it was doing before I started setting the additivity property) and still prevent the duplicate log messages?
Please note, this is a simplified summary of the problem. In the real world scenario there are multiple loggers and more than two log files
This problem can be solved in two parts.
1. Prevent duplicate log messages
The log messages were written twice because we listed the FOO appender in both the rootLogger and the log4j.logger.foobar category. So we must remove the appender and only define the logging level in category:
log4j.rootLogger = WARN, FOO, BAR
log4j.logger.foobar = INFO
This means that the INFO level messages from log4j.logger.foobar are propagated upwards to ALL of the loggers the appenders in rootLogger, but will only be written to each log file once.
2. Prevent INFO level message being written to bar.log
Since all of the INFO level log messages for the log4j.logger.foobar category are being inherited by the appenders in rootLogger, we need to stop the BAR appender for recording the INFO level messages.
We can achieve this by setting the Threshold property on the BAR appender itself:
log4j.appender.BAR.Threshold = WARN
This will prevent the INFO level statements being logged in the bar.log file as it will only accept levels of WARN and upwards.
So the complete log4j properties file would be as follows:
log4j.rootLogger = WARN, FOO, BAR
log4j.logger.foobar = INFO
log4j.appender.FOO = org.apache.log4j.RollingFileAppender
log4j.appender.FOO.layout = org.apache.log4j.PatternLayout
log4j.appender.FOO.layout.ConversionPattern = %d{ISO8601} %-5p %c ~ %m%n
log4j.appender.FOO.File = foo.log
log4j.appender.FOO.Threshold = INFO
log4j.appender.BAR = org.apache.log4j.RollingFileAppender
log4j.appender.BAR.layout = org.apache.log4j.PatternLayout
log4j.appender.BAR.layout.ConversionPattern = %d{ISO8601} %-5p %c ~ %m%n
log4j.appender.BAR.File = bar.log
log4j.appender.BAR.Threshold = WARN