This is Log4j 1.2.17. I have file:/C:/workspaces/gitlab/QMT/bin/log4j.properties on the classpath. Its contents are:
log4j.rootLogger=DEBUG, stdout, queuesLog
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Threshold=INFO
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %C{1}:%M:%L - %m%n
log4j.appender.queuesLog=org.apache.log4j.RollingFileAppender
log4j.appender.queuesLog.Threshold=DEBUG
log4j.appender.queuesLog.MaxFileSize=1MB
log4j.appender.queuesLog.MaxBackupIndex=10
log4j.appender.queuesLog.layout=org.apache.log4j.PatternLayout
log4j.appender.queuesLog.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %C{1}:%M:%L - %m%n
log4j.appender.queuesLog.File=C:/QEU/logs/queues.log
So when I saw that all of the archives (e.g. queues.log.1) are missing, I concluded that Log4j wasn't obeying this configuration file. I checked several things, whose details I don't believe are necessary for this question. Those checks told me that Log4j should be obeying the above configuration file. Finally, I learned about -Dlog4j.debug.
log4j: Trying to find [log4j.xml] using context classloader sun.misc.Launcher$AppClassLoader#659e0bfd.
log4j: Trying to find [log4j.xml] using sun.misc.Launcher$AppClassLoader#659e0bfd class loader.
log4j: Trying to find [log4j.xml] using ClassLoader.getSystemResource().
log4j: Trying to find [log4j.properties] using context classloader sun.misc.Launcher$AppClassLoader#659e0bfd.
log4j: Using URL [file:/C:/workspaces/gitlab/QMT/bin/log4j.properties] for automatic log4j configuration.
log4j: Reading configuration from URL file:/C:/workspaces/gitlab/QMT/bin/log4j.properties
log4j: Parsing for [root] with value=[DEBUG, stdout, queuesLog].
log4j: Level token is [DEBUG].
log4j: Category root set to DEBUG
log4j: Parsing appender named "stdout".
log4j: Parsing layout options for "stdout".
log4j: Setting property [conversionPattern] to [%d{yyyy-MM-dd HH:mm:ss} %-5p %C{1}:%M:%L - %m%n].
log4j: End of parsing for "stdout".
log4j: Setting property [threshold] to [INFO].
log4j: Parsed "stdout" options.
log4j: Parsing appender named "queuesLog".
log4j: Parsing layout options for "queuesLog".
log4j: Setting property [conversionPattern] to [%d{yyyy-MM-dd HH:mm:ss} %-5p %C{1}:%M:%L - %m%n].
log4j: End of parsing for "queuesLog".
log4j: Setting property [file] to [C:/QEU/logs/queues.log].
log4j: Setting property [maxBackupIndex] to [10].
log4j: Setting property [maxFileSize] to [1MB].
log4j: Setting property [threshold] to [DEBUG].
log4j: setFile called: C:/QEU/logs/queues.log, true
log4j: setFile ended
log4j: Parsed "queuesLog" options.
log4j: Finished configuring.
log4j: rolling over count=1025467879
log4j: maxBackupIndex=10
log4j: Renaming file C:\QEU\logs\queues.log to C:\QEU\logs\queues.log.1
log4j: setFile called: C:/QEU/logs/queues.log, true
log4j: setFile ended
This confirms that file:/C:/workspaces/gitlab/QMT/bin/log4j.properties is in use. Furthermore,
log4j: Renaming file C:\QEU\logs\queues.log to C:\QEU\logs\queues.log.1
is explicit, but the archive files aren't there!
queues.log grows until the operating system doesn't like it anymore, which happens at 4 GiB.
Related
In a Cloudera Spark-on-YARN installation, I encounter an application that has multiple log4j APIs in its classpath:
log4j:1.2
log4j-1.2-api:2.x
log4j-core:2.x
When the application is launched. I found the following error:
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
It cannot be fixed because the vendor of the platform exclusively used the log4j:1.2 configuration everywhere: it has integrated a managed log4j.properties file (not the log4j2.properties), which is in the classpath, used by many components, and cannot be tampered.
How do I tell the classloader or log4j implementation to not trying to find log4j2 file if it is missing?
UPDATE 1: I've tried the following Spark arguments to customise the driver JVM. Effects are as follows:
--conf spark.driver.extraJavaOptions="-XX:+UseG1GC -Dlog4j1.compatibility=true -Dlog4j.configuration=./conf_5/log4j.properties -Dlog4j.debug"
result is:
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
log4j: setFile called: /tmp/spark-b190d9df-64bf-4ff6-b72d-2cb8de1d888a/__driver_logs__/driver.log, true
log4j: setFile ended
log4j: Finalizing appender named [_DriverLogAppender].
--conf spark.driver.extraJavaOptions="-XX:+UseG1GC -Dlog4j1.compatibility=true -Dlog4j.configurationFile=./conf_5/log4j.properties -Dlog4j.debug"
result is:
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
log4j: setFile called: /tmp/spark-817bc6eb-bcc0-4cee-ba32-dbd4d89048e5/__driver_logs__/driver.log, true
log4j: setFile ended
log4j: Finalizing appender named [_DriverLogAppender].
So none of them is working properly. There seems to be no way to either use the right version or use the wrong configuration file
In my k8s spark application, I would like to change the log4j log LEVEL in executor pods.
In log4j properties file, I have set rootLogger to WARN, but still in executors pods I can see it is parsing to INFO.
log4j.properties:
log4j.rootLogger=WARN,console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d %-5p [%t] %c{2}(%L): %m%n
spark-submit:
spark-submit \
--master k8s://.. \
--deploy-mode client \
--conf spark.driver.extraJavaOptions='-Dlog4j.debug=true -Dlog4j.configuration=file:///opt/log4j.properties'\
--conf spark.executor.extraJavaOptions='-Dlog4j.debug=true -Dlog4j.configuration=file:///opt/log4j.properties'\
--class org.log4jTestRunner \
logger-test-1.0.jar
Driver logs:
log4j: Using URL [file:/opt/log4j.properties] for automatic log4j configuration.
log4j: Reading configuration from URL file:/opt/log4j.properties
log4j: Parsing for [root] with value=[WARN,console].
log4j: Level token is [WARN].
log4j: Category root set to WARN
log4j: Parsing appender named "console".
log4j: Parsing layout options for "console".
log4j: Setting property [conversionPattern] to [%d %-5p [%t] %c{2}(%L): %m%n].
log4j: End of parsing for "console".
log4j: Setting property [target] to [System.err].
log4j: Parsed "console" options.
Executor logs:
log4j: Using URL [file:/opt/log4j.properties] for automatic log4j configuration.
log4j: Reading configuration from URL file:/opt/log4j.properties
log4j: Parsing for [root] with value=[INFO,console].
log4j: Level token is [INFO].
log4j: Category root set to INFO
log4j: Parsing appender named ""console"".
log4j: Parsing layout options for ""console"".
log4j: Setting property [conversionPattern] to [%d %-5p [%t] %c{2}(%L): %m%n].
log4j: End of parsing for ""console"".
log4j: Setting property [target] to [System.err].
log4j: Parsed ""console"" options.
I can see in Driver it parsing correctly and log4j log level is respected.
Not sure, why in executors it is working differently.
I am using k8s with spark 3.x version.
Thank you in advance.
i have webapplication hosted in Websphere App Server. it uses log4j version 1.2.15 for logging and below is my log4j config. the log file is not rotating as soon as it reaches threshold, though log4j debugger says renaming to new file. but the same configuration works fine in another environment.Please note single instance of JVM only running in both of my server environments. So there is no chance for locking the same log file by different JVM is not possible.
log4j.properties
log4j.appender.local=org.apache.log4j.RollingFileAppender
log4j.appender.local.MaxFileSize=1MB
log4j.appender.local.MaxBackupIndex=10
log4j.appender.local.File=${applogs.home}\\web-app.log
log4j.appender.local.layout=org.apache.log4j.PatternLayout
log4j.appender.local.layout.ConversionPattern=%d\t%r\t%p\t%c\t%m%n
log4j.appender.local.Threshold=DEBUG
system.out log
[1/9/20 15:41:42:520 EST] 00000086 SystemOut O log4j: rolling over count=1048745
[1/9/20 15:41:42:536 EST] 00000086 SystemOut O log4j: maxBackupIndex=10
[1/9/20 15:41:42:551 EST] 00000086 SystemOut O log4j: Renaming file <nas_path_of_server>\web-app.log to <nas_path_of_server>\web-app.log.1
[1/9/20 15:41:42:551 EST] 00000086 SystemOut O log4j: setFile called: <nas_path_of_server>\web-app.log, true
[1/9/20 15:41:42:551 EST] 00000086 SystemOut O log4j: setFile ended
Resolved
I was able to fix the issue. it turned out that nas_path_of_server did not have modify permission for the application service id which caused this issue.
The property has standalone.xml
<System-properties>
<Property name = "eclipselink.archive.factory" value = "org.jipijapa.eclipselink.JBossArchiveFactoryImpl" />
<Property name = "-Dlog4j.configuration" value = "file:///d:Work/Options/log4j.xml" />
</ System-properties>
But in the logs is a warning
log4j: WARN No appenders could be found for logger (org.jboss.logging).
log4j: WARN Please initialize the log4j system properly.
log4j: WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Connected to server
What I did not = (
I have my own log4j1.12.17 in modules
I am pretty new to using the Kafka stream.
In a particular requirement I have to push my log4j logs directly to Kafka topic.
I have a standalone kafka installation running on centos and i have verified it with the kafka publisher and consumer clients. Also i am using the bundled zookeeper instance.
Now i have also created a standalone java app with log4j logging enabled. Also i have edited the log4j.properties file as below -
log4j.rootCategory=INFO
log4j.appender.file=org.apache.log4j.DailyRollingFileAppender
log4j.appender.file.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.file.File=/home/edureka/Desktop/Anurag/logMe
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd'T'HH:mm:ss.SSS'Z'}{UTC} %p %C %m%n
log4j.logger.com=INFO,file,KAFKA
#Kafka Appender
log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
log4j.appender.KAFKA.layout.ConversionPattern=%d{yyyy-MM-dd'T'HH:mm:ss.SSS'Z'}{UTC} %p %C %m%n
log4j.appender.KAFKA.ProducerType=async
log4j.appender.KAFKA.BrokerList=localhost:2181
log4j.appender.KAFKA.Topic=test
log4j.appender.KAFKA.Serializer=kafka.test.AppenderStringSerializer
Now when i am running the application, all the logs are going into the local log file but the consumer is still not showing any entry happening.
The topic i am using is test in either scenario.
Also no error log is being generated the the detailed logs of the log4j library are as below -
log4j: Trying to find [log4j.xml] using context classloader sun.misc.Launcher$AppClassLoader#a1d92a.
log4j: Trying to find [log4j.xml] using sun.misc.Launcher$AppClassLoader#a1d92a class loader.
log4j: Trying to find [log4j.xml] using ClassLoader.getSystemResource().
log4j: Trying to find [log4j.properties] using context classloader sun.misc.Launcher$AppClassLoader#a1d92a.
log4j: Using URL [file:/home/edureka/workspace/TestKafkaLog4J/bin/log4j.properties] for automatic log4j configuration.
log4j: Reading configuration from URL file:/home/edureka/workspace/TestKafkaLog4J/bin/log4j.properties
log4j: Parsing for [root] with value=[DEBUG, stdout, file].
log4j: Level token is [DEBUG].
log4j: Category root set to DEBUG
log4j: Parsing appender named "stdout".
log4j: Parsing layout options for "stdout".
log4j: Setting property [conversionPattern] to [%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n].
log4j: End of parsing for "stdout".
log4j: Setting property [target] to [System.out].
log4j: Parsed "stdout" options.
log4j: Parsing appender named "file".
log4j: Parsing layout options for "file".
log4j: Setting property [conversionPattern] to [%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n].
log4j: End of parsing for "file".
log4j: Setting property [file] to [/home/edureka/Desktop/Anurag/logMe].
log4j: Setting property [maxBackupIndex] to [10].
log4j: Setting property [maxFileSize] to [5MB].
log4j: setFile called: /home/edureka/Desktop/Anurag/logMe, true
log4j: setFile ended
log4j: Parsed "file" options.
log4j: Finished configuring.
2015-05-11 19:44:40 DEBUG TestMe:19 - This is debug : anurag
2015-05-11 19:44:40 INFO TestMe:23 - This is info : anurag
2015-05-11 19:44:40 WARN TestMe:26 - This is warn : anurag
2015-05-11 19:44:40 ERROR TestMe:27 - This is error : anurag
2015-05-11 19:44:40 FATAL TestMe:28 - This is fatal : anurag
2015-05-11 19:44:40 INFO TestMe:29 - message from log4j appender
Any help will be really great.
Thanks,
AJ
In your output, I don't see the KAFKA appender being created, so no wonder nothing is logged to Kafka. I'm guessing the reason for that is that you only log from a class named TestMe (probably in the default package), while the KAFKA appender is only added to the logger named "com".