Log4j filter by package - log4j

I would like to have log4j (v1.2.17) send ...
... to fileAppender
INFO (and above) from all packages
... to console
if from "my.project" then INFO (and above)
else WARN (and above) for all other packages
How can I do this? I'd prefer a properties file if possible, but would switch to XML if necessary.
I've tried combinations of loggers, non-additivity, Threshold, & LevelMatchFilter but can't figure it out.

This can definitely be done. Below you will find some example code and log4j properties that will do what you want.
Here is the first example class:
package my;
import org.apache.log4j.Logger;
public class Example1 {
private static final Logger logger = Logger.getLogger(Example1.class);
public static void main(String[] args){
logger.debug("debug from my.Example1 - should display nowhere!");
logger.info("info from my.Example1 - should display in log file only!");
logger.warn("warning from my.Example1 - should display in console and log file!");
logger.error("error from my.Example1 - should display in console and log file!");
logger.fatal("fatal from my.Example1 - should display in console and log file!");
}
}
Here is the second example class:
package my.project;
import org.apache.log4j.Logger;
public class Example2 {
private static final Logger logger = Logger.getLogger(Example2.class);
public static void main(String[] args){
logger.debug("debug from my.project.Example2 - should display nowhere!");
logger.info("info from my.project.Example2 - should display in console and log file!");
logger.warn("warning from my.project.Example2 - should display in console and log file!");
logger.error("error from my.project.Example2 - should display in console and log file!");
logger.fatal("fatal from my.project.Example2 - should display in console and log file!");
}
}
Here is the log4j.properties file:
# The root logger will accept INFO messages or higher and send them to the log file
# and to a console appender that filters out any messages below WARN level
log4j.rootLogger=INFO, R, warnStdout
# Configure a console appender that will be used for messages of any level
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
# Pattern to output the caller's file name and line number.
log4j.appender.stdout.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n
#This will be used to print WARN level or higher messages to console
log4j.appender.warnStdout=org.apache.log4j.ConsoleAppender
log4j.appender.warnStdout.layout=org.apache.log4j.PatternLayout
log4j.appender.warnStdout.Threshold=WARN
# Pattern to output the caller's file name and line number.
log4j.appender.warnStdout.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n
log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=test.log
log4j.appender.R.MaxFileSize=100KB
# Keep one backup file
log4j.appender.R.MaxBackupIndex=1
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n
# Classes in the my.project package will accept messages of INFO level or higher
# and send those messages to the console and to the log file
log4j.logger.my.project=INFO, stdout, R
# Need to set additivity to false or else both the my.project and root loggers
# will accept messages from classes in package my.project
log4j.additivity.my.project=false
Here is the console output after running Example1 followed immediately by Example2:
WARN [main] (Example1.java:11) - warning from my.Example1 - should display in console and log file!
ERROR [main] (Example1.java:12) - error from my.Example1 - should display in console and log file!
FATAL [main] (Example1.java:13) - fatal from my.Example1 - should display in console and log file!
INFO [main] (Example2.java:11) - info from my.project.Example2 - should display in console and log file!
WARN [main] (Example2.java:12) - warning from my.project.Example2 - should display in console and log file!
ERROR [main] (Example2.java:13) - error from my.project.Example2 - should display in console and log file!
FATAL [main] (Example2.java:14) - fatal from my.project.Example2 - should display in console and log file!
Here is the test.log file content after running Example1 followed by Example2:
INFO main my.Example1 - info from my.Example1 - should display in log file only!
WARN main my.Example1 - warning from my.Example1 - should display in console and log file!
ERROR main my.Example1 - error from my.Example1 - should display in console and log file!
FATAL main my.Example1 - fatal from my.Example1 - should display in console and log file!
INFO main my.project.Example2 - info from my.project.Example2 - should display in console and log file!
WARN main my.project.Example2 - warning from my.project.Example2 - should display in console and log file!
ERROR main my.project.Example2 - error from my.project.Example2 - should display in console and log file!
FATAL main my.project.Example2 - fatal from my.project.Example2 - should display in console and log file!

Related

Pyspark and Log4J configuration

I'm trying to have some decent logging through Python using log4j.
I want to have all logs written to a DB, only errors written to an error.log file, and only info written to an info.log file.
logger = sc._jvm.org.apache.log4j
lg = logger.LogManager.getRootLogger()
lg.info('test')
lg.error('test')
lg.debug('test')
lg.fatal('test')
and my log4j.properties file is as follow:
# Set everything to be logged to the console
log4j.rootLogger=INFO, ria_info, ria_error, ria_mysql, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.out
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=[%p] %d %c %M - %m%n
# Set info logs to be written to info file
log4j.appender.ria_info=org.apache.log4j.RollingFileAppender
log4j.appender.ria_info.filter.RangeFilter=org.apache.log4j.varia.LevelMatchFilter
log4j.appender.ria_info.filter.RangeFilter.LevelToMatch=INFO
log4j.appender.ria_info.filter.RangeFilter.AcceptOnMatch=true
log4j.appender.ria_info.layout=org.apache.log4j.PatternLayout
log4j.appender.ria_info.layout.ConversionPattern=[%p] %d %c %M - %m%n
log4j.appender.ria_info.File=/home/data/logs/info.log
log4j.appender.ria_info.MaxFileSize=10MB
log4j.appender.ria_info.MaxBackupIndex=10
log4j.appender.ria_error=org.apache.log4j.RollingFileAppender
log4j.appender.ria_error.Append=false
log4j.appender.ria_error.filter.RangeFilter=org.apache.log4j.varia.LevelMatchFilter
log4j.appender.ria_error.filter.RangeFilter.LevelToMatch=ERROR
log4j.appender.ria_error.filter.RangeFilter.AcceptOnMatch=true
log4j.appender.ria_error.layout=org.apache.log4j.PatternLayout
log4j.appender.ria_error.layout.ConversionPattern=[%p] %d %c %M - %m%n
log4j.appender.ria_error.File=/home/data/logs/error.log
log4j.appender.ria_error.MaxFileSize=10MB
log4j.appender.ria_error.MaxBackupIndex=10
log4j.appender.ria_mysql=org.apache.log4j.jdbc.JDBCAppender
log4j.appender.ria_mysql.URL=jdbc:mysql://localhost/DB
log4j.appender.ria_mysql.driver=com.mysql.jdbc.Driver
log4j.appender.ria_mysql.user=xxxx
log4j.appender.ria_mysql.password=xxxxx
log4j.appender.ria_mysql.sql=INSERT INTO LOGS VALUES('%p','%d{yyyy-MM-dd HH:mm:ss}','%t','%x','%c','%m')
log4j.appender.ria_mysql.layout=org.apache.log4j.PatternLayout
# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=INFO
# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
Now I get the error and fatal 'test' message in my DB and error.log. But the info and debug message gets completely lost for some reason. Also lg.isInfoEnabled() returns False.
I tried a lot of stuff around additivity but it didn't seem to solve the problem.

Silence Debug Logging in HTTPComponents

How do I silence the debug logging in http componenets?
09:23:22.145 [main] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: default
09:23:22.145 [main] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
09:23:22.145 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://example.org:80][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 200]
09:23:22.145 [main] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {}->http://diagnostics-uat.corp.apple.com:80][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 200]
09:23:22.145 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Executing request POST /evt/detect HTTP/1.1
09:23:22.145 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Target auth state: UNCHALLENGED
09:23:22.145 [main] DEBUG o.a.h.impl.execchain.MainClientExec - Proxy auth state: UNCHALLENGED
09:23:22.145 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> POST /evt/detect HTTP/1.1
09:23:22.145 [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Content-Length: 92
I have tried a code based solution:
static {
System.setProperty("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.SimpleLog");
System.setProperty("org.apache.commons.logging.simplelog.log.org.apache.http", "warn");
}
public static void main(String[] args) throws Exception {
And this log4j.properties
log4j.rootLogger=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%5p [%c] %m%n
log4j.logger.org.apache.http=WARN
log4j.logger.org.apache.http.wire=WARN
But neither have worked. Help is appreciated, thanks!
You settings are correct possible causes may be another log4j config file is in the classpath and overriding your config. So I will recommended this.
- Ensure you log4j.properties file is in classpath . Use below command to see open files by a process (assuming unix)
lsof -p java-process-pid | grep log4j
Search for log4j.properties in your runtime folders and get rid of all duplicate files (if possible) - otherwise change them all.
Search for log4j.xml and do the same.
Sometimes I have seen third party jars that ship with a log4j.xml or log4j.properties so in worst case check for all jars using below command
jar -tvf a.jar | grep log4j
Programmatically, it can be set silent with:
(Apache HttpComponents 4.3.5)
import org.apache.log4j.Level;
import org.apache.log4j.Logger;
...
Logger.getLogger("org.apache.http") .setLevel(Level.WARN);
Logger.getLogger("org.apache.http.wire") .setLevel(Level.WARN);
Logger.getLogger("org.apache.http.headers").setLevel(Level.WARN);

Rexster refuses to start with extension but does not display errors

I have a small Rexster/Titan cluster using Cassandra. A Rexster extension is used to query the graph. I did some benchmarking and did start and stop Rexster/Titan many times. But now I run into a strange issue: Rexster refuses to start but does not display any error message.
I tried to figure out what is causing this and reduced the cluster to a single node 192.168.0.4.
If I remove my extension Rexster manages to start up.
# console output
Forking Cassandra...
Running `nodetool statusthrift`..... OK
(returned exit status 0 and printed string "running").
Forking Titan + Rexster...
Connecting to Titan + Rexster (127.0.0.1:8184)...... OK
(connected to 127.0.0.1:8184).
Run rexster-console.sh to connect.
but when I place my extension uber JAR in the ext folder Rexster refuses to start.
# console output
Forking Cassandra...
Running `nodetool statusthrift`..... OK
(returned exit status 0 and printed string "running").
Forking Titan + Rexster...
Connecting to Titan + Rexster (127.0.0.1:8184)............................
timeout exceeded (60 seconds): could not connect to 127.0.0.1:8184
See /var/lib/titan/bin/../log/rexstitan.log for Rexster log output.
If I now check rexstitan.log, as suggested by the console output, I can not find any error message.
# rexstitan.log
0 [main] INFO com.tinkerpop.rexster.Application - .:Welcome to Rexster:.
73 [main] INFO com.tinkerpop.rexster.server.RexsterProperties -
Using [/var/lib/titan/rexhome/../conf/rexster-cassandra-cluster.xml]
as configuration source.
78 [main] INFO com.tinkerpop.rexster.Application - Rexster is watching
[/var/lib/titan/rexhome/../conf/rexster-cassandra-cluster.xml] for change.
244 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=ClusterTitanConnectionPool,ServiceType=connectionpool
252 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
537 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=KeyspaceTitanConnectionPool,ServiceType=connectionpool
538 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
1951 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration -
Set cluster.partition=false from store features
1971 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration -
Set default timestamp provider MICRO
2019 [main] INFO com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration -
Generated unique-instance-id=7f0000012902-node1
2045 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=ClusterTitanConnectionPool,ServiceType=connectionpool
2046 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
2053 [main] INFO com.netflix.astyanax.connectionpool.impl.ConnectionPoolMBeanManager -
Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,
name=KeyspaceTitanConnectionPool,ServiceType=connectionpool
2054 [main] INFO com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor -
AddHost: 192.168.0.4
2228 [main] INFO com.thinkaurelius.titan.diskstorage.Backend -
Initiated backend operations thread pool of size 4
6619 [main] INFO com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog -
Loaded unidentified ReadMarker start time Timepoint[1423479705116000 μs]
into com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller#212f3ff1
6625 [main] INFO com.tinkerpop.rexster.RexsterApplicationGraph -
Graph [graph] - configured with allowable namespace [*:*]
The only entry that looks strange to me is the one concerning the log:
6619 [main] INFO com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog -
Loaded unidentified ReadMarker start time Timepoint[1423479705116000 μs]
into com.thinkaurelius.titan.diskstorage.log.kcvs.KCVSLog$MessagePuller#212f3ff1
My exception uses the logger for debugging. You can see the instantiation an usage on github: https://github.com/sebschlicht/titan-graphity-kribble/blob/master/src/main/java/de/uniko/sebschlicht/titan/extensions/GraphityExtension.java#L22
Though Rexster failed to start there is a process with the PID displayed in the console but curl fails to connect to Rexster:
$ curl 192.168.0.4:8182
curl: (7) Failed to connect to 192.168.0.4 port 8182: Connection refused
Why doesn't Rexster throw an exception? How can I debug this situation?
edit:
I removed any log messages in my code. I removed all exceptions that may be thrown during startup. Still Rexster refuses to start with my extension and the only hint in the log files is the unidentified read marker. I have to clue what prevents Rexster from starting.
The log message is nothing to worry about.
After rebuilding the application in another project step-by-step Rexster is now able to start with the extension. During this rebuild I noticed two situations, that can cause the behaviour described:
Missing dependency
If your project depends on a second project you might use Maven to inject it as a dependency. However, if you use
mvn clean package
to build the extension's JAR file it does not contain this dependency by default. You need to use a Maven plugin (e.g. maven-shade-plugin) to create a shaded JAR that contains all the dependencies your extension needs. Set the dependency scope to provided for all Titan/Rexster/Blueprints related dependencies. Use the shaded uber-JAR to deploy the extension to Rexster.
However, this was not new to me and should not have caused the problem in my case. There might be more situations that cause this problem or maybe there was a problem with Maven that messed up the shaded JAR. Feel free to browse the commit on github to catch this voodoo.
Missing extension
Another cause of this behaviour is a missing extension.
If you specify an extension in the com.tinkerpop.rexster.extension.RexsterExtension resource file, that is not present on startup, Rexster does neither log nor throw an exception, but refuses to start.

log4j configuration level error

My log4j configuration is as follows
log4j.rootLogger=INFO, CA, FA, DA
#Console Appender
log4j.appender.CA=org.apache.log4j.ConsoleAppender
log4j.appender.CA.layout=org.apache.log4j.PatternLayout
log4j.appender.CA.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
#File Appender
log4j.appender.FA=org.apache.log4j.FileAppender
log4j.appender.FA.File=/home/admin/logs/sysout.log
log4j.appender.FA.layout=org.apache.log4j.PatternLayout
log4j.appender.FA.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
log4j.appender.FA.Threshold = WARN
#File Appender 2
log4j.appender.DA=org.apache.log4j.FileAppender
log4j.appender.DA.File=/home/admin/logs/debug.log
log4j.appender.DA.layout=org.apache.log4j.PatternLayout
log4j.appender.DA.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
log4j.appender.DA.Threshold = TRACE
My understading is
INFO will be logged to console
WARN will be logged to sysout.log
TRACE will logged to debug.log
But WARN is getting logged to both debug.log and sysout.log. Also TRACE is not logging in any of the file.
Console is having TRACE and WARN both.
Can you please tell me what am I doing wrong
You need to separate the logger and appender concepts in your mind.
For the three appenders, remember that the threshold is the lowest level of message that the appender will process. An appender will process messages at its threshold level or any higher level.
CA has no threshold set, so it will log all messages that are sent to it regardless of level. Similarly DA has a threshold of TRACE so it will also log everything that is sent to it (since TRACE is the lowest level). FA has a threshold of WARN so it will filter out any messages at levels below WARN - it will contain only WARN, ERROR and FATAL messages.
The important part of that previous paragraph is "all messages that are sent to it". Since you have configured your root logger with a level of INFO and have not configured any specific loggers to a lower level, only messages at INFO and above will be sent to the appenders - DEBUG and TRACE messages will be silently dropped. This is why you see no TRACE output in any of your loggers.

How to configure chainsaw bundle to view log4j log file?

I want to view my log4j log file in chainsaw bundle. My log file pattern is
LOG:%x : %d{ISO8601} : %-5p %c{1} - %m%n
The corresponding pattern to view in chainsaw bundle is needed. I have tried many things, i couldn't represent the 'Log' ':' in chainsaw bundle pattern.
Thanks.
Try the latest developer snapshot here instead:
http://people.apache.org/~sdeboy
I was able to process these log messages:
LOG:{blah1=blah2} : 1/1/2011 : DEBUG LOGGER1 - MESSAGE1
LOG:{blah1=blah2} : 1/1/2011 : DEBUG LOGGER2 - MESSAGE2
LOG:{blah1=blah2} : 1/1/2011 : DEBUG LOGGER3 - MESSAGE3
Using this logformat and a timestampformat of M/d/yyyy
LOG:PROP(BLAH1) : TIMESTAMP : LEVEL LOGGER - MESSAGE

Resources