I have a myapp.py that looks like:
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
fh = logging.FileHandler('out.log')
fh.setLevel(logging.INFO)
logger.addHandler(fh)
import ext_library
logger.info('Starting...')
ext_library.do_something()
.do_something() outputs a bunch of INFO messages from different modules/functions of ext_library. It looks like:
[INFO 05-10 19:40:59] ext_library.model.utils: Some information here.
[INFO 05-10 19:40:59] ext_library.service: Started to run inner function.
[INFO 05-10 19:40:59] ext_library.service: Finishing...
All of ext_library has loggers following the logger = logging.getLogger(__name__) format. I would like everything to be logged to out.log. What I'm finding is that:
Starting... and all the INFO messages coming from ext_library are displayed on the terminal.
None of the INFO messages from ext_library are getting logged to out.log.
What am I doing wrong? I also find that even if I change the log level of logger to logging.ERROR, all the INFO messages from ext_library are still appearing on the terminal which is not what I expected... am I not propagating the setLevel and addHandler correctly?
Thanks.
Related
I'm setting log level to 'debug' which I recall is the most verbose, however I'm only getting lines like this, even when an exception is thrown:
[2018-04-18 22:08:21 +0000] [23394] [DEBUG] POST /json
My startup command is this:
gunicorn --log-level debug --error-logfile gunicorn_error.log -D -b 0.0.0.0:5000 forward_to_es:app
Thanks for any suggestions.
I recommend that you create a gunicorn config file. for example:
# /path-to-your-project/gunicorn_conf.py
bind = '0.0.0.0:8811'
worker_class = 'sync'
loglevel = 'debug'
accesslog = '/var/log/gunicorn/access_log_yourapp'
acceslogformat ="%(h)s %(l)s %(u)s %(t)s %(r)s %(s)s %(b)s %(f)s %(a)s"
errorlog = '/var/log/gunicorn/error_log_yourapp'
In the documentation you may find all possible identifiers for your access log.
Then just do
/path-to-your-project/gunicorn -c gunicorn_conf.py forward_to_es:app
This way you may have one or more configurations or even create various logs depending on the configuration you are trying.
I'm working on an PySpark application and I deploy it yarn-cluster mode. I have given stdout as the logging streamhandler. I could see the log in YARN UI. However, I couldn't find stdout logs under /var/log/sparkapp/yarn. I see only stderr logs there. What could be the reason for this?
This is my logging part in the application
import logging
import sys
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
lsh = logging.StreamHandler(sys.stdout)
lsh.setLevel(logging.INFO)
lformat = logging.Formatter(fmt='%(asctime)s.%(msecs)03d %(levelname)s :%(name)s - %(message)s', datefmt='%m/%d/%Y %I:%M:%S')
lsh.setFormatter(lformat)
logger.addHandler(lsh)
log4j.properties
log4jspark.root.logger=INFO,console
log4jspark.log.dir=.
log4jspark.log.file=spark.log
log4jspark.log.maxfilesize=1024MB
log4jspark.log.maxbackupindex=10
# Define the root logger to the system property "spark.root.logger".
log4j.rootLogger=${log4jspark.root.logger}, EventCounter
# Set everything to be logged to the console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
log4j.appender.console.Threshold=INFO
# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
Try this instead to get the logger for your spark job:
log4jLogger = sc._jvm.org.apache.log4j
logger = log4jLogger.LogManager.getLogger(__name__)
You can modify log4j.propertiesto change the targetfile:
log4j.appender.console.target=System.out
I'm trying to have some decent logging through Python using log4j.
I want to have all logs written to a DB, only errors written to an error.log file, and only info written to an info.log file.
logger = sc._jvm.org.apache.log4j
lg = logger.LogManager.getRootLogger()
lg.info('test')
lg.error('test')
lg.debug('test')
lg.fatal('test')
and my log4j.properties file is as follow:
# Set everything to be logged to the console
log4j.rootLogger=INFO, ria_info, ria_error, ria_mysql, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.out
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=[%p] %d %c %M - %m%n
# Set info logs to be written to info file
log4j.appender.ria_info=org.apache.log4j.RollingFileAppender
log4j.appender.ria_info.filter.RangeFilter=org.apache.log4j.varia.LevelMatchFilter
log4j.appender.ria_info.filter.RangeFilter.LevelToMatch=INFO
log4j.appender.ria_info.filter.RangeFilter.AcceptOnMatch=true
log4j.appender.ria_info.layout=org.apache.log4j.PatternLayout
log4j.appender.ria_info.layout.ConversionPattern=[%p] %d %c %M - %m%n
log4j.appender.ria_info.File=/home/data/logs/info.log
log4j.appender.ria_info.MaxFileSize=10MB
log4j.appender.ria_info.MaxBackupIndex=10
log4j.appender.ria_error=org.apache.log4j.RollingFileAppender
log4j.appender.ria_error.Append=false
log4j.appender.ria_error.filter.RangeFilter=org.apache.log4j.varia.LevelMatchFilter
log4j.appender.ria_error.filter.RangeFilter.LevelToMatch=ERROR
log4j.appender.ria_error.filter.RangeFilter.AcceptOnMatch=true
log4j.appender.ria_error.layout=org.apache.log4j.PatternLayout
log4j.appender.ria_error.layout.ConversionPattern=[%p] %d %c %M - %m%n
log4j.appender.ria_error.File=/home/data/logs/error.log
log4j.appender.ria_error.MaxFileSize=10MB
log4j.appender.ria_error.MaxBackupIndex=10
log4j.appender.ria_mysql=org.apache.log4j.jdbc.JDBCAppender
log4j.appender.ria_mysql.URL=jdbc:mysql://localhost/DB
log4j.appender.ria_mysql.driver=com.mysql.jdbc.Driver
log4j.appender.ria_mysql.user=xxxx
log4j.appender.ria_mysql.password=xxxxx
log4j.appender.ria_mysql.sql=INSERT INTO LOGS VALUES('%p','%d{yyyy-MM-dd HH:mm:ss}','%t','%x','%c','%m')
log4j.appender.ria_mysql.layout=org.apache.log4j.PatternLayout
# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=INFO
# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
Now I get the error and fatal 'test' message in my DB and error.log. But the info and debug message gets completely lost for some reason. Also lg.isInfoEnabled() returns False.
I tried a lot of stuff around additivity but it didn't seem to solve the problem.
I would like to have log4j (v1.2.17) send ...
... to fileAppender
INFO (and above) from all packages
... to console
if from "my.project" then INFO (and above)
else WARN (and above) for all other packages
How can I do this? I'd prefer a properties file if possible, but would switch to XML if necessary.
I've tried combinations of loggers, non-additivity, Threshold, & LevelMatchFilter but can't figure it out.
This can definitely be done. Below you will find some example code and log4j properties that will do what you want.
Here is the first example class:
package my;
import org.apache.log4j.Logger;
public class Example1 {
private static final Logger logger = Logger.getLogger(Example1.class);
public static void main(String[] args){
logger.debug("debug from my.Example1 - should display nowhere!");
logger.info("info from my.Example1 - should display in log file only!");
logger.warn("warning from my.Example1 - should display in console and log file!");
logger.error("error from my.Example1 - should display in console and log file!");
logger.fatal("fatal from my.Example1 - should display in console and log file!");
}
}
Here is the second example class:
package my.project;
import org.apache.log4j.Logger;
public class Example2 {
private static final Logger logger = Logger.getLogger(Example2.class);
public static void main(String[] args){
logger.debug("debug from my.project.Example2 - should display nowhere!");
logger.info("info from my.project.Example2 - should display in console and log file!");
logger.warn("warning from my.project.Example2 - should display in console and log file!");
logger.error("error from my.project.Example2 - should display in console and log file!");
logger.fatal("fatal from my.project.Example2 - should display in console and log file!");
}
}
Here is the log4j.properties file:
# The root logger will accept INFO messages or higher and send them to the log file
# and to a console appender that filters out any messages below WARN level
log4j.rootLogger=INFO, R, warnStdout
# Configure a console appender that will be used for messages of any level
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
# Pattern to output the caller's file name and line number.
log4j.appender.stdout.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n
#This will be used to print WARN level or higher messages to console
log4j.appender.warnStdout=org.apache.log4j.ConsoleAppender
log4j.appender.warnStdout.layout=org.apache.log4j.PatternLayout
log4j.appender.warnStdout.Threshold=WARN
# Pattern to output the caller's file name and line number.
log4j.appender.warnStdout.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n
log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=test.log
log4j.appender.R.MaxFileSize=100KB
# Keep one backup file
log4j.appender.R.MaxBackupIndex=1
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n
# Classes in the my.project package will accept messages of INFO level or higher
# and send those messages to the console and to the log file
log4j.logger.my.project=INFO, stdout, R
# Need to set additivity to false or else both the my.project and root loggers
# will accept messages from classes in package my.project
log4j.additivity.my.project=false
Here is the console output after running Example1 followed immediately by Example2:
WARN [main] (Example1.java:11) - warning from my.Example1 - should display in console and log file!
ERROR [main] (Example1.java:12) - error from my.Example1 - should display in console and log file!
FATAL [main] (Example1.java:13) - fatal from my.Example1 - should display in console and log file!
INFO [main] (Example2.java:11) - info from my.project.Example2 - should display in console and log file!
WARN [main] (Example2.java:12) - warning from my.project.Example2 - should display in console and log file!
ERROR [main] (Example2.java:13) - error from my.project.Example2 - should display in console and log file!
FATAL [main] (Example2.java:14) - fatal from my.project.Example2 - should display in console and log file!
Here is the test.log file content after running Example1 followed by Example2:
INFO main my.Example1 - info from my.Example1 - should display in log file only!
WARN main my.Example1 - warning from my.Example1 - should display in console and log file!
ERROR main my.Example1 - error from my.Example1 - should display in console and log file!
FATAL main my.Example1 - fatal from my.Example1 - should display in console and log file!
INFO main my.project.Example2 - info from my.project.Example2 - should display in console and log file!
WARN main my.project.Example2 - warning from my.project.Example2 - should display in console and log file!
ERROR main my.project.Example2 - error from my.project.Example2 - should display in console and log file!
FATAL main my.project.Example2 - fatal from my.project.Example2 - should display in console and log file!
Hi i am runnig the jar of this open source Ex-Crawler
But i always receive this error :
og4j:WARN No appenders could be found for logger (eu.medsea.mimeutil.TextMimeDetector).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info
The application you're running uses log4j to produce log files. And log4j needs a configuration file, usually named log4j.properties, to be available in the application's class path, in order to start properly.
This is sample of default configuration you might start with:
log4j.rootLogger=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.conversionPattern=%5p [%t] (%F:%L) - %m%n