I want to see the INFO level logs that Ignite prints during its running(so that it will help me on what Ignite is doing). I am using the following log4j.properties to make Ignite print INFO level logs,but the logs are not printed out.
Are there special configuration to make Ignite logs printed out?
Thanks.
log4j.rootCategory=INFO, stdout , R
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%p %d - [TS] %c %M(%L) - %m%n
log4j.appender.R=org.apache.log4j.DailyRollingFileAppender
log4j.appender.R.File=c:/ioc.ignite.log
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %d - [TS] %c %M(%L) - %m%n
log4j.logger.org.apache.ignite=INFO
log4j.logger.org.springframework=WARN
Please follow below steps:
e.g. if you are using log4j for logging
configure ignite/config/ignite-log4j.xml in gridLogger of IgniteConfiguration
e.g.
.........
uncomment "CONSOLE" appender in ignite/config/ignite-log4j.xml
copy ignite/libs/optional/ignite-log4j/log4j.jar and ignite/libs/optional/ignite-log4j/ignite-log4j.jar in ignite/libs/ folder
you can also set IGNITE_LOG_HOME environment variable to redirect the logs to any file
You should use <CLASSPATH>/config/ignite-log4j.xml file instead of log4j.properties.
Related
I am running a PySpark job using Cloud Dataproc, and want to log info using the logging module of Python. The goal is to then push these logs to Cloud Logging.
From this question, I learned that I can achieve this by adding a logfile to the fluentd configuration, which is located at /etc/google-fluentd/google-fluentd.conf.
However, when I look at the log files in /var/log, I cannot find the files that contain my logs. I've tried using the default python logger and the 'py4j' logger.
logger = logging.getLogger()
logger = logging.getLogger('py4j')
Can anyone shed some light as to which logger I should use, and which file should be added to the fluentd configuration?
Thanks
tl;dr
This is not natively supported now but will be natively supported in a future version of Cloud Dataproc. That said, there is a manual workaround in the interim.
Workaround
First, make sure you are sending the python logs to the correct log4j logger from the spark context. To do this declare your logger as:
import pyspark
sc = pyspark.SparkContext()
logger = sc._jvm.org.apache.log4j.Logger.getLogger(__name__)
The second part involves a workaround that isn't natively supported yet. If you look at the spark properties file under
/etc/spark/conf/log4j.properties
on the master of your cluster, you can see how log4j is configured for spark. Currently it looks like the following:
# Set everything to be logged to the console
log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c: %m%n
# Settings to quiet third party logs that are too verbose
...
Note that this means log4j logs are sent only to the console. The dataproc agent will pick up this output and return it as the job driver ouput. However in order for fluentd to pick up the output and send it to Google Cloud Logging, you will need log4j to write to a local file. Therefore you will need to modify the log4j properties as follows:
# Set everything to be logged to the console and a file
log4j.rootCategory=INFO, console, file
# Set up console appender.
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c: %m%n
# Set up file appender.
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=/var/log/spark/spark-log4j.log
log4j.appender.file.MaxFileSize=512KB
log4j.appender.file.MaxBackupIndex=3
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.conversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c: %m%n
# Settings to quiet third party logs that are too verbose
...
If you set the file to /var/log/spark/spark-log4j.log as shown above, the default fluentd configuration on your Dataproc cluster should pick it up. If you want to set the file to something else you can follow the instructions in this question to get fluentd to pick up that file.
I have a problem where log4j is not logging to one of my files and i'm not sure why. I have some code that runs scripts, the scripts can add logging which is logged to a file using log4j, I am trying to create an appender that only logs for a particular script.
log4j.logger.com.my.class=INFO, JS_LOG
log4j.appender.JS_LOG.layout=org.apache.log4j.PatternLayout
log4j.appender.JS_LOG.Encoding=UTF-8
log4j.appender.JS_LOG.File=${log.outputdir}/js_service.log
log4j.appender.JS_LOG.MaxFileSize=2MB
log4j.appender.JS_LOG.MaxBackupIndex=10
log4j.appender.JS_LOG.Append=true
log4j.appender.JS_LOG=org.apache.log4j.RollingFileAppender
log4j.appender.JS_LOG.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n
the above appender works and i get this in my log file
2012-04-18 11:25:52,043 [<MD> Inc Msg Dispatch-1 New] INFO com.my.class.file:/myfile - info
when the script logs something it logs to a logger of the form
com.my.class.file:/myfile this appears in the log using the above config in my log4j.properties file.
if I change the above to:
log4j.logger.com.my.class.file:/myfile=INFO, JS_LOG
log4j.appender.JS_LOG.layout=org.apache.log4j.PatternLayout
log4j.appender.JS_LOG.Encoding=UTF-8
log4j.appender.JS_LOG.File=${log.outputdir}/js_service.log
log4j.appender.JS_LOG.MaxFileSize=2MB
log4j.appender.JS_LOG.MaxBackupIndex=10
log4j.appender.JS_LOG.Append=true
log4j.appender.JS_LOG=org.apache.log4j.RollingFileAppender
log4j.appender.JS_LOG.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n
nothing is logged to the file. I would have thought this should work? I retrieve the file
I managed to solve this problem, its something to do with the file: in my log4j.properties I changed the appender to file_ then in my code I do some manipulation to the file url to make it match the appender.
i have the following log4j.properties file. what i want to do is log everything at the level INFO or above to the console. however, for anything that falls in the demo.* package, i want to log everything at the DEBUG level to a file. my log4j.properties below does not seem to work. what happens is that any DEBUG message from demo.* still gets output to the console. any idea what i am doing wrong?
how can i exclude demo.* from logging to the console?
log4j.rootLogger=INFO, C
log4j.logger.demo=DEBUG, R1
log4j.appender.C=org.apache.log4j.ConsoleAppender
log4j.appender.C.target=System.err
log4j.appender.C.layout=org.apache.log4j.PatternLayout
log4j.appender.C.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
log4j.appender.R1=org.apache.log4j.RollingFileAppender
log4j.appender.R1.File=output.log
log4j.appender.R1.MaxFileSize=5MB
log4j.appender.R1.MaxBackupIndex=5
log4j.appender.R1.layout=org.apache.log4j.PatternLayout
log4j.appender.R1.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
sorry, never mind. i solved it by setting the threshold of the console appender.
log4j.appender.C.threshold=INFO
I have a small problem using log4j and FileAppender. When I am using ConsoleAppender, everything is working well but FileAppender won't ever produce a file (or write into a file). Moreover, I have no errors concerning log4j in the osgi console.
Here's my configuration :
# main link (console or file, depending on what we want)
# log4j.category.com.foo=DEBUG, console
log4j.category.com.foo=DEBUG, file
# log4j.category.com.foo=DEBUG, console, file
# console logging
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%-4r %-5p %c %x - %m%n
# file logging
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.File=C:\\Users\\foo\\some_file.log
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%-4r %-5p %c %x - %m%n
Note that the ConversionPattern attributes are exactly the same in the two appenders.
I have tried to change the File argument to write the log in the "run" directory but it didn't do anything.
I also tried to add log4j.appender.file.ImmediateFlush=true without success.
Have I missed something (obvious ?) somewhere ? It sound strange that log4j isn't even able to write a simple log in a file....
Thanks a lot in advance for your help.
Replace :
log4j.category.com.foo=DEBUG, console
By :
log4j.category.com.foo=DEBUG, console, file
I know this is old but for other Googlers...
For me it was simply using wrong package.
I used org.apache.logging.log4j instead org.apache.log4j
If you are using Maven use this:
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
I'm trying to print my log messages to a file. At this point, it only logs to the console which is not ideal for future support.
My log4j.properties file looks like this:
log4j.rootCategory=DEBUG, stdout, logfile
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - <%m>%n
log4j.appender.logfile=org.apache.log4j.RollingFileAppender
log4j.appender.logfile.File:/apps/wlserver10/bpdomain/servers/HRServer/logs/HRServer.log
log4j.appender.logfile.MaxFileSize=4MB
log4j.appender.logfile.MaxBackupIndex=10
log4j.appender.logfile.Append=true
log4j.appender.logfile.layout=org.apache.log4j.PatternLayout
log4j.appender.logfile.layout.ConversionPattern=%d %p [%c] - %m%n
This works fine on my local weblogic instance, but when I try to deploy it to the server it doesn't write the log file.
Can anyone please help me find why?
Thanks in advance
Yols
Is it the missing = after File instead of : ?
log4j.appender.logfile.File:/apps/wlserver10/bpdomain/servers/HRServer/logs/HRServer.log
Did you miss '=' in this part of your configuration:
log4j.appender.logfile.File**:/**apps/wlserver10/bpdomain/servers/HRServer/logs/HRServer.log
Regards