Here's my log4j configuration:
log4j.rootLogger=INFO, loggerId
log4j.appender.loggerId=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.loggerId.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.loggerId.rollingPolicy.ActiveFileName=${spark.yarn.app.container.log.dir}/spark.log
log4j.appender.loggerId.rollingPolicy.FileNamePattern=${spark.yarn.app.container.log.dir}/spark_%d{dd-HHmm}.log.gz
log4j.appender.loggerId.layout=org.apache.log4j.PatternLayout
log4j.appender.loggerId.layout.ConversionPattern=%d %p %t %c usecase=${spark.yarn.appMasterEnv.usecase} usecase2=${spark.executorEnv.usecase} logdir=${spark.yarn.app.container.log.dir} - %m%n
log4j.appender.loggerId.encoding=UTF-8
In the ConversionPattern key, I can see logdir value but not usecase or usecase2. When I go to Spark UI, I can see spark.yarn.appMasterEnv.usecase & spark.executorEnv.usecase values are set. These values are set during spark-submit. My only guess is these values are set after log4j is instantiated, that's why there are no values. If that's the case, how can I set spark environment variables sooner? Or is there another way around this?
Related
Its multi tenancy application and generates lots of logs.
I want to see tenant information in an individual log statement.
I have the tenant information in my thread context.
How can i configure log4j to add tenant information to log statements by default.
I saw Conversion pattern says the pattern of log4j messages like %d [%t] %-5p %c - %m%n.
It didnt helped, not able to print thread context in it.
Say CurrentThread.getTenantName() gives me the current tenant, how could add it to log4j.
In log4j , patterns parsed by PatternParser
You can write your own parser by overriding it and parse custom literal like %i where "i" will denote tenant id in your case.
Please refer below blog for creating custom literal and parser
http://fw-geekycoder.blogspot.in/2010/07/creating-log4j-custom-patternlayout.html
I am migrating to Logback from log4j. Log4j has the %l format specifier which will print out the fully qualified name of the calling method followed by the callers source the file name and line number between parentheses. Example: com.my.company.MyClass.doSomething(MyClass.java:54)
I want the same output using Logback. They don't have the %l format specified. They do have the %caller format specifier and when you provide the {0} option you will get the first level of the call stack. Example: Caller+0 at com.my.company.MyClass.doSomething(MyClass.java:54)
I don't want the prefix "Caller+0 at ". Is there a way to do this without having to create my own format specifier in which I would just remove that part of the string?
try the following command in your compiling
javac -g
I am facing some problem regarding log message.
When I generate log report, the first line was not shown in proper alignment but rest of the log message was in proper order.
I checked log patterns, but didn't find any clue to the issue.
Can anybody suggest me the how I can resolve this issue?
Output:
INFO|------------------------------- Start Control Information --------------------------
INFO|***********
INFO|**********************
INFO|***** ************************
INFO|Doc***** Version : 6.7.0004.0217 Win64.SQLServer
INFO|------------------------------ End Control Information -----------------------------
INFO|
INFO|******************
INFO|*************** :
INFO|Version identifiers :
INFO|***********
here , first line(INFO|------------------------------- Start Control Information --------------------------
Only the first line is not properly aligned.
Logger messages are generated based on the pattern layout of log4j.properties. If you wanted to do alignmnet of your logger message, you have use pattern layout as follows.
log4j.rootLogger=DEBUG, CA
log4j.appender.CA=org.apache.log4j.ConsoleAppender
log4j.appender.CA.layout=org.apache.log4j.PatternLayout
log4j.appender.CA.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
I am using the latest ChainSaw developer snapshot
I'm trying get logs from a file with logs in pattern :
[%d] [%t] [%c] [%m]%n
and I don't know why it doesn't work - whole log is read as %m. When I use pattern without %c every log is displayed correctly.
Try this pattern instead
[TIMESTAMP] [THREAD] [LOGGER] [MESSAGE]
(I'm assuming your logs also have those "[]", otherwise remove them)
i am trying to use a log viewer (doesn't matter which one) to parse my log files.
my log4j pattern is this.
%p [%t] (%C{1}:%M():%L) %d{dd/MM/yyyy-HH:mm:ss,SSS} S:%X{serviceType} N:%X{requestID}- %m%n
the log viewers (at least the open source ones) need you to implement a pattern so they will be able to read the file.
for example:
for the log4j pattern: %p [%t] (%C{1}:%M():%L) %d{dd/MM/yyyy-HH:mm:ss,SSS} - %m%n
the log viewer pattern would be:
pattern= pattern=LEVEL [THREAD] (CLASS:METHOD():LINE) TIMESTAMP - MESSAGE
the example works well.
but i have not been able to parse the %X property in any way. i have seen there are property types NDC and PROP(key) but i seem to either miss use them or they are not related to the %X
so the question is how to implement the pattern so it will read the %X parameter.
thanks.
Ok, i think i see the problem.
Your application use the log4J MDC since it use the %X in the pattern layout. Your log viewer seems to support only NDC.
log4j pattern layout for NDC is %x (lowercase).
If you have control on the application, you have to change MDC -> NDC and modifiy the log4j.xml to use %x instead of %X. That may be a big task if the app is huge...
Another solution would be to find a log viewer that support MDC(%X)
I tried to look around for the PROP(key), but there is not much doc on it ;-(
Good luck