Write Spark logs to STDOUT instead of STDERR - apache-spark

Question

I have a service that reads the logs from STDOUT for further analysis. It seems like there has been struggle with writing spark logs to STDOUT, by default, log4j sends any kind of log to STDERR.
Is there a way to change this behavior?
What changes need to be made specifically to move logs from STDERR to STDOUT?
Here's what my log4j file looks like:
log4j.rootLogger=INFO, FILE
log4j.appender.FILE=org.apache.log4j.RollingFileAppender
log4j.appender.FILE.File=stderr
log4j.appender.FILE.ImmediateFlush=true
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.conversionPattern=....
log4j.appender.FILE.MaxFileSize=5248997
log4j.appender.FILE.MaxBackupIndex=10
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

Answer 1

When you do a spark submit add 2>&1 at the the end. This means combine stderr(2) and stdout(1) into the stdout stream.
To do it through log4j.properties file, try adding the below properties.
# Log everything INFO and above to stdout
log4j.rootLogger=INFO,console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.Threshold=INFO
log4j.appender.console.Target=System.out
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=[%d] %-5p %.26c [%X{testName}] [%X{akkaSource}] - %m%n