Spark Custom Logging - apache-spark

Question

I have multiple spark projects in my IDE. By default spark is picking log4j.properties file in spark/conf folder.
As I have multiple spark projects, I want have multiple log4j.properties files(per project one). Probably as part of the project code(resources folder)
Is there a way we can pickup specified log4j.properries instead of default log4j.properties.
Note:
I tried this
--driver-java-options "-Dlog4j.configuration=file:///usr/local/Cellar/apache-spark/2.4.1/libexec/conf/driver_log4j.properties"
and it worked without any issues, however I'm looking for something like below.
however I want to load log4j.properties file which is in resource folder while creating the spark logger.
class SparkLogger():
def __init__(self, app_name, sparksession = None):
self._spark = sparksession
self.log4jLogger = None
if self._spark is not None:
sparkContext =self._spark.sparkContext
self.log4jLogger = sparkContext._jvm.org.apache.log4j
self.log4jLogger = self.log4jLogger.LogManager.getLogger(app_name)
def info(self, info):
if self.log4jLogger:
self.log4jLogger.info(str(info))
def error(self, info):
if self.log4jLogger:
self.log4jLogger.error(str(info))
def warn(self, info):
if self.log4jLogger:
self.log4jLogger.warn(str(info))
def debug(self, info):
if self.log4jLogger:
self.log4jLogger.debug(str(info))

Answer 1

You have to define application_name log logger properties in log4j file. When you call get logger method using applicaiton_name, you will able to access customized application basis logs generation.

Answer 2

I have attempted to build my custom Logging just like what u described in your question but failed at last. I have to say it was totally a waste.
Finally I chose java.util.logging instead of log4j. Actually it is an original Logging util within JDK. The purpose I use it is that I wanna log information only for myself into a specified file.
So the class is like below.
package org.apache.spark.internal
import java.io.File
import java.text.SimpleDateFormat
import java.util.Date
import java.util.logging._
import scala.collection.mutable
protected [spark] object YLogger extends Serializable with Logging {
private var ylogs_ = new mutable.HashMap[String, Logger]()
private def initializeYLogging(className: String): Unit = {
// Here we set log file onto user's home.
val rootPath = System.getProperty("user.home")
val logPath = rootPath + File.separator + className + ".log"
logInfo(s"Create ylogger for class [${className}] with log file named [${logPath}]")
val log_ = Logger.getLogger(className)
val fileHandler = new FileHandler(logPath, false)
val formatter = new Formatter {
override def format(record: LogRecord): String = {
val time = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date)
new StringBuilder()
.append("[")
.append(className)
.append("]")
.append("[")
.append(time)
.append("]")
.append(":: ")
.append(record.getMessage)
.append("\r\n")
.toString
}
}
fileHandler.setFormatter(formatter)
log_.addHandler(fileHandler)
ylogs_.put(className, log_)
}
private def ylog(logName: String): Logger = {
if (!ylogs_.contains(logName)) {
initializeYLogging(logName)
}
ylogs_.get(logName).get
}
def ylogInfo(logName: String)(info: String): Unit = {
if (ylog(logName).isLoggable(Level.INFO)) ylog(logName).info(info)
}
def ylogWarning(logName: String)(warning: String): Unit = {
if (ylog(logName).isLoggable(Level.WARNING)) ylog(logName).warning(warning)
}
}
And you can use it like below.
YLogger.ylogInfo("logFileName") ("This is a log.")
It's quite simple to use, I hope my answer could help u.