How to reduce logs for Apache Spark in EMR? - apache-spark

Question

I have a question regarding the Apache Spark job running on AWS EMR. Each time when I executed the Spark job it generated a lot of logs, in my case the logs size around 5-10GB, but the 80% of the logs is information(useless), how can I reduce those logs?
I was used log4j2 for Spark to change the log level to "warn" to avoid the unnecessary logs but as those logs from different components in spark some of theose logs from YARN, some of the logs from EMR, it merged together. so how to fix this issue? Does anyone have such experiences? because for me I don't want to re-configuration each node in the cluster.
I have tried the below solution, seems it doesn't work in the EMR
Logger logger = LogManager.getLogger("sparklog");
logger.setlevel()
xml configuration below.
String used to match the log4j2.xml configuration files
<Configuration status="WARN" monitorInterval="300">////reload the configuration file each 300 seconds
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" /> //// control output format
</Console>
</Appenders>
<Loggers>
<Logger name="sparklog" level="warn" additivity="false">//// configuration the myloger loger level
<AppenderRef ref="Console" />
</Logger>
<Root level="error">
<AppenderRef ref="Console" />
</Root>
</Loggers>
</Configuration>

Answer 1

Since no one answers my question, here I got solutions by myself.
1.upload the configuration file to your master node.
scp -i ~/.ssh/emr_dev.pem /Users/x/log4j_files/log4j.properties hadoop#ec2-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com:/usr/tmp/
2.In your submit script just attach
"--files": "/usr/tmp/log4j.properties"
This above solution is working properly for me.

Answer 2

Configuring Applications - Amazon EMR
when creating EMR - log level should be set to INFO in config.json
...
[
{
"Classification": "spark-log4j",
"Properties": {
"log4j.rootCategory": "INFO, console"
}
}
]
...
use config.json when creating EMR
aws emr create-cluster --release-label emr-5.27.0 --applications Name=Spark \
--instance-type m4.large --instance-count 2 --service-role EMR_DefaultRole --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole --configurations https://s3.amazonaws.com/mybucket/myfolder/config.json
pyspark example to set WARN as default DEBUG when troubleshooting
from org.apache.spark.sql import SparkSession
spark = SparkSession.builder().master("/emr/spark/master").getOrCreate()
#normal run
spark.sparkContext.setLogLevel("WARN")
#troubleshooting
spark.sparkContext.setLogLevel("DEBUG")

Answer 3

EMR 6.9.0 with Spark 3.3.0 uses log4j2 so this is the configuration you need to provide in order to change the log output. You can do this for the entire Spark cluster and steps by setting a custom "spark-log4j2" configuration. The example below demonstrates setting a "KafkaConsumer" logger to "warn" level, any valid log4j2 properties can be appended as necessary.
In the EMR console you do this under "Software settings - optional" when creating the cluster and then amend the JSON configuration something like this:
[
{
"Classification": "spark-log4j2",
"Properties": {
"logger.KafkaConsumer.name": "org.apache.kafka.clients.consumer.KafkaConsumer",
"logger.KafkaConsumer.level": "warn"
}
}
]
If you're creating an EMR cluster from the command line then do something like this:
aws emr create-cluster \
--name "Your cluster name" \
--release-label "emr-6.9.0" \
--configurations '[{"Classification":"spark-log4j2","Properties":{"logger.KafkaConsumer.name":"org.apache.kafka.clients.consumer.KafkaConsumer","logger.KafkaConsumer.level":"warn"}}]' \
.... other arguments go here