Suppress INFO-log in Spark by changing log4j.properties file - apache-spark

Question

I read that to suppress the plethora of INFO-log messages in Spark I need to change the line
log4j.rootCategory=INFO, console
to
log4j.rootCategory=ERROR, console
in my log4j.properties, which in my case I found in
/usr/local/Cellar/apache-spark/3.3.0/libexec/conf/log4j.properties
Yet the structure of my file seems to be different to what I found from other users:
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
log4j.logger.org.apache.spark.api.python.PythonGatewayServer=error
# Set everything to be logged to the console
rootLogger.level = error
rootLogger.appenderRef.stdout.ref = console
# In the pattern layout configuration below, we specify an explicit `%ex` conversion
# pattern for logging Throwables. If this was omitted, then (by default) Log4J would
# implicitly add an `%xEx` conversion pattern which logs stacktraces with additional
# class packaging information. That extra information can sometimes add a substantial
# performance overhead, so we disable it in our default logging config.
# For more information, see SPARK-39361.
appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
# Set the default spark-shell/spark-sql log level to WARN. When running the
# spark-shell/spark-sql, the log level for these classes is used to overwrite
# the root logger's log level, so that the user can have different defaults
# for the shell and regular Spark apps.
logger.repl.name = org.apache.spark.repl.Main
logger.repl.level = warn
logger.thriftserver.name = org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
logger.thriftserver.level = warn
# Settings to quiet third party logs that are too verbose
logger.jetty1.name = org.sparkproject.jetty
logger.jetty1.level = warn
logger.jetty2.name = org.sparkproject.jetty.util.component.AbstractLifeCycle
logger.jetty2.level = error
logger.replexprTyper.name = org.apache.spark.repl.SparkIMain$exprTyper
logger.replexprTyper.level = info
logger.replSparkILoopInterpreter.name = org.apache.spark.repl.SparkILoop$SparkILoopInterpreter
logger.replSparkILoopInterpreter.level = info
logger.parquet1.name = org.apache.parquet
logger.parquet1.level = error
logger.parquet2.name = parquet
logger.parquet2.level = error
# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
logger.RetryingHMSHandler.name = org.apache.hadoop.hive.metastore.RetryingHMSHandler
logger.RetryingHMSHandler.level = fatal
logger.FunctionRegistry.name = org.apache.hadoop.hive.ql.exec.FunctionRegistry
logger.FunctionRegistry.level = error
# For deploying Spark ThriftServer
# SPARK-34128: Suppress undesirable TTransportException warnings involved in THRIFT-4805
appender.console.filter.1.type = RegexFilter
appender.console.filter.1.regex = .*Thrift error occurred during processing of message.*
appender.console.filter.1.onMatch = deny
appender.console.filter.1.onMismatch = neutral
So I simply added said line into the script. This does still not change anything.
When I type into my mac terminal spark-submit somepythonscript.py, right at the beginning of the following output, I read:
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Unfortunately I have no idea where that "org"-folder even is. This is confusing because , as stated above, my properties folder is entirely elsewhere.
I further learnt here that I should add
--conf spark.driver.extraJavaOptions=-Dlog4j.configuration=file:///usr/local/Cellar/apache-spark/3.3.0/libexec/conf/log4j.properties
to my spark-submit. In whole that'd be in my case:
spark-submit somepythonscript.py --conf
spark.driver.extraJavaOptions=-Dlog4j.configuration=file:///usr/local/Cellar/apache-spark/3.3.0/libexec/conf/log4j.properties
Yet that does not do anything. Script works, but the INFO-logging is still there. What am I doing wrong?

Answer 1

This answer has been helpful for me while I was running Spark in local mode, with the only difference being that it is log4j2.properties:
https://stackoverflow.com/a/52648367/11951587