getting CSV sink metrics files from spark-submit at run time - apache-spark

Question

Having metrics.properties in /conf (enabling CSV sink) as follows (see configuration below), collects metrics every time you submit a job (using spark-submit) and it works by saving it to /tmp/
# Enable CsvSink for all instances
*.sink.csv.class=org.apache.spark.metrics.sink.CsvSink
# Polling period for CsvSink
*.sink.csv.period=1
*.sink.csv.unit=minutes
# Polling directory for CsvSink
*.sink.csv.directory=/tmp/
# Worker instance overlap polling period
worker.sink.csv.period=1
worker.sink.csv.unit=minutes
Now I want to give metrics.properties file at run time (using the same configuration as above), and I gave the arguments for spark-submit as follows:
$spark_home/bin/spark-submit --files=file:///home/log_properties/metrics.properties --conf spark.metrics.conf=./metrics.properties --class com.myClass job1.jar
And I get the following warning and I don't have any Graphite configuration in my metrics.properties file (I just used the metrics.template and enabled the above csv configurations only)
WARN graphite.GraphiteReporter: Unable to report to Graphite
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:244)
at javax.net.DefaultSocketFactory.createSocket(SocketFactory.java:277)
at com.codahale.metrics.graphite.Graphite.connect(Graphite.java:118)
at com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:167)
at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162)
at org.apache.spark.metrics.sink.GraphiteSink.report(GraphiteSink.scala:91)
at org.apache.spark.metrics.MetricsSystem$$anonfun$report$1.apply(MetricsSystem.scala:114)
at org.apache.spark.metrics.MetricsSystem$$anonfun$report$1.apply(MetricsSystem.scala:114)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.metrics.MetricsSystem.report(MetricsSystem.scala:114)
at org.apache.spark.SparkContext$$anonfun$stop$3.apply$mcV$sp(SparkContext.scala:1715)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1219)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1714)
at org.apache.spark.SparkContext$$anonfun$3.apply$mcV$sp(SparkContext.scala:596)
at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:267)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:239)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:239)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:239)
Is it defaulted to report to Graphite and is ignoring my metrics.properties (which only is enabled for CSV sink)????

Answer 1

pass the conf like this -Dspark.metrics.conf=metrics.properties not via --conf spark.metrics.conf=./metrics.properties
Thats the reason why even though your file is added it is not used for the metrics config, it instead uses the default metrics.properties

Answer 2

yeah I realized I had metrics.properties file locally (from the directory where I run the spark-submit ) but what I passed i.e.
--files=file:///home/log_properties/metrics.properties in the spark-submit doesn't... while I resolved the issue by updating the local file (removing the Graphite flags). I am still puzzled on why it should care about the local file (metrics.properties) when I have already passed the metrics.properties that I want to use for my job.