Why is saveAsTextFile hanging when I try to output an RDD to HDFS? - apache-spark

Question

I'm trying to write an RDD[String] to HDFS (in spark-shell) using:
output.saveAsTextFile("hdfs://localhost:9000/datasets/result")
However, it just hangs - and the job doesn't even appear in the web UI. I have to kill the SparkSubmit process.
I'm reading data successful from HDFS using:
val input = sc.textFile("hdfs://localhost:9000/datasets/data.csv")
I can output.collect successfully, and writing to local files works as expected.
I'm using Spark 1.4 and Hadoop 2.6. Everything is running on a local machine.
Any ideas?
Comments made me realize that, I should switch on DEBUG level logs. Initial log extract below. There's something about the connection being closed. But, I read the data from HDFS at the start of the short script, so I'm confused.
Solution (kind of)
Must be something to do with local routing. I replaced localhost with 127.0.0.1 and it's working now.
Debug logs
15/06/30 17:04:42 DEBUG ClosureCleaner: +++ Cleaning closure <function1> (org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1$$anonfun$30}) +++
15/06/30 17:04:42 DEBUG ClosureCleaner: + declared fields: 1
15/06/30 17:04:42 DEBUG ClosureCleaner: public static final long org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1$$anonfun$30.serialVersionUID
15/06/30 17:04:42 DEBUG ClosureCleaner: + declared methods: 2
15/06/30 17:04:42 DEBUG ClosureCleaner: public final java.lang.Object org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1$$anonfun$30.apply(java.lang.Object)
15/06/30 17:04:42 DEBUG ClosureCleaner: public final scala.collection.Iterator org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1$$anonfun$30.apply(scala.collection.Iterator)
15/06/30 17:04:42 DEBUG ClosureCleaner: + inner classes: 1
15/06/30 17:04:42 DEBUG ClosureCleaner: org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1$$anonfun$30$$anonfun$apply$49
15/06/30 17:04:42 DEBUG ClosureCleaner: + outer classes: 0
15/06/30 17:04:42 DEBUG ClosureCleaner: + outer objects: 0
15/06/30 17:04:42 DEBUG ClosureCleaner: + populating accessed fields because this is the starting closure
15/06/30 17:04:42 DEBUG ClosureCleaner: + fields accessed by starting closure: 0
15/06/30 17:04:42 DEBUG ClosureCleaner: + there are no enclosing objects!
15/06/30 17:04:42 DEBUG ClosureCleaner: +++ closure <function1> (org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1$$anonfun$30) is now cleaned +++
15/06/30 17:04:42 DEBUG BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false
15/06/30 17:04:42 DEBUG BlockReaderLocal: dfs.client.read.shortcircuit = false
15/06/30 17:04:42 DEBUG BlockReaderLocal: dfs.client.domain.socket.data.traffic = false
15/06/30 17:04:42 DEBUG BlockReaderLocal: dfs.domain.socket.path =
15/06/30 17:04:42 DEBUG DFSClient: No KeyProvider found.
15/06/30 17:04:43 DEBUG Client: IPC Client (832019786) connection to localhost/127.0.0.1:9000 from user: closed
15/06/30 17:04:43 DEBUG Client: IPC Client (832019786) connection to localhost/127.0.0.1:9000 from user: stopped, remaining connections 0
15/06/30 17:04:48 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#4eb1e256,BlockManagerId(driver, localhost, 51349)),true) from Actor[akka://sparkDriver/temp/$L]
15/06/30 17:04:48 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#4eb1e256,BlockManagerId(driver, localhost, 51349)),true)
15/06/30 17:04:48 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.290877 ms) AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#4eb1e256,BlockManagerId(driver, localhost, 51349)),true) from Actor[akka://sparkDriver/temp/$L]
15/06/30 17:04:48 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 51349)),true) from Actor[akka://sparkDriver/temp/$M]
15/06/30 17:04:48 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 51349)),true)
15/06/30 17:04:48 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.257991 ms) AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 51349)),true) from Actor[akka://sparkDriver/temp/$M]
15/06/30 17:04:57 DEBUG RetryUtils: multipleLinearRandomRetry = null
15/06/30 17:04:57 DEBUG Client: getting client out of cache: org.apache.hadoop.ipc.Client#7d1fc150
15/06/30 17:04:57 DEBUG DataTransferSaslUtil: DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
15/06/30 17:04:57 DEBUG PairRDDFunctions: Saving as hadoop file of type (NullWritable, Text)
15/06/30 17:04:57 DEBUG Client: The ping interval is 60000 ms.
15/06/30 17:04:57 DEBUG Client: Connecting to localhost/81.200.64.50:9000
15/06/30 17:04:58 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#7aba4180,BlockManagerId(driver, localhost, 51349)),true) from Actor[akka://sparkDriver/temp/$N]
15/06/30 17:04:58 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#7aba4180,BlockManagerId(driver, localhost, 51349)),true)
15/06/30 17:04:58 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.277965 ms) AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#7aba4180,BlockManagerId(driver, localhost, 51349)),true) from Actor[akka://sparkDriver/temp/$N]
15/06/30 17:04:58 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 51349)),true) from Actor[akka://sparkDriver/temp/$O]
15/06/30 17:04:58 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 51349)),true)
15/06/30 17:04:58 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.258037 ms) AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 51349)),true) from Actor[akka://sparkDriver/temp/$O]