PySpark EMR step fails with exit code 1 - apache-spark

Question

I'm learning to use AWS EMR for the first time to submit my Spark jobs. The script I'm using is very short (restaurant.py):
from pyspark import SparkContext, SQLContext
from pyspark.sql import SparkSession
class SparkRawConsumer:
def __init__(self):
self.sparkContext = SparkContext.getOrCreate()
self.sparkContext.setLogLevel("ERROR")
self.sqlContext = SQLContext(self.sparkContext)
self.df = self.sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load('zomato.csv')
if __name__ == "__main__":
sparkConsumer = SparkRawConsumer()
print(sparkConsumer.df.count())
sparkConsumer.df.groupBy("City").agg({"Average Cost for two": "avg", "Aggregate rating": "avg"})
I used the AWS GUI to submit my step, but the CLI export is
spark-submit --deploy-mode cluster s3://data-pipeline-testing-yu-chen/dependencies/restaurant.py -files s3://data-pipeline-testing-yu-chen/dependencies/zomato.csv
However, the step will run for a few minutes, and then return an exit code of 1. I'm pretty confused what exactly is going on, and finding it difficult to interpret the output of my syserr:
18/07/28 06:40:10 INFO Client: Application report for application_1532756827478_0012 (state: RUNNING)
18/07/28 06:40:11 INFO Client: Application report for application_1532756827478_0012 (state: RUNNING)
18/07/28 06:40:12 INFO Client: Application report for application_1532756827478_0012 (state: RUNNING)
18/07/28 06:40:13 INFO Client: Application report for application_1532756827478_0012 (state: FINISHED)
18/07/28 06:40:13 INFO Client:
client token: N/A
diagnostics: User application exited with status 1
ApplicationMaster host: myip
ApplicationMaster RPC port: 0
queue: default
start time: 1532759825922
final status: FAILED
tracking URL: http://myip.compute.internal:20888/proxy/application_1532756827478_0012/
user: hadoop
18/07/28 06:40:13 INFO Client: Deleted staging directory hdfs://myip.compute.internal:8020/user/hadoop/.sparkStaging/application_1532756827478_0012
Exception in thread "main" org.apache.spark.SparkException: Application application_1532756827478_0012 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1165)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/07/28 06:40:13 INFO ShutdownHookManager: Shutdown hook called
18/07/28 06:40:13 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-dedwd323x
18/07/28 06:40:13 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-dedwd323x
Command exiting with ret '1'
I am able to run the script by SSH-ing into my master instance, and then issuing spark-submit restaurant.py. I've loaded a CSV file into my master instance by using:
[hadoop#my-ip ~]$ aws s3 sync s3://data-pipeline-testing-yu-chen/dependencies/ .
Then I load my restaurant.csv file into HDFS:
hadoop fs -put zomato.csv ./zomato.csv
My guess is that the -files option I am passing in isn't getting used the way I intend it to be used, but I'm really at a loss for how to interpret the console output and begin debugging.