Only one SparkContext may be running in this JVM - Flask - apache-spark

Question

Can anyone provide guidance on why this simple Flask app complains about Only one SparkContext may be running in this JVM. I'm not attempting to load more than one context, obviously.
Code:
import flask
from pyspark import SparkContext
from operator import itemgetter
app = flask.Flask(__name__)
#app.route('/')
def homepage():
return 'Example: /dt/140'
#app.route('/dt/<int:delaythreshold>')
def dt(delaythreshold):
global flights_rdd
flights_dict = \
flights_rdd \
.filter( lambda (day, delay): delay >= threshold ) \
.countByValue()
sorted_flight_tuples = \
sorted( flights_dict.items(), key=itemgetter(1), reverse=True )
return render_template('delays.html', tuples=sorted_flight_tuples[:5])
if __name__ == '__main__':
global flights_rdd
sc = SparkContext()
flights_rdd = \
sc.textFile('/tmp/flights.csv', 4) \
.map( lambda s: s.split(',') ) \
.map( lambda l: ( l[0][:4], int(lst[1]) ) ) \
.cache()
app.config['DEBUG'] = True
app.run(host='0.0.0.0')
Thanks in advance.

Answer 1

You probably shouldn't create "global" resources such as the SparkContext in the __main__ section.
In particular, if you run your app in debug mode the module is instantly reloaded a second time upon start - hence the attempt to create a second SparkContext. (Add e.g. print 'creating sparkcontext' to your __main__ section before creating the SparkContext - you'll see it twice).
Check the flask documenation for proposals on how to cache global resources.
Following http://flask.pocoo.org/docs/0.10/appcontext/#context-usage you could e.g. retrieve the SparkContext as follows:
from flask import g
def get_flights():
flights_rdd = getattr(g, '_flights_rdd', None)
if flights_rdd is None:
# create flights_rdd on the fly
sc = g._sc = SparkContext()
flights_rdd = \
sc.textFile('/tmp/flights.csv', 4) \
.map( lambda s: s.split(',') ) \
.map( lambda l: ( l[0][:4], int(lst[1]) ) ) \
.cache()
g._flights_rdd = flights_rdd
return flights_rdd
#app.teardown_appcontext
def teardown_sparkcontext(exception):
sc = getattr(g, '_sc', None)
if sc is not None:
sc.close()
Then use flights_rdd = get_flights() instead of the global flights_rdd.