Using Keras Model in Flask App with Threading - multithreading

Question

I have 3 working Keras models and each of those I saved it with 2 file structure.json (saving Model Structure) and weight.h5.
I've built a Flask App to load these model. But currently I can't add threading=True to option of run() like this:
api.run(threaded=True)
So I am only able to use:
api.run()
In this flask app I have two different api (1 GET and 1 POST). Because it runs on one thread so it works too slowly. My system has over 100 connections by seconds, and each connection I have to load another Model.
Note that all of my keras model have the same structure. I only have to load one structure, and whenever a new connection is coming, I load weight to that structure.
My code in api likes this:
# Compile model
json_file = open(formal_model_path, 'r')
loaded_model_json = json_file.read()
json_file.close()
model = model_from_json(loaded_model_json)
model.compile(loss='mean_absolute_error', optimizer='adam')
#api.route("/predict", methods=["POST"])
try:
model.load_weights(os.path.join(cointainer_folder, 'weight.h5'))
except Exception:
return jsonify(
error_code='model_file_reading_failed'
)
My code ran errors when enabling threading=True at line model.load_weights[...].
Are there any solutions to build multithreading API with loading many different Keras model?

Answer 1

I think you are running into two separate problems:
You are loading model weights per request. That's a bad idea and will make every request very slow.
flask uses multiple threads. Tensorflow models loaded in one thread, must be used in that same thread.
The right place to load models would be an init method. You also need to use tf.get_default_graph() to make sure that you are loading models and predicting in the same thread.
Here's what your code might look like
def init():
global models
models = {}
for idx, model_path in enumerate(model_paths):
with open(model_path, "r") as fp:
model = model_from_json(json.load(fp))
model.compile(loss='mean_absolute_error', optimizer='adam')
model.load_weights(os.path.join(model_path, "weights.h5"))
models[idx] = model
# save default graph in a global var
global graph
graph = tf.get_default_graph()
And inside your request handler
#api.route("/predict", methods=["POST"])
def predict():
# select your model based on something inside the request
# making up func !
model = models[func(request)]
with graph.as_default():
model.predict(..)