MongoEngine ReplicaSet Connection fails on stepdown - python-3.x

Question

Recently I have been trying to use Mongoengine and Flask with a Replica set. I can connect but, when the primary node changes, connection is lost and there is a break.
Here's a snippet where you can test the behavior. It is using the very useful http://flip-flop.mlab.com/ site to debug replica-set problems
from flask import Flask
from mongoengine import connect
from flask_mongoengine import MongoEngine
import os
db = MongoEngine()
app = Flask(__name__)
class TestDoc(db.Document):
texto = db.StringField()
class ProductionConfig:
def get_conn_data(self):
conn = {
'host':"mongodb://testdbuser:testdbpass#flip.mongolab.com:53117,flop.mongolab.com:54117/testdb?replicaSet=rs-flip-flop",
'replicaSet': 'rs-flip-flop'
}
return conn
import time
app.config['MONGODB_SETTINGS'] = ProductionConfig().get_conn_data()
db.init_app(app)
if __name__ == '__main__':
with app.test_client() as c:
while True:
time.sleep(1)
print(TestDoc.objects().count())
TestDoc(texto="1").save()
I get every time that the primary changes an error: pymongo.errors.AutoReconnect: connection closed
.
Many thanks! I have tried a couple of different pyMongo versions but without success. Any help will be really, really appreciated!

Answer 1

The issue here is that the election of a new primary is not instantaneous. From the docs:
It varies, but a replica set will generally select a new primary within a
minute.
For instance, it may take 10-30 seconds for the members of a replica
set to declare a primary inaccessible (see electionTimeoutMillis). One
of the remaining secondaries holds an election to elect itself as a
new primary. During the election, the cluster is unavailable for
writes.
The election itself may take another 10-30 seconds.
In the time between the primary going down and the replica being elected as the new primary then there is no connection that will accept writes (because they have to go to primaries).
However, there are some things you can do to your code to make it more resilient in these situations.
Firstly, you should set a read preference on the connection (more info here):
conn = {
'host':"mongodb://testdbuser:testdbpass#flip.mongolab.com:53117,flop.mongolab.com:54117/testdb",
'replicaSet': 'rs-flip-flop',
'read_preference': ReadPreference.SECONDARY_PREFERRED
}
This means that reads should be pretty robust during the election.
Unfortunately, short of wrapping all of your writes in try blocks your code will fall over if it is trying to write during the election.
This should be less of a problem than it seems in your question's example because (assuming you are doing your writes in a flask route) the webserver will throw a 500 error response. By the time you request the route again from flask the election should be complete and mongoengine will be writing to the new primary.