ReactiveMongo with Cluster DocumentDB throw internal server error after warning timeout connection - reactivemongo

Question

We are using Playframework 2.7.x, Scala 2.13.x, ReactiveMongo 0.19.2 and a Cluster DocumentDB, when We put in the connection string pointing to cluster
mongodb://user:pwd#clusterhost:27017/db-name?replicaSet=rs0
We had this error
Dec 5 22:50:23 i-00f33fa3398767595 xxxxx INFO [warn] r.a.MongoConnection - [Supervisor-1/Connection-1] Timeout after 8200 milliseconds while probing the connection monitor: IsPrimaryAvailable#560128359?
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO [error] application -
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO ! #7e4b6iijd - Internal server error, for (GET) [/api/user-sessions/user/session] ->
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO play.api.UnexpectedException: Unexpected exception[MongoException: unexpected failed when read info to userId : 12345]
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:347)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:267)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at play.core.server.AkkaHttpServer$$anonfun$1.applyOrElse(AkkaHttpServer.scala:448)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at play.core.server.AkkaHttpServer$$anonfun$1.applyOrElse(AkkaHttpServer.scala:446)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:453)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO Caused by: com.rappi.infrastructure.repositories.mongo.exceptions.MongoException: unexpected failed when read info to userId : 12345
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at com.rappi.infrastructure.repositories.mongo.UserRepositoryMongo$$anonfun$loadUser$4.applyOrElse(UserRepositoryMongo.scala:40)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at com.rappi.infrastructure.repositories.mongo.UserRepositoryMongo$$anonfun$loadUser$4.applyOrElse(UserRepositoryMongo.scala:36)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:453)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO Caused by: reactivemongo.core.actors.Exceptions$PrimaryUnavailableException: MongoError['No primary node is available! (Supervisor-1/Connection-1)']
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at play.modules.reactivemongo.DefaultReactiveMongoApi.database(DefaultReactiveMongoApi.scala:54)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at reactivemongo.api.MongoConnection.database(MongoConnection.scala:96)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO Caused by: reactivemongo.core.actors.Exceptions$InternalState:
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at reactivemongo.ChannelDisconnected(fa5db447, {{NodeSet Some(rs0) Node[documentdb-dev.cluster-cn18vvzkqual.us-east-1.docdb.amazonaws.com:27017: Unknown (20/20/20 available connections), latency=7517999ns, authenticated={user-sessions#user-sessions}] }})(<time:16609211522842>)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at reactivemongo.ChannelDisconnected(229a0250, {{NodeSet Some(rs0) Node[documentdb-dev.cluster-cn18vvzkqual.us-east-1.docdb.amazonaws.com:27017: Primary (20/20/20 available connections), latency=7517999ns, authenticated={user-sessions#user-sessions}] }})(<time:16609164193108>)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at reactivemongo.SetUnavailable({{NodeSet Some(rs0) Node[documentdb-dev.cluster-cn18vvzkqual.us-east-1.docdb.amazonaws.com:27017: Unknown (20/20/20 available connections), latency=7517999ns, authenticated={user-sessions#user-sessions}] }})(<time:16609164187318>)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at reactivemongo.ConnectAll({{NodeSet Some(rs0) Node[documentdb-dev.cluster-cn18vvzkqual.us-east-1.docdb.amazonaws.com:27017: Primary (20/20/20 available connections), latency=7517999ns, authenticated={user-sessions#user-sessions}] }})(<time:16608976741545>)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at reactivemongo.ConnectAll$IsMaster(53, {{NodeSet Some(rs0) Node[documentdb-dev.cluster-cn18vvzkqual.us-east-1.docdb.amazonaws.com:27017: Primary (20/20/20 available connections), latency=7517999ns, authenticated={user-sessions#user-sessions}] }})(<time:16608974830690>)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at reactivemongo.IsMaster(true, 53, {{NodeSet Some(rs0) Node[documentdb-dev.cluster-cn18vvzkqual.us-east-1.docdb.amazonaws.com:27017: Primary (20/20/20 available connections), latency=16608966983407ns, authenticated={user-sessions#user-sessions}] }})(<time:16608974658158>)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at reactivemongo.RefreshAll({{NodeSet Some(rs0) Node[documentdb-dev.cluster-cn18vvzkqual.us-east-1.docdb.amazonaws.com:27017: Primary (20/20/20 available connections), latency=4699542ns, authenticated={user-sessions#user-sessions}] }})(<time:16608967488204>)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at reactivemongo.ConnectAll({{NodeSet Some(rs0) Node[documentdb-dev.cluster-cn18vvzkqual.us-east-1.docdb.amazonaws.com:27017: Primary (20/20/20 available connections), latency=4699542ns, authenticated={user-sessions#user-sessions}] }})(<time:16598976125426>)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at reactivemongo.ConnectAll$IsMaster(52, {{NodeSet Some(rs0) Node[documentdb-dev.cluster-cn18vvzkqual.us-east-1.docdb.amazonaws.com:27017: Primary (20/20/20 available connections), latency=4699542ns, authenticated={user-sessions#user-sessions}] }})(<time:16598972081992>)
Dec 5 22:50:23 i-00f33fa3398767595 user-sessions-dev INFO at reactivemongo.IsMaster(true, 52, {{NodeSet Some(rs0) Node[documentdb-dev.cluster-cn18vvzkqual.us-east-1.docdb.amazonaws.com:27017: Primary (20/20/20 available connections), latency=16598966982725ns, authenticated={user-sessions#user-sessions}] }})(<time:16598971885250>)
Dec 5 22:51:54 i-00f33fa3398767595 user-sessions-dev INFO [warn] r.c.a.MongoDBSystem - [Supervisor-1/Connection-1] The entire node set is unreachable, is there a network problem?
Dec 5 22:53:34 i-00f33fa3398767595 user-sessions-dev INFO [warn] r.c.a.MongoDBSystem - [Supervisor-1/Connection-1] The entire node set is unreachable, is there a network problem?
Dec 5 22:55:14 i-00f33fa3398767595 user-sessions-dev INFO [warn] r.c.a.MongoDBSystem - [Supervisor-1/Connection-1] The entire node set is unreachable, is there a network problem?
We try to change our string connection to pointing directly to primary node (writer) like this:
mongodb://user:pwd#primarynode:27017/db-name?replicaSet=rs0
This work properly, but, the url of primary node could change constantly, and that solution doesn't work for us.