Eclipse ditto forwarding connection fails from time to time - eclipse-ditto

Question

I have connected Eclipse hono with Eclipse ditto using the Connectivity api. When I set it up, this works fine. However, after some time the forwarding connection fails. When I retrieve the metrics, I'm getting following response:
{
"?": {
"?": {
"type": "connectivity.responses:aggregatedResponse",
"status": 200,
"connectionId": "<connectionId>",
"responsesType": "connectivity.responses:retrieveConnectionMetrics",
"responses": {
"connectivity-7cc7b5dc4c-6nn59": {
"type": "connectivity.responses:retrieveConnectionMetrics",
"status": 200,
"connectionId": "<connectionId>",
"connectionMetrics": {
"connectionStatus": "open",
"connectionStatusDetails": "Connected at 2019-03-19T08:28:53.211Z",
"inConnectionStatusSince": "2019-03-19T08:28:53.211Z",
"clientState": "CONNECTED",
"sourcesMetrics": [],
"targetsMetrics": [
{
"addressMetrics": {
"gw/{{ thing:namespace }}/{{ thing:id }}": {
"status": "failed",
"statusDetails": "Producer closed at 2019-03-19T21:00:16.466Z",
"messageCount": 2048,
"lastMessageAt": "2019-03-19T21:00:05.361Z"
}
},
"publishedMessages": 4070
}
]
}
}
}
}
}
}
I've been checking the logs around the time mentioned, but I'm not getting any errors. The logs I'm posting here are the last one before and the first one after the mentioned timestamp (2019-03-19T21:00:16.466Z).
2019-03-19 21:00:11,771 DEBUG [ID:AMQP_NO_PREFIX:TelemetrySenderImpl-42872] o.e.d.s.c.m.a.AmqpPublisherActor akka://ditto-cluster/system/sharding/connection/7/tenant_aloxy_consumer-aloxy-forward/pa/$a/c1/amqpPublisherActor3
- Message JmsTextMessage { org.apache.qpid.jms.provider.amqp.message.AmqpJmsTextMessageFacade#9bc051af } sent successfully.
2019-03-19 21:01:11,733 DEBUG [ID:AMQP_NO_PREFIX:TelemetrySenderImpl-42872] o.e.d.s.c.m.a.AmqpClientActor akka://ditto-cluster/system/sharding/connection/1/tenant_aloxy_consumer-aloxy/pa/$a/c1 - Inbound message: JmsInboundMessageDispatch { sequence = 38885, messageId = TelemetrySenderImpl-42873, consumerId = ID:a4925b59-1bb4-4cd8-9151-96ad422c36df:1:1:1 }
Although the log levels for all ditto services are set to debug, I'm not getting any useful logging.
Does any of you have any idea how I can get the loggging to investigate this problem or, even better, have any idea on what the problem might be and how to fix it?
When I delete the connection and recreate it, everything works as expected again. Maybe ditto can do this under the hood automatically?
UPDATE
When retrieving the connection via the API, I'm getting following response (including the failoverEnabled property which is set to true). This also indicates that the connection uses AMQP 1.0. The broker used is Enmasse.
{
"?": {
"?": {
"type": "connectivity.responses:retrieveConnection",
"status": 200,
"connection": {
"id": "<connectionId>",
"name": null,
"connectionType": "amqp-10",
"connectionStatus": "open",
"uri": "amqp://<consumer>:<password>#<amqp-host>:5672",
"sources": [],
"targets": [
{
"address": "gw/{{ thing:namespace }}/{{ thing:id }}",
"topics": [
"_/_/things/twin/events?filter=exists(features/alp)"
],
"authorizationContext": [
"<auth-context>"
]
}
],
"clientCount": 1,
"failoverEnabled": true,
"validateCertificates": true,
"processorPoolSize": 5,
"tags": []
}
}
}
}

Answer 1

Eclipse Ditto does an automatic failover if configured to so do (see https://www.eclipse.org/ditto/basic-connections.html - "failoverEnabled" property in the model).
It could however be that this was improved since the release 0.8.0 you are using.
The Ditto team is currently working towards a 0.9.0-M1 release which would contain an improved reconnection behavior.
Does the connection to Eclipse Hono automatically reconnect?
You described that the "forwarding connection" fails from time to time. Which technology (broker, etc.) is as endpoint for that gw/{{ thing:namespace }}/{{ thing:id }} address?