Hazelcast OperationTimeoutException: GetOperation invocation failed to complete due to operation-heartbeat-timeout - hazelcast

Question

I am using hazelcast 3.8.3. Occasionally, I have Operation TimeoutException (GetOperation). I have 4 nodes in the same Hz cluster.
Application throws error
..
Caused by: java.util.concurrent.ExecutionException: com.hazelcast.core.OperationTimeoutException: GetOperation invocation failed to complete due to operation-heartbeat-timeout. Current time: 2020-04-23 19:17:37.414. Start time: 2020-04-23 19:14:16.456. Total elapsed time: 200958 ms. Last operation heartbeat: never. Last operation heartbeat from member: 2020-04-23 19:16:59.335. Invocation{op=com.hazelcast.map.impl.operation.GetOperation{serviceName='hz:impl:mapService', identityHash=300649867, partitionId=30, replicaIndex=0, callId=-10707, invocationTime=1587665497325 (2020-04-23 19:11:37.325), waitTimeout=-1, callTimeout=180000, name=ENRICHMENT-CACHE-DEFAULT}, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=180000, firstInvocationTimeMs=1587665656456, firstInvocationTime='2020-04-23 19:14:16.456', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 01:00:00.000', target=[10.195.40.212]:5701, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=0, connection=Connection[id=56, /10.195.40.210:5702->/10.195.40.212:60231, endpoint=[10.195.40.212]:5701, alive=true, type=MEMBER]}
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at com.experian.eda.ea.core.enrichmentpack.MultiThreadEnrichmentPackManager.evaluateAllPreEnrichmentRulesWithSizePopulator(MultiThreadEnrichmentPackManager.java:123)
... 90 common frames omitted
Caused by: com.hazelcast.core.OperationTimeoutException: GetOperation invocation failed to complete due to operation-heartbeat-timeout. Current time: 2020-04-23 19:17:37.414. Start time: 2020-04-23 19:14:16.456. Total elapsed time: 200958 ms. Last operation heartbeat: never. Last operation heartbeat from member: 2020-04-23 19:16:59.335. Invocation{op=com.hazelcast.map.impl.operation.GetOperation{serviceName='hz:impl:mapService', identityHash=300649867, partitionId=30, replicaIndex=0, callId=-10707, invocationTime=1587665497325 (2020-04-23 19:11:37.325), waitTimeout=-1, callTimeout=180000, name=ENRICHMENT-CACHE-DEFAULT}, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=180000, firstInvocationTimeMs=1587665656456, firstInvocationTime='2020-04-23 19:14:16.456', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 01:00:00.000', target=[10.195.40.212]:5701, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=0, connection=Connection[id=56, /10.195.40.210:5702->/10.195.40.212:60231, endpoint=[10.195.40.212]:5701, alive=true, type=MEMBER]}
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.newOperationTimeoutException(InvocationFuture.java:151)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:101)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:75)
at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:155)
at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:373)
at com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:304)
at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:114)
Here is the setting I use.
<map name="ENRICHMENT-CACHE-*">
<in-memory-format>BINARY</in-memory-format>
<backup-count>0</backup-count>
<max-idle-seconds>600</max-idle-seconds>
<eviction-policy>LRU</eviction-policy>
<max-size policy="PER_NODE">5000</max-size>
<eviction-percentage>25</eviction-percentage>
</map>
<properties>
<property name="hazelcast.logging.type">slf4j</property>
<property name="hazelcast.client.invocation.timeout.seconds">360</property>
<property name="hazelcast.operation.call.timeout.millis">180000</property>
</properties>
I see the code is written this way.
#Override
public final V get() throws InterruptedException, ExecutionException {
Object response = registerWaiter(Thread.currentThread(), null);
if (response != VOID) {
// no registration was done since a value is available.
return resolveAndThrowIfException(response);
}
boolean interrupted = false;
try {
for (; ; ) {
park();
if (isDone()) {
return resolveAndThrowIfException(state);
} else if (Thread.interrupted()) {
interrupted = true;
onInterruptDetected();
}
}
} finally {
restoreInterrupt(interrupted);
}
}
The current thread is parked, waiting for interruption. Could it be the heartbeat timeout exception interrupted the thread, and woke it in an errorneous state?
If yes, would increment of the hazelcast.max.no.heartbeat.seconds help?
If this is not the case, do you have any idea why the error occured & how to solve it?
Thank you.
Regards,
Khairul