How to handle error being thrown by ClusterListenerThread during the instantiation (by Spring) of my Hazelcast Client bean? - hazelcast

Question

I am using Spring to configure a (lazy loaded) Hazelcast Client that connects to a 2 member cluster.
<hz:client id="hazelcast" lazy-init="true">
<hz:group name="${HzName}" password="${HzPassword}"/>
<hz:properties>
<hz:property name="hazelcast.client.connection.timeout">10000</hz:property>
<hz:property name="hazelcast.client.retry.count">600</hz:property>
<hz:property name="hazelcast.jmx">true</hz:property>
<hz:property name="hazelcast.logging.type">slf4j</hz:property>
</hz:properties>
<hz:network smart-routing="true" redo-operation="true" connection-attempt-period="5000"
connection-attempt-limit="2">
<hz:member>${HzMember1}</hz:member>
<hz:member>${HzMember2}</hz:member>
</hz:network>
</hz:client>
My issue is:
If, at the time of my application starting, BOTH of the cluster members happen to be unavailable then I am seeing ClusterListenerThread throwing a SEVERE exception:
WARNING: Unable to get alive cluster connection, try in 4945 ms later, attempt 1 of 2.
16-Apr-2015 14:57:34 com.hazelcast.client.spi.impl.ClusterListenerThread
WARNING: Unable to get alive cluster connection, try in 4987 ms later, attempt 2 of 2.
16-Apr-2015 14:57:39 com.hazelcast.client.spi.impl.ClusterListenerThread
SEVERE: Error while connecting to cluster!
java.lang.IllegalStateException: Unable to connect to any address in the config!
at com.hazelcast.client.spi.impl.ClusterListenerThread.connectToOne(ClusterListenerThread.java:273)
at com.hazelcast.client.spi.impl.ClusterListenerThread.run(ClusterListenerThread.java:79)
Caused by: com.hazelcast.spi.exception.RetryableIOException: java.util.concurrent.ExecutionException: com.hazelcast.core.HazelcastException: java.net.ConnectException: Connection refused
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl$OwnerConnectionFuture.createNew(ClientConnectionManagerImpl.java:649)
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl$OwnerConnectionFuture.access$300(ClientConnectionManagerImpl.java:605)
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.ownerConnection(ClientConnectionManagerImpl.java:268)
at com.hazelcast.client.spi.impl.ClusterListenerThread.connectToOne(ClusterListenerThread.java:245)
...which subsequently results in my Hazelcast Client bean not being instantiated and therefore everything downstream that relies upon it's existence blowing up as well.
SEVERE: Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcast': Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public static com.hazelcast.core.HazelcastInstance com.hazelcast.client.HazelcastClient.newHazelcastClient(com.hazelcast.client.config.ClientConfig)] threw exception; nested exception is java.lang.IllegalStateException: Cannot get initial partitions!
How can I detect and handle the error being thrown by ClusterListenerThread during the instantiation (by Spring) of my Hazelcast Client bean?
nb. I'm frustrated by the fact that the Client bean is not being constructed (despite there being no available members) because I know for a fact that an already constructed Client can handle having all of it's member's become unavailable and will happily start working again when one or more of those members become available again.

Answer 1

What you need to do is configure connectionAttemptLimit and connectionAttemptPeriod.
If you set connectionAttemptLimit to INT_MAX and pick a reasonable connectiontAttemptPeriod, the client will practically try to connect to given memberlist forever every X seconds. This is currently more like a workaround, because when you first open HazelcastClient it will block until one of the members become available. A further workaround, you can try to open the client in another dedicated thread.
About making a fully supported feature, there was a discussion going on by the hazelcast team here.
https://github.com/hazelcast/hazelcast/issues/552
I am adding a question as a reference to the issue.