Hazelcast 3.6 - java.io.IOException: No available connection to address - hazelcast

Question

i am using hazelcast 3.6 cluster consist of 2 nodes .
my client configuration is :
ClientConfig clientConfig = new ClientConfig();
clientConfig.getGroupConfig().setName("dev").setPassword("dev-pass");
String[] list = hazelcastServerList.toString().split(" ");
clientConfig.getNetworkConfig().addAddress(list);
clientConfig.getNetworkConfig().setConnectionAttemptLimit(5);
clientConfig.getNetworkConfig().setSmartRouting(true);
HazelcastInstance client = HazelcastClient.newHazelcastClient(clientConfig);
i see that sometimes i get this error :
error is :java.io.IOException: No available connection to address Address[{node1_address}]:5701
i wonder :
why it happens
why it is not failed over to the second node, this is the whole purpose of the cluster , isn't it ?
i don't know if it is related or not , but the address of the hazelcast servers is recognized behind a VPN network , and is resolved to a private IP.
member config is :
Copyright (C) 2012.
Olaf Bergner.
Hamburg, Germany. olaf.bergner#gmx.de
All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an "AS IS"
BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied. See the License for the specific language
governing permissions and limitations under the License.
-->
<hazelcast
xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.6.xsd"
xmlns="http://www.hazelcast.com/schema/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
id="hazelcast-server.defaultInstance">
<properties>
<property
name="hazelcast.logging.type">slf4j</property>
<property
name="hazelcast.version.check.enabled">false</property>
<property
name="hazelcast.mancenter.enabled">false</property>
<property
name="hazelcast.memcache.enabled">true</property>
<property
name="hazelcast.rest.enabled">true</property>
<property
name="hazelcast.log.state">true</property>
<property
name="hazelcast.jmx">true</property>
<property
name="hazelcast.jmx.detailed">true</property>
<property
name="hazelcast.executor.client.thread.count">100</property>
</properties>
<group>
<name>dev</name>
<password>dev-pass</password>
</group>
<management-center
enabled="false">http://localhost:8080/mancenter</management-center>
<network>
<port
auto-increment="true">5701</port>
<join>
<multicast
enabled="false">
<multicast-group>IP</multicast-group>
<multicast-port>54327</multicast-port>
<multicast-timeout-seconds>3</multicast-timeout-seconds>
</multicast>
<tcp-ip connection-timeout-seconds="60"
enabled="true">
<!-- <connection-timeout-seconds>60</connection-timeout-seconds> -->
<interface>hostname1:5701</interface>
<interface>hostname2:5701</interface>
</tcp-ip>
</join>
<interfaces
enabled="false">
<interface>10.10.1.*</interface>
</interfaces>
<ssl
enabled="false" />
<socket-interceptor
enabled="false" />
</network>
<partition-group
enabled="false" />
<executor-service name="exec">
<pool-size>16</pool-size>
<!--Queue capacity. 0 means Integer.MAX_VALUE.-->
<queue-capacity>0</queue-capacity>
<statistics-enabled>true</statistics-enabled>
<!-- <core-pool-size>50</core-pool-size>
<max-pool-size>200</max-pool-size>
<keep-alive-seconds>60</keep-alive-seconds> -->
</executor-service>
<map name="default">
<!--
Number of backups. If 1 is set as the backup-count for example, then all entries of
the map will be copied to another JVM for fail-safety. 0 means no backup.
-->
<backup-count>1</backup-count>
<!--
Maximum number of seconds for each entry to stay in the map. Entries that are
older than <time-to-live-seconds> and not updated for <time-to-live-seconds>
will get automatically evicted from the map.
Any integer between 0 and Integer.MAX_VALUE. 0 means infinite. Default is 0.
-->
<time-to-live-seconds>86400</time-to-live-seconds>
<!--
Maximum number of seconds for each entry to stay idle in the map. Entries that are
idle(not touched) for more than <max-idle-seconds> will get
automatically evicted from the map. Entry is touched if get, put or containsKey is called.
Any integer between 0 and Integer.MAX_VALUE. 0 means infinite. Default is 0.
-->
<max-idle-seconds>86400</max-idle-seconds>
<!--
Valid values are:
NONE (no eviction),
LRU (Least Recently Used),
LFU (Least Frequently Used).
NONE is the default.
-->
<eviction-policy>LFU</eviction-policy>
<!--
Maximum size of the map. When max size is reached,
map is evicted based on the policy defined.
Any integer between 0 and Integer.MAX_VALUE. 0 means
Integer.MAX_VALUE. Default is 0.
-->
<max-size policy="PER_NODE">100000</max-size>
<!--
When max. size is reached, specified percentage of
the map will be evicted. Any integer between 0 and 100.
If 25 is set for example, 25% of the entries will
get evicted.
-->
<eviction-percentage>15</eviction-percentage>
<!--
Minimum time in milliseconds which should pass before checking
if a partition of this map is evictable or not.
Default value is 100 millis.
-->
<min-eviction-check-millis>100</min-eviction-check-millis>
<!--
While recovering from split-brain (network partitioning),
map entries in the small cluster will merge into the bigger cluster
based on the policy set here. When an entry merge into the
cluster, there might an existing entry with the same key already.
Values of these entries might be different for that same key.
Which value should be set for the key? Conflict is resolved by
the policy set here. Default policy is PutIfAbsentMapMergePolicy
There are built-in merge policies such as
com.hazelcast.map.merge.PassThroughMergePolicy; entry will be
overwritten if merging entry exists for the key.
com.hazelcast.map.merge.PutIfAbsentMapMergePolicy ; entry will be added if the merging entry doesn't exist in the cluster.
com.hazelcast.map.merge.HigherHitsMapMergePolicy ; entry with the higher hits wins.
com.hazelcast.map.merge.LatestUpdateMapMergePolicy ; entry with the latest update wins.
-->
<merge-policy>com.hazelcast.map.merge.LatestUpdateMapMergePolicy</merge-policy>
</map>
<map name="local">
<!--
Number of backups. If 1 is set as the backup-count for example,
then all entries of the map will be copied to another JVM for
fail-safety. Valid numbers are 0 (no backup), 1, 2, 3.
-->
<backup-count>1</backup-count>
<!--
Maximum number of seconds for each entry to stay in the map. Entries
that are
older than <time-to-live-seconds> and not updated for <time-to-live-
seconds>
will get automatically evicted from the map.
Any integer between 0 and Integer.MAX_VALUE. 0 means infinite.
Default is 0.
-->
<time-to-live-seconds>86400</time-to-live-seconds>
<!--
Maximum number of seconds for each entry to stay idle in the map. Entries that are
idle(not touched) for more than <max-idle-seconds> will get
automatically evicted from the map.
Entry is touched if get, put or containsKey is called.
Any integer between 0 and Integer.MAX_VALUE.
0 means infinite. Default is 0.
-->
<max-idle-seconds>86400</max-idle-seconds>
<!--
Valid values are:
NONE (no extra eviction, <time-to-live-seconds> may still apply),
LRU (Least Recently Used),
LFU (Least Frequently Used).
NONE is the default.
Regardless of the eviction policy used, <time-to-live-seconds> will still apply.
-->
<eviction-policy>LRU</eviction-policy>
<!--
Maximum size of the map. When max size is reached,
map is evicted based on the policy defined.
Any integer between 0 and Integer.MAX_VALUE. 0 means
Integer.MAX_VALUE. Default is 0.
-->
<!-- <max-size policy="cluster_wide_map_size">0</max-size> -->
<max-size policy="PER_NODE">100000</max-size>
<!--
When max. size is reached, specified percentage of
the map will be evicted. Any integer between 0 and 100.
If 25 is set for example, 25% of the entries will
get evicted.
-->
<eviction-percentage>15</eviction-percentage>
<!--
Specifies when eviction will be started. Default value is 3.
So every 3 (+up to 5 for performance reasons) seconds
eviction will be kicked of. Eviction is costly operation, setting
this number too low, can decrease the performance. -->
<!--
Minimum time in milliseconds which should pass before checking
if a partition of this map is evictable or not.
Default value is 100 millis.
-->
<min-eviction-check-millis>100</min-eviction-check-millis>
<!--
While recovering from split-brain (network partitioning),
map entries in the small cluster will merge into the bigger cluster
based on the policy set here. When an entry merge into the
cluster, there might an existing entry with the same key already.
Values of these entries might be different for that same key.
Which value should be set for the key? Conflict is resolved by
the policy set here. Default policy is PutIfAbsentMapMergePolicy
There are built-in merge policies such as
com.hazelcast.map.merge.PassThroughMergePolicy; entry will be
overwritten if merging entry exists for the key.
com.hazelcast.map.merge.PutIfAbsentMapMergePolicy ; entry will be
added if the merging entry doesn't exist in the cluster.
com.hazelcast.map.merge.HigherHitsMapMergePolicy ; entry with the
higher hits wins.
com.hazelcast.map.merge.LatestUpdateMapMergePolicy ; entry with the
latest update wins.
-->
<merge-policy>com.hazelcast.map.merge.LatestUpdateMapMergePolicy</merge-
policy>
</map>
</hazelcast>

Answer 1

In your member config, you need to change tcp joiner config like
<tcp-ip connection-timeout-seconds="60" enabled="true">
<!--connection-timeout-seconds>60</connection-timeout-seconds -->
<member>hostname1:5701</member>
<member>hostname2:5701</member>
</tcp-ip>
In this case, client config should look like
ClientConfig clientConfig = new ClientConfig();
// those are default values, it's not necessary to explicitly set it
clientConfig.getGroupConfig().setName("dev").setPassword("dev-pass");
String hazelcastServerList = "hostname1:5701 hostname2:5701";
String[] list = hazelcastServerList.split(" ");
clientConfig.getNetworkConfig().addAddress(list);
clientConfig.getNetworkConfig().setConnectionAttemptLimit(5);
// enabled by default
clientConfig.getNetworkConfig().setSmartRouting(true);
HazelcastInstance client = HazelcastClient.newHazelcastClient(clientConfig);
p.s. for the best performance, a client and the members should be on the same local network. To understand different ways to configure which network interfaces Hazelcast will use / listen, kindly, consult with documentation
Best,
Vik
p.p.s if you have any questions, write them in comments below.