Hazelcast Scheduled Executor always starts task on newest member - hazelcast

Question

I am using Hazelcast Scheduled Executor Service in order to run a specific task, on one service instance only. In order to achieve this behavior we take advantage of com.hazelcast.scheduledexecutor.TaskUtils.named(java.lang.String, java.lang.Runnable) decorator to avoid duplicate tasks.
Hazelcast dependencies used:
implementation "com.hazelcast:hazelcast-all:4.2"
implementation "com.hazelcast:hazelcast-kubernetes:2.2.2" // DNS Lookup
We use a hazelcast cluster with one member per service instance
Code example:
public void scheduleTask() {
IScheduledExecutorService es = hazelcastInstance.getScheduledExecutorService("myScheduledExecutor");
try {
es.scheduleAtFixedRate(named("taskName", task)), 0, 30, SECONDS);
} catch (DuplicateTaskException ex) {
System.out.println("Task was already scheduled!");
}
}
The above example manages to partially achieve the desired behavior. The only issue is that every time a new instance spins up, the scheduled executor will run the task on that specific instance. This is not ideal since we would like the task to execute, for example, once every 6 hours.
Is there any way to configure the scheduled executor so that it maintains running the task on the original instance it started and transfers to another instance only if the original one goes down?

Answer 1

This is not directly possible, but there is an approximation.
IScheduledExecutorService does not provide a method to provide an ordering for which members to use.
You can submit a task to a specific member, but this won't failover to a member of your choosing if the original specific member dies.
... various other options ...
Or the option in your code, scheduleAtFixedRate which will pick a member, and this may be a different member each time the cluster changes size. It should not always be the newest unless by co-incidence.
What you could do is have a scheduled task that selects a member to run an ordinary task upon. One task that launches another.
In the scheduled task, the run() could call hazelcastInstance.getCluster().getMembers() to get the list of members in the cluster. All it needs is some logic to pick a member, and then do hazelcastInstance.getExecutorService("default").executeOnMember(runnable, member).
You might pick a member with a specific attribute that you configure. Or a member with a specific IP. Perhaps simplest is to pick the oldest, since this doesn't change until the oldest leaves, in which case 2nd olest is now oldest, easy failover.

Answer 2

The workaround I used was to dispose already scheduled task and schedule again with an initial delay which I calculate based on the last time the task was scheduled.
Something like:
public void scheduleTask() {
IScheduledExecutorService es = hazelcastInstance.getScheduledExecutorService("myScheduledExecutor");
try {
disposePreviousScheduledTasks(es);
es.scheduleAtFixedRate(named("taskName", task)), initialDelay, 30, SECONDS);
} catch (DuplicateTaskException ex) {
System.out.println("Task was already scheduled!");
}
}