I had following deployment of sentinel - 3 redis instances on different servers, 3 sentinels on each of these servers.
Now, I realized that the current master does not have much memory, so I stopped sentinel and redis instance on this particular server. And did the same setup on a new machine. SO, still I have the same deployment, 3 redis instances and 3 sentinels.
The issue is that, now sentinels are saying, master is down, as they think the master is the server which I removed. What should I do to tell sentinel that it need not include that server in loop.
From the docs about Redis Sentinel, under the chapter Adding or removing Sentinels:
Removing a Sentinel is a bit more complex: Sentinels never forget already seen Sentinels, even if they are not reachable for a long time, since we don't want to dynamically change the majority needed to authorize a failover and the creation of a new configuration number. So in order to remove a Sentinel the following steps should be performed in absence of network partitions:
Stop the Sentinel process of the Sentinel you want to remove.
Send a SENTINEL RESET * command to all the other Sentinel instances (instead of * you can use the exact master name if you want to reset just a single master). One after the other, waiting at least 30 seconds between instances.
Check that all the Sentinels agree about the number of Sentinels currently active, by inspecting the output of SENTINEL MASTER mastername of every Sentinel.
Further:
Removing the old master or unreachable slaves.
Sentinels never forget about slaves of a given master, even when they are unreachable for a long time. This is useful, because Sentinels should be able to correctly reconfigure a returning slave after a network partition or a failure event.
Moreover, after a failover, the failed over master is virtually added as a slave of the new master, this way it will be reconfigured to replicate with the new master as soon as it will be available again.
However sometimes you want to remove a slave (that may be the old master) forever from the list of slaves monitored by Sentinels.
In order to do this, you need to send a SENTINEL RESET mastername command to all the Sentinels: they'll refresh the list of slaves within the next 10 seconds, only adding the ones listed as correctly replicating from the current master INFO output.
Related
I have setup redis on three seperate instances and have configured them in such a way that 1 instance is a master and 2 are replicas of master. I have used sentinels to make sure there is high availability of the setup. I have a nodejs application which needs to use the redis. How do i achieve the read and write splitting in my application as incase my redis master goes down one of my read replica becomes the master and the writes need to go to it.
As far has I know, ioredis is the only node redis client that supports sentinels.
"ioredis guarantees that the node you connected to is always a master even after a failover. When a failover happens, instead of trying to reconnect to the failed node (which will be demoted to slave when it's available again), ioredis will ask sentinels for the new master node and connect to it. All commands sent during the failover are queued and will be executed when the new connection is established so that none of the commands will be lost."
I've been looking into using a local entry listener instead of a normal entry listener so that an event is only processes by a single listener.
I've found various posts on this topic such as this, this, this, this and this. It seems that a local entry listener is indeed the way to go for handling an event only once in a multi node cluster.
However, I'm not sure how such a local entry listener would function under failure conditions. For instance, what happens to an evicted-event if the node which is the master for that entry is unavailable. Will the backup pick this up in time? Or could the event be missed due to hazelcast needing some time to figure out the master is down and a new master should be elected? Is this different between the older AP-system and the new CP-subsystem?
We've refrained from using a local entry listener. Instead we are now using the executorservice from hazelcast to schedule a named task. In this way, we can correctly respond to changes in the cluster. It does seem like hazelcast has a preferred member on which a task is executed, but that isn't an issue for us.
From Hazelcast docs:
Note that entries in distributed map are partitioned across the
cluster members; each member owns and manages the some portion of the
entries. Owned entries are called local entries. This listener will be
listening for the events of local entries. Let's say your cluster has
member1 and member2. On member2 you added a local listener and from
member1, you call {#code map.put(key2, value2)}. If the key2 is owned
by member2 then the local listener will be notified for the add/update
event. Also note that entries can migrate to other nodes for load
balancing and/or membership change.
The key part is: "Also note that entries can migrate to other nodes for load balancing and/or membership change.”
I guess that if an original partition owner fails, then some other node will become a new owner of those entries (or part of them, depending on cluster state after the repartitioning is done) and then it, the new owner, will run local entry listener.
According to this answer from an Azure Redis Cache team member, the Azure Redis Cache exposes a single endpoint. That endpoint is automatically routed to either the master or the slave node (on failover I assume). That answer also states that:
Azure... requires checks on the client side to ensure that the node is
indeed Master or Slave
So clients see a single endpoint and have to sometime check which instance they're talking to - that raises some questions:
When should a Redis client care whether it talks to the master or the slave node? Is it only to prevent inconsistency during failover, or are there other concerns here?
How (and when) should a client check whether it's connected to the master or the slave instance? Is it by running info replication?
From the docs:
When the master node is rebooted, Azure Redis Cache fails over to the replica node and promotes it to master. During this failover, there may be a short interval in which connections may fail to the cache.
My understanding is you never connect to the slave because it is never exposed to you. If the master goes out, the slave is promoted to master and that's what you reconnect to.
I am trying to set up a collection of docker containers and trying to emulate a cluster election process, by using etcd to store the current leader hostname with a ttl of 5 mins. I'm running a looped shell script inside every container which sleeps for 5min + random 1-9 sec, and try to create the leader key, which fails if there is an existing value, or sets a new value and the successful container becomes the new leader.
My immediate concern is, are etcdctl connections to the etcd host atomic ? What if by chance two containers connect at the same time ? And also I would appreciate suggestions on a more elegant way to implement the whole scenario.
I am looking for a way to have a "Singleton" module over multiple worker role instances.
I would like to have a parallel execution model with Queues and multiple worker roles in Azure.
The idea is that would like to have a "master" instance, that is let's say checking for new data, and is scheduling it by adding it to a queue, processing all messages from a special queue, that is not processed by nobody else, and has mounted blob storage as a virtual drive, with read/write access.
I will always have only one "master instance". When that master instance goes down for some reason, another instance from the one already instantiated should very quickly be "elected" for a master instance (couple of seconds). This should happen before the broken instance is replaced by a new one by the Azure environment (about 15 min).
So it will be some kind of self-organizing, dynamic environment.
I was thinking of having some locking, based on a storage or table data. the opportunity to set lock timeouts and some kind of "watchdog" timer if we can talk with microprocessor terminology.
There is general approach to what you seek to achieve.
First, your master instance. You could do your check based on instance ID. It is fairly easy. You need RoleEnvironment.CurrentRoleInstance to get the "Current instance", now compare the Id property with what you get out of RoleEnvironment.CurrentRoleInstance.Role.Instances first member ordered by Id. Something like:
var instance = RoleEnvironment.CurrentRoleInstance;
if(instance.Id.Equals(instance.Role.Instances.OrderBy(ins => ins.Id).First().Id))
{
// you are in the single master
}
Now you need to elect master upon "Healing"/recycling.
You need to get the RoleEnvironment's Changed event. Check if it is TopologyChange (just check whether it is topology change, you don't need the exact change in topology). And if it is Topology Change - elect the next master based on the above algorithm. Check out this great blog post on how to exactly perform events hooking and change detection.
Forgot to add.
If you like locks - blob lease is the best way to acquire / check locks. However working with just the RoleEnvironment events and the simple master election based on Instance ID, I don't think you'll need that complicated locking mechanism. Besides - everything lives in the Queue until it is successfully processed. So if the master dies before it processes something, the "next master" will process it.