Master-Slave Cluster - How to make sure the master is really dead for the slave to take over?
I have a in-house messaging system, similar to a message broker. We have one master message broker and one slave message broker. A message broker just receives messages and sends them to all nodes. The slave is acting as a node, receiving messages from master and building state so it can take over in case of master failure. Now my problem is: how can I detect, if possible and without human intervention, that the master is dead!? The master may look dead, and the slave might be tempted to take over, but then you might end up in the situation of two masters in your system. I'm trying to understand how clustering systems implement master-dead detection. Until now it looks like a human has to manually kill the master and turn on a slave, but it would be much more preferable for this process to be automatic.
I have a in-house messaging system, similar to a message broker. We have one master message broker and one slave message broker. A message broker just receives messages and sends them to all nodes. The slave is acting as a node, receiving messages from master and building state so it can take over in case of master failure.
Now my problem is: how can I detect, if possible and without human intervention, that the master is dead!? The master may look dead, and the slave might be tempted to take over, but then you might end up in the situation of two masters in your system.
I'm trying to understand how clustering systems implement master-dead detection. Until now it looks like a human has to manually kill the master and turn on a slave, but it would be much more preferable for this process to be automatic.
