Split-Brain Resolution Process

Overview

A Split Brain situation is a state where each of the ScaleArc nodes in a HA setup believes it is the only active surviving node. 

This article provides the steps to quickly recover from a HA Split-Brain situation if it occurs.

 

Prerequisites

  • Applicable Releases: ScaleArc up to 3.11.0.2, does not apply to 3.11.0.4, 3.12 or 20XX.X and later releases
  • Access to the ScaleArc appliance via SSH
  • Credentials for the idb user

 

Solution

Split-brain resolution is achieved using a witness server which is not a ScaleArc node.

Witness server is the system that will keep track of the current Primary node and in case of a split-brain, determine the node most capable of taking over resources.

Follow these steps to resolve the split-brain situation:

  1.  Try to identify Secondary ScaleArc prior to the split-brain situation.
    • If you are not sure, try the two methods below to identify which node was secondary:
    • cat /etc/ha.d/haresources

      The host displayed here is usually the primary

      cat /logs/ha_idb.log

      This could reveal primary / secondary stance prior to the split-brain situation
  2. Login to the Secondary ScaleArc UI and navigate to "SETTINGS > HA Settings" and click on the Restart button. This will restart the heartbeat on the Secondary ScaleArc and try to auto-resolve the split-brain.
  3. If the above step does NOT resolve the issue, then perform the following:
    • SSH into the Secondary ScaleArc and stop the heartbeat service using the below command:
      • # sudo service heartbeat stop
    • Login to the Primary ScaleArc UI and restart heartbeat
    • The above 2 steps should resolve the split-brain situation
    • Now start the heartbeat service on the Secondary ScaleArc using this command:
      • sudo service heartbeat start

Back to top

Comments

0 comments

Please sign in to leave a comment.