You've built a redundant system. Two nodes, leader-follower, automatic failover. The network blips for 30 seconds, the follower can't reach the leader, decides the leader is dead, and promotes itself. Now you have two leaders both accepting writes. When the network heals, you have two divergent histories and no clean way to merge them. Congratulations: you have a split brain.

Split brain happens when a network partition makes nodes disagree about who's in charge. Each side thinks the other is dead. Both make decisions. Reality forks.

The real-world horror story: GitHub's October 2018 outage. A 43-second network partition between their US East Coast and West Coast data centers caused MySQL Orchestrator to fail over to the West Coast. When connectivity returned, both coasts had accepted writes. It took 24 hours and 11 minutes to reconcile — they had to manually replay binary logs and reconcile divergent data. Webhooks were delayed by days.

How to prevent it:

• Quorum (majority voting): Require N/2 + 1 nodes to agree before any action. With 3 nodes, you need 2. A partition that isolates 1 node leaves it powerless — it can't form a majority alone, so it won't promote itself.
• Fencing (STONITH — "Shoot The Other Node In The Head"): Before a new leader takes over, it forcibly disables the old one. Power off via IPMI, revoke storage access, kill the network port. Brutal but effective.
• Leases with monotonic tokens: Each leader gets a token that increases. Downstream systems reject writes with stale tokens. Even if the old leader doesn't know it's been replaced, its writes get refused.

The odd-number rule of thumb: Always run quorum-based systems with an odd number of nodes. Three tolerates one failure. Five tolerates two. A 2-node cluster is the worst possible choice — losing either node loses quorum, and you can't distinguish "the other node is dead" from "I'm partitioned." Adding a third cheap node (even just an arbiter) doubles your fault tolerance.

The fencing math: For N nodes to tolerate F failures, you need N ≥ 2F + 1. Want to survive 2 simultaneous failures? You need 5 nodes minimum. Want to survive 3? You need 7. This is why etcd, Consul, and ZooKeeper docs all push you toward 3, 5, or 7 node clusters — never 4 or 6, which give you no extra tolerance over 3 or 5 but cost more.

If you find yourself reasoning "well, the other node is probably dead" — stop. Probability isn't a partition tolerance strategy. Quorum is.