Leader Election Explained: How a Cluster Picks Who's in Charge

Run three replicas of a service for availability and you’ve solved one problem and created another. Availability wants every replica doing real work. But some jobs, running a scheduled cleanup task, writing to a replicated log, coordinating which replica owns which shard, break if more than one replica does them at the same time. Leader election is the mechanism that resolves that tension: it picks exactly one node out of an otherwise identical group to hold a specific role, and keeps that guarantee true even as nodes crash, restart, and lose network connectivity.

The problem in concrete terms

Say you’re running three instances of a service for redundancy, and one part of that service needs to poll an external API every minute and write the results to a database. Run that poll on all three instances and you’ve tripled your API usage and written three copies of every result. Run it on none of them because you’re worried about duplication and nothing happens at all. What you actually want is exactly one instance doing it, with automatic failover to another instance if that one goes down.

That’s the shape of every leader election problem: multiple equally capable processes, a job that must run in exactly one place, and a requirement that the system keep working correctly even when the current leader disappears without warning.

How Raft elects a leader

Raft, the consensus algorithm underneath etcd and HashiCorp Consul, is the mechanism most production leader election ultimately builds on, so it’s worth understanding even if you never implement it directly.

Every node in a Raft cluster is in one of three states: follower, candidate, or leader. Followers passively wait for heartbeats from a leader. If a follower doesn’t receive a heartbeat within a randomized election timeout, typically somewhere between 150 and 300 milliseconds, it assumes the leader is gone, becomes a candidate, increments a counter called the term, and requests votes from every other node.

Term 4: Node A is leader, sending heartbeats every 50ms
  -> Network partition isolates Node A from the rest of the cluster
  -> Nodes B, C, D stop receiving heartbeats
  -> Node B's election timeout fires first (randomized, so it wins the race)
  -> Node B becomes a candidate, increments to Term 5, requests votes
  -> Nodes C and D vote for B (they haven't heard from a Term-5-or-higher
     leader and B's log is at least as up to date as theirs)
  -> Node B wins a majority (3 of 4 nodes: B, C, D) and becomes leader
  -> Node A, still isolated, keeps believing it's the Term 4 leader
  -> When the partition heals, Node A sees Term 5 and steps down immediately

The term number is what makes this safe. Every message in Raft carries the term it was sent in, and any node that sees a higher term than its own immediately recognizes the sender as more current and defers to it. That’s exactly how Node A, still convinced it’s the leader after the partition heals, steps down the moment it observes Term 5: a higher term is unambiguous proof that an election happened without it. The randomized timeout matters too, because if every follower waited the same fixed interval, they’d all become candidates simultaneously, split the vote evenly, and repeat the same failed election indefinitely.

Winning requires a majority of the full cluster, not just of nodes that happen to be reachable, which is the property that prevents split brain: only one side of a network partition can ever contain a majority.

What you actually build on top of

Almost nobody implements Raft’s election logic themselves, for the same reason almost nobody implements TCP’s congestion control themselves. etcd and ZooKeeper already run this protocol internally and expose leader election as a primitive on top of it.

The etcd pattern uses a renewable lease. A candidate tries to create a key under a lease with a short TTL; whichever candidate’s write succeeds first holds the lease and is the leader, and it renews the lease periodically to signal it’s still alive.

import etcd3

client = etcd3.client()
lease = client.lease(ttl=10)  # leader must renew every 10s or lose it

def try_become_leader(node_id):
    # put() with create-only semantics: succeeds only if the key
    # doesn't already exist, which is what makes this a real election
    # rather than just an overwrite.
    success, _ = client.transaction(
        compare=[client.transactions.create("/election/leader") == 0],
        success=[client.transactions.put("/election/leader", node_id, lease)],
        failure=[],
    )
    return success

if try_become_leader("node-b"):
    # This process is now the leader. Keep the lease alive with periodic
    # keepalives; if the process crashes or hangs, the lease expires and
    # the key disappears, opening the field for another candidate.
    for _ in lease.refresh():
        run_leader_duties()

If the leader crashes, its lease stops being renewed, the key expires after the TTL, and the next candidate’s write succeeds. No manual failover step, no human paging themselves at 3am to promote a replica. The cluster resolves it on its own, typically within one TTL window.

Kubernetes controllers use the same underlying idea through the client-go leaderelection package, backed by a Lease object in the Kubernetes API rather than raw etcd access. This is why you can safely run three replicas of a Kubernetes controller for availability: exactly one holds the lease and does the reconciliation work at any moment, and losing that pod triggers automatic failover to a standby replica within seconds.

Leader election vs. a distributed lock

These two patterns get confused because they’re often built from the same primitive, a lease, but they solve different problems. A distributed lock protects one critical section for the duration of a single operation, then releases. Leader election grants a standing role that persists across many operations until the leader actively fails, and the group continuously works to keep exactly one leader designated, not just to prevent simultaneous access during one operation.

	Distributed lock	Leader election
Duration	One operation	Ongoing, until failure or step-down
Typical use	Prevent concurrent writes to one resource	Designate one node to own a recurring role
Failure handling	Lock expires, next request acquires it	Lease expires, cluster elects a new leader
Underlying primitive	Often the same: a renewable lease	Often the same: a renewable lease

When you actually need this

Reach for leader election when a job genuinely must run in exactly one place: a scheduled task that shouldn’t fire three times, a coordinator that assigns work to other nodes, a singleton connection to a system that only tolerates one writer. Skip it when idempotency would solve the same problem more cheaply. If duplicate execution is merely wasteful rather than actually harmful, for example three replicas polling the same read-only API and discarding duplicates, a lightweight dedup check is often less operational overhead than standing up leader election for a job that could tolerate running more than once.

The pattern is worth the setup cost specifically when “ran twice” means real damage: double-charged customers, corrupted state, conflicting writes to the same record. For everything short of that, it’s usually not the simplest tool that solves the actual problem.