Load Balancing Algorithms Explained: Round Robin, Least Connections, and Consistent Hashing

Two backend instances, identical hardware, sitting behind a round robin load balancer. One of them ends up handling requests that take ten times longer than the other’s, on average, because round robin distributes by request count, not by actual load. The instance stuck with the slow requests backs up while its neighbor idles. Nothing about the load balancer is broken; it’s doing exactly what round robin does. It’s just the wrong algorithm for traffic that isn’t uniform.

What a load balancer is actually deciding

Every request that hits a load balancer needs an answer to one question: which backend handles this one? The algorithm behind that answer is the load balancing strategy, and the differences between strategies matter a lot more than they look on paper, because real traffic is rarely uniform. Request durations vary, backend capacity varies, and some requests need to land on the same backend as a previous request from the same client. Different algorithms optimize for different versions of that problem.

Round robin: simple, and simply wrong for uneven load

Round robin sends each new request to the next backend in a fixed rotation: server A, then B, then C, then back to A. It’s the default in a lot of load balancer configurations because it’s trivial to implement and guarantees an even count of requests per backend over time.

The failure mode is that an even count isn’t an even load. If requests vary in cost, a slow database query on one, a cached lookup on another, round robin can leave one backend consistently overloaded while others sit comfortably under capacity, purely because of which requests happened to land where in the rotation. Weighted round robin helps when backends have different capacities (a bigger instance gets proportionally more requests in the rotation), but it doesn’t address the deeper issue: the algorithm still doesn’t know or care how busy each backend actually is right now.

Least connections: routing by actual load

Least connections tracks how many active connections each backend currently holds and routes each new request to whichever has the fewest:

def pick_backend_least_connections(backends):
    # backends: list of (backend_id, active_connection_count)
    return min(backends, key=lambda b: b[1])

This adapts to reality instead of a fixed rotation. If one backend is stuck processing a batch of slow requests, its connection count stays elevated, and new requests get steered toward backends that are actually free to handle them. Least response time extends this further by factoring in observed latency alongside connection count, favoring backends that are both lightly loaded and responding quickly, which catches degradation that connection count alone can miss, a backend that’s slow but not yet backed up.

The trade is that least connections needs the load balancer to track live state per backend, which round robin doesn’t. That’s a small cost for most setups, and it’s why least connections is a reasonable default over plain round robin for most stateless HTTP services with variable request cost.

Consistent hashing: when the same key needs the same backend

Round robin and least connections both assume any healthy backend can serve any request equally well. That assumption breaks for caching layers and sharded data stores, where routing request A and request B for the same cache key to different backends means duplicated cache entries at best and inconsistent reads at worst. What you actually need there is: the same key always routes to the same backend, consistently, across requests and across time.

Naively, you might hash the key and mod by the number of backends: hash(key) % num_backends. This works until a backend is added or removed, at which point num_backends changes and nearly every key’s target shifts, because the modulo of most keys changes even if only one server left the pool. For a cache, that’s a mass cache invalidation event triggered by a routine scaling operation, exactly when you can least afford it.

Consistent hashing solves this by mapping both backends and keys onto the same hash ring, and assigning each key to the next backend clockwise from its position:

import hashlib
import bisect

class ConsistentHashRing:
    def __init__(self, backends, replicas=100):
        self.replicas = replicas
        self.ring = {}
        self.sorted_keys = []
        for backend in backends:
            self.add_backend(backend)

    def _hash(self, key):
        return int(hashlib.md5(key.encode()).hexdigest(), 16)

    def add_backend(self, backend):
        for i in range(self.replicas):
            h = self._hash(f"{backend}:{i}")
            self.ring[h] = backend
            bisect.insort(self.sorted_keys, h)

    def remove_backend(self, backend):
        for i in range(self.replicas):
            h = self._hash(f"{backend}:{i}")
            del self.ring[h]
            self.sorted_keys.remove(h)

    def get_backend(self, key):
        h = self._hash(key)
        idx = bisect.bisect(self.sorted_keys, h) % len(self.sorted_keys)
        return self.ring[self.sorted_keys[idx]]

Each backend gets placed at multiple points on the ring (the replicas parameter, sometimes called virtual nodes), which smooths out the distribution so one backend doesn’t end up owning a disproportionate arc of the ring purely by hash chance. When a backend is removed, only the keys that were mapped to its specific ring positions need to move, to whichever backend is now next clockwise, not the entire keyspace. That’s the property that makes consistent hashing worth its extra complexity: scaling a cache or sharded store up or down remaps a small, bounded fraction of keys instead of nearly all of them.

Matching the algorithm to what you’re actually balancing

Algorithm	Optimizes for	Best fit
Round robin (weighted)	Even request count across backends	Stateless services with uniform, cheap requests
Least connections	Even active load across backends	Stateless services with variable request duration
Least response time	Load and observed latency together	Services where degraded-but-not-down backends need to be avoided
Consistent hashing	Stable key-to-backend mapping	Caches, sharded databases, anywhere session or data locality matters

This is the same tradeoff our database sharding strategies guide covers from the data layer’s side: the algorithm you pick depends entirely on whether “any healthy node can serve this” is true for your workload, or whether specific data needs to consistently land on a specific node. Get that distinction wrong in either direction, using consistent hashing where any backend would do, or round robin where key locality matters, and you’ve added complexity or lost correctness for nothing.

The part that makes any of this actually work

None of these algorithms function correctly without health checks feeding them accurate backend state. A load balancer that doesn’t know a backend is failing will happily route it “its fair share” under round robin, or worse, route it extra traffic under least connections if that backend is failing fast and clearing its connection count quickly, which looks like capacity rather than the problem it actually is. Pair whichever algorithm fits your workload with active health checks that pull a backend out of rotation the moment it stops responding correctly, and bring it back only after it’s demonstrated it’s actually healthy again, not just reachable.