Retries and Exponential Backoff: How to Fail Without Making It Worse

A service goes down for ninety seconds. During that window, every client that was calling it starts retrying, each with its own copy of the same naive logic: try again immediately, and if that fails, try again in exactly one second. The service comes back up. All of those retries land in the same fraction of a second, at a volume the just-recovered service can’t absorb, and it goes back down. This is a self-inflicted outage, and the fix isn’t retrying less, it’s retrying smarter.

Why retries need a shape, not just a count

Retrying on failure is the right instinct. Networks drop packets, servers get momentarily overloaded, load balancers occasionally route a request to an instance that’s mid-restart. Most of these are transient: the same request sent a second later would probably succeed. The mistake isn’t retrying, it’s retrying with no regard for what a swarm of simultaneous retries does to the thing being retried against.

Two problems compound each other in a naive retry loop. First, retrying immediately gives a struggling dependency no time to recover; you’re adding load to something that’s already failing under load. Second, if every client computes the same delay before retrying, for example a flat one-second wait, all of them retry at effectively the same instant, recreating the exact traffic spike that caused the failure in the first place. That second problem is the thundering herd, and it’s the reason “just add a delay” isn’t actually a fix on its own.

Exponential backoff: spreading retries out over time

Exponential backoff increases the wait between attempts geometrically instead of linearly or not at all:

def backoff_delay(attempt, base=0.5, cap=30):
    return min(cap, base * (2 ** attempt))

# attempt 0: 0.5s, attempt 1: 1s, attempt 2: 2s, attempt 3: 4s, attempt 4: 8s...

Each failed attempt roughly doubles the wait before the next one, up to a cap so a client doesn’t end up waiting minutes between tries. That shape matters: it means the retry traffic that follows a failure tapers off exponentially instead of hammering the dependency at a constant rate. A dependency that’s overloaded for a few seconds sees a burst of near-immediate retries, then rapidly less, giving it room to recover instead of getting reset every second by fresh retry pressure.

Why the formula above is still wrong without jitter

Exponential backoff alone doesn’t solve the thundering herd. If every client runs the exact same formula, every client that failed at the same moment computes the exact same delay, and they all retry in lockstep, just at longer and longer intervals instead of a single instant. You’ve spread the herd out in time, but it’s still a herd, arriving in synchronized waves instead of one wave.

Jitter fixes this by adding randomness to the delay so different clients’ retries land at different times instead of the same computed instant:

import random

def backoff_with_full_jitter(attempt, base=0.5, cap=30):
    max_delay = min(cap, base * (2 ** attempt))
    return random.uniform(0, max_delay)

This is “full jitter,” one of the approaches AWS’s architecture team documented in their widely-cited writeup on this exact problem: instead of waiting the full computed delay, wait a random amount between zero and that delay. Two clients that failed at the same instant now retry at different, unpredictable times, which is what actually breaks the synchronization rather than just changing its period.

The chart shows why this matters in practice. Without jitter, retries from many clients cluster into visible spikes at each backoff interval, the herd, just delayed. With full jitter, that same retry volume spreads across each window instead of clustering, so the dependency sees a smoother, lower peak load instead of a series of coordinated bursts.

Knowing what not to retry

A retry is only safe when the underlying operation is either naturally idempotent or protected to be idempotent. Retrying a GET is fine by default. Retrying a POST that charges a card, without an idempotency key or equivalent protection, risks the exact double-execution problem that pattern exists to prevent. Before adding automatic retries to any call, confirm the operation is safe to run more than once, or make it safe rather than assuming.

Status codes matter too. A 500 or a connection timeout is usually worth retrying, it’s plausibly transient. A 400 Bad Request is not; the request was malformed, and sending the identical malformed request again will fail identically every time, wasting an attempt and adding pointless load. 429 Too Many Requests and 408 Request Timeout are worth retrying but deserve special handling: if the response includes a Retry-After header, honor it directly instead of computing your own backoff, since the server is telling you exactly how long it wants you to wait.

def should_retry(status_code):
    if status_code in (429, 408):
        return True
    if 500 <= status_code < 600:
        return True
    return False  # 4xx other than 408/429 means don't retry, the request itself is the problem

Bounding the loop on both axes

A retry loop needs two limits, not one. A maximum attempt count alone can still leave a client waiting an unreasonably long time if the backoff cap is generous; five attempts with a 30-second cap could mean minutes of total wait. A maximum total elapsed time alone can still allow an unreasonable number of attempts if the backoff base is small. Bound both:

import time

def retry_with_backoff(fn, max_attempts=5, max_elapsed=60, base=0.5, cap=30):
    start = time.monotonic()
    for attempt in range(max_attempts):
        try:
            return fn()
        except TransientError:
            if attempt == max_attempts - 1:
                raise
            if time.monotonic() - start > max_elapsed:
                raise
            time.sleep(backoff_with_full_jitter(attempt, base, cap))
    raise TransientError("exhausted retries")

This is also where retries and a circuit breaker work together rather than substitute for each other. Backoff and jitter shape how a single client retries a single call. A circuit breaker tracks the health of a dependency across many calls and stops sending traffic to it at all once it’s clearly down, which is what actually protects a dependency during an extended outage instead of just a brief blip. Retries handle the transient case; circuit breakers handle the sustained one.

The one-sentence version

Retry failures that are plausibly transient and safe to repeat, back off exponentially so retry pressure tapers instead of staying constant, add jitter so clients don’t retry in lockstep, and cap both the attempt count and the total time spent retrying. Skip any one of those and you’ve built a retry loop that looks correct in a test with a single client, then makes a real outage worse the first time thousands of clients hit it at once.