Idempotency Keys: How to Make API Endpoints Safe to Retry

A payment request times out. The client doesn’t know if the card was charged or not, so it retries. Without protection against this exact scenario, that retry can charge the customer twice. Idempotency keys are the standard fix: a unique value the client attaches to a request so the server can recognize “I’ve already processed this” and return the original result instead of repeating the side effect.

The failure mode this actually fixes

The problem isn’t that clients retry, retrying is the correct behavior when you don’t know if a request succeeded. The problem is that a naive retry re-executes the operation, and for anything with a side effect (charging a card, creating an order, sending a notification, incrementing a counter) that means doing it twice.

Client                          Server
  |--- POST /charge ------------->|
  |                                | charges card, saves order
  |<---------- (timeout, no response) 
  |
  |--- POST /charge (retry) ----->|   <- without idempotency protection,
  |                                |      this charges the card again

The client has no reliable way to distinguish “the server never got my request” from “the server processed it and the response got lost.” Both look identical from the client’s side: silence. An idempotency key moves the decision of “have I seen this exact logical request before” onto the server, where it can actually be answered.

How the pattern works

The client generates a unique key, typically a UUID, once per logical operation, and sends it on every attempt including retries.

POST /api/charges HTTP/1.1
Idempotency-Key: 8f14e45f-ceea-467e-bd3f-8e7cd377f123
Content-Type: application/json

{"amount": 4999, "currency": "usd", "customer_id": "cus_123"}

If the client has to retry, it sends the exact same key with the exact same request. The server’s job is to recognize the key, check whether it already has a stored outcome for it, and if so, return that outcome instead of executing the operation again.

A real implementation against Postgres

Here’s a working pattern, not pseudocode. It uses a dedicated table with a unique constraint on the key to close the race condition where two concurrent requests with the same key both try to proceed.

CREATE TABLE idempotency_keys (
    key             TEXT PRIMARY KEY,
    status          TEXT NOT NULL DEFAULT 'in_progress',
    response_body   JSONB,
    response_status INT,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at      TIMESTAMPTZ NOT NULL
);

import json
from datetime import datetime, timedelta, timezone
from psycopg2 import IntegrityError

IDEMPOTENCY_WINDOW = timedelta(hours=24)

def handle_charge_request(idempotency_key, request_body, db):
    # Try to claim the key atomically. If another request already
    # claimed it, this insert fails on the unique constraint.
    try:
        db.execute(
            """INSERT INTO idempotency_keys (key, status, expires_at)
               VALUES (%s, 'in_progress', %s)""",
            (idempotency_key, datetime.now(timezone.utc) + IDEMPOTENCY_WINDOW),
        )
        db.commit()
    except IntegrityError:
        db.rollback()
        return get_existing_result(idempotency_key, db)

    # This request won the race. Do the actual work.
    try:
        result = charge_card(request_body)
        db.execute(
            """UPDATE idempotency_keys
               SET status = 'completed', response_body = %s, response_status = %s
               WHERE key = %s""",
            (json.dumps(result), 200, idempotency_key),
        )
        db.commit()
        return result, 200
    except Exception as e:
        db.execute("UPDATE idempotency_keys SET status = 'failed' WHERE key = %s", (idempotency_key,))
        db.commit()
        raise

def get_existing_result(idempotency_key, db):
    row = db.execute(
        "SELECT status, response_body, response_status FROM idempotency_keys WHERE key = %s",
        (idempotency_key,),
    ).fetchone()
    if row["status"] == "in_progress":
        # Another request is still processing this key. A 409 tells the
        # client to back off and retry, rather than racing further.
        return {"error": "request already in progress"}, 409
    return json.loads(row["response_body"]), row["response_status"]

The unique constraint on key is what actually closes the race. Two nearly-simultaneous requests with the same key both attempt the insert; the database guarantees only one succeeds, and the loser reads the winner’s eventual result instead of proceeding independently.

Why a Redis SETNX version is sometimes better

If your throughput or latency requirements make a database round-trip for key claiming too slow, Redis’s SETNX (set if not exists) gives you the same atomic claim semantics with lower latency:

import redis
r = redis.Redis()

def claim_key(idempotency_key, ttl_seconds=86400):
    # SETNX-style atomic claim: returns True only for the first caller
    claimed = r.set(f"idem:{idempotency_key}", "in_progress", nx=True, ex=ttl_seconds)
    return claimed is not None

The tradeoff is durability. Redis is fast but typically configured for performance over strict durability, so a Redis-only implementation risks losing a claimed-but-not-yet-persisted key on a crash. A common pattern is Redis for the fast claim path with the authoritative result still written to your primary database, giving you speed on the common path and durability on the record that matters.

What idempotency keys don’t do

They protect exactly the operation they’re scoped to, nothing more. A multi-step checkout flow, reserve inventory, charge the card, create the order, needs either a separate idempotency key per step or a higher-level orchestration pattern that tracks the whole workflow’s state, not one key wrapping the entire sequence. Treating one key as coverage for a multi-step process is a common design mistake; it protects the first step from duplication and does nothing for the rest.

They also don’t replace input validation or authorization checks. A replayed request with a stored idempotency key still needs to go through the same authorization checks on the way in, in case the key has expired or the requester’s permissions changed between the original request and the retry.

For workflows more complex than a single-step operation, the durable execution patterns discussed in our background jobs and workflow orchestration guide build on the same core idea, tracking state so a retry resumes correctly instead of restarting, just extended across multiple steps instead of one request.

Where to actually put this

Add idempotency key support to any endpoint with a side effect that a client might retry: payments, order creation, sending a notification, or any operation triggered by a webhook (webhooks are retried by design on many providers, which makes this pattern close to mandatory for webhook handlers). Skip it for genuinely read-only or naturally idempotent operations, adding the machinery there is complexity with nothing to show for it.

The pattern is small to implement and prevents a category of bug that’s expensive precisely because it tends to surface in production, under real network conditions, against a real customer’s payment method, not in a test suite where requests rarely time out.