Webhook Design: Signatures, Retries, and Idempotency Done Right

A webhook receiver is one of the only places in most systems where you’re building a server that has to trust a request it didn’t ask for, at a time it can’t control, possibly more than once. That inversion, someone else’s server calling yours instead of the other way around, is where most webhook bugs come from: teams build the endpoint like a normal API route and skip the parts that only matter because the caller isn’t a client you control.

Signature verification isn’t optional

Every legitimate webhook provider, Stripe, GitHub, Shopify, Twilio, signs its requests with an HMAC computed over the raw request body using a secret only you and the provider know. Verifying that signature is the only thing confirming a request to your webhook endpoint actually came from the provider and not from anyone who found or guessed the URL.

const crypto = require('crypto');

function verifyWebhookSignature(rawBody, signatureHeader, secret) {
  const expectedSignature = crypto
    .createHmac('sha256', secret)
    .update(rawBody)          // the RAW bytes, before any JSON parsing
    .digest('hex');

  // Constant-time comparison: prevents a timing side-channel that could
  // let an attacker guess the correct signature one byte at a time
  return crypto.timingSafeEqual(
    Buffer.from(expectedSignature),
    Buffer.from(signatureHeader)
  );
}

Two details here matter more than they look like they should. First, the HMAC has to run over the exact raw body bytes, not a re-serialized version of the parsed JSON. If your framework parses the body before your handler sees it, and you then re-stringify it to compute the signature, whitespace or key-ordering differences between the original bytes and your re-serialized version will produce a different hash, and every legitimate request will fail verification. Keep the raw body available to your signature check, separate from the parsed object you use for business logic.

Second, use a constant-time comparison function (crypto.timingSafeEqual in Node, hmac.compare_digest in Python), not === or ==. A naive string comparison returns as soon as it finds the first mismatched character, and the tiny timing difference between failing on character 1 versus character 30 is, in principle, an exploitable side channel for guessing a valid signature byte by byte. It’s a subtle bug to introduce and a one-line fix to avoid.

Retries mean duplicates, and duplicates are your problem

Webhook providers universally offer at-least-once delivery, not exactly-once. If your endpoint doesn’t respond successfully, or times out, the provider retries, often on a backoff schedule stretching over hours or days. That’s the correct behavior on the provider’s side: they’d rather you receive an event twice than not at all. It does mean your receiver has to be built assuming duplicates will arrive, not as an edge case, as a routine part of normal operation.

The fix is an idempotency key, almost always the event ID the provider includes in the payload:

async function handleWebhookEvent(event) {
  // Check whether we've already processed this exact event
  const alreadyProcessed = await db.processedEvents.findOne({
    eventId: event.id,
  });

  if (alreadyProcessed) {
    // Not an error. This is the expected, safe response to a duplicate delivery.
    return { status: 'already_processed' };
  }

  // Record the event ID BEFORE doing side-effecting work, inside the same
  // transaction where practical, so a crash mid-processing doesn't leave a
  // gap where a retry re-triggers the side effect.
  await db.processedEvents.insertOne({ eventId: event.id, receivedAt: new Date() });

  await applyBusinessLogic(event);
}

The ordering matters: record the event ID as processed at the same time as, or just before, the actual side-effecting work, not after. If you do the work first and record the ID last, a crash between those two steps leaves the event fully applied but not marked as seen, and the inevitable retry reapplies it. This is the same at-least-once-versus-exactly-once tradeoff we’ve covered in message queue delivery guarantees, just showing up at the HTTP boundary instead of inside a queue.

Respond fast, process later

A webhook handler that does its real work inline, calling three other APIs, sending emails, running a report, before returning a response, is setting itself up for exactly the retry storm idempotency handling exists to survive. Most providers apply a response timeout in the single-digit seconds; miss it, and the provider marks the delivery failed and retries, even though your server may have actually finished (or half-finished) the work.

The reliable pattern is to acknowledge fast and defer the actual processing:

app.post('/webhooks/provider', async (req, res) => {
  const isValid = verifyWebhookSignature(req.rawBody, req.headers['x-signature'], WEBHOOK_SECRET);
  if (!isValid) {
    return res.status(401).send('Invalid signature');
  }

  // Durably enqueue the event, then respond immediately.
  // The actual business logic runs in a background worker, not inline here.
  await eventQueue.enqueue(req.body);
  return res.status(200).send('ok');
});

This pushes the actual work onto a background job queue, the same pattern behind most production job queue setups, and it also means a slow downstream dependency (a flaky third-party API your processing logic calls) can’t turn into a webhook timeout and a duplicate delivery. The two problems, slow processing and webhook retries, would otherwise compound each other.

A minimal reference flow

Step	What happens	Why it matters
1. Receive	Capture the raw request body before any parsing	Signature verification needs the exact original bytes
2. Verify	Recompute HMAC, compare with constant-time equality	Confirms the request is genuinely from the provider
3. Deduplicate	Check the event ID against previously processed events	Handles the at-least-once retry guarantee safely
4. Acknowledge	Return 200 quickly, before doing slow work	Prevents timeout-triggered duplicate retries
5. Process	Run the actual business logic asynchronously	Keeps the webhook response fast regardless of downstream latency

Skipping any one of these five steps is where most production webhook incidents start: an unverified endpoint accepting forged requests, a handler that double-charges a customer on a retried payment event, or a slow synchronous handler that times out and gets replayed by the provider until it succeeds, sometimes applying the same side effect several times over.

The takeaway

A webhook endpoint is a small piece of infrastructure with an outsized failure surface, because it’s the one part of your API that has to accept requests from a caller you don’t control, on a schedule you don’t control, with a delivery guarantee that explicitly allows duplicates. Verify every signature with a constant-time comparison, treat every event as a possible duplicate until proven otherwise, and get out of the request-response cycle before doing real work. Those three habits cover the overwhelming majority of what goes wrong with webhooks in production. If you’re building or auditing webhook infrastructure for a client integration, that kind of reliability review is part of the backend work our team does regularly.