How to Rotate Secrets in Production Without Downtime

Somebody eventually asks the question that kills a secrets rotation program before it starts: “what happens to the three services still using the old key while we’re rotating it?” Nobody has a good answer, so the ticket gets deprioritized, and the API key from 2023 keeps working in production exactly as it did before anyone raised the question.

That’s not laziness. It’s a reasonable reaction to a genuinely dangerous default. Rotating secrets by changing the value and redeploying breaks every consumer still holding the old one, without warning, the moment the change lands. That’s not a risk you accept for better security hygiene, it’s production breaking with a security label on it, and it’s exactly why so many teams quietly stop rotating secrets at all rather than risk it.

The quick answer

Rotate secrets with an overlap window, not a swap. Provision the new credential so it’s valid alongside the old one, update consumers to the new value, confirm every consumer has actually switched, and only then revoke the old credential. The old value keeps working the entire time you’re migrating off it. Nothing breaks because nothing was ever invalidated while still in use.

Why “rotate in place” is the anti-pattern

Picture the naive version: a Stripe key gets regenerated in the dashboard, the old key is deactivated immediately, and the new value ships in an environment variable on the next deploy. Sounds fine, until you remember your billing webhook handler, your admin tool, and a nightly refund-sync job all read that same variable from three deploy targets that don’t redeploy on the same schedule. Two pick up the new key within minutes. The third doesn’t redeploy until next week’s release train, and for six days it throws 401s against a key that no longer exists, added eighteen months ago by someone who’s since left, its existence documented nowhere anyone thought to search.

This isn’t rare. The credential doesn’t fail because rotation is risky. It fails because “rotate” got treated as “replace,” and replacement leaves no room for the gap between when you change something and when every consumer notices. It’s the same shape of mistake covered in patterns for storing secrets in production: the tool matters less than whether the process around it assumes a clean, instant cutover that production never actually gives you.

The pattern: provision, overlap, confirm, revoke

Step 1: provision the new secret alongside the old one

Generate the new credential without touching the old one: a second API key rather than a regenerated one, a new database user (or a rotation tool’s shadow credential), or a new key pair whose public component gets added wherever tokens are verified before you sign anything with it.

What goes wrong here: teams skip this step and regenerate in place because it’s one click instead of several, and some providers don’t even offer a two-valid-values option without you architecting for it. Check that before you start. If a service genuinely can’t hold two live credentials, you need a maintenance window, not a silent rotation.

Step 2: run both in parallel

This is the overlap window. Both credentials are valid at once. Update consumers to the new one, one deploy target at a time, and leave the old one alone. Nothing revokes yet.

The window’s length depends on deploy cadence and discovery risk, not a fixed rule. A service redeploying continuously might need hours; a partner integration you don’t control the deploy schedule for might need weeks. Size it to the slowest consumer you know about, then pad it for the one you don’t.

Step 3: confirm every consumer is on the new value

This is the step people skip, and the one that matters most. “Confirm” doesn’t mean “I redeployed the services I know about.” It means checking, per consumer, that traffic against the old credential has actually dropped to zero: query pg_stat_activity (or your database’s equivalent) for connections still authenticating as the old database user, or check your provider’s usage dashboard for the old API key’s ID over the last 24-48 hours.

What goes wrong here: you find nonzero traffic from something you can’t place. Don’t push through, that’s the whole point of the step. Stop, identify the source, migrate it, re-check. This is where the undocumented CI job or the mobile build still shipping the old key surfaces. Finding it here costs a delay. Finding it after step 4 costs an incident.

Step 4: revoke the old credential

Once traffic against the old value is confirmed at zero for a reasonable observation period, deactivate it. Not before. The urge to revoke early because the rotation “should” be done is exactly what turns a clean rotation into an outage.

How the overlap differs by secret type

The four-step shape holds across secret types, but what “running both in parallel” means changes.

API keys are the easy case. Most providers let a service account hold multiple valid keys at once, each independently revocable. Add a second key, migrate callers, watch usage on the old key’s ID drop to zero, then delete it. Stripe, GitHub, and most SaaS APIs support this directly.

Database credentials are harder, because a lot of databases authenticate as one user with one password, and changing that password is a hard cutover for every connection pool, not an overlap. The fix is a second database user rather than a second password on the first one: create app_user_v2 with its own grants, point new connections at it, drain the old pool, then drop app_user_v1. This “alternating users” pattern is exactly what AWS RDS and other managed databases lean on for built-in rotation.

Signing keys are the trickiest case, because the overlap isn’t about the credential being independently valid, it’s about verification accepting two keys at once. During rollover, you issue new tokens signed with the new key while verification still accepts either (commonly via a kid header in a JWT pointing at the right key in a JWKS endpoint). Only once every token signed with the old key has naturally expired can you drop old-key verification. That’s why signing-key rotation tracks token lifetime, not a calendar date: a 30-day token means the old key stays verifiable for at least 30 more days.

Automating it: an AWS Secrets Manager rotation Lambda

Doing this by hand works for a handful of secrets. Past that, the discipline only survives if it’s automated, because a human “confirm every consumer” step gets skipped under deadline pressure exactly when it matters most.

AWS Secrets Manager’s built-in rotation formalizes this pattern with version staging labels: AWSPENDING (new value, being validated), AWSCURRENT (live value), AWSPREVIOUS (last known-good, kept for rollback). The rotation Lambda implements four steps that map onto the runbook above:

def lambda_handler(event, context):
    step = event["Step"]
    secret_id = event["SecretId"]
    token = event["ClientRequestToken"]

    if step == "create_secret":
        # Step 1: provision. Generate a new credential and store it
        # under the AWSPENDING label -- the old one is untouched.
        new_value = generate_credential()
        client.put_secret_value(
            SecretId=secret_id,
            ClientRequestToken=token,
            SecretString=new_value,
            VersionStages=["AWSPENDING"],
        )

    elif step == "set_secret":
        # Step 2: overlap. Apply the AWSPENDING value to the actual
        # database or service -- e.g. create the second DB user, or
        # register the second API key -- without touching AWSCURRENT.
        pending = get_secret_value(secret_id, "AWSPENDING")
        apply_credential_to_target(pending)

    elif step == "test_secret":
        # Step 3: confirm. Use the AWSPENDING value to make a real
        # call against the target. If this fails, rotation halts here
        # and AWSCURRENT never moves -- nothing in production changed.
        pending = get_secret_value(secret_id, "AWSPENDING")
        assert can_authenticate(pending)

    elif step == "finish_secret":
        # Step 4: revoke. Move AWSCURRENT to the version that was
        # AWSPENDING; Secrets Manager automatically relabels the old
        # AWSCURRENT version as AWSPREVIOUS instead of deleting it.
        client.update_secret_version_stage(
            SecretId=secret_id,
            VersionStage="AWSCURRENT",
            MoveToVersionId=token,
            RemoveFromVersionId=get_current_version_id(secret_id),
        )

Notice finish_secret never deletes the old credential outright. It relabels it AWSPREVIOUS and leaves it retrievable, so a dependency that surfaces after cutover gets a known-good fallback instead of a forensic investigation. That’s the automated version of “don’t revoke until you’ve confirmed.”

Vault’s dynamic secrets: sidestepping the problem

There’s a second approach that avoids most of this: don’t have long-lived secrets to rotate. HashiCorp Vault’s database secrets engine generates a unique username and password per request, with a default lease (commonly one hour) and a hard maximum lease (commonly 24 hours). A service asks Vault for credentials, gets a value good for that window, and either renews it before expiry or lets it lapse and requests a new one.

$ vault read database/creds/readonly-app
Key                Value
---                -----
lease_id            database/creds/readonly-app/2f6a8b...
lease_duration       1h
username             v-readonly-app-8f3k2p1
password             A1b2C3d4E5f6G7h8!

Nobody schedules a quarterly rotation for this credential, because it’s already gone in an hour whether anyone rotates it or not. A leaked value is only useful for the remainder of its lease, a fundamentally different risk profile than a static password sitting in a .env file since 2024. We covered the tradeoffs of adopting Vault against AWS Secrets Manager and Doppler elsewhere; short version, dynamic secrets make the rotation problem mostly disappear, at the cost of a stateful service your infrastructure now depends on just to hand out credentials.

A rotation runbook worth actually using

For a secret you’re rotating manually or semi-manually, work through this in order:

1. List every known consumer, by service name, not memory. Grep your infrastructure-as-code and CI configs for the secret’s name if there’s no list yet.
2. Provision the new value without deactivating the old one.
3. Set an overlap window sized to your slowest consumer’s deploy cadence, then pad it for the consumer you haven’t found yet.
4. Migrate consumers one at a time, starting with the ones you’re most confident about.
5. Check usage or access logs for the old value at the end of the window. Nonzero traffic means stop and investigate, not push through.
6. Revoke the old value only after confirmed zero usage for a full observation cycle, at least 24 hours.
7. Document what you found. An undocumented consumer surfacing in step 5 is the most valuable output of the whole rotation. Write it down where the next person rotating this secret will see it.

The parts nobody warns you about

Three failure modes show up more than the rest, and none are about the rotation mechanics.

Secrets baked into client-side code don’t rotate on your schedule at all. A mobile app with an API key compiled into the binary keeps using it as long as anyone has that build installed, which for iOS and Android can mean months after you’ve “rotated.” Long-lived CI caches are the same problem in a different costume: a GitHub Actions cache or a self-hosted runner’s persistent environment can hold an old secret value long after your secrets manager thinks it’s gone, injecting it into builds until the cache expires or gets manually cleared.

And the one that actually causes incidents: a secret shared across more services than anyone remembers granting access to. That’s why step 1, listing consumers, is worth real time. A zero-trust approach to service identity helps structurally here, because per-service credentials with explicit grants are easier to enumerate than one shared secret three teams have quietly depended on since before anyone currently on-call joined. That’s the credential that gets revoked confidently, on schedule, by someone who did everything right on paper, and takes down a system nobody remembered was still reading it.

Rotation isn’t dangerous because the mechanism is hard. It’s the same discipline behind migrating a live database table without downtime: build the new thing alongside the old one, cut over once you’ve actually confirmed nothing still depends on the old path, and only then remove it. Rotation is dangerous when it skips straight to removal. Build the overlap window instead, and the question that ends most rotation programs (who, exactly, is using this) stops being the thing that ends the program before it starts.