Distributed Locks Explained: Redis, ZooKeeper, and Where They Quietly Break

Two worker processes both think they’re the only one processing order #4821. Both read the inventory count, both decrement it, both write back. The count is now off by one, and nothing in either process’s logs looks wrong, because from each process’s point of view, nothing was. This is the problem a distributed lock exists to prevent: making sure that when multiple processes could touch the same resource at the same time, only one of them actually does.

Why an ordinary lock doesn’t help

A mutex or a threading.Lock solves mutual exclusion within a single process, sometimes within a single machine. It works because the operating system or language runtime can guarantee only one thread holds the lock at a time, because everything checking that lock lives in the same address space. Once your system spans multiple processes on multiple machines, that guarantee disappears. There’s no shared memory to hold the lock state, so you need something external that every process can see and agree on: a shared datastore, a coordination service, or a consensus-based system acting as the single source of truth for who currently holds the lock.

That’s what a distributed lock is: a lock whose state lives outside any single process, in a system all the contending processes can reach and agree with.

The Redis pattern, and what it actually buys you

The simplest and most common implementation uses Redis’s atomic SET with NX (only set if not already set) and a PX expiry:

import redis
import uuid

r = redis.Redis()

def acquire_lock(resource_name, ttl_ms=10000):
    token = str(uuid.uuid4())
    acquired = r.set(f"lock:{resource_name}", token, nx=True, px=ttl_ms)
    return token if acquired else None

The NX flag makes the claim atomic: if two processes race to set the same key, only one succeeds, and Redis guarantees that at the protocol level rather than leaving it to application logic. The TTL matters just as much as the atomic claim. Without it, a process that crashes while holding the lock leaves it held forever, and every other process is locked out permanently. With a TTL, the lock releases itself if its holder disappears.

Releasing the lock needs the same care as acquiring it. A naive DEL is wrong, because it can delete a lock some other process has since acquired after your TTL expired:

RELEASE_SCRIPT = """
if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
else
    return 0
end
"""

def release_lock(resource_name, token):
    r.eval(RELEASE_SCRIPT, 1, f"lock:{resource_name}", token)

Running the check-and-delete as a single Lua script matters here, not as a style preference. Doing the GET and the DEL as two separate round trips reopens the exact race the lock exists to close: another process could acquire the lock in the gap between your check and your delete, and your delete would remove its lock instead of your own.

The gap the TTL doesn’t close

Here’s the failure mode that catches teams who stop at the pattern above. A process acquires the lock with a 10-second TTL, then pauses, a garbage collection stop-the-world pause, a slow disk write, a scheduler delay, anything that stalls execution for longer than 10 seconds. The TTL expires while the process is paused. Another process acquires the lock and starts its own critical section. Then the first process resumes, still believing it holds the lock, since it never got a chance to notice the expiry, and proceeds to act concurrently with the second process. The lock did its job of preventing two processes from acquiring it at once. It did nothing to prevent two processes from acting as if they held it at once, which is the guarantee you actually wanted.

This isn’t a hypothetical edge case invented for a blog post; it’s the core argument Martin Kleppmann made against relying on Redis’s Redlock algorithm for correctness-critical locking, and it holds for the simple single-instance pattern too. The lock is a coordination hint, not a hard guarantee, unless something else closes this specific gap.

Fencing tokens close the gap

The fix is to stop trusting “I hold the lock” as evidence of anything on its own, and instead have the protected resource check a token that only increases:

def acquire_lock_with_fence(resource_name, ttl_ms=10000):
    token = r.incr(f"fence:{resource_name}")  # monotonically increasing
    r.set(f"lock:{resource_name}", token, nx=True, px=ttl_ms)
    return token

def write_with_fence(storage, fence_token):
    last_seen = storage.get_last_fence_token()
    if fence_token <= last_seen:
        raise StaleLockError("a newer lock holder has already written")
    storage.write(fence_token=fence_token)

Now the paused process from the earlier scenario comes back with an old fencing token, tries to write, and gets rejected, because the resource it’s writing to has already recorded a higher token from whoever acquired the lock after the TTL expired. The lock itself no longer has to be perfectly correct under every pause and network partition; the resource being protected enforces the final guarantee. This is the same principle behind idempotency keys: don’t trust the caller’s belief about what state it’s in, verify against a value the receiving side controls.

When Redis isn’t enough

The Redis pattern above is good enough for locks where an occasional false acquisition, extremely rare, and now further mitigated by fencing, is an acceptable risk relative to the operational simplicity of running one more thing on your existing Redis instance. For locks where correctness genuinely can’t bend, leader election for a system that would corrupt data with two leaders, for example, ZooKeeper or etcd are the standard choices, because they’re built on a consensus protocol (ZAB for ZooKeeper, Raft for etcd) rather than a single instance’s atomic operations.

The trade is operational weight. A consensus-based coordination service needs a cluster of its own, typically three or five nodes, to tolerate node failures without losing the ability to make decisions. You’re taking on a new piece of infrastructure to get a stronger guarantee, and that’s worth it exactly when the cost of a rare false lock acquisition is genuinely severe, not when it’s merely annoying.

Approach	Guarantee strength	Operational cost	Good fit
Redis SET NX PX	Best-effort, needs fencing for correctness	Low, one existing Redis instance	Rate limiting, deduplication, low-stakes mutual exclusion
Redis + fencing tokens	Strong, if the protected resource enforces the token	Low-medium	Anything above where a stale writer must be provably rejected
ZooKeeper / etcd	Strong, backed by consensus	Higher, needs its own cluster	Leader election, configuration coordination, anything where a split-brain is unacceptable

The question to ask before reaching for any of this

A lot of problems that look like they need a distributed lock actually need idempotency instead, and idempotency is usually cheaper and fails more gracefully. If the question is “how do I stop this operation from happening twice,” an idempotency key that makes the operation safe to repeat is often simpler than coordinating exclusive access to prevent it from starting twice in the first place. Reach for a distributed lock when the actual requirement is mutual exclusion during a window of time, two processes genuinely cannot both be running this section concurrently, not just “this operation shouldn’t have duplicate side effects.” Those are different problems, and the second one usually has a cheaper solution than the first.