Distributed Locks with Redis and ZooKeeper, Explained

Two worker processes both think they’re the only one processing order #4821. Both read the inventory count, both decrement it, both write back, and the count ends up off by one with nothing in either process’s logs looking wrong. This is exactly the problem distributed locks exist to prevent: making sure that when multiple processes could touch the same resource at once, only one of them actually does. Redis and ZooKeeper are the two systems teams reach for most often to build one, and they close the gap in different ways, which is the difference this post walks through.

Why an ordinary lock doesn’t help

A distributed lock is a lock whose state lives outside any single process, in a system every contending process can reach and agree with, unlike a threading.Lock, which only works within one process’s memory. Once your system spans multiple machines, no shared memory exists to hold that state, so you need an external datastore, coordination service, or consensus system to act as the single source of truth for who currently holds the lock.

An ordinary mutex depends on the operating system or language runtime guaranteeing only one thread holds it at a time, which only works because everything checking that lock lives in the same address space. That guarantee disappears the moment two processes on two machines both think they own the same critical section.

The Redis pattern, and what it actually buys you

The simplest and most common implementation uses Redis’s atomic SET with NX (only set if not already set) and a PX expiry:

import redis
import uuid

r = redis.Redis()

def acquire_lock(resource_name, ttl_ms=10000):
    token = str(uuid.uuid4())
    acquired = r.set(f"lock:{resource_name}", token, nx=True, px=ttl_ms)
    return token if acquired else None

The NX flag makes the claim atomic: if two processes race to set the same key, only one succeeds, and Redis guarantees that at the protocol level rather than leaving it to application logic. The TTL matters just as much as the atomic claim. Without it, a process that crashes while holding the lock leaves it held forever, and every other process is locked out permanently. With a TTL, the lock releases itself if its holder disappears.

Releasing the lock needs the same care as acquiring it. A naive DEL is wrong, because it can delete a lock some other process has since acquired after your TTL expired:

RELEASE_SCRIPT = """
if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
else
    return 0
end
"""

def release_lock(resource_name, token):
    r.eval(RELEASE_SCRIPT, 1, f"lock:{resource_name}", token)

Running the check-and-delete as a single Lua script matters here, not as a style preference. Doing the GET and the DEL as two separate round trips reopens the exact race the lock exists to close: another process could acquire the lock in the gap between your check and your delete, and your delete would remove its lock instead of your own.

The gap the TTL doesn’t close

Here’s the failure mode that catches teams who stop at the pattern above. A process acquires the lock with a 10-second TTL, then pauses, a garbage collection stop-the-world pause, a slow disk write, a scheduler delay, anything that stalls execution for longer than 10 seconds. The TTL expires while the process is paused. Another process acquires the lock and starts its own critical section. Then the first process resumes, still believing it holds the lock, since it never got a chance to notice the expiry, and proceeds to act concurrently with the second process. The lock did its job of preventing two processes from acquiring it at once. It did nothing to prevent two processes from acting as if they held it at once, which is the guarantee you actually wanted.

This isn’t a hypothetical edge case invented for a blog post; it’s the core argument Martin Kleppmann made against relying on Redis’s Redlock algorithm for correctness-critical locking, and it holds for the simple single-instance pattern too. The lock is a coordination hint, not a hard guarantee, unless something else closes this specific gap.

Fencing tokens close the gap

The fix is to stop trusting “I hold the lock” as evidence of anything on its own, and instead have the protected resource check a token that only increases:

def acquire_lock_with_fence(resource_name, ttl_ms=10000):
    token = r.incr(f"fence:{resource_name}")  # monotonically increasing
    r.set(f"lock:{resource_name}", token, nx=True, px=ttl_ms)
    return token

def write_with_fence(storage, fence_token):
    last_seen = storage.get_last_fence_token()
    if fence_token <= last_seen:
        raise StaleLockError("a newer lock holder has already written")
    storage.write(fence_token=fence_token)

Now the paused process from the earlier scenario comes back with an old fencing token, tries to write, and gets rejected, because the resource it’s writing to has already recorded a higher token from whoever acquired the lock after the TTL expired. The lock itself no longer has to be perfectly correct under every pause and network partition; the resource being protected enforces the final guarantee. This is the same principle behind idempotency keys: don’t trust the caller’s belief about what state it’s in, verify against a value the receiving side controls.

When Redis isn’t enough

The Redis pattern above is good enough for locks where an occasional false acquisition, extremely rare, and now further mitigated by fencing, is an acceptable risk relative to the operational simplicity of running one more thing on your existing Redis instance, the same tradeoff that makes Redis a reasonable fit for rate limiting patterns elsewhere in a system. For locks where correctness genuinely can’t bend, leader election for a system that would corrupt data with two leaders, for example, ZooKeeper or etcd are the standard choices, because they’re built on a consensus protocol (ZAB for ZooKeeper, Raft for etcd) rather than a single instance’s atomic operations.

The trade is operational weight. A consensus-based coordination service needs a cluster of its own, typically three or five nodes, to tolerate node failures without losing the ability to make decisions. You’re taking on a new piece of infrastructure to get a stronger guarantee, and that’s worth it exactly when the cost of a rare false lock acquisition is genuinely severe, not when it’s merely annoying. Leader election specifically is the same problem a service mesh’s control plane has to solve internally, which is part of why those systems lean on the same consensus protocols rather than a simpler datastore lock.

Approach	Guarantee strength	Operational cost	Good fit
Redis SET NX PX	Best-effort, needs fencing for correctness	Low, one existing Redis instance	Rate limiting, deduplication, low-stakes mutual exclusion
Redis + fencing tokens	Strong, if the protected resource enforces the token	Low-medium	Anything above where a stale writer must be provably rejected
ZooKeeper / etcd	Strong, backed by consensus	Higher, needs its own cluster	Leader election, configuration coordination, anything where a split-brain is unacceptable

The question to ask before reaching for any of this

A lot of problems that look like they need a distributed lock actually need idempotency instead, and idempotency is usually cheaper and fails more gracefully. If the question is “how do I stop this operation from happening twice,” an idempotency key that makes the operation safe to repeat is often simpler than coordinating exclusive access to prevent it from starting twice in the first place. Reach for a distributed lock when the actual requirement is mutual exclusion during a window of time, two processes genuinely cannot both be running this section concurrently, not just “this operation shouldn’t have duplicate side effects.” Those are different problems, and the second one usually has a cheaper solution than the first, the same distinction that shapes whether a multi-step workflow needs the saga pattern instead of a lock, and whether your message processing needs exactly-once or at-least-once delivery guarantees.