Read Replicas and Replication Lag: What Breaks in Production

Adding a read replica is usually the first move once a database’s read load starts crowding out its ability to serve writes quickly. It works, and it’s the standard scaling pattern for exactly that problem. What it also does, quietly, is introduce a gap between when a write is committed and when that write is visible everywhere else, and that gap is where a specific, recurring class of production bug lives.

What replication actually is

A primary-replica setup has one database (the primary) that accepts all writes, and one or more replicas that receive a continuous stream of changes from the primary and apply them to keep their own copy of the data current.

In the common case, that replication is asynchronous: the primary commits a write, returns success to the client, and separately streams the change to replicas, which apply it on their own schedule. That last part is the source of everything discussed below. The primary doesn’t wait for a replica to confirm it applied the change before telling the client the write succeeded. It can’t, without giving up the latency and availability benefits asynchronous replication exists to provide in the first place.

Primary commits write at t=0
Client receives "write successful" at t=0
Replica receives the change at t=0 + network delay
Replica applies the change at t=0 + network delay + apply time
                              ^
                    this whole gap is replication lag

The failure that actually shows up

The lag itself is usually small, single-digit to double-digit milliseconds under normal load. The problem isn’t the typical case, it’s that the lag is unbounded and situational, and the moment it matters is exactly the moment a user is watching.

The classic sequence:

1. A user submits a form (updates a profile field, places an order, posts a comment). The write goes to the primary and succeeds.
2. The application redirects the user to a page that displays that same data, and that page’s read query goes to a replica, because reads are routed to replicas by default to keep primary load down.
3. The replica hasn’t caught up yet. The user sees the old value, or the new record missing entirely.

To the user, this looks exactly like data loss: “I just saved this and it’s gone.” The write actually succeeded. The read just happened to hit a replica that was, for a few hundred milliseconds, behind. This is one of the most common sources of “intermittent, can’t reproduce it” bug reports in systems that added read replicas without a consistency strategy to go with them.

The fix: read-your-writes, not fewer replicas

The answer isn’t to stop using replicas, it’s to route reads based on whether staleness is acceptable for that specific read, not apply a single global rule to every query.

// Naive: every read goes to a replica, including the one right after a write
async function updateProfile(userId, data) {
  await primaryDb.query('UPDATE users SET ... WHERE id = ?', [userId, data]);
  return replicaDb.query('SELECT * FROM users WHERE id = ?', [userId]); // may be stale
}

// Read-your-writes: reads that immediately follow this user's own write
// stay on the primary for a short window
async function updateProfile(userId, data) {
  await primaryDb.query('UPDATE users SET ... WHERE id = ?', [userId, data]);
  markRecentWrite(userId); // e.g. cache a flag with a few seconds' TTL
  return primaryDb.query('SELECT * FROM users WHERE id = ?', [userId]);
}

async function getProfile(userId) {
  const db = hasRecentWrite(userId) ? primaryDb : replicaDb;
  return db.query('SELECT * FROM users WHERE id = ?', [userId]);
}

This is the read-your-writes consistency guarantee: a user always sees the effect of their own write, even though other users viewing the same data might briefly see a slightly stale version from a replica. It’s a narrower, cheaper guarantee than “every read everywhere is always fully consistent,” and it’s the one that actually matches what users notice and complain about.

Some frameworks build a version of this in already; a common pattern is “stickiness,” where a session is pinned to the primary for a short window after any write it performs, then falls back to normal replica routing. If your ORM or data layer doesn’t support this natively, the flag-based approach above works in application code.

Deciding what goes where

Read type	Route to	Why
Data the current user just wrote	Primary (or a confirmed-caught-up replica)	Read-your-writes; staleness here reads as a bug
An admin actively debugging live state	Primary	Staleness undermines the entire point of looking
Public content listings, feeds, dashboards	Replica	Users have no baseline for “how fresh should this be”
Analytics and reporting queries	Replica (often a dedicated one)	Long-running and can tolerate real staleness; isolates load from user-facing replicas

A useful pattern for the last row: a dedicated replica just for analytics and reporting workloads, separate from the ones serving user-facing reads. Long-running aggregate queries are exactly the kind of load that can itself cause a replica to fall behind, and isolating that load means an expensive report doesn’t degrade the read replica ordinary users are hitting.

Monitoring the thing you’re relying on

Every major database surfaces a replication lag metric: PostgreSQL through pg_stat_replication, MySQL through Seconds_Behind_Master (or Seconds_Behind_Source on newer versions), and managed services like Amazon RDS, Cloud SQL, and Aurora expose it directly as a dashboard metric. This deserves the same alerting treatment as any other production health signal. A replica whose lag is normally under 50ms and climbs into multi-second territory is telling you something concrete: it’s under-provisioned for current read load, or a long-running write on the primary is backing up the replication stream. Catching that before it produces a wave of “my data disappeared” support tickets is exactly what the metric is for.

Replication lag isn’t a bug in your setup, it’s an inherent property of asynchronous replication that every system using read replicas has to design around explicitly. The teams that get bitten by it are the ones who added replicas purely for connection pooling and read scaling without treating consistency as a separate design decision. Route by read type, implement read-your-writes for the cases where staleness is visible to the user who caused it, and monitor lag as a first-class production signal rather than an implementation detail you only think about after something breaks.