Change Data Capture: Streaming Changes, No Polling

A polling job that checks WHERE updated_at > last_check every thirty seconds looks fine in a demo and quietly breaks in three ways once it hits production: it never sees a hard delete, it queries a live table on a fixed schedule whether anything changed or not, and it’s always at least thirty seconds stale. Change data capture fixes all three by reading the same log the database already writes for its own crash recovery, and turning it into an event stream.

The problem with asking the table what changed

Polling treats the database like a black box you interrogate periodically. That’s workable for low-stakes freshness requirements, but it has a structural flaw: a SELECT query can only see rows that currently exist. Delete a row, and there’s nothing left to select. Teams work around this with soft deletes, a deleted_at column instead of an actual DELETE, but that means retrofitting every table and every query in the system to filter out soft-deleted rows forever.

The load problem compounds it. A poll interval tight enough to feel real-time, say every second, means a query running every second against a table whether one row changed or ten thousand did. Loosen the interval to reduce load, and you’ve traded query cost for staleness. There’s no interval that’s both cheap and fresh, because polling is fundamentally guessing how often to check.

Reading the log instead of the table

Every transactional database already keeps a durable, ordered record of every change, for its own crash recovery. Postgres calls it the write-ahead log (WAL). MySQL calls it the binary log (binlog). MongoDB calls it the oplog. This log is the actual source of truth the database replays to reconstruct state after a crash, and it’s also, conveniently, an exact, ordered record of every insert, update, and delete that ever happened.

Change data capture attaches a reader to that log instead of querying the table.

This is structurally the same attachment point a streaming replica uses. Postgres’s logical replication, the mechanism behind its native CDC support, literally decodes the WAL into a stream of row-level changes, which is exactly what a replica consumes to stay in sync, and exactly what a CDC connector consumes to publish events. You’re not adding a new kind of load to the database, you’re reading a stream it already produces.

What a CDC event actually looks like

A tool like Debezium turns each committed row change into a structured event with the before and after state:

{
  "op": "u",
  "before": { "id": 42, "status": "pending", "total_cents": 4999 },
  "after":  { "id": 42, "status": "shipped", "total_cents": 4999 },
  "source": {
    "table": "orders",
    "lsn": 24601984,
    "ts_ms": 1756345200000
  }
}

op tells you whether it was a create, update, or delete (c, u, d). before and after give you the actual row state on each side of the change, which is what lets a consumer compute a diff without a second query back to the source. source.lsn is the log sequence number, the same ordering token the database itself uses, which is what guarantees consumers see changes in true commit order, not an order reconstructed from timestamps that can collide or drift.

Setting up Debezium against Postgres is mostly configuration, not code:

{
  "name": "orders-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "db.internal",
    "database.dbname": "storefront",
    "table.include.list": "public.orders,public.order_items",
    "plugin.name": "pgoutput",
    "topic.prefix": "storefront"
  }
}

That connector creates a logical replication slot on Postgres, tails it, and publishes one Kafka topic per table, storefront.public.orders, with every change as it commits.

Where this earns its complexity

Cache invalidation that actually keeps up. Instead of a TTL-based cache that’s wrong for up to N seconds after every write, a CDC-driven invalidator sees the exact row that changed and evicts precisely that key, close to the moment of commit.

Search index sync. Elasticsearch or a similar index needs to reflect the database without the application code remembering to also call the index’s API on every write path. A CDC connector reading the database’s own log does that consistently, including for changes made by a script, a migration, or a different service that writes to the same table.

Feeding an analytics warehouse or a different database engine. Moving operational data into a warehouse without a nightly batch ETL job, and without asking the application to double-write, is one of CDC’s most common production uses, and it’s the backbone of a lot of “real-time” data pipeline architectures.

Keeping two systems in sync during a migration. Strangler-pattern migrations, where you’re moving off a legacy database gradually, often use CDC to replicate writes from the old system to the new one during the transition, so both stay consistent without a risky big-bang cutover.

CDC versus the outbox pattern

These solve adjacent problems and get confused often enough to spell out directly. The outbox pattern is for when your application needs to atomically commit a business change and an event about that change in the same transaction, so it writes both to an outbox table it controls, and a relay process publishes from there afterward. CDC is for capturing changes to tables that already exist, often ones you don’t own the write path for at all, a third-party ORM, a legacy service, an admin panel doing direct SQL, without touching application code.

The two combine well: point a CDC connector at your outbox table specifically, and you get the outbox pattern’s transactional guarantee plus CDC’s low-overhead, log-based delivery instead of a custom polling relay.

When to skip it

CDC adds real operational surface: a connector to run and monitor, a message broker if you don’t already have one, schema evolution to manage as source tables change shape. For a low-write table that a nightly batch export already serves well enough, or a freshness requirement measured in hours rather than seconds, that infrastructure is overhead without a matching payoff. It’s also worth checking your database’s replication setup first; if you’re already fighting replication lag on read replicas, adding a CDC connector to the same WAL stream is one more consumer competing for the same log, worth sizing before you add it.

Reach for CDC specifically when the cost of staleness or a missed delete is concrete, an out-of-date search index, a cache serving wrong prices, a downstream system silently drifting from the source of truth, and polling can’t close that gap without either falling behind or hammering the database. When that’s the actual failure mode you’re solving for, reading the log the database already writes is a better trade than querying it again and again and hoping nothing slipped through.