Wall Clock vs Monotonic Clock: Measure Time Right

Use a monotonic clock to measure how long something took, and the wall clock only to say when something happened. If you subtract two Date.now() or time.Now() readings to get a duration, you have a bug that is simply waiting for the clock to move.

It sounds pedantic until you read what happened to Cloudflare’s DNS on New Year’s Day 2017.

The bug: a negative duration

At midnight UTC on January 1, 2017, a leap second was applied. Cloudflare’s RRDNS service, written in Go, measured how fast upstream resolvers answered by subtracting two time.Now() values:

rtt := time.Now().Sub(start)

Cloudflare’s post-mortem puts it plainly: rtt could be negative if time.Now() was earlier than start. Their code then smoothed many such measurements, and eventually the smoothed value went negative. That number was fed into weighted server selection, which called rand.Int63n(), and rand.Int63n panics when its argument is negative.

The blast radius was small, which is the only reason it’s a story we can laugh at. About 0.2% of DNS queries and under 1% of HTTP requests saw errors at the peak, and the impact ended at 06:45 UTC. The cause, in Cloudflare’s own words, was the belief that time cannot go backwards.

Two clocks, two jobs

Your machine keeps two different notions of time, and they exist because the jobs conflict.

	Wall clock	Monotonic clock
Answers	”What time is it?"	"How much time has passed?”
Can jump backward	Yes (NTP step, admin change, leap second)	No
Meaningful on its own	Yes, it’s a calendar time	No, the starting point is arbitrary
Use for	Log timestamps, created_at, certificates, schedules	Timeouts, latency, retries, rate limits, TTLs in memory

Python’s documentation describes time.monotonic() as a clock “that cannot go backwards” and “not affected by system clock updates,” with an undefined reference point so that only the difference between two calls is valid. The same docs warn that time.time() “can return a lower value than a previous call if the system clock has been set back between the two calls.”

That one sentence is the entire argument.

The fix, language by language

Python

import time

start = time.perf_counter()
result = do_work()
elapsed = time.perf_counter() - start     # seconds, never negative

time.monotonic() works the same way for timeouts and deadlines. perf_counter() is the one for short measurements, because it has the highest available resolution. On CPython, both use the same monotonic clock.

A deadline loop is where this matters most:

deadline = time.monotonic() + 30
while time.monotonic() < deadline:
    if try_connect():
        break
    time.sleep(0.5)

Write that with time.time() and a clock step of an hour turns a 30-second retry into either an instant failure or a wait of an hour.

JavaScript and Node

const t0 = performance.now();
await doWork();
const ms = performance.now() - t0;        // milliseconds, fractional

performance.now() exists in browsers and Node. For nanosecond-scale work in Node, process.hrtime.bigint() returns a BigInt you subtract the same way. Keep Date.now() for timestamps you store or display.

Go

Modern Go reads both clocks when you call time.Now() and uses the monotonic reading automatically when you subtract:

start := time.Now()
doWork()
elapsed := time.Since(start) // uses the monotonic reading

The trap is that the monotonic reading is stripped when you do things like Round(0), convert to Unix time, or serialise and deserialise the value. Compare a Time that crossed a process boundary and you’re back on wall-clock subtraction.

Rust and others

std::time::Instant is monotonic and SystemTime is wall clock. The names are the pattern: nearly every language gives you both, and the bug is almost always reaching for the familiar one.

What to store, what to compare

A rule that prevents most of these bugs:

1. Store wall-clock time (as UTC) in databases, logs and tokens. It’s the only kind that means anything to another machine.
2. Never persist a monotonic reading and compare it after a restart. The starting point is arbitrary and different on every boot.
3. Convert at the edge. Compute expires_at = now_wall + ttl when you create a token. Inside one process, enforce the TTL with the monotonic clock.
4. Don’t assume two machines agree. Even with a perfect single-machine clock, ordering events across servers needs another tool entirely, such as vector clocks.

Are leap seconds going away?

Yes, eventually. Resolution 4 of the 27th CGPM in 2022 decided that the maximum permitted difference between UT1 and UTC will be increased in or before 2035, and it asks for a plan so that UTC stays continuous for at least a century. The resolution leaves the new maximum value to a later proposal.

Good news, but don’t relax. Leap seconds were never the only reason a wall clock moves. An NTP daemon can step a drifting clock, a virtual machine can resume from a pause with stale time, and an administrator can type date -s. A bug that only fires every few years is still worth removing, and a bug that fires when someone resumes a laptop lid will not wait for 2035.

The takeaway

Search your codebase for subtraction between two wall-clock reads: Date.now() - , time.time() - , time.Now().Sub. Each hit that measures a duration, timeout or rate should move to a monotonic clock. It’s a small diff, and it’s a bug class you can remove completely. If your team wants a review of reliability basics like this across a production codebase, the services page is the place to start.