The ripgrep Segfault That Wasn't a musl Bug, or a ripgrep Bug

A thread writes a byte to memory. Ten instructions later, the same thread reads that byte back and sees zero. That is not supposed to be possible, and it is the centre of one of the more instructive bug reports of the year.

The symptom was mundane. ripgrep 15.2.0, the official x86_64-unknown-linux-musl build, segfaults during very large searches. The reporter hit it in roughly a minute on a 24-core openSUSE Tumbleweed box, searching about 20 GiB across 1.8M files in a loop. glibc-linked builds did not reproduce it.

Three layers took the blame in turn, and all three were wrong.

The crash lands in the allocator

The stack is short and points somewhere embarrassing for musl:

get_meta()            meta.h:141
__malloc_allzerop()   malloc.c:384
calloc()              calloc.c:41
opendir()             opendir.c:15

ripgrep’s parallel directory walker calls std::fs::read_dir, which calls opendir, which calls calloc, and musl’s mallocng trips a heap metadata integrity assertion. Andrew Gallant, ripgrep’s author, confirmed the crash sits inside musl’s allocator rather than ripgrep’s code.

At that point the obvious readings are: musl has an allocator bug, or ripgrep’s aggressive parallelism is corrupting the heap somehow. The 145-comment discussion that followed spent a lot of energy on exactly that fork. Both readings are reasonable. Both turned out to be wrong.

What the analysis found

Daniel Fox Franke published a root-cause writeup, and the conclusion is that this is a kernel bug. Two concurrent paths inside the kernel race:

One path handles an anonymous page fault. Using the per-VMA-lock fast path, it installs a fresh page table entry via set_ptes(). The other path is a concurrent munmap teardown, clearing PTEs and sending TLB-shootdown inter-processor interrupts.

The shootdown from the second path invalidates the entry the first path just installed. For a moment, the virtual address translates to the kernel’s zero page instead of the page the thread just wrote to. Any read in that window returns zeros.

That is the mechanism. Now look back at where the crash surfaced.

Why mallocng is the one that notices

mallocng’s enframe writes to two adjacent bytes of heap metadata, p[-3] and p[-2], then reads them back. On a freshly faulted page, in a process doing heavy concurrent mapping and unmapping, that store-then-reload is precisely the shape that catches a transient remap. Both reads come back zero, the integrity assertion fires, and the process dies with a message implicating the allocator.

mallocng did its job. It detected memory that did not contain what was written to it and refused to continue. The metadata check that made it look guilty is the only reason anyone noticed at all. A less paranoid allocator would have kept running on corrupt state.

This is worth sitting with, because it is a general property of debugging. The component with the best internal consistency checks reports the failure, which makes it the first suspect, and it is frequently the least likely culprit precisely because it is the one that checks.

The three probes

The method is the part I would want a junior engineer to read, more than the conclusion.

Establishing “the kernel remapped my page” over “my program has a data race” needs evidence that distinguishes them, and a single crash dump does not. The analysis used three probes, each ruling out one explanation.

An immediate re-read against a delayed re-read. Reading the bytes back straight away saw the correct values. Reading them back about ten instructions later saw zeros. That kills the “the store was lost” theory: the store landed. Something changed the memory afterwards.

A pagemap capture at the moment of mismatch. The page was backed by the kernel’s zero page, pfn=0. Not a plausible outcome of any userspace race. It is what a translation looks like when the mapping has gone away underneath you.

Prefaulting to suppress the fault path. Touching the page before the allocator did, so the page was no longer freshly faulted, eliminated the crashes entirely. That points at the fault path itself rather than at anything about concurrent access.

Three probes, three eliminated layers. That progression, rather than the specific commits, is the reusable part. The same structure works on far less exotic problems, including the heap growth investigations that look like leaks and are not.

Which kernels

The analysis reports the crash on Linux 7.0.12 and no reproduction on 6.19.10, attributing the widened race window to the PTE-table-reclaim rework in zap_empty_pte_table() that landed in v7.0.

Treat that as one reporter’s careful finding on one machine, not a vetted advisory. The ripgrep issue was still open when this was written, and kernel review may reach a different conclusion about the fix or the affected range. What is durable is the signature: mallocng metadata assertions, under heavy parallel filesystem work, on a recent kernel, absent on an older one.

What this changes for anyone shipping musl containers

Not much, and that is the honest answer. If your workload is a normal service handling requests, this does not describe you. Reproduction wanted a huge tree, many cores, sustained concurrency, and a specific kernel range.

Where it does apply, the first move is diagnostic rather than corrective. Record the exact kernel version, then try the same workload on an older kernel and on a glibc build. If the failure tracks the kernel and not the libc, you have this bug and no amount of rewriting your own code will fix it.

Switching to a glibc base image will probably make the symptom go away, and it is worth being clear about what that buys you. It does not fix the race. It removes the allocator whose store-reload pattern happens to notice. If the analysis is right, the underlying window is still there, and something else may eventually step in it more quietly. That is a real tradeoff, not a fix, and it is the same kind of tradeoff that runs through choosing a container base image at all, where small images and predictable runtime behaviour pull in different directions.

The takeaway

The stack trace pointed at musl. The reporting tool was ripgrep. Neither was at fault. A crash location tells you where a problem became visible, which is a different question from where it came from, and the gap between those two can span the entire stack down to page table management.

When a component with strong internal checks starts failing on code that has not changed, widen the search before you narrow it. Ask what else moved. Here the answer was the kernel, and only a deliberately designed sequence of probes could have shown it.