Distributed Tracing Explained: Following a Request Across Every Service It Touches

A checkout request touches five services and takes 420 milliseconds longer than it should. Which one is slow? Logs from each service show normal-looking entries. Metrics show nothing alarming, average latency per service is fine. The problem only exists in the specific combination of calls this one request happened to make, and no single service’s logs or dashboards can show you that combination. This is exactly the gap distributed tracing closes.

What a trace actually is

A trace is the complete path one request takes through a system, identified by a single trace ID generated the moment the request enters. Every unit of work along that path, an HTTP call, a database query, a cache lookup, gets recorded as a span: a record with a name, a start time, a duration, and a parent span ID pointing back to whichever span caused it to happen.

Read left to right, that waterfall answers the question directly: the gateway’s total request time is 420ms, but payments: charge card alone accounts for 260ms of it, starting at the 165ms mark, well after orders: create order returns. No log-grepping across five services required. The trace already has the answer laid out on one timeline.

Spans, and how they connect

Each span carries, at minimum:

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "parent_span_id": "b7ad6b7169203331",
  "name": "orders: create order",
  "start_time": "2026-08-08T14:02:11.045Z",
  "duration_ms": 210
}

The trace_id is shared across every span in the request. The parent_span_id is what turns a flat list of spans into a tree: it says “this span happened because that other span called it.” A tracing backend uses that parent-child structure to render the waterfall, and to compute derived views like which service accounts for the most total time across many traces, not just one.

The part that requires actual code: context propagation

Generating a span inside one service is the easy part; most tracing SDKs do it automatically for common frameworks. The part that requires a deliberate integration choice is context propagation: passing the current trace ID and span ID to whatever service gets called next, so that service’s span gets created as a child of the right parent instead of starting a disconnected trace of its own.

In practice, this means an outgoing HTTP call needs a header like:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

which encodes the trace ID and current span ID per the W3C Trace Context spec. Most OpenTelemetry SDKs inject this automatically for HTTP clients they instrument, but it breaks silently at any boundary the SDK doesn’t know about: a message queue, a gRPC call using a library the SDK doesn’t wrap, or a call made through a raw socket. When a trace looks “broken” into unconnected fragments, the propagation step at exactly one hop is almost always where to look first.

Setting it up with OpenTelemetry

A minimal Node.js example, instrumenting an Express service:

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({ url: 'http://localhost:4318/v1/traces' }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

getNodeAutoInstrumentations() patches common libraries (Express, http, popular database clients) to generate spans and propagate context automatically. For anything the auto-instrumentation doesn’t cover, a manual span looks like:

const { trace } = require('@opentelemetry/api');
const tracer = trace.getTracer('orders-service');

async function createOrder(payload) {
  return tracer.startActiveSpan('orders: create order', async (span) => {
    try {
      const result = await db.orders.insert(payload);
      return result;
    } finally {
      span.end();
    }
  });
}

The exporter sends completed spans to a backend, Jaeger, Grafana Tempo, Honeycomb, or a vendor’s hosted product, over the OTLP protocol. Because OpenTelemetry decouples instrumentation from backend, switching which backend receives those spans later is a configuration change, not a re-instrumentation project.

Tracing versus logging versus metrics

These three answer genuinely different questions, and teams that set up only one of them tend to hit a wall the other two would have caught:

Tool	Answers	Doesn’t answer
Metrics	Is the system healthy right now, in aggregate?	Which specific request was slow, and why?
Logs	What happened inside one service?	How did that fit into the rest of the request across services?
Traces	Which specific hop, in which specific request, caused the slowness?	Broad health trends across thousands of requests (better suited to metrics)

A trace that shows payments: charge card as the long pole in a slow checkout still benefits from that service’s own logs to explain why the charge took 260ms, a downstream card network timeout, a retry, a lock wait. Tracing narrows down where to look; logs and metrics fill in what happened there.

When it’s actually worth the setup cost

Below two or three services in a request’s path, tracing is usually more infrastructure than the problem calls for; solid logging covers most of what a trace would tell you. The value curve bends sharply once a request routinely crosses three or more service boundaries and a slow response could plausibly originate at any of them, similar to how reasoning about failure isolation with circuit breakers only pays off once one slow dependency can actually take others down with it. At that point, the alternative to a trace isn’t “no debugging effort,” it’s manually correlating timestamps across five sets of logs by hand, which takes real engineering time on every single incident instead of a one-time setup cost paid once.

If a team is already running microservices and still triaging cross-service latency by opening five log dashboards side by side, that’s the concrete signal that tracing has been overdue for a while, not a nice-to-have for later.