What Is Distributed Tracing? – Definition and How It Works

Informational P3 what is distributed tracing

Learn how distributed tracing follows one unit of work across services, and why queue wait is what it usually exposes.

Distributed tracing follows a single unit of work as it crosses process and service boundaries, producing one connected view of where its time went and where it failed.

1. The Problem It Solves

In a system of several components, each one's logs and metrics describe only its own part. When end-to-end latency is bad, per-component data cannot say which component caused it — every component looks locally fine.

Tracing stitches the parts together, so one record shows the whole path.

2. The Model

Concept	Meaning
Trace	The whole unit of work
Span	One operation within it
Parent span	The operation that caused this one
Trace id	Shared by every span in the trace
Span id	Identifies this operation
Attributes	Key-value detail on a span
Events	Timestamped points within a span

Spans form a tree, and the tree's shape is the information — parallel spans show concurrency, deep chains show sequential dependency.

3. Context Propagation Is the Whole Trick

A trace only spans services if the context travels between them. Over HTTP that is a header:

traceparent: 00-<trace-id>-<parent-span-id>-<flags>

Through a queue, the context must be carried in the message:

# producer
msg = {"url": url, "traceparent": current_traceparent()}
queue.put(json.dumps(msg))

# consumer
ctx = extract_context(msg.get("traceparent"))
with tracer.start_as_current_span("process", context=ctx):
    ...

Forgetting the queue case is the most common way tracing half-works: HTTP calls are traced, queue hops are not, and every worker produces orphan traces.

4. What It Reveals in a Collection Pipeline

The stages that compete for time:

trace: collect https://example.com/p/42          total 4,210ms
├── queue_wait                                          2,900ms   ← the finding
├── fetch                                                 940ms
│   ├── dns                                                12ms
│   ├── connect                                            88ms
│   ├── tls                                               104ms
│   └── ttfb + transfer                                   736ms
├── parse                                                 310ms
└── store                                                  60ms

That shape is extremely common and almost never what people expect. Two-thirds of the elapsed time is waiting for a worker, meaning the fix is concurrency or capacity — not a faster parser, and not a closer proxy.

Without tracing, the same system reports "average latency 4.2 seconds" and invites optimisation of the wrong stage.

5. Errors Become Attributable

A span carrying an error status shows exactly where a failure originated, and its parents show what was being attempted. That turns "the run had 400 failures" into "384 failures were read timeouts in the fetch span for one domain, from one proxy region".

That level of attribution is what makes a failure actionable rather than merely counted.

6. Sampling and Cost

Tracing every unit of work at scale is expensive, so sampling is required. The strategy that suits collection:

Tail sampling, keeping every trace that contains an error or a zero-record outcome.
A low baseline rate for successful traces, enough to see the latency distribution.
Parent-based propagation, so a sampled trace stays whole across services.

Keeping all error traces and few success traces gives you the ones worth reading at a fraction of the cost.

7. Tracing Versus Logs Versus Metrics

Signal	Answers	Cardinality
Metrics	How much, how often	Low — few labels
Logs	What happened, in detail	High
Traces	Where the time went, across services	High, sampled

They are complementary rather than alternatives. Metrics detect that something changed, traces locate it, logs explain it. A system with only one of the three has a predictable blind spot.

8. Summary

Distributed tracing follows one unit of work across services as a tree of spans sharing a trace id, with context propagated in HTTP headers and, crucially, inside queue messages.

For collection pipelines it routinely reveals that queue wait rather than the network dominates elapsed time — a finding no per-stage metric produces — and tail sampling that retains every error and zero-record trace gives the traces worth reading without tracing everything.