What Is OpenTelemetry? – Definition and How It Works

Learn what OpenTelemetry standardises, how traces spans and context propagation work, and how to instrument a crawler.

OpenTelemetry is a vendor-neutral standard and set of libraries for producing traces, metrics, and logs. Its purpose is to decouple instrumentation from whichever backend stores it.

1. What It Standardises

The decoupling is the point: instrument once against the API, and change backends by reconfiguring the exporter rather than rewriting instrumentation.

2. Traces and Spans

A trace is one unit of work end to end. A span is one operation within it, with a parent, a duration, and attributes.

from opentelemetry import trace
tracer = trace.get_tracer(__name__)

def collect(url):
    with tracer.start_as_current_span("collect_page") as span:
        span.set_attribute("url.full", url)
        with tracer.start_as_current_span("fetch") as fetch:
            html = http_get(url)
            fetch.set_attribute("http.response.status_code", 200)
            fetch.set_attribute("http.response.body.size", len(html))
        with tracer.start_as_current_span("parse"):
            records = parse(html)
        span.set_attribute("collector.records", len(records))
        return records

The nesting is what makes the trace useful: it shows where the time actually went, which in a collection pipeline is routinely surprising.

3. Why Traces Beat Metrics for Diagnosis

Metrics tell you the p95 latency rose. A trace tells you which stage it rose in.

For a collection pipeline the stages that compete are queue wait, DNS, connect, TLS, server response, transfer, parse, and store. A trace attributes time to each, and the common finding is that a pipeline believed to be network-bound is actually spending most of its time waiting for a worker slot — which no per-stage metric reveals.

4. Context Propagation

The mechanism that makes distributed tracing work: trace context travels with the request in headers.

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

A scheduler that starts a trace, a fetch worker that continues it, and a parse worker that adds spans all contribute to one trace, because the context was propagated between them.

For a queue-based crawler that means putting the trace context into the message alongside the URL — otherwise each stage produces an orphan trace and the end-to-end view is lost.

5. Semantic Conventions Matter More Than They Look

OpenTelemetry defines standard attribute names — http.request.method, server.address, url.full, http.response.status_code.

Using them rather than inventing your own is what makes backend tooling work out of the box: dashboards, alerts, and service maps keyed on the conventions. Custom names mean building all of that yourself.

Add your own domain attributes under your own prefix — collector.records, collector.selector_version, collector.proxy_region — alongside the standard ones rather than instead of them.

6. Sampling Is Not Optional

Tracing every request at volume is expensive in both cost and overhead. Sampling strategies:

• Head sampling — decide at the start, cheap, may miss the interesting cases.
• Tail sampling — decide after seeing the whole trace, so errors and slow traces are kept preferentially. Requires a collector.
• Parent-based — respect the upstream decision, keeping traces whole.

For collection work, tail sampling that always keeps traces containing errors or zero-record outcomes is the configuration that matters: you keep the traces you will actually want to look at.

7. Instrumenting a Crawler Well

1. One trace per URL processed, not per run.
2. Spans for queue wait, fetch, parse, and store at minimum.
3. Propagate context through the queue, or traces fragment.
4. Record the extracted record count as an attribute — it is the health signal.
5. Record proxy region and selector version, so failures are attributable.
6. Set span status to error on failures, so tail sampling retains them.
7. Use standard attribute names plus a namespaced set of your own.

8. Summary

OpenTelemetry standardises traces, metrics, and logs behind a vendor-neutral API so instrumentation survives a change of backend, with spans forming a tree that attributes time to each stage of a unit of work.

For a crawler it answers the question metrics cannot — which stage the latency is in, frequently queue wait rather than the network — provided trace context is propagated through the queue, standard attribute names are used, and tail sampling is configured to keep the traces containing errors and zero-record outcomes.

Potential Related Glossary

• Concurrency vs Parallelism
• Web Crawling
• Data Structure
• Parsing