What Is Structured Logging? – Definition and How It Works

Learn why structured logs are queryable, which fields a collection pipeline needs, and how to keep secrets out.

Structured logging emits log lines as machine-readable records — typically JSON with named fields — instead of prose sentences.

1. The Difference

2026-08-18 14:00:02 INFO  Parsed https://example.com/p/42 and found 0 items in 812ms

{"ts":"2026-08-18T14:00:02Z","level":"info","event":"page_parsed",
 "url":"https://example.com/p/42","records":0,"duration_ms":812,
 "status":200,"bytes":48210,"proxy_region":"es","run_id":"r-8891"}

Both say the same thing. Only the second can answer "which domains returned zero records yesterday, grouped by proxy region" without writing a parser for your own logs.

2. Why It Matters More for Collection Than Most Systems

The characteristic failure of a scraper is success with no data. Detecting it requires querying across dimensions — domain, selector version, proxy region, time — and prose logs cannot be queried that way.

The specific question you will need to ask, repeatedly:

SELECT domain, count(*) FROM logs
WHERE event = 'page_parsed' AND records = 0 AND ts > now() - interval '1 day'
GROUP BY domain ORDER BY 2 DESC;

That is trivial against structured logs and effectively impossible against sentences.

3. The Fields a Collection Pipeline Needs

records and selector_version together are what let you attribute an extraction failure to a specific change.

4. Use a Closed Vocabulary for Events and Errors

Free-text messages defeat aggregation as surely as prose. event and error_kind should come from a fixed set:

EVENTS = {"fetch_start","fetch_ok","fetch_failed","page_parsed","record_stored","run_finished"}
ERRORS = {"connect_timeout","read_timeout","dns_failure","http_4xx","http_5xx",
          "challenge_page","empty_extraction","parse_error","validation_failed"}

challenge_page and empty_extraction as first-class error kinds is the important part — those are the silent failures, and naming them makes them countable.

5. Never Log Secrets

Structured logging makes this both easier and more dangerous: easier because fields are explicit, more dangerous because it is tempting to log a whole request or response object.

Rules that hold:

• Never log headers wholesale. They contain cookies and authorisation.
• Never log request bodies from authenticated flows.
• Never log proxy credentials, including inside a proxy URL — log a proxy identifier instead.
• Redact by allowlist, not by denylist. Log the fields you chose, not everything minus known-bad keys.
• Do not log full response bodies. Log a hash, a length, and a few extracted fields.

An allowlist is the only approach that survives someone adding a new sensitive field later.

6. Keep One Line Per Event

Multi-line log entries break most collection tooling, so serialise to a single line and avoid embedded newlines. That is also why NDJSON is the natural on-disk format — one JSON object per line, appendable, streamable.

7. Levels Still Matter

Structure does not remove the need for levels. A useful convention:

• debug — per-request detail, off in production.
• info — lifecycle events and per-page outcomes.
• warning — recoverable anomalies: a retry, a fallback selector matching.
• error — a failed unit of work.
• critical — the run cannot continue.

A fallback selector matching at warning is the early-warning signal that primary extraction is degrading.

8. Summary

Structured logging emits named fields rather than sentences, which is what makes logs queryable across domain, region, time, and selector version.

For collection the essential fields are the extracted records count, selector_version, and a closed error_kind vocabulary that names challenge_page and empty_extraction explicitly — and secrets stay out by logging an allowlist of chosen fields rather than whole request or response objects.

Potential Related Glossary

• Data Structure
• Web Crawling
• Data Wrangling
• Parsing