What Is JSON Lines? – Definition and How It Works

Informational P2 what is json lines

Learn what JSON Lines is, why one object per line enables streaming and appending, and why it became the default for collected data.

JSON Lines stores one complete JSON object per line, separated by newlines. It is not a new format so much as a convention, and it solves the specific problems a single large JSON array creates.

1. The Format

{"url": "https://example.com/1", "title": "Widget", "price": 49.9}
{"url": "https://example.com/2", "title": "Gadget", "price": 12.0}

Rules: one object per line, no enclosing array, no commas between records, newline-delimited, UTF-8.

2. What It Fixes

Problem with one big array	With JSON Lines
Must be fully parsed before use	Each line parses independently
Cannot be appended without rewriting	Append a line
One malformed record breaks everything	Only that line fails
Cannot be split across files	Split anywhere on a line boundary
Memory scales with file size	Memory scales with one record

Appendability is the property that matters most in collection: a crawler can write records as it finds them, and an interrupted run leaves a valid file.

3. Processing It

import json

with open("records.jsonl", encoding="utf-8") as handle:
    for line in handle:
        record = json.loads(line)
        process(record)

Memory stays constant regardless of file size, and standard line-based command-line tools work on it directly — counting, filtering, splitting, and sampling without a parser.

4. Why It Suits Collected Data

Streaming writes as records are extracted.
Resumable runs — an interrupted file is still valid up to its last complete line.
Partial failure tolerance — one bad record does not invalidate the batch.
Trivial sharding across workers or files.
Schema flexibility, since records need not have identical keys.

That last point cuts both ways: flexibility is useful during collection and a liability downstream, which is why validation still belongs in the pipeline.

5. Compression

JSON Lines compresses well, since records repeat keys and structure. Gzip is the usual choice and remains streamable — a compressed file can still be read line by line without full decompression.

6. When to Use Something Else

Analytical queries at scale — Parquet is smaller and far faster to filter.
A single configuration object — plain JSON is the right shape.
Handing data to a non-technical colleague — CSV opens in a spreadsheet.
API responses — consumers expect a JSON document.

7. Practical Conventions

Use the .jsonl extension, write UTF-8 without a byte order mark, keep one record type per file, avoid embedded newlines inside string values, and include an identifier and a collection timestamp in every record.

8. Summary

JSON Lines puts one JSON object per line, making files streamable, appendable, splittable, and tolerant of individual malformed records.

Those properties match how collection actually works, which is why it has become the default intermediate format — with Parquet taking over once the data settles into analysis.