What Is NDJSON? – Definition and How It Works

Learn what NDJSON is, why one JSON object per line suits streaming and append-only collection, and the rules that keep it valid.

NDJSON — newline-delimited JSON — stores one complete JSON value per line. It is the standard format for streaming records, append-only logs, and datasets too large to hold in memory.

1. The Format

{"url":"https://example.com/1","title":"First","price":19.99}
{"url":"https://example.com/2","title":"Second","price":24.50}
{"url":"https://example.com/3","title":"Third","price":null}

Three rules cover it: one JSON value per line, no newlines inside a value, and no enclosing array or commas between lines.

2. Why Not a JSON Array

For collection specifically, the append and partial-file properties are decisive. A crawler writing an array must hold everything or rewrite the tail; a crawler writing NDJSON appends each record and a crash costs at most the final line.

3. Writing It

import json

with open("out.ndjson", "a", encoding="utf-8") as f:
    for record in records:
        f.write(json.dumps(record, ensure_ascii=False) + "\n")
        f.flush()

Two details matter more than they look. json.dumps must not be given indent, since pretty-printing inserts the newlines the format forbids. And flushing per record is what makes the file recoverable after a crash — buffered writes lose whatever had not reached disk.

4. Reading It

import json

with open("out.ndjson", encoding="utf-8") as f:
    for line_no, line in enumerate(f, 1):
        line = line.strip()
        if not line:
            continue
        try:
            record = json.loads(line)
        except json.JSONDecodeError:
            print(f"malformed line {line_no}")
            continue
        process(record)

Skipping blank lines and tolerating one malformed line are what make a reader usable against real files, where the last line of an interrupted write is frequently truncated.

5. Command-Line Work

The format's other advantage is that ordinary tools apply:

wc -l out.ndjson                                    # record count
jq -c 'select(.price != null)' out.ndjson           # filter
jq -r '[.url, .title] | @csv' out.ndjson            # to CSV
split -l 100000 out.ndjson part-                    # chunk
sort -u out.ndjson > deduped.ndjson                 # exact dedupe

Counting records with wc -l is a genuinely useful property: no parser, no memory, constant time in practice.

6. Compression and Storage

NDJSON compresses very well, because records share keys. Gzip is universal but not splittable; block formats such as bzip2 or zstd with frames allow parallel processing of one file.

For long-term analytical storage, converting to a columnar format costs far less space and reads far faster. The usual arrangement keeps compressed NDJSON as the raw append-only record and a columnar copy for querying.

7. Naming and Content Type

Both .ndjson and .jsonl are in wide use for the same format; application/x-ndjson is the conventional media type. Consistency within a project matters more than the choice.

8. Summary

NDJSON is one JSON value per line, with no wrapping array and no embedded newlines.

That makes it appendable, streamable, resilient to interruption, trivially concatenated, and usable with line-based tools — so write compact JSON with a flush per record, tolerate blank and truncated lines when reading, and keep a columnar copy when the data will be queried repeatedly.

Potential Related Glossary

• Data Structure
• Parsing
• Data Wrangling
• Unstructured Data