What Is BM25? – Definition and How It Works

Learn what BM25 is, how term frequency saturation and length normalisation work, and why it still beats embeddings on some queries.

BM25 is a lexical ranking function that scores how well a document matches a query based on the query's terms. It is the default relevance model in most search engines and the lexical half of hybrid search.

1. What It Improves On

Naive term counting says a document containing a word twenty times is twice as relevant as one containing it ten times, and that a long document is more relevant simply because it contains more words.

BM25 corrects both, and those two corrections are essentially the whole algorithm:

• Term frequency saturation — the tenth occurrence of a word adds far less than the second. Relevance flattens rather than growing linearly.
• Length normalisation — a term appearing in a short document counts for more than the same term in a long one.
• Inverse document frequency — a rare term is worth more than a common one.

2. The Components

b = 0 disables length normalisation entirely; b = 1 applies it fully. The conventional 0.75 is a compromise that holds up across most collections.

3. Where It Beats Embeddings

This is the reason BM25 has not been replaced. It wins decisively on:

• Exact identifiers — a product code, an error code, a SKU. An embedding blurs ERR_1015 toward similar-looking codes; BM25 matches it exactly.
• Rare proper nouns the embedding model never saw in training.
• Keyword queries where the user typed the literal term they want.
• Negation and precision, where semantic similarity actively hurts.

An embedding model asked for cloudflare error 1020 will happily return documents about error 1015, because they are semantically near-identical. BM25 will not.

4. Where It Loses

• Synonyms. A query for "car" does not match a document saying "automobile".
• Paraphrase. Nothing lexical connects "how do I stop getting blocked" to "avoiding detection".
• Multilingual queries against differently worded content.
• Vocabulary mismatch generally — the user's words and the document's words differ.

5. Hybrid Search Is the Practical Answer

Because the two failure modes are complementary, production retrieval usually runs both and fuses the results. Reciprocal rank fusion is the common method, since it needs no score calibration between the two systems:

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:                 # each is a list of doc ids, best first
        for rank, doc in enumerate(ranking, start=1):
            scores[doc] = scores.get(doc, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

Fusing ranks rather than scores is what makes this robust — BM25 scores and cosine similarities are not on a comparable scale, and normalising them is fragile.

6. Tokenisation Decides Everything

BM25 operates on tokens, so the analyser matters more than the parameters. Stemming, case folding, stopword handling, and how identifiers are split all change what can be matched.

The common defect: an aggressive tokeniser splits ERR_1015 into err and 1015, destroying the exact-match property that was the reason for choosing BM25. Keep identifiers intact, or index them in a separate unanalysed field.

7. Using It on Collected Data

For a corpus of scraped pages, BM25 is the cheap, strong baseline — no embedding model, no vector index, no inference cost, and it runs well from Postgres, SQLite FTS, OpenSearch, or a small local index.

Two practical notes. Index the extracted main content, not the raw HTML, or navigation boilerplate dominates the term statistics. And index title and body as separate fields with different weights, since a query term in a title is a much stronger signal.

8. Summary

BM25 scores lexical matches using term frequency with saturation, inverse document frequency, and length normalisation, tuned by k1 and b.

It remains the right tool for exact identifiers, rare terms, and literal keyword queries where embeddings blur distinctions — and because its weaknesses are synonyms and paraphrase, the standard production answer is hybrid search fusing BM25 with vector results by rank rather than by score.

Potential Related Glossary

• Parsing
• Data Structure
• Unstructured Data
• Data Fusion