What Is Deduplication? – Definition and How It Works
Learn what deduplication is, how exact and fuzzy approaches differ, how to choose an identity key, and why order of operations matters.
Deduplication removes records that represent the same thing. The difficulty is rarely the removal — it is deciding what "the same thing" means.
1. Where Duplicates Come From
| Source | Cause |
|---|---|
| Pagination overlap | Consecutive pages repeat rows |
| Re-runs | The same collection executed twice |
| Multiple entry points | One item reachable from several listings |
| Syndication | The same content republished across sites |
| Retries | A request that succeeded but appeared to fail |
| Genuine near-duplicates | Two sellers listing one product differently |
The last is a matching problem rather than a duplication problem, and conflating the two causes most deduplication mistakes.
2. Exact Deduplication
Pick an identity key and keep one record per key.
seen, unique = set(), []
for record in records:
key = (record["source"], record["external_id"])
if key not in seen:
seen.add(key)
unique.append(record)
The key must be genuinely identifying. Using a title alone merges different products with the same name; using the whole record deduplicates nothing, because any trivial difference makes it unique.
3. Choosing the Key
- A source identifier where one exists — the strongest option.
- A canonical URL, after normalising away tracking parameters and fragments.
- A composite of stable fields, such as source plus brand plus model.
- A content hash for detecting whether a document changed, distinct from identity.
Normalise before hashing: differences in whitespace, casing, or parameter order otherwise produce false uniqueness.
4. Near-Duplicates
Records that refer to the same thing without sharing a key require similarity rather than equality — string distance on names, embedding proximity for text, or blocking on a coarse key before comparing within groups.
These decisions are probabilistic, so store a confidence score and keep the originals. A merge is much harder to undo than to defer.
5. Order of Operations
Deduplicate after normalising and before aggregating.
Deduplicating raw records misses duplicates that differ only in whitespace or parameter order. Aggregating before deduplicating produces counts inflated by repeats — and those wrong numbers are rarely questioned.
6. Deduplication Across Runs
Within a run, an in-memory set suffices. Across runs it needs persistence: a stored set of seen keys, or an upsert into a table with a unique constraint.
The database approach is usually simpler and safer, because the constraint enforces the property rather than relying on application code being correct every time.
7. What to Keep
When two records share a key, decide deliberately: the newest, the most complete, or a merge of non-empty fields.
Record which policy was applied. "Keep the first seen" and "keep the most recent" produce different datasets, and a consumer needs to know which one they have.
8. Summary
Deduplication removes records representing the same thing, and its whole difficulty lies in defining identity.
Normalise first, choose a genuinely identifying key, treat near-duplicate matching as a separate probabilistic problem, deduplicate before aggregating, and enforce uniqueness in storage rather than only in code.