How to Deduplicate Scraped Data – A Practical Method

Learn where duplicates come from in collection, how to choose an identity key, and where in the pipeline deduplication belongs.

Duplicates are not an edge case in collection; they are the normal state of raw output. Pagination overlaps, runs repeat, and the same item is reachable from several places.

1. Know Where They Come From

The last is a matching problem rather than a duplication one, and conflating them is the usual mistake.

2. Normalise Before Comparing

def normalise_url(url):
    parts = urlsplit(url)
    query = [(k, v) for k, v in parse_qsl(parts.query)
             if not k.startswith(("utm_", "gclid", "fbclid", "sid"))]
    return urlunsplit((parts.scheme, parts.netloc.lower(), parts.path.rstrip("/"),
                       urlencode(sorted(query)), ""))

Without this, the same page at two URLs is two records. Strip tracking parameters, sort the remainder, lowercase the host, drop the fragment, and settle a trailing-slash convention.

3. Choose a Genuine Identity Key

In order of preference:

1. A source identifier — a product code, listing id, or SKU.
2. The canonical URL the page declares.
3. The normalised URL.
4. A composite of stable fields, such as source plus brand plus model.

A key that is too broad merges different items; one that is too narrow — the whole record — deduplicates nothing, because any trivial difference makes it unique.

4. Deduplicate at Write Time, in the Database

INSERT INTO observations (source, external_id, collected_at, price, raw)
VALUES (...)
ON CONFLICT (source, external_id, collected_at) DO UPDATE
SET price = EXCLUDED.price, raw = EXCLUDED.raw;

A unique constraint enforces the property; application code merely intends it. This also makes re-runs idempotent, which is the same problem viewed differently.

5. Within a Run, Use a Set

seen = set()
for record in records:
    key = (record["source"], record["external_id"])
    if key in seen:
        continue
    seen.add(key)
    write(record)

Cheap and sufficient for one run. Across runs, persistence is required — either a stored key set or the database constraint above.

6. Decide What to Keep on Conflict

Newest, most complete, or a merge of non-empty fields. Any is defensible; the requirement is to choose deliberately and record which policy applied.

"Keep the first seen" and "keep the most recent" produce different datasets, and a consumer needs to know which one they have.

7. Order of Operations

Normalise, then deduplicate, then aggregate.

Deduplicating raw records misses duplicates differing only in whitespace or parameter order. Aggregating first produces counts inflated by repeats — and those wrong numbers are rarely questioned.

8. Summary

Deduplicate collected data by normalising URLs and values, choosing a genuinely identifying key, and enforcing uniqueness in storage rather than only in code.

Use an in-memory set within a run and a database constraint across runs, decide the conflict policy explicitly, and always deduplicate before aggregating.

Potential Related Glossary

• Data Wrangling
• Data Fusion
• Web Crawling
• Data Structure