Deduplicating Records with Fuzzy Matching

The same product routinely appears twice under different text, and this page — part of Cleaning and Validating Scraped Data — covers how to find those pairs without comparing every record to every other one.

Work down the rungs in order. Exact and normalised keys are cheap and safe; fuzzy matching only has to cover the residue the first two rungs missed.

Work down three rungs in order. Group on an exact key such as a SKU or a canonical URL first, because it is free and never produces a false merge. Then group on a normalised key — casefolded, punctuation stripped, tokens sorted — which catches most of the remainder at the same zero risk. Only the residue goes to similarity scoring with rapidfuzz, and even then you bucket records into blocks first so the comparison count stays linear-ish rather than quadratic.

Why Ordering the Rungs Matters

Similarity scoring is the interesting part and the wrong place to start. It is expensive, it needs a threshold that has to be tuned per dataset, and every threshold admits some false positives. Exact matching has none of those properties: two records with the same SKU are the same product, full stop.

On a typical scraped product set the first rung finds most duplicates on its own, the second rung — which only needs a normalising function you already wrote for the cleaning stage — captures most of what remains, and fuzzy matching handles the residue where the text genuinely differs: Acme Kettle 1.7L Stainless against ACME Kettle 1.7 L, stainless. Running fuzzy matching first would produce the same answer at hundreds of times the cost, with the added risk of merging two genuinely different variants.

Deduplication also has to run after normalising. Comparing raw strings means "Acme Kettle " and "acme kettle" score below any sane threshold, so the duplicate survives — and it is worth being precise about which duplication problem this is. Skipping a URL you have already fetched is a crawl-frontier concern handled by Deduplicating URLs with Bloom Filters. What follows is about records that were legitimately fetched from different pages and describe the same thing.

Rungs One and Two: Keys

The normalised key does four things: casefold (which is stricter than lower() and handles German ß and Turkish dotted forms correctly), strip combining accents via NFKD decomposition, remove punctuation, and sort the remaining tokens so word order stops mattering. Dropping a small stop-word list removes the marketing filler that varies between listings of the same item.

import re
import unicodedata
from collections import defaultdict

PUNCT_RE = re.compile(r"[^\w\s]", re.UNICODE)
NOISE = {"the", "a", "an", "of", "and", "with", "new", "pack", "size"}


def normalise_key(text: str) -> str:
    """Order-insensitive, case-insensitive, accent-insensitive key."""
    folded = unicodedata.normalize("NFKD", text).casefold()
    folded = "".join(c for c in folded if not unicodedata.combining(c))
    folded = PUNCT_RE.sub(" ", folded)
    tokens = [t for t in folded.split() if t not in NOISE]
    return " ".join(sorted(tokens))


def exact_groups(records: list[dict[str, str]], field: str) -> dict[str, list[int]]:
    """Group record ids by an exact field value, keeping only real collisions."""
    groups: dict[str, list[int]] = defaultdict(list)
    for record in records:
        value = (record.get(field) or "").strip()
        if value:
            groups[value].append(record["id"])
    return {key: ids for key, ids in groups.items() if len(ids) > 1}


RECORDS = [
    {"id": 1, "brand": "Acme", "title": "Acme Kettle 1.7L Stainless", "sku": "AK-17S"},
    {"id": 2, "brand": "Acme", "title": "ACME  Kettle 1.7 L, stainless", "sku": ""},
    {"id": 3, "brand": "Acme", "title": "Acme Kettle 1.7L Stainless", "sku": "AK-17S"},
    {"id": 4, "brand": "Acme", "title": "Acme Toaster 2-Slice", "sku": "AT-2"},
    {"id": 5, "brand": "Bolt", "title": "Bolt Kettle 1.7L Stainless", "sku": "BK-17"},
]

print(exact_groups(RECORDS, "sku"))        # {'AK-17S': [1, 3]}
for record in RECORDS:
    print(record["id"], repr(normalise_key(record["title"])))

Records 1 and 3 collapse on the SKU alone. Records 1 and 3 also produce identical normalised keys ('1 7l acme kettle stainless'), so rung two would have caught them too — but record 2 does not, because 1.7 L tokenises as 1 7 l while 1.7L tokenises as 1 7l. That single space is exactly the residue fuzzy matching exists for.

Sorting the tokens is what makes the key order-insensitive, and it is a real trade-off: Kettle Acme and Acme Kettle become the same key, which is almost always what you want for product titles and almost never what you want for addresses or people's names.

Rung Three: Blocking, Then Similarity

Comparing every pair of n records is n(n−1)/2 comparisons. At 10,000 records that is just under 50 million calls, which at roughly 2 microseconds each is around 40 minutes of pure CPU. At 100,000 records it is 5 billion, which is not a run you finish.

Blocking is what makes fuzzy matching finish. Comparing every pair of 10,000 rows costs 50 million similarity calls; bucketing first cuts that by two orders of magnitude.

Blocking fixes this. Partition the records into buckets on a cheap key that any true duplicate pair is very likely to share, then compare only within a bucket. The cost becomes the sum of the squares of the bucket sizes, which for a few hundred small buckets is a rounding error next to the original figure.

The blocking key is the design decision. It must be cheap, and it must be coarse enough that true duplicates land together — a key that is too specific silently discards real matches, and no amount of threshold tuning gets them back. Brand plus the first few letters of the longest word is a reasonable default for product data, because brand rarely varies and the longest word is usually the distinguishing noun.

from rapidfuzz import fuzz


def blocking_key(record: dict[str, str]) -> str:
    """Coarse bucket: same brand and same opening letters of the longest word."""
    brand = normalise_key(record.get("brand") or "")
    words = [w for w in normalise_key(record.get("title") or "").split() if w.isalpha()]
    anchor = max(words, key=len)[:4] if words else ""
    return f"{brand}|{anchor}"


def fuzzy_pairs(
    records: list[dict[str, str]], threshold: float = 88.0
) -> list[tuple[int, int, float]]:
    """Candidate duplicate pairs, comparing only within blocks."""
    blocks: dict[str, list[dict[str, str]]] = defaultdict(list)
    for record in records:
        blocks[blocking_key(record)].append(record)

    pairs: list[tuple[int, int, float]] = []
    for members in blocks.values():
        if len(members) < 2:
            continue
        keys = [normalise_key(m["title"]) for m in members]
        for i in range(len(members)):
            for j in range(i + 1, len(members)):
                score = fuzz.token_set_ratio(keys[i], keys[j])
                if score >= threshold:
                    pairs.append((members[i]["id"], members[j]["id"], round(score, 1)))
    return pairs


for record in RECORDS:
    print(record["id"], repr(blocking_key(record)))
print(fuzzy_pairs(RECORDS))
# [(1, 2, 98.1), (1, 3, 100.0), (2, 3, 98.1)]

Records 1, 2 and 3 share the block 'acme|stai'; the toaster gets 'acme|toas' and the Bolt kettle gets 'bolt|stai', so neither is ever compared against the Acme kettles. Three comparisons instead of ten — a small saving on five records, and the difference between minutes and days at scale.

token_set_ratio is the right scorer for product text. It splits both strings into token sets and compares their intersection against each remainder, which makes it insensitive to word order and, crucially, to one string carrying extra words the other lacks — the common case where one listing includes the colour and the other does not. fuzz.ratio would penalise those extra words heavily; partial_ratio swings too far the other way and rates any substring match highly, which produces false positives on short titles.

rapidfuzz is a C++ implementation, roughly an order of magnitude faster than the pure-Python fuzzywuzzy it replaces, and it is MIT-licensed rather than GPL. Install with pip install rapidfuzz. If numpy is available, rapidfuzz.process.cdist computes a full score matrix per block with released threads and workers=-1, which is worth the extra dependency once blocks routinely exceed a few hundred members.

Choosing a Threshold From a Labelled Sample

A threshold picked by intuition is a guess about your data that nobody has checked. Label a few hundred pairs by hand — drawn from the candidate pairs, not at random, or almost every pair will be a trivial non-match — then sweep the threshold and read the precision and recall off the sample.

LABELLED = [
    ("Acme Kettle 1.7L Stainless", "ACME  Kettle 1.7 L, stainless", True),
    ("Acme Kettle 1.7L Stainless", "Acme Kettle 1.7L Stainless Steel", True),
    ("Acme Kettle 1.7L Stainless", "Bolt Kettle 1.7L Stainless", False),
    ("Acme Kettle 1.7L Stainless", "Acme Kettle 1.0L Stainless", False),
    ("Acme Toaster 2-Slice", "Acme Toaster 2 Slice", True),
    ("Acme Toaster 2-Slice", "Acme Toaster 4-Slice", False),
]


def sweep(pairs: list[tuple[str, str, bool]], lo: int = 70, hi: int = 100) -> None:
    scored = [
        (fuzz.token_set_ratio(normalise_key(a), normalise_key(b)), same)
        for a, b, same in pairs
    ]
    print(f"{'thresh':>6} {'prec':>6} {'recall':>7}")
    for threshold in range(lo, hi + 1, 5):
        tp = sum(1 for score, same in scored if score >= threshold and same)
        fp = sum(1 for score, same in scored if score >= threshold and not same)
        fn = sum(1 for score, same in scored if score < threshold and same)
        precision = tp / (tp + fp) if tp + fp else 1.0
        recall = tp / (tp + fn) if tp + fn else 1.0
        print(f"{threshold:>6} {precision:>6.2f} {recall:>7.2f}")


sweep(LABELLED)

Read the table and pick the lowest threshold that still holds precision where you need it. The asymmetry is what matters: a false positive destroys data by merging two distinct products into one row, while a false negative merely leaves a duplicate you can find later. Set the threshold high, and send the band just below it — typically five points wide — to a review queue rather than discarding it.

Note the two negative pairs that differ only in a number: 1.7L against 1.0L, and 2-Slice against 4-Slice. Those score high on any text-similarity measure while being genuinely different products, which is the single most common false-positive class in product deduplication. The fix is not a higher threshold — it is a hard rule that any numeric token appearing in one title and contradicted in the other blocks the match regardless of score.

Merging a Duplicate Group

Finding the pairs is half the job. Merging them means picking a winner per field rather than per row, because the whole reason two records survived is that each has something the other lacks.

Merging is per field, not per row. Each field is taken from whichever record actually has it, and the timestamps of both observations are kept.

from typing import Any

FIELDS = ("title", "price", "brand", "url")


def is_filled(value: Any) -> bool:
    return value not in (None, "", [], {})


def shoutiness(title: str) -> float:
    letters = [c for c in title if c.isalpha()]
    return sum(1 for c in letters if c.isupper()) / len(letters) if letters else 1.0


def rank_key(record: dict[str, Any]) -> tuple[int, float]:
    filled = sum(1 for field in FIELDS if is_filled(record.get(field)))
    return (filled, -shoutiness(str(record.get("title") or "")))


def merge_group(records: list[dict[str, Any]]) -> dict[str, Any]:
    """Merge duplicates field by field, preferring the most complete record."""
    ranked = sorted(records, key=rank_key, reverse=True)
    merged: dict[str, Any] = {}
    field_source: dict[str, Any] = {}

    for field in FIELDS:
        for record in ranked:
            if is_filled(record.get(field)):
                merged[field] = record[field]
                field_source[field] = record["record_id"]
                break
        else:
            merged[field] = None
            field_source[field] = None

    seen = sorted(r["scraped_at"] for r in records)
    merged["first_seen"], merged["last_seen"] = seen[0], seen[-1]
    merged["source_ids"] = sorted(r["record_id"] for r in records)
    merged["field_source"] = field_source
    return merged


GROUP = [
    {"record_id": "A", "title": "Acme Kettle 1.7L", "price": "24.99", "brand": "",
     "url": "https://example.com/p/1001", "scraped_at": "2026-08-01T09:02:00Z"},
    {"record_id": "B", "title": "ACME KETTLE 1.7 L", "price": "", "brand": "Acme",
     "url": "https://example.com/p/2477", "scraped_at": "2026-08-01T11:40:00Z"},
]

for key, value in merge_group(GROUP).items():
    print(f"{key:12} {value}")

The output takes title, price and url from A and brand from B. Both records populate three of the four fields, so the completeness score ties and the tiebreak decides: shoutiness measures the fraction of letters in upper case, and an all-caps title is nearly always a nav label, a promotional banner or a badly templated listing rather than the real product name.

Keeping field_source, source_ids, first_seen and last_seen costs four columns and makes the merge auditable. Without them a wrong merge is unrecoverable; with them you can identify every affected row, revert it, and re-run with a corrected threshold.

Edge Cases and Caveats

Transitivity is not guaranteed. A matches B and B matches C does not mean A matches C. Either build connected components with a union-find structure and accept some drift, or require every pair inside a group to clear the threshold.
Numbers inside titles. Sizes, capacities and model numbers dominate meaning while barely moving a similarity score. Extract them and compare them exactly before trusting the score.
Short titles score erratically. Two-token names produce a coarse score distribution where a single character shifts the result by twenty points. Raise the threshold for short strings or fall back to exact key matching.
Blocking loses recall you cannot see. A pair split across two blocks is never scored, so it never appears in any evaluation. Measure recall by running an unblocked comparison on a sample of a few thousand records and counting what blocking would have missed.
Non-Latin text. casefold and NFKD handle European scripts well; CJK text has no word boundaries, so token-based scorers degrade badly. Use character n-grams for those languages.
Merging changes row counts. Anything downstream that asserts on the number of scraped items needs to see the pre-merge count as well, or deduplication looks identical to a broken crawl.

Frequently Asked Questions

Which rapidfuzz scorer should I use for product titles?token_set_ratio. It ignores word order and tolerates one string having extra tokens the other lacks, which is the normal shape of the problem. ratio punishes the extra words too hard, and partial_ratio rates any substring match highly, which produces false positives whenever one title is short.

What threshold should I start with? Around 88 on normalised strings is a reasonable starting point for product titles, but treat it as a placeholder until you have swept a labelled sample. The right value depends on title length, how much boilerplate the source adds, and how expensive a wrong merge is in your dataset.

How do I keep deduplication fast on millions of rows? Block first, and make the blocking key selective enough that no bucket exceeds a few hundred records. Beyond that, drop text similarity entirely in favour of MinHash with locality-sensitive hashing, which finds near-duplicates in roughly linear time at the cost of a tunable false-negative rate.

Should duplicates be removed during the crawl or afterwards? Afterwards, as a batch step over the validated rows. During a crawl you have only seen part of the data, so an early decision is made with less information than a later one, and any merge you get wrong is already written. Keeping every raw row and merging into a separate table also means a threshold change is a re-run rather than a re-crawl.

Cleaning and Validating Scraped Data — the parent guide and where this step belongs
Normalizing Prices, Dates and Units — the field parsers that have to run before any matching
Validating Scraped Data with Pydantic — rejecting bad rows before they reach the matcher
Storing and Exporting Scraped Data — writing merged records and keeping their provenance