What Is Unicode Normalization? – Definition and How It Works

Learn what NFC, NFD, NFKC, and NFKD do, why identical-looking strings compare unequal, and where to normalize in a pipeline.

Unicode normalization converts text to a canonical form so that strings which should be equal actually compare equal. Without it, two visually identical strings can have different byte sequences.

1. The Problem

The character é has two valid encodings: one precomposed code point, or e followed by a combining acute accent.

a = "café"        # precomposed
b = "café"       # e + combining accent
print(a == b, len(a), len(b))   # False 4 5

They render identically. They are not equal, they have different lengths, and a database lookup with one will not find the other. Collected data contains both, because different sources emit different forms.

2. The Four Forms

3. Which to Use

NFC is the default. It is the form the web recommends, the most compact, and preserves meaning exactly.

NFD is useful as an intermediate step, because with the accents separated they can be stripped by category to produce an unaccented comparison key.

NFKC/NFKD are lossy in a way that matters. They fold characters that look similar but are not the same: superscripts become digits, ligatures split, full-width forms become ASCII. Excellent for search keys, wrong for stored values — x² becoming x2 changes the meaning.

4. Where to Normalize

At the ingestion boundary, immediately after decoding, before anything else:

import unicodedata

def clean(text):
    if text is None:
        return None
    text = unicodedata.normalize("NFC", text)
    text = text.replace(" ", " ")        # non-breaking space
    text = "".join(c for c in text if unicodedata.category(c) != "Cc" or c in "\t\n")
    return " ".join(text.split())

Normalising once at the boundary means everything downstream — comparison, deduplication, keys, indexes — operates on one form. Normalising at comparison time instead means every call site has to remember, and one that forgets produces a silent miss.

5. Separate Storage From Matching

Two values per field, for different purposes:

stored = clean(raw)                                     # NFC, meaning preserved
key    = unicodedata.normalize("NFKC", stored).casefold()
key    = "".join(c for c in unicodedata.normalize("NFD", key)
                 if unicodedata.category(c) != "Mn")    # strip accents

The stored value is what you display and export. The key is what you match, deduplicate, and index on. Collapsing them into one loses either accuracy or matching power.

6. Casefold, Not Lower

str.casefold() is the correct case-insensitive operation. str.lower() misses cases such as German ß, which casefolds to ss but lowercases to itself.

Combined with normalisation, casefolding is what makes case-insensitive comparison actually work across languages.

7. Invisible Characters

Beyond normalization, real text carries zero-width spaces, byte order marks mid-string, soft hyphens, directional marks, and non-breaking spaces. All are invisible and all break exact matching.

Normalization does not remove them — it is a separate cleaning step, and one worth doing at the same boundary.

8. Summary

Unicode normalization makes visually identical strings byte-identical by converting to a canonical form: NFC for storage, NFD as a step toward accent stripping, and NFKC/NFKD only for matching keys because their folding is lossy.

Normalise once at the ingestion boundary, keep a separate NFKC-casefolded key for matching, use casefold rather than lower, and strip invisible characters in the same pass.

Potential Related Glossary

• Data Wrangling
• Parsing
• Unstructured Data
• Data Structure