What Is Character Encoding? – Definition and How It Works

Learn what character encoding is, how UTF-8 works, why mojibake happens, how encoding is declared on the web, and how to handle it correctly.

Character encoding is the mapping between characters and the bytes that represent them. Every text file and every HTTP response is bytes; the encoding is what turns those bytes back into readable text.

1. The Core Concepts

• Character set — the collection of characters, each with a numeric code point.
• Encoding — the rule for turning code points into bytes and back.
• Unicode — the universal character set covering essentially every script.
• UTF-8 — the dominant encoding of Unicode on the web.

Unicode says which characters exist; UTF-8 says how to store them.

2. Why UTF-8 Won

• Backwards compatible with ASCII: the first 128 characters encode identically.
• Variable width — one to four bytes per character, so common text stays compact.
• Self-synchronising, so a corrupted byte does not destroy everything after it.
• No byte order ambiguity, unlike wider fixed-width encodings.

3. Where Mojibake Comes From

Garbled text is bytes decoded with the wrong encoding — a UTF-8 sequence read as a single-byte encoding, most commonly.

"café"  encoded UTF-8   → 63 61 66 C3 A9
        decoded latin-1 → "café"

The bytes are intact; only the interpretation is wrong. This is why the fix is almost always to decode correctly rather than to repair the text afterwards.

4. How Encoding Is Declared on the Web

The HTTP header takes precedence in browsers. When nothing is declared, clients guess — and guesses are frequently wrong on non-English content.

5. Handling It in Code

• Decode explicitly where the encoding is known, rather than relying on library guessing.
• Use the declared charset from the header, falling back to the document's own declaration.
• Detect as a last resort, and record that detection was used.
• Store as UTF-8 internally, converting at the boundary.
• Compare normalised text — the same character can have multiple Unicode representations.

6. Normalisation

Unicode allows some characters to be composed in more than one way — a letter with an accent as a single code point, or as a base letter plus a combining mark. The two look identical and are not equal as strings.

Normalising before comparison, deduplication, or use as a key prevents entries that appear identical from behaving as different values.

7. Practical Traps

• Assuming the platform default. It differs between systems and CI environments.
• A byte order mark breaking strict parsers, particularly for JSON and CSV.
• Length in bytes versus characters, which differ for anything non-ASCII.
• Truncating by bytes, which can split a character and corrupt it.
• Databases with a non-Unicode column encoding, silently replacing characters on write.

8. Summary

Character encoding maps characters to bytes; UTF-8 is the web's default and handles every script. Garbled text is almost always correct bytes decoded with the wrong rule.

Read the declared charset, decode explicitly, normalise before comparing, store UTF-8 internally, and never assume the platform default — most encoding bugs are one missing explicit decision.

Potential Related Glossary

• Parsing
• Data Wrangling
• Unstructured Data
• Data Structure