What Is a BOM (Byte Order Mark)? β Definition and How to Handle It
Learn what a byte order mark is, why it exists, which parsers it breaks, and when including one is deliberate rather than a bug.
A byte order mark is a special character at the start of a text file signalling its encoding and, for wider encodings, its byte order. In UTF-8 it serves only as an encoding signal, and its presence causes more problems than it solves.
1. What It Is
The mark is the Unicode character U+FEFF, encoded at the start of the file. In UTF-8 it appears as three bytes:
EF BB BF
Nothing in the text requires it. UTF-8 has no byte-order ambiguity, so in UTF-8 it functions purely as a hint that the file is UTF-8.
2. Why It Exists
For UTF-16 and UTF-32 it is genuinely useful: those encodings can be little-endian or big-endian, and the mark disambiguates them.
Its use in UTF-8 is a convention borrowed from that purpose, adopted mainly so that software guessing at encodings guesses correctly.
3. What It Breaks
| Consumer | Effect |
|---|---|
| Strict JSON parsers | Reject the document β the first character is not { |
| CSV readers | The first column name gains invisible characters |
| Shell scripts | The shebang is no longer at byte zero |
| XML parsers | Some reject it before the declaration |
| String comparison | A header that looks equal is not |
The CSV case is the most insidious: a header named id becomes ο»Ώid, and code looking up that column finds nothing while the file looks perfect.
4. Detecting One
head -c 3 file.csv | xxd
# 00000000: efbb bf ...
In Python, opening with encoding utf-8-sig strips it transparently, which is the pragmatic fix when consuming files from elsewhere:
with open(path, encoding="utf-8-sig") as handle:
...
5. When Including One Is Deliberate
Some spreadsheet versions detect UTF-8 more reliably when a mark is present, and without it they may interpret a file using a legacy single-byte encoding β producing garbled accented characters.
So for a CSV destined for a spreadsheet user, including it can be the right choice. For a CSV destined for a program, it is a bug.
6. The Rule of Thumb
- Producing for machines β write UTF-8 without a mark.
- Producing for spreadsheet users β consider including one, deliberately.
- Consuming β always strip it, since you cannot control what arrives.
- JSON β never include one.
7. Why It Surfaces in Collection Work
Collected files arrive from many sources, and some include the mark. A pipeline that does not strip it on read will eventually fail on a column name that appears correct in every log and does not match.
Stripping on ingestion is one line and removes the entire class of problem.
8. Summary
A byte order mark is a leading character signalling encoding, essential for UTF-16 and merely a hint in UTF-8 β where it breaks strict JSON parsers, CSV header matching, and shell scripts.
Write UTF-8 without one for machine consumers, consider one for spreadsheet users, and always strip it when reading files you did not produce.