Unicode and UTF-8

Text is bytes too. Every letter you see is stored as a number, and the encoding is the rule that turns that number into bytes. When the writer and the reader use different rules, you get bugs like a customer named José. Unicode and UTF-8 are the rules that prevent most of them.

The big idea

The old world used a separate 8-bit code page for each region. Latin-1, Windows-1252 and Shift JIS reused the same byte values for different letters. A file was unreadable unless you knew which code page it used, and often you didn't.

Unicode fixes this by giving every character one number, called a code point. Code points are written U+ plus hex: A is U+0041, é is U+00E9, € is U+20AC, 😀 is U+1F600. Unicode says which number a character is. It does not say how to store that number. That is the job of an encoding.

Walking through UTF-8

UTF-8 stores each code point in 1 to 4 bytes. Small characters take fewer bytes. The bit pattern in the first byte says how many bytes follow.

Encode € (U+20AC) by hand:

U+20AC in binary (16 bits):   0010 0000 1010 1100
split into 4 + 6 + 6 bits:    0010 | 000010 | 101100
fill the 3-byte template:     1110 0010 | 10 000010 | 10 101100
in hex:                       E2        82         AC

The design has three useful properties.

• ASCII-compatible. Every ASCII file is already valid UTF-8, byte for byte.
• Self-synchronizing. Continuation bytes always start with 10. You can jump into a string and find the next character boundary.
• No byte-order issues, because the unit is one byte.

UTF-8 is the right default for files, databases, APIs and source code.

UTF-16 and surrogate pairs

UTF-16 uses 2-byte units. Code points up to U+FFFF take one unit. Larger ones take two, a surrogate pair. The emoji 😀 becomes D83D DE00. Java, JavaScript, C# and the Windows API use UTF-16 internally, so their string length counts units, not characters.

"😀".length          // 2 (two UTF-16 units)
[..."😀"].length     // 1 (iterates by code point)

What is a "length"?

One visible symbol can have several lengths. Take the family emoji 👨‍👩‍👧, which is three people joined by zero-width joiners.

Pick the measure that matches your limit. A database VARCHAR(255) counts characters or bytes depending on the database. A byte budget for a queue counts UTF-8 bytes. To count graphemes, use Intl.Segmenter in JavaScript or the regex module's \X in Python. Truncating by bytes or UTF-16 units can cut an emoji in half and produce invalid text.

Normalization: two ways to write é

The same visible text can have different code points. é can be the single code point U+00E9, or e followed by a combining accent.

>>> a = "café"
>>> b = "cafe\u0301"
>>> a == b, len(a), len(b)
(False, 4, 5)
>>> import unicodedata
>>> unicodedata.normalize("NFC", b) == a
True

Normalization picks one canonical form. NFC combines characters where it can, and is the usual choice for storage. Normalize usernames, emails and filenames before comparing them or using them as keys. Otherwise two accounts can exist for what looks like one name.

Mojibake and the MySQL trap

Mojibake is text decoded with the wrong encoding. The name José is 4A 6F 73 C3 A9 in UTF-8. Decode C3 A9 as Windows-1252 and you get é, so the name shows as José. An Ã followed by an odd character usually means "UTF-8 read as Latin-1".

MySQL's utf8 character set is really utf8mb3, which stores at most 3 bytes per character. Every emoji needs 4, so it fails to insert or gets cut off. Use utf8mb4, which is the default in MySQL 8.0 and later.

Common mistakes

• Using string length as a byte limit. Count bytes for storage limits and graphemes for user-facing limits.
• Cutting strings at a fixed index. It can split a multi-byte character or an emoji.
• Skipping normalization. Two visually identical usernames can be different keys.
• Reading bytes without naming the encoding. Declare UTF-8 explicitly in files, HTTP headers and databases.

Check yourself

1. Why does len("👨‍👩‍👧") in Python return 5, but a user sees one symbol? Python counts code points. The family emoji is three person code points joined by two zero-width joiners, which makes 5 code points and 1 grapheme cluster.
2. A name shows as José on screen. What happened? The UTF-8 bytes were decoded as a single-byte encoding such as Windows-1252.

Key takeaways

• Unicode assigns each character a code point. UTF-8 stores code points in 1 to 4 bytes and is the right default.
• UTF-16 uses surrogate pairs above U+FFFF, and it is the internal format of Java and JavaScript.
• "Length" can mean bytes, UTF-16 units, code points or graphemes. Choose the one your limit needs.
• Normalize identifiers to NFC, and always use utf8mb4 in MySQL.

Further learning

Tom Scott's Computerphile video explains why UTF-8 is designed the way it is, including the self-synchronizing trick.

The Absolute Minimum Every Software Developer Must Know About Unicode (Joel Spolsky)Ever wonder about that mysterious Content-Type tag? You know, the one you’re supposed to put in HTML and you never quite know what it should be? Did you ever get an email from your friends in…Joel on Software

Next, read base64 and URL encoding for the encodings that carry text in web protocols.